Data monitoring method and device, computer equipment and storage medium

Abstract

The invention provides a data monitoring method and device, computer equipment and a storage medium, and the data monitoring method comprises the steps: for any data interface in a plurality of data interfaces corresponding to a database for application program data updating, when the current time meets the preset triggering time requirement of the data interface, the preset triggering time requirement of the data interface is met; screening out at least one data monitoring rule corresponding to the preset triggering time from a plurality of data monitoring rules corresponding to the data interface; the plurality of data interfaces are interfaces for the data warehouse to import update data into the database; and for any data monitoring rule in the at least one data monitoring rule, performing data monitoring on the data interface according to the data monitoring rule. According to the embodiment of the invention, the data synchronized from the data warehouse to the database can be effectively monitored, and the data updating conditions of a plurality of function modules with inconsistent data updating time can be effectively monitored.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to a data monitoring method, apparatus, computer equipment, and storage medium. Background Technology

[0002] When performing data analysis and processing, data is typically stored in a data warehouse (such as Hive), and then the data in Hive is synchronized to a database more suitable for fast querying and analysis (such as the distributed database StarRocks for online analytical processing) through Extract-Transform-Load (ETL) tasks. As the number of Hive instances increases, ensuring the timeliness and accuracy of data updates has become a pressing issue.

[0003] Currently, existing technologies can support monitoring data in Hive, but there is no effective monitoring of data synchronized from Hive to StarRocks. It relies on users to discover data anomalies in StarRocks and trigger alarms, which results in a lag in monitoring data in StarRocks and the discovery of abnormal data, leading to a poor user experience. Summary of the Invention

[0004] In view of this, this disclosure proposes a data monitoring method, apparatus, computer equipment, and storage medium to solve the problems of low alarm efficiency and poor user experience caused by the lag in data monitoring and abnormal data discovery in related technologies.

[0005] A first aspect of this disclosure provides a data monitoring method, the method comprising:

[0006] For any one of the multiple data interfaces corresponding to the database that updates application data, when the preset trigger time requirement of the data interface is met at the current time, at least one data monitoring rule corresponding to the preset trigger time is selected from the multiple data monitoring rules corresponding to the data interface; the multiple data interfaces are the interfaces through which the data warehouse imports updated data into the database;

[0007] For any one of the at least one data monitoring rules, data monitoring is performed on the data interface according to the data monitoring rule.

[0008] This embodiment of the disclosure, when the current time meets the preset trigger time requirement of the data interface, selects at least one data monitoring rule corresponding to the preset trigger time from multiple data monitoring rules corresponding to the data interface, and monitors the data interface according to the data monitoring rule. This enables effective monitoring of data synchronized from the data warehouse to the database, as well as effective monitoring of the data update status of multiple functional modules with inconsistent data update times, and achieves the purpose of automated data monitoring and anomaly alarm.

[0009] In this embodiment of the disclosure, data monitoring of the data interface is performed according to the data monitoring rules, including:

[0010] Determine the verification parameters corresponding to the data monitoring rules;

[0011] The verification parameters are sent to the data interface so that the data interface returns response data associated with the verification parameters;

[0012] If the response data does not meet the data monitoring rules, then it is determined that there is abnormal data in the data interface.

[0013] This embodiment of the disclosure sends verification parameters corresponding to the data monitoring rules to the data interface, enabling the data interface to return response data associated with the verification parameters. This helps the data monitoring rules to verify the response data, thereby improving the accuracy of anomaly detection of the data interface.

[0014] In this embodiment of the disclosure, the data monitoring rules include rules for monitoring at least one of the following: data type, data value range, data arrangement order, and data business requirements.

[0015] In this embodiment of the disclosure, if the response data does not meet the data monitoring rules, it is determined that the data interface contains abnormal data, including any of the following:

[0016] If the data type of the response data does not belong to the preset data type, then it is determined that there is abnormal data in the data interface;

[0017] If the value of the response data does not fall within the preset value range, it is determined that there is abnormal data in the data interface;

[0018] If multiple sub-response data in the response data do not meet the preset arrangement order, it is determined that there is abnormal data in the data interface;

[0019] If the response data does not meet the preset business requirements, it is determined that there is abnormal data in the data interface.

[0020] This disclosure uses different data monitoring rules to verify different data in the data interface, which can ensure the comprehensiveness of the data in the data interface and thus improve the accuracy of anomaly detection in the data interface.

[0021] In this embodiment of the disclosure, after determining that abnormal data exists in the data interface, the method further includes:

[0022] The abnormal function module corresponding to the data interface is determined according to the first mapping relationship; the first mapping relationship includes multiple function modules of the application and at least one data interface corresponding to each function module.

[0023] The alarm information is sent to the client of the person in charge of the abnormal function module; the alarm information includes the abnormal data and the data interface.

[0024] This embodiment of the disclosure determines the abnormal functional module corresponding to the data interface through the first mapping relationship, which can accurately locate the functional module with the problem, rather than the entire system or multiple modules. This can reduce the time and workload of troubleshooting. Sending alarm information to the client of the person in charge of the abnormal functional module can ensure that the relevant person in charge can be notified immediately after the anomaly occurs, reducing the delay and error of human notification.

[0025] In this embodiment of the disclosure, the method further includes:

[0026] For any data interface, receive the configuration information corresponding to the data interface;

[0027] Configure the data monitoring task for the data interface according to the configuration information;

[0028] The data monitoring task includes at least one of the following: the interface address of each data interface, at least one data monitoring rule associated with each data interface, a preset trigger time associated with each data monitoring rule, and a verification parameter associated with each data monitoring rule.

[0029] This embodiment of the disclosure enables automatic data monitoring of multiple data interfaces with different data update times by configuring corresponding data monitoring tasks for each data interface.

[0030] In this embodiment of the disclosure, the method further includes:

[0031] Monitor the data update volume of multiple functional modules of the application;

[0032] For any one of the multiple functional modules, the time range during which the data update volume of the functional module is greater than a preset threshold is defined as the first time range; the first time range is used to monitor the data of each data interface of the functional module using the first data monitoring rule;

[0033] The time range within which the data update volume of the functional module is equal to or less than the preset threshold is defined as the second time range; the second time range is used to monitor the data of each data interface of the functional module using the second data monitoring rules.

[0034] This disclosure embodiment monitors the data update volume of multiple functional modules of an application and selects a corresponding time range as a first or second time range based on the data update volume. This enables the use of higher-standard first data monitoring rules to monitor the corresponding data interfaces during periods of high data update volume, such as peak periods like Spring Festival and Double Eleven. By strengthening data monitoring during high-traffic periods, data anomalies can be captured more accurately, thereby ensuring the accuracy and reliability of the data at the data interface.

[0035] In this embodiment of the disclosure, the preset trigger time requirement is a first time range; the plurality of data monitoring rules include at least one first data monitoring rule and at least one second data monitoring rule; when the preset trigger time requirement of the data interface is met at the current time, selecting at least one data monitoring rule corresponding to the preset trigger time from the plurality of data monitoring rules corresponding to the data interface includes:

[0036] If the preset trigger time is within the first time range, then a first target data monitoring rule corresponding to the preset trigger time is selected from the at least one first data monitoring rule;

[0037] If the preset trigger time is outside the first time range, then a second target data monitoring rule corresponding to the preset trigger time is selected from the at least one second data monitoring rule.

[0038] In this embodiment of the disclosure, the step of filtering out the second target data monitoring rule corresponding to the preset trigger time from the at least one second data monitoring rule includes:

[0039] When the preset trigger time is outside the first time range, determine whether the preset trigger time is within the second time range;

[0040] If the preset trigger time is within the second time range, then a second target data monitoring rule corresponding to the preset trigger time is selected from the at least one second data monitoring rule.

[0041] In this embodiment of the disclosure, the step of filtering out the first target data monitoring rule corresponding to the preset trigger time from the at least one first data monitoring rule includes:

[0042] When the preset trigger time is simultaneously within the first time range and the second time range, a third target data monitoring rule corresponding to the preset trigger time is selected from the at least one first data monitoring rule.

[0043] In this embodiment of the disclosure, the method further includes:

[0044] A preset page is launched at preset time intervals, and the preset page includes multiple page elements;

[0045] If a display error occurs during the loading process of the preset page, the display error is sent to the client of the corresponding administrator; wherein, the display error includes page element data loading error, page element rendering error, and page loading speed exceeding the preset loading speed.

[0046] In this embodiment, a preset page is launched at preset time intervals. If a display error occurs during the loading process of the preset page, the error is sent to the client of the corresponding administrator. By periodically and automatically checking the page, problems can be proactively discovered instead of waiting for user reports. When a display error is detected, the administrator can be notified immediately, which speeds up the response and resolution of problems.

[0047] A second aspect of this disclosure provides a data monitoring apparatus, the apparatus comprising:

[0048] The data monitoring rule filtering module is used to filter at least one data monitoring rule corresponding to the preset trigger time from multiple data monitoring rules corresponding to the data interface when the current time meets the preset trigger time requirement of the data interface; the multiple data interfaces are the interfaces through which the data warehouse imports updated data into the database;

[0049] The data monitoring module is used to monitor the data interface according to any one of the at least one data monitoring rules.

[0050] An embodiment of the third aspect of this disclosure provides a computer device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the data monitoring method described in the first aspect above.

[0051] An embodiment of the fourth aspect of this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the data monitoring method described in the first aspect above.

[0052] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description or may be learned by practice of this disclosure. Attached Figure Description

[0053] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0054] In the attached diagram:

[0055] Figure 1 A flowchart illustrating a data monitoring method provided in an embodiment of this disclosure is shown.

[0056] Figure 2 It shows Figure 1 A flowchart illustrating the specific implementation of step S102 in this embodiment;

[0057] Figure 3 It shows Figure 2 A flowchart illustrating the specific implementation method following step S1023 in the embodiment;

[0058] Figure 4 A flowchart illustrating another data monitoring method provided in an embodiment of this disclosure is shown;

[0059] Figure 5 A flowchart illustrating another data monitoring method provided in an embodiment of this disclosure is shown;

[0060] Figure 6 It shows Figure 1 A flowchart illustrating the specific implementation of step S101 in the embodiment;

[0061] Figure 7 It shows Figure 6 A flowchart illustrating the specific implementation of step S1012 in the embodiment;

[0062] Figure 8 A flowchart illustrating another data monitoring method provided in an embodiment of this disclosure is shown;

[0063] Figure 9A schematic diagram of the structure of a data monitoring device provided in an embodiment of this disclosure is shown;

[0064] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure is shown;

[0065] Figure 11 A schematic diagram of a storage medium provided according to an embodiment of the present disclosure is shown. Detailed Implementation

[0066] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0067] It should be noted that, unless otherwise stated, the technical or scientific terms used in this disclosure shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure pertains.

[0068] The following describes the relevant technical scenarios involved in the embodiments of this disclosure.

[0069] In existing technologies, when application modules update data, they write new data to the corresponding Hive tables in the Hive data warehouse. Then, an automated ETL (Extract-Transform-Load) task script extracts the latest data from the Hive tables and loads it into the StarRocks database through the appropriate data interface. During this process, because the data update times of different application modules are inconsistent (e.g., module A updates data daily, module B updates data weekly), and the number of modules increases, it becomes difficult to effectively monitor the data updates of multiple modules with inconsistent update times.

[0070] Furthermore, existing technologies only support monitoring data updates in Hive, and cannot effectively monitor data synchronized from Hive to StarRocks.

[0071] To address the aforementioned technical issues, this invention provides a data monitoring method based on a scheduled task platform. This method enables effective monitoring of data synchronized from Hive to StarRocks, as well as effective monitoring of data updates from multiple functional modules with inconsistent update times.

[0072] According to an embodiment of this disclosure, a data monitoring method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0073] This embodiment provides a data monitoring method that can be used in a scheduled task platform. Figure 1 This is a flowchart of a data monitoring method according to an embodiment of the present disclosure, such as... Figure 1 As shown, the process includes the following steps:

[0074] Step S101: For any one of the multiple data interfaces corresponding to the database that updates application data, when the preset trigger time requirement of the data interface is met at the current time, at least one data monitoring rule corresponding to the preset trigger time is selected from the multiple data monitoring rules corresponding to the data interface.

[0075] Among them, the database used for application data updates can be StarRocks. StarRocks is an open-source, next-generation, high-speed, full-scenario MPP (Massive Parallel Processing) database. It adopts a new generation of elastic MPP architecture, which can support real-time data analysis and enable efficient querying of real-time updated data.

[0076] In some embodiments, the database has multiple data interfaces, each used to synchronize updated data of the corresponding Hive table in the data warehouse to the database; wherein, each Hive table corresponds to at least one data interface, and the application includes multiple functional modules, each functional module corresponding to at least one Hive table.

[0077] In some embodiments, the preset trigger time requirement includes the preset trigger time corresponding to the data interface, which can be understood as the start time for data monitoring of the data interface, such as: 7:00 AM every day, 8:00 AM every Monday, and 9:00 AM on the 1st of each month. The preset trigger time for each data interface, as well as the number of preset trigger times, can be set according to the actual situation and are not specifically limited here.

[0078] In some embodiments, each data interface has at least one preset trigger time, and each preset trigger time corresponds to at least one data monitoring rule. For example, data interface A has two preset trigger times: 8:00 AM and 9:00 AM daily. "8:00 AM daily" corresponds to three data monitoring rules (hereinafter referred to as rules): rule 1, rule 2, and rule 3; "9:00 AM daily" corresponds to two data monitoring rules: rule 4 and rule 5. Monitoring the data of the data interface using data monitoring rules ensures data integrity and accuracy.

[0079] In this step, when the current time reaches any preset trigger time of any data interface, at least one data monitoring rule corresponding to that preset trigger time can be selected from multiple data monitoring rules corresponding to that data interface. For example, in the example above: assuming the current time is 8 o'clock, then rule 1, rule 2, and rule 3 corresponding to the preset trigger time "8 o'clock every day" can be selected.

[0080] Step S102: For any one of the at least one data monitoring rules, perform data monitoring on the data interface according to the data monitoring rule.

[0081] This invention, when the preset trigger time requirement of the data interface is met at the current time, selects at least one data monitoring rule corresponding to the preset trigger time from multiple data monitoring rules corresponding to the data interface, and monitors the data interface according to the data monitoring rule. This can ensure effective monitoring of the data update status of multiple functional modules with inconsistent data update times, and achieve the purpose of automated data monitoring and anomaly alarm.

[0082] In some specific embodiments, such as Figure 2 As shown, step S102 above includes steps S1021-S1023:

[0083] Step S1021: Determine the verification parameters corresponding to the data monitoring rules.

[0084] Step S1022: Send the verification parameters to the data interface so that the data interface returns response data associated with the verification parameters.

[0085] Step S1023: If the response data does not meet the data monitoring rules, then it is determined that there is abnormal data in the data interface.

[0086] In steps S1021-S1023 above, each data monitoring rule has a corresponding verification parameter, and the correspondence between the data monitoring rule and the verification parameter is preset. The verification parameter is mainly used to make the data interface return response data associated with the verification parameter, so that the data monitoring rule can effectively verify the response data and thus determine whether there is abnormal data in the data interface. For example, when the data monitoring rule is "determine whether the product ID (Identification) conforms to the preset format (e.g., 8-digit number)," the corresponding verification parameter can be determined to be "product ID information." The scheduled task platform will send its verification parameter to the data interface so that the data interface returns the product ID information, so that the data monitoring rule can determine whether the returned product ID information conforms to the preset format.

[0087] In some specific embodiments, the data monitoring rules include rules for monitoring at least one of the following: data type, data value range, data arrangement order, and data business requirements.

[0088] In this embodiment of the disclosure, the data monitoring rule is a data type, which can be understood as: determining whether the response data returned by the data interface belongs to a preset data type; if the data type of the response data does not belong to the preset data type, then it is determined that there is abnormal data in the data interface.

[0089] The data monitoring rule is the numerical range of the data, which can be understood as: determining whether the response data returned by the data interface belongs to the preset numerical range; if the value of the response data does not belong to the preset numerical range, it is determined that there is abnormal data in the data interface;

[0090] The data monitoring rule is the order in which the data is arranged. It can be understood as: determining whether multiple word response data returned by the data interface meet the preset order; if multiple sub-response data in the response data do not meet the preset order, then it is determined that there is abnormal data in the data interface.

[0091] Data monitoring rules are business requirements for data, which can be understood as: determining whether the response data returned by the data interface meets the preset business requirements; if the response data does not meet the preset business requirements, it is determined that there is abnormal data in the data interface.

[0092] In some specific embodiments, such as Figure 3 As shown, after step S1023, the method further includes steps S301-S302:

[0093] Step S301: Determine the abnormal function module corresponding to the data interface according to the first mapping relationship.

[0094] In this step, the first mapping relationship includes multiple functional modules of the application and at least one data interface corresponding to each functional module.

[0095] Step S302: Send the alarm information to the client of the person in charge of the abnormal function module.

[0096] In steps S301-S302 above, the alarm information includes the abnormal data and the data interface. When an abnormal functional module corresponding to the data interface containing abnormal data is determined, the client of the person in charge of the abnormal functional module can be identified, and the alarm information, i.e., the abnormal data and the interface information of the data interface where the abnormal data is located, is sent to the client of the person in charge, so that the person in charge can perform abnormal handling on the data interface according to the abnormal data and the interface information.

[0097] In some specific embodiments, the method further includes:

[0098] Step a1: For any data interface, receive the configuration information corresponding to the data interface.

[0099] In this step, the configuration information received by the scheduled task platform includes the interface address of the data interface to be monitored (e.g., Figure 4 The data includes domain name 1, domain name 2 and domain name 3, at least one data monitoring rule corresponding to the data interface to be monitored, the preset trigger time of each data monitoring rule, and the verification parameters associated with each data monitoring rule.

[0100] Step a2: Configure the data monitoring task of the data interface according to the configuration information.

[0101] In this step, the data monitoring task includes at least one of the following: the interface address of each data interface, at least one data monitoring rule associated with each data interface, a preset trigger time associated with each data monitoring rule, and a verification parameter associated with each data monitoring rule.

[0102] In steps a1-a2 above, after the data monitoring task of the data interface is configured, the scheduled task platform will send the verification parameters corresponding to the triggered data monitoring rule to the data interface according to the interface address of the data interface when the preset trigger time of any data monitoring rule associated with the data interface is reached at the current time, so that the data interface returns response data associated with the verification parameters, and performs data monitoring on the response data according to the triggered data monitoring rule.

[0103] In some specific embodiments, such as Figure 5 As shown, the method further includes steps S501-S503:

[0104] Step S501: Monitor the data update volume of multiple functional modules of the application.

[0105] In this step, the data update volume of the functional module can be determined according to a preset time interval, such as determining the data update volume of the functional module every hour or every day. The preset time interval can be set according to the actual situation and is not specifically limited here.

[0106] Step S502: For any one of the plurality of functional modules, the time range in which the data update amount of the functional module is greater than a preset threshold is taken as the first time range.

[0107] In this step, the preset threshold can be set according to the actual situation, and no specific limitation is made here.

[0108] In this step, the first time range is used to monitor the data of each data interface of the functional module using the first data monitoring rule. The first time range can be understood as special periods such as Double Eleven and Spring Festival. During these special periods, the data update volume of the functional module is much greater than in other time ranges. Therefore, a higher standard of data monitoring rule, namely the first data monitoring rule, is needed within the first time range to monitor the data of each data interface.

[0109] Step S503: The time range in which the data update amount of the functional module is equal to or less than the preset threshold is taken as the second time range.

[0110] In this step, the second time range is used to monitor the data of each data interface of the functional module using the second data monitoring rule. The second time range can also be understood as a user-defined time range excluding special periods such as Double Eleven and Spring Festival. Within these time ranges, a less standardized data monitoring rule, namely the second data monitoring rule, can be used to monitor the data of each data interface.

[0111] In some specific embodiments, such as Figure 6 As shown, step S101 above includes steps S1011-S1012:

[0112] Step S1011: If the preset trigger time is within the first time range, then select the first target data monitoring rule corresponding to the preset trigger time from the at least one first data monitoring rule.

[0113] In this step, the preset trigger time requirement is a first time range, and the multiple data monitoring rules include at least one first data monitoring rule and at least one second data monitoring rule.

[0114] In this step, the first time range corresponds to multiple first data monitoring rules. When the preset trigger time requirement of the data interface is the first time range, it is necessary to select the first target data monitoring rule corresponding to the preset trigger time from the multiple first data monitoring rules corresponding to the data interface. For example, if the current time is 8:00 AM on Singles' Day, this time falls within the Singles' Day time range. Therefore, it is necessary to select the first data monitoring rule corresponding to "8:00 AM" from the multiple first data monitoring rules corresponding to the data interface as the first target data monitoring rule, and then monitor the data using this first target data monitoring rule.

[0115] Step S1012: If the preset trigger time is outside the first time range, then select the second target data monitoring rule corresponding to the preset trigger time from the at least one second data monitoring rule.

[0116] In this step, if the preset trigger time of the data interface is outside the first time range, it means that the first data monitoring rule corresponding to the preset trigger time cannot be selected from multiple first data monitoring rules. Therefore, it is necessary to select the second target data monitoring rule corresponding to the preset trigger time from multiple second data monitoring rules.

[0117] In some specific embodiments, such as Figure 7 As shown, step S1012 above includes steps S701-S702:

[0118] Step S701: When the preset trigger time is outside the first time range, determine whether the preset trigger time is within the second time range.

[0119] Step S702: If the preset trigger time is within the second time range, then select the second target data monitoring rule corresponding to the preset trigger time from the at least one second data monitoring rule.

[0120] For example, let's illustrate steps S701-S702 above: If the current time is 8:00 AM on June 8th, this time is not during peak periods such as Singles' Day or Chinese New Year (i.e., the first time range). Therefore, it is necessary to determine whether "8:00 AM on June 8th" is within the user-defined time range (i.e., the second time range). If it is determined that "8:00 AM on June 8th" is within the user-defined time range, then it is only necessary to select the second data monitoring rule corresponding to "8:00 AM on June 8th" from the multiple second data monitoring rules corresponding to the data interface as the second target data monitoring rule, and perform data monitoring through this second target data monitoring rule.

[0121] In some specific embodiments, step S1011 above further includes step S10111:

[0122] Step S10111: When the preset trigger time is simultaneously within the first time range and the second time range, a third target data monitoring rule corresponding to the preset trigger time is selected from the at least one first data monitoring rule.

[0123] Considering an exemplary application scenario where the preset trigger time of a data interface falls within both a special period and a user-defined time range, how should data monitoring rules be selected to monitor the data interface? For example, a special period might include agreed-upon promotional days, such as Singles' Day (November 11th). The user-defined time range might include 8:00 AM to 9:00 AM daily. If the current time is 8:00 AM on Singles' Day, it means that the current time falls within both the special period (i.e., the first time range) and the user-defined time range (i.e., the second time range). In this case, the first data monitoring rule corresponding to the preset trigger time (i.e., 8:00 AM) should be selected from among multiple first data monitoring rules corresponding to the first time range and then used as the third target data monitoring rule.

[0124] In some specific embodiments, such as Figure 8 As shown, the method further includes steps S801-S802:

[0125] Step S801: Launch the preset page at preset time intervals.

[0126] In this step, the preset page includes multiple page elements. The preset page can be a page specified by the user. The preset time interval can be set according to the actual situation, such as 1 hour, but no specific limitation is made here.

[0127] Step S802: If the preset page encounters a display error during loading, the display error is sent to the client of the corresponding administrator.

[0128] In this step, if the preset page encounters display anomalies such as page element data loading error, page element rendering error, or page loading speed exceeding the preset loading speed during page loading, the display anomaly is sent to the corresponding administrator's client so that the administrator can perform the appropriate anomaly handling operation based on the display anomaly.

[0129] In some specific embodiments, such as Figure 4As shown, different data monitoring rules can be adopted to monitor different data interfaces at different times. For example, rule 1 and rule 2 monitor interface 1 daily through domain 1 of interface 1; rule 3 and rule 5 monitor interface 2 weekly through domain 2 of interface 2; and rule 6 monitors interface 3 monthly through domain 3 of interface 3.

[0130] Corresponding to the above implementation methods for data monitoring, this disclosure also provides a data monitoring device for performing the above-described data monitoring methods. Figures 1 to 8 The data monitoring method is illustrated in any of the suggested embodiments. Figure 9 As shown, the data monitoring device includes:

[0131] The data monitoring rule filtering module is used to filter at least one data monitoring rule corresponding to the preset trigger time from multiple data monitoring rules corresponding to the data interface when the current time meets the preset trigger time requirement of the data interface; the multiple data interfaces are the interfaces through which the data warehouse imports updated data into the database;

[0132] The data monitoring module is used to monitor the data interface according to any one of the at least one data monitoring rules.

[0133] Optionally, the data monitoring module is further configured to: determine a verification parameter corresponding to the data monitoring rule; send the verification parameter to the data interface so that the data interface returns response data associated with the verification parameter; if the response data does not meet the data monitoring rule, then determine that there is abnormal data in the data interface.

[0134] Optionally, the data monitoring rules include rules that monitor at least one of the following: data type, data value range, data arrangement order, and data business requirements.

[0135] Optionally, the data monitoring module is further configured to: determine that the data interface has abnormal data if the data type of the response data does not belong to a preset data type; determine that the data interface has abnormal data if the value of the response data does not belong to a preset value range; determine that the data interface has abnormal data if multiple sub-response data in the response data do not meet a preset arrangement order; and determine that the data interface has abnormal data if the response data does not meet preset business requirements.

[0136] Optionally, the device also includes:

[0137] An abnormal function module determination module is used to determine the abnormal function module corresponding to the data interface according to a first mapping relationship after determining that there is abnormal data in the data interface; the first mapping relationship includes multiple function modules of the application and at least one data interface corresponding to each function module.

[0138] An alarm module is used to send alarm information to the client of the person in charge of the abnormal function module; the alarm information includes the abnormal data and the data interface.

[0139] Optionally, the device further includes: a data monitoring task configuration module, used to receive configuration information corresponding to any data interface; and configure a data monitoring task for the data interface according to the configuration information; wherein the data monitoring task includes at least one of the following: the interface address of each data interface, at least one data monitoring rule associated with each data interface, a preset trigger time associated with each data monitoring rule, and a verification parameter associated with each data monitoring rule.

[0140] Optionally, the device also includes:

[0141] The data update volume monitoring module is used to monitor the data update volume of multiple functional modules of the application;

[0142] The first time range determination module is used to determine the time range in which the data update volume of any one of the plurality of functional modules is greater than a preset threshold as the first time range; the first time range is used to monitor the data of each data interface of the functional module using the first data monitoring rule.

[0143] The second time range determination module is used to define the time range in which the data update volume of the functional module is equal to or less than the preset threshold as the second time range; the second time range is used to monitor the data of each data interface of the functional module using the second data monitoring rules.

[0144] Optionally, the data monitoring rule filtering module is further configured to: if the preset trigger time is within the first time range, filter out a first target data monitoring rule corresponding to the preset trigger time from the at least one first data monitoring rule; if the preset trigger time is outside the first time range, filter out a second target data monitoring rule corresponding to the preset trigger time from the at least one second data monitoring rule.

[0145] Optionally, the data monitoring rule filtering module is further configured to: determine whether the preset trigger time is within the second time range when the preset trigger time is outside the first time range; if the preset trigger time is within the second time range, then filter out the second target data monitoring rule corresponding to the preset trigger time from the at least one second data monitoring rule.

[0146] Optionally, the data monitoring rule filtering module is further configured to: when the preset trigger time is simultaneously located within the first time range and the second time range, filter out a third target data monitoring rule corresponding to the preset trigger time from the at least one first data monitoring rule.

[0147] Optionally, the device also includes:

[0148] A page launch module is used to launch a preset page at preset time intervals, the preset page including multiple page elements;

[0149] The display error reporting module is used to send the display error to the corresponding administrator's client if a display error occurs during the loading process of the preset page; wherein, the display error includes page element data loading error, page element rendering error, and page loading speed exceeding the preset loading speed.

[0150] The data monitoring device and the data monitoring method provided in the above embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0151] This disclosure also provides a computer device for performing the above-described data monitoring method. Please refer to... Figure 10 This illustrates a schematic diagram of a computer device provided by some embodiments of the present disclosure. For example... Figure 10 As shown, the computer device 10 includes: a processor 1000, a memory 1001, a bus 1002, and a communication interface 1003. The processor 1000, the communication interface 1003, and the memory 1001 are connected via the bus 1002. The memory 1001 stores a computer program that can run on the processor 1000. When the processor 1000 runs the computer program, it executes the aforementioned provisions of this disclosure. Figures 1 to 8 The data monitoring method provided by any of the illustrated embodiments.

[0152] The memory 1001 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 1003 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0153] Bus 1002 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 1001 is used to store programs, and the processor 1000 executes the programs after receiving execution instructions. Figures 1 to 8 The data monitoring method disclosed in any of the illustrated embodiments can be applied to the processor 1000, or implemented by the processor 1000.

[0154] The processor 1000 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 1000 or by instructions in software form. The processor 1000 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software functional modules in the decoding processor. The software functional modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1001. Processor 1000 reads the information in memory 1001 and, in conjunction with its hardware, completes the steps of the above method.

[0155] The computer equipment and data monitoring method provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0156] This disclosure also provides a computer-readable storage medium corresponding to the data monitoring method provided in the foregoing embodiments. Please refer to... Figure 11 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the data monitoring method provided in any of the foregoing embodiments.

[0157] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0158] The computer-readable storage medium provided in the above embodiments of this disclosure and the data monitoring method provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0159] It should be noted that:

[0160] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0161] Similarly, it should be understood that, in order to simplify this disclosure and aid in understanding one or more of the various inventive aspects, in the foregoing description of exemplary embodiments of this disclosure, various features of this disclosure are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting the following: that is, the claimed disclosure requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this disclosure.

[0162] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this disclosure and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0163] The above description is merely a preferred embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A data monitoring method, characterized in that, The method includes: For any one of the multiple data interfaces corresponding to the database that updates application data, when the preset trigger time requirement of the data interface is met at the current time, at least one data monitoring rule corresponding to the preset trigger time is selected from the multiple data monitoring rules corresponding to the data interface; the multiple data interfaces are the interfaces through which the data warehouse imports updated data into the database; For any one of the at least one data monitoring rules, data monitoring is performed on the data interface according to the data monitoring rule.

2. The method according to claim 1, characterized in that, According to the data monitoring rules, data monitoring is performed on the data interface, including: Determine the verification parameters corresponding to the data monitoring rules; The verification parameters are sent to the data interface so that the data interface returns response data associated with the verification parameters; If the response data does not meet the data monitoring rules, then it is determined that there is abnormal data in the data interface.

3. The method according to claim 2, characterized in that, The data monitoring rules include rules that monitor at least one of the following: data type, data value range, data arrangement order, and data business requirements.

4. The method according to claim 3, characterized in that, If the response data does not meet the data monitoring rules, then the data interface is determined to contain abnormal data, including any of the following: If the data type of the response data does not belong to the preset data type, then it is determined that there is abnormal data in the data interface; If the value of the response data does not fall within the preset value range, it is determined that there is abnormal data in the data interface; If multiple sub-response data in the response data do not meet the preset arrangement order, it is determined that there is abnormal data in the data interface; If the response data does not meet the preset business requirements, it is determined that there is abnormal data in the data interface.

5. The method according to claim 3 or 4, characterized in that, After determining that the data interface contains abnormal data, the method further includes: The abnormal function module corresponding to the data interface is determined according to the first mapping relationship; the first mapping relationship includes multiple function modules of the application and at least one data interface corresponding to each function module. The alarm information is sent to the client of the person in charge of the abnormal function module; the alarm information includes the abnormal data and the data interface.

6. The method according to claim 1 or 2, characterized in that, The method further includes: For any data interface, receive the configuration information corresponding to the data interface; Configure the data monitoring task for the data interface according to the configuration information; The data monitoring task includes at least one of the following: the interface address of each data interface, at least one data monitoring rule associated with each data interface, a preset trigger time associated with each data monitoring rule, and a verification parameter associated with each data monitoring rule.

7. The method according to claim 1 or 2, characterized in that, The method further includes: Monitor the data update volume of multiple functional modules of the application; For any one of the multiple functional modules, the time range during which the data update volume of the functional module is greater than a preset threshold is defined as the first time range; the first time range is used to monitor the data of each data interface of the functional module using the first data monitoring rule; The time range within which the data update volume of the functional module is equal to or less than the preset threshold is defined as the second time range; the second time range is used to monitor the data of each data interface of the functional module using the second data monitoring rules.

8. The method according to claim 7, characterized in that, The preset trigger time requirement is a first time range; the multiple data monitoring rules include at least one first data monitoring rule and at least one second data monitoring rule; when the preset trigger time requirement of the data interface is met at the current time, at least one data monitoring rule corresponding to the preset trigger time is selected from the multiple data monitoring rules corresponding to the data interface, including: If the preset trigger time is within the first time range, then a first target data monitoring rule corresponding to the preset trigger time is selected from the at least one first data monitoring rule; If the preset trigger time is outside the first time range, then a second target data monitoring rule corresponding to the preset trigger time is selected from the at least one second data monitoring rule.

9. The method according to claim 8, characterized in that, The step of filtering out the second target data monitoring rule corresponding to the preset trigger time from the at least one second data monitoring rule includes: When the preset trigger time is outside the first time range, determine whether the preset trigger time is within the second time range; If the preset trigger time is within the second time range, then a second target data monitoring rule corresponding to the preset trigger time is selected from the at least one second data monitoring rule.

10. The method according to claim 9, characterized in that, The step of filtering out the first target data monitoring rule corresponding to the preset trigger time from the at least one first data monitoring rule includes: When the preset trigger time is simultaneously within the first time range and the second time range, a third target data monitoring rule corresponding to the preset trigger time is selected from the at least one first data monitoring rule.

11. The method according to claim 1 or 2, characterized in that, The method further includes: A preset page is launched at preset time intervals, and the preset page includes multiple page elements; If a display error occurs during the loading process of the preset page, the display error is sent to the client of the corresponding administrator; wherein, the display error includes page element data loading error, page element rendering error, and page loading speed exceeding the preset loading speed.

12. A data monitoring device, characterized in that, The device includes: The data monitoring rule filtering module is used to filter at least one data monitoring rule corresponding to the preset trigger time from multiple data monitoring rules corresponding to the data interface when the current time meets the preset trigger time requirement of the data interface; the multiple data interfaces are the interfaces through which the data warehouse imports updated data into the database; The data monitoring module is used to monitor the data interface according to any one of the at least one data monitoring rules.

13. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the data monitoring method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the data monitoring method according to any one of claims 1 to 11.

Images