A data monitoring method and device

Abstract

The application relates to the field of Fintech (Fintech), and discloses a data monitoring method and device, which queries a data table where a to-be-monitored field is located; the to-be-monitored field is obtained through user configuration; for any data table containing the to-be-monitored field, to-be-monitored data corresponding to the to-be-monitored field is acquired from the data table; a monitoring rule corresponding to the to-be-monitored field is acquired from a preset corresponding relationship between to-be-monitored fields and monitoring rules; and whether each to-be-monitored data is abnormal is determined according to the monitoring rule corresponding to the to-be-monitored field. The method can automatically determine whether to-be-monitored data is abnormal, and the monitored data is all data tables in a database, so the method also has universality.

Description

Technical Field

[0001] This application relates to the field of financial technology (Fintech), and more particularly to a data monitoring method and apparatus. Background Technology

[0002] With the development of computer technology, more and more technologies (such as big data, cloud computing, or blockchain) are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology. In the context of big data technology, it is necessary to store business data based on databases.

[0003] When the business volume is low, that is, when there is not much business data stored in the database, it is possible to check whether the business data is abnormal by manual means. However, this method of manually checking whether the business data is abnormal becomes very inefficient when the business volume increases, and cannot meet the needs of checking the business data well.

[0004] To address this, monitoring code with specific rules can be developed and run to monitor the business data that needs to be checked. However, this method of checking for anomalies in business data by running monitoring code only works for existing data tables and fields. When the data tables in the system database are updated, the monitoring code also needs to be updated synchronously, resulting in high maintenance costs. In addition, due to differences in developers' development habits, the same object may be presented in different business systems in a way that is not entirely consistent, such as different field names. Therefore, monitoring code developed for a specific object that needs to be checked is difficult to use directly in different business systems; that is, it needs to be adapted and modified before it can be used.

[0005] In summary, there is an urgent need for a universal and automated method for checking data. Summary of the Invention

[0006] This application provides a data monitoring method and apparatus for automatically checking the correctness of data in all tables of any object that needs to be checked in the system database.

[0007] In a first aspect, embodiments of this application provide a data monitoring method, which includes: querying a data table containing a field to be monitored; the field to be monitored is obtained through user configuration; for any data table containing the field to be monitored, obtaining the data to be monitored corresponding to the field to be monitored from the data table; obtaining the monitoring rule corresponding to the field to be monitored from a preset correspondence between the field to be monitored and the monitoring rule; and determining whether each piece of data to be monitored is abnormal based on the monitoring rule corresponding to the field to be monitored.

[0008] Based on this scheme, for fields that need to be monitored to check data correctness, the monitoring fields are configured by the user. When querying the data table containing the monitored field, the corresponding monitored data can be obtained. Then, by matching monitoring rules, it can be determined whether the monitored data is abnormal. This method can automatically determine whether the monitored data is abnormal, and since the monitored data is located in all data tables in the database, this method also has universality.

[0009] In one possible implementation method, determining whether each piece of data to be monitored is abnormal according to the monitoring rules corresponding to the field to be monitored includes: for any piece of data to be monitored, determining whether the data to be monitored exists in the local cache; if it does not exist, then detecting the data to be monitored according to the monitoring rules corresponding to the field to be monitored to obtain a monitoring result; and storing the monitoring result of the data to be monitored in the local cache.

[0010] Based on this scheme, for data to be monitored, it is matched against the local cache. If the data exists in the local cache, a conclusion on whether the data is normal can be quickly reached. If the data is not stored in the local cache, it indicates that the data is new and needs to be tested using the corresponding monitoring rules to obtain monitoring results. Finally, by storing the monitoring results in the local cache, it helps to quickly determine whether the data is normal during subsequent testing by matching the monitoring results in the local cache. This method improves the efficiency of monitoring data in subsequent data testing processes by storing the monitoring results of the data to be monitored in the local cache.

[0011] In one possible implementation, storing the monitoring result of the data to be monitored into the local cache includes: determining the hash value of the data to be monitored; storing the hash value of the data to be monitored and the monitoring result of the data to be monitored into the local cache as key-value pairs; determining whether the data to be monitored exists in the local cache includes: determining the hash value of the data to be monitored; and determining whether the data to be monitored exists in the local cache based on the hash value of the data to be monitored.

[0012] Based on this scheme, after obtaining the corresponding monitoring results by matching the data to be monitored with monitoring rules, the hash value of the data to be monitored can be determined, and the hash value and the corresponding monitoring results can be stored in the local cache as key-value pairs. Therefore, in the subsequent data detection process, for a piece of data to be monitored, by determining its hash value and then matching the hash value with all hash values ​​in the local cache, it can be determined whether the data to be monitored exists in the local cache. If it is determined that it exists, a conclusion on whether the data to be monitored is normal data can be quickly output.

[0013] In one possible implementation method, determining whether the data to be monitored exists in the local cache and storing the monitoring results of the data to be monitored in the local cache includes: determining whether the data to be monitored exists in the local cache by using the @Cacheable annotation method and storing the monitoring results of the data to be monitored in the local cache.

[0014] Based on this approach, when determining whether the data to be monitored exists in the local cache, it can be determined by using the @Cacheable annotation, and the monitoring results can be stored in the local cache.

[0015] In one possible implementation, the field to be monitored includes different field names for the same object; querying the data table where the field to be monitored is located includes: for any field name, determining the table name of the data table containing the field name by querying the data table under the information_schema schema.

[0016] Based on this scheme, querying data tables in the information_schema mode can retrieve all data tables in the current database that contain the field names to be queried, including both existing and new data tables, thus achieving the goal of querying all data tables in the database.

[0017] In one possible implementation method, obtaining the data to be monitored corresponding to the field to be monitored from the data table includes: constructing an SQL query statement based on the table name and the field name of the data table using dynamic SQL functionality; and obtaining the data to be monitored corresponding to the field name from the data table based on the SQL query statement.

[0018] Based on this scheme and leveraging the dynamic SQL functionality, for any set of data tables with corresponding relationships, a corresponding SQL query statement can be dynamically constructed. By running the SQL statement, the monitored data corresponding to the indicated field can be retrieved from the data table specified by the SQL statement. This method, based on dynamic SQL functionality, can dynamically generate the SQL statement to retrieve the monitored data when any set of data tables with corresponding relationships exists, without hardcoding the query statement, thus improving the efficiency of retrieving the monitored data.

[0019] In one possible implementation method, obtaining the monitoring rule corresponding to the field to be monitored includes: obtaining the monitoring rule corresponding to the field to be monitored by using the getBean method.

[0020] Based on this solution, for a field to be monitored, its corresponding monitoring rules can be obtained through the getBean method; and, if the monitoring rules are adjusted during subsequent data detection, the update requirements for the monitoring rules can be met by modifying the configuration, thus improving the flexibility of the program.

[0021] Secondly, embodiments of this application provide a data monitoring device, which includes: a data table query unit for querying a data table containing a field to be monitored; the field to be monitored is obtained through user configuration; a data acquisition unit for acquiring data to be monitored corresponding to any data table containing the field to be monitored; a monitoring rule determination unit for acquiring the monitoring rule corresponding to the field to be monitored from a preset correspondence between the field to be monitored and the monitoring rule; and a judgment unit for determining whether each piece of data to be monitored is abnormal based on the monitoring rule corresponding to the field to be monitored.

[0022] Based on this scheme, for fields that need to be monitored to check data correctness, the monitoring fields are configured by the user. When querying the data table containing the monitored field, the corresponding monitored data can be obtained. Then, by matching monitoring rules, it can be determined whether the monitored data is abnormal. This method can automatically determine whether the monitored data is abnormal, and since the monitored data is located in all data tables in the database, this method also has universality.

[0023] In one possible implementation method, the determination unit is specifically used to: determine whether any of the monitored data exists in the local cache; if it does not exist, detect the monitored data according to the monitoring rules corresponding to the monitored field to obtain the monitoring result; and store the monitoring result of the monitored data into the local cache.

[0024] Based on this scheme, for data to be monitored, it is matched against the local cache. If the data exists in the local cache, a conclusion on whether the data is normal can be quickly reached. If the data is not stored in the local cache, it indicates that the data is new and needs to be tested using the corresponding monitoring rules to obtain monitoring results. Finally, by storing the monitoring results in the local cache, it helps to quickly determine whether the data is normal during subsequent testing by matching the monitoring results in the local cache. This method improves the efficiency of monitoring data in subsequent data testing processes by storing the monitoring results of the data to be monitored in the local cache.

[0025] In one possible implementation, the determination unit is further configured to: determine the hash value of the data to be monitored; store the hash value of the data to be monitored and the monitoring result of the data to be monitored in the local cache as key-value pairs; determine the hash value of the data to be monitored; and determine whether the data to be monitored exists in the local cache based on the hash value of the data to be monitored.

[0026] Based on this scheme, after obtaining the corresponding monitoring results by matching the data to be monitored with monitoring rules, the hash value of the data to be monitored can be determined, and the hash value and the corresponding monitoring results can be stored in the local cache as key-value pairs. Therefore, in the subsequent data detection process, for a piece of data to be monitored, by determining its hash value and then matching the hash value with all hash values ​​in the local cache, it can be determined whether the data to be monitored exists in the local cache. If it is determined that it exists, a conclusion on whether the data to be monitored is normal data can be quickly output.

[0027] In one possible implementation, the determination unit is further configured to: determine whether the data to be monitored exists in the local cache by using the @Cacheable annotation method, and store the monitoring result of the data to be monitored into the local cache.

[0028] Based on this approach, when determining whether the data to be monitored exists in the local cache, it can be determined by using the @Cacheable annotation, and the monitoring results can be stored in the local cache.

[0029] In one possible implementation, the field to be monitored includes different field names for the same object; the data table query unit is specifically used to: for any field name, determine the table name of the data table containing the field name by querying the data table under the information_schema schema.

[0030] Based on this scheme, querying data tables in the information_schema mode can retrieve all data tables in the current database that contain the field names to be queried, including both existing and new data tables, thus achieving the goal of querying all data tables in the database.

[0031] In one possible implementation method, the data acquisition unit is specifically used to: construct an SQL query statement based on the table name and field name of the data table using dynamic SQL functionality; and obtain the data to be monitored corresponding to the field name from the data table based on the SQL query statement.

[0032] Based on this scheme and leveraging the dynamic SQL functionality, for any set of data tables with corresponding relationships, a corresponding SQL query statement can be dynamically constructed. By running the SQL statement, the monitored data corresponding to the indicated field can be retrieved from the data table specified by the SQL statement. This method, based on dynamic SQL functionality, can dynamically generate the SQL statement to retrieve the monitored data when any set of data tables with corresponding relationships exists, without hardcoding the query statement, thus improving the efficiency of retrieving the monitored data.

[0033] In one possible implementation, the monitoring rule determination unit is specifically used to: obtain the monitoring rule corresponding to the field to be monitored by using the getBean method.

[0034] Based on this solution, for a field to be monitored, its corresponding monitoring rules can be obtained through the getBean method; and, if the monitoring rules are adjusted during subsequent data detection, the update requirements for the monitoring rules can be met by modifying the configuration, thus improving the flexibility of the program.

[0035] Thirdly, embodiments of this application provide a computing device, including:

[0036] Memory, used to store program instructions;

[0037] A processor is configured to invoke program instructions stored in the memory and execute any implementation method of the first aspect according to the obtained program.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform any implementation method as described in the first aspect. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A data monitoring method provided in this application embodiment;

[0041] Figure 2 This application provides an embodiment of a schematic diagram illustrating the result of obtaining data tables under the information_schema schema.

[0042] Figure 3 A schematic diagram illustrating a dynamic SQL concatenation query statement provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram illustrating the process of acquiring data to be monitored, provided in an embodiment of this application.

[0044] Figure 5 A data monitoring device provided in the embodiments of this application;

[0045] Figure 6 A computing device provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] Currently, the inspection of business data stored in databases is mainly carried out manually, which is obviously extremely inefficient when business data continues to increase. Another approach is to develop and run monitoring code with specific rules to monitor the business data to be inspected; however, this method lacks versatility, cannot be applied to all data tables in the database, and monitoring code for the same object is difficult to directly apply to different business systems.

[0048] To address the aforementioned technical problems, embodiments of this application provide a data monitoring method, such as... Figure 1 As shown, the method includes the following steps:

[0049] Step 101: Query the data table containing the field to be monitored; the field to be monitored is obtained through user configuration.

[0050] In this step, for a company, different business departments often have overlapping records of user information. For example, different business departments may simultaneously record a user's mobile phone number, or they may simultaneously record a user's ID card number. This embodiment of the application uses the example of different business departments recording user ID card numbers for illustration. During the operation of a company, since the technical personnel responsible for different business departments are not the same, it is difficult to ensure the consistency of the representation of the same object in the data tables corresponding to each business department. For example, the user ID card number can be represented in different data tables, such as being recorded as 'legal_cert_id', 'cert_id', etc., i.e., fields. For the user ID card number object, the representation methods of each business department within the company can be collected in advance, and then the various representation methods of the user ID card number object can be configured. For example, the 'legal_cert_id' field and the 'cert_id' field can be pre-configured to represent the user ID card number. For example, the object of a user's mobile phone number can be represented in different ways in different data tables. It can be recorded as 'phone', 'phone_no', 'mobile', etc., i.e., as a field. Based on the same configuration logic as the user's ID number, for the object of a user's mobile phone number, we can pre-collect information from various business departments within the enterprise regarding how it should be represented, and then configure the various representations of the user's mobile phone number object. For example, we can pre-configure the 'phone' field, 'phone_no' field, and 'mobile' field to represent the user's mobile phone number.

[0051] This embodiment uses monitoring a user's ID number as an example. The field to be monitored is how the user's ID number is represented in the data tables corresponding to different business departments within the enterprise. Based on the aforementioned example, when monitoring the user's ID number, the fields to be monitored may include the user's pre-configured 'legal_cert_id' and 'cert_id' fields. After obtaining the fields to be monitored, data tables containing the monitored fields can be queried, such as querying the data table containing the 'legal_cert_id' field and querying the data table containing the 'cert_id' field.

[0052] Step 102: For any data table containing the field to be monitored, obtain the data to be monitored corresponding to the field to be monitored from the data table.

[0053] In this step, based on the data table with the fields to be monitored that has been queried in step 101, the data corresponding to the fields to be monitored can be obtained from the data table, that is, the data to be monitored can be obtained.

[0054] Continuing with the previous example of monitoring user ID numbers, for instance, if a query for a user ID number is performed based on the 'legal_cert_id' field, and both table A and table B in the system database have this field, then the data to be monitored corresponding to the 'legal_cert_id' field can be obtained from both table A and table B. Similarly, if a query for a user ID number is performed based on the 'cert_id' field, and only table C in the system database has this field, then the data to be monitored corresponding to the 'cert_id' field can be obtained from table C.

[0055] Step 103: Obtain the monitoring rule corresponding to the field to be monitored from the preset correspondence between the field to be monitored and the monitoring rule.

[0056] Because the objects of interest differ, the monitoring rules for monitoring these objects will also differ. Specifically, the monitoring rules will vary depending on the fields to be monitored. For example, when the object of interest is a company ID number, and monitoring is required for all fields of that object, the monitoring rule might be whether the format is correct. When the object of interest is a user ID number, and monitoring is required for all fields of that object, the monitoring rule might be whether the length is correct. When the object of interest is a loan amount / interest rate / term, and monitoring is required for all fields of that object, the monitoring rule might be whether it conforms to the prescribed regulatory range. When the object of interest is a product number, and monitoring is required for all fields of that object, the monitoring rule might be whether it conforms to the enumerated range. The above monitoring of the fields to be monitored can be called a general check, which may include checks on format, length, size, enumeration, etc.

[0057] In this step, when the object of focus is the user's ID card number, i.e., the fields to be monitored are the 'legal_cert_id' and 'cert_id' fields, the corresponding monitoring rule can be to verify whether the ID card number length is between 15 and 18 digits. Furthermore, different monitoring rule identifiers can be set for different monitoring rules to distinguish them. For example, in this embodiment, any monitoring rule and its corresponding monitoring rule identifier can be pre-configured in the rule configuration table for use in the data monitoring process. For instance, the monitoring rule identifier for each field to be monitored for the user's ID card number can be set to: ruleId = 00001. Similarly, when the object of focus is the user's mobile phone number, i.e., the fields to be monitored are the 'phone', 'phone_no', and 'mobile' fields, the corresponding monitoring rule can be to verify whether the mobile phone number is purely numeric, and this monitoring rule is recorded in the rule configuration table; furthermore, the monitoring rule identifier for each field to be monitored for the user's mobile phone number can be set to: ruleId = 00002 in the rule configuration table. It should be noted that the setup process for the monitoring rule identifiers for other objects of interest will not be listed here.

[0058] Step 104: Determine whether each piece of data to be monitored is abnormal based on the monitoring rules corresponding to the fields to be monitored.

[0059] In this step, based on the monitoring rules obtained in step 103, the data to be monitored obtained in step 102 can be checked to determine whether any abnormalities have occurred in the data to be monitored.

[0060] Continuing with the previous example of monitoring user ID numbers, since this monitoring focuses on the user ID number itself, the corresponding monitoring rule identifier is ruleId = 00001. Therefore, by querying the preset rule configuration table and based on the correspondence between ruleId = 00001 and the monitoring rules, it can be determined that this time, it is necessary to verify whether each user ID number corresponding to the 'legal_cert_id' field in data table A, the 'legal_cert_id' field in data table B, and the 'cert_id' field in data table C is 15-18 characters long. The verification result may include:

[0061] Case 1: If the length of the user's ID number is 15-18 digits, then the record of this user's ID number is considered normal, and the verification of the next user's ID number is performed.

[0062] Scenario 2: If the user's ID number is not 15-18 digits long, the record for that user's ID number is considered abnormal. In this case, business personnel can be notified for investigation based on preset alarm methods. These alarm methods can be configured by the user and include various methods such as email, SMS, and WeChat.

[0063] Based on this scheme, for fields that need to be monitored to check data correctness, the monitoring fields are configured by the user. When querying the data table containing the monitored field, the corresponding monitored data can be obtained. Then, by matching monitoring rules, it can be determined whether the monitored data is abnormal. This method can automatically determine whether the monitored data is abnormal, and since the monitored data is located in all data tables in the database, this method also has universality.

[0064] The following will provide a detailed explanation of some of the steps mentioned above, with examples.

[0065] In one implementation of step 101 above, the field to be monitored includes different field names for the same object; querying the data table where the field to be monitored is located includes: for any field name, determining the table name of the data table containing the field name by querying the data table under the information_schema schema.

[0066] In this embodiment, the monitoring time points for each data point corresponding to the field to be monitored can be set. For example, monitoring can be performed every 1 minute or every 10 minutes. The specific time can be set according to the actual needs of the technical personnel. After the monitoring system starts, the monitoring scope configured by the user is read. For example, if the user's configured monitoring scope is to monitor the user's ID number, and the user's ID number can be represented as 'legal_cert_id' and 'cert_id', then it means that the fields to be monitored can be read as 'legal_cert_id' and 'cert_id'.

[0067] The monitoring system can dynamically retrieve all relevant data tables, including the most recently added tables, by querying the COLUMNS table in the MySQL database under the information_schema schema, based on the field to be monitored. For example... Figure 2The diagram shown illustrates the results of retrieving data tables under the information_schema schema according to an embodiment of this application. Specifically, the data tables named "enterprise_info" and "white_info" can be retrieved based on the 'legal_cert_id' field, and the data table named "enterprise_staff_bind" can be retrieved based on the 'cert_id' field.

[0068] It should be noted that the `information_schema` schema is a database included with the MySQL system, providing methods for accessing database metadata. The `COLUMNS` table, in the `information_schema` schema, stores all entity objects in the database. It allows you to retrieve the names of all tables in the current database, including `TABLE_NAME` and the `COLUMN_NAME` field.

[0069] In some embodiments of this application, obtaining the data to be monitored corresponding to the field to be monitored from the data table includes: constructing an SQL query statement based on the table name and the field name of the data table using dynamic SQL functionality; and obtaining the data to be monitored corresponding to the field name from the data table based on the SQL query statement.

[0070] Continuing with the previous example, based on the 'legal_cert_id' field, we can query tables named "enterprise_info" and "white_info" respectively, and based on the 'cert_id' field, we can query a table named "enterprise_staff_bind". Then, using the Mybatis framework's dynamic SQL concatenation query, we can retrieve the records corresponding to the 'legal_cert_id' field in the "enterprise_info" table, the records corresponding to the 'legal_cert_id' field in the "white_info" table, and the records corresponding to the 'cert_id' field in the "enterprise_staff_bind" table, thus obtaining the data to be monitored. Figure 3The diagram illustrates a dynamic SQL concatenation query statement provided in an embodiment of this application. The variable ${}, relating to the aforementioned three sets of table field and table name correspondences—including <'legal_cert_id', enterprise_info>, <'cert_id', enterprise_staff_bind>, and <'legal_cert_id', white_info>—can be replaced one by one with each set of table fields and table names. Thus, for each set of corresponding table fields and table names, a set of data to be monitored can be obtained, and this data can then be output to a file to be verified. For example... Figure 4 The diagram illustrates a process for acquiring monitored data according to an embodiment of this application. Specifically, the 'legal_cert_id' field in the "enterprise_info" table, the 'cert_id' field in the "enterprise_staff_bind" table, and the 'legal_cert_id' field in the "white_info" table are replaced in the 'SELECT ${columns} FROM ${tablename}' statement, resulting in three data tables. The three data tables are shown below. Figure 4 The Excel spreadsheet is shown in the image.

[0071] In one implementation of step 104 above, determining whether each piece of data to be monitored is abnormal according to the monitoring rules corresponding to the field to be monitored includes: for any piece of data to be monitored, determining whether the data to be monitored exists in the local cache; if it does not exist, then detecting the data to be monitored according to the monitoring rules corresponding to the field to be monitored to obtain a monitoring result; and storing the monitoring result of the data to be monitored in the local cache.

[0072] Based on the aforementioned example, records corresponding to the 'legal_cert_id' field in the "enterprise_info" data table, the 'legal_cert_id' field in the "white_info" data table, and the 'cert_id' field in the "enterprise_staff_bind" data table were obtained—three sets of data to be monitored. Each set of data was compiled into a file to be verified, and the data in the files to be verified will be monitored subsequently. In this embodiment, the files to be verified corresponding to the three sets of data to be monitored can be designated as File A, File B, and File C, respectively. Imagine: File A contains 10 million ID card numbers; File B contains 8 million ID card numbers; and File C contains 5 million ID card numbers.

[0073] The current method for monitoring the data to be monitored is as follows: for any given file, the user ID numbers are parsed and obtained. Then, each user ID number in the file is monitored using pre-written ID number monitoring rules, and the corresponding monitoring results are output. Clearly, for the three sets of data to be monitored, totaling 23 million user ID numbers, the ID number monitoring rules need to be executed 23 million times.

[0074] The current methods for monitoring data, as described above, have at least two problems:

[0075] Issue 1: Performance problem. A user's ID number may exist in multiple data tables within the same system. In scenarios with a large amount of duplicate data, such as repeatedly executing monitoring rules, it will cause a significant waste of system resources.

[0076] Question 2: Monitoring is performed using pre-written ID number monitoring rules based on the user's ID number. However, this approach is not flexible enough if the monitoring rules are adjusted or expanded in the future.

[0077] To address issue 1 above, this embodiment utilizes a method of caching monitoring results. Specifically, the monitoring data to be monitored, generated for the first time using pre-written monitoring rules, is cached along with its monitoring results. In subsequent data monitoring processes, for any piece of data to be monitored, it is first compared with the cached monitoring results. If the data to be monitored can be matched in the cache, the monitoring results for that data are directly output. If the data to be monitored cannot be matched in the cache, it indicates that the data to be monitored is entirely new relative to the cached data. In this case, the pre-written monitoring rules can be used to obtain monitoring results, and the data to be monitored and its monitoring results are then entered into the cache.

[0078] In some embodiments of this application, storing the monitoring result of the data to be monitored into the local cache includes: determining the hash value of the data to be monitored; storing the hash value of the data to be monitored and the monitoring result of the data to be monitored into the local cache as key-value pairs; determining whether the data to be monitored exists in the local cache includes: determining the hash value of the data to be monitored; determining whether the data to be monitored exists in the local cache based on the hash value of the data to be monitored.

[0079] In some embodiments of this application, determining whether the data to be monitored exists in the local cache and storing the monitoring results of the data to be monitored in the local cache includes: determining whether the data to be monitored exists in the local cache by using the @Cacheable annotation method and storing the monitoring results of the data to be monitored in the local cache.

[0080] For example, in this embodiment of the application, Caffeine can be used as a local cache. Caffeine is a high-performance Java caching library, and the cached data structure is a MAP structure, i.e.<key,value> The format is as follows: For any piece of data to be monitored, the hash value of the data to be monitored and the corresponding monitoring rule identifier is used as the key value, and the monitoring result obtained after processing the data to be monitored by the monitoring rule is used as the value value.

[0081] Specifically, this can be illustrated using the examples of files A, B, and C from the previous section:

[0082] First, let's say the first file to be parsed is file A, obtaining each user's ID number. For any given user ID number, determine the hash value between the user ID number and the monitoring rule identifier. Using the `@Cacheable` annotation in the Spring Boot framework, we can automatically determine if this hash value hits the local cache. Since file A is the first file in this verification, all user ID numbers in file A are appearing for the first time, and obviously, none of them hit the local cache (understandably, the local cache is empty at this point). Therefore, each user ID number in file A needs to be processed by a specific business method. That is, for any user ID number in file A, it needs to be processed using a pre-written monitoring rule, and the user ID number and its monitoring result need to be stored in the local cache. More specifically, the user ID number is hashed with `ruleId = 00001`, and the calculated hash value is stored as the key in the local cache, while the corresponding monitoring result is stored as the value in the local cache.

[0083] Next, let's say the second file being parsed is file B, and we obtain the user's ID number for each record. For any given user ID number, we determine the hash value between that user ID number and the monitoring rule identifier. Using the `@Cacheable` annotation in the Spring Boot framework, we can automatically determine whether this hash value hits the local cache. Since the user ID numbers in file A were monitored before monitoring the user ID numbers in file B, and all user ID numbers in file A along with their corresponding monitoring results have been stored in the local cache, the Spring Boot framework's `@Cacheable` annotation can be used to determine if the user ID numbers in file B are cached. If they are cached, the monitoring results corresponding to the user ID numbers in the local cache are directly used as the monitoring results for that user ID number in file B and output, without needing to use pre-written monitoring rules. If they are not cached, it means that the user ID numbers in file B are completely new compared to the local cache. In this case, pre-written monitoring rules need to be used to output the monitoring results for that user ID number, and the user ID number and its monitoring results are stored in the local cache.

[0084] Finally, the method and process for monitoring user ID numbers in file C can be found in file B, and will not be elaborated here.

[0085] In some implementations of this application, obtaining the monitoring rules corresponding to the field to be monitored includes: obtaining the monitoring rules corresponding to the field to be monitored through the getBean method.

[0086] Continuing with the aforementioned example of monitoring user ID numbers, the business method can query the rule configuration table based on the monitoring rule identifier, such as ruleId=0001, to obtain the rule Service corresponding to ruleId=0001, and then obtain the specific service object through the getBean method of the ApplicationContext object under the Spring framework of Java.

[0087] It should be noted that, regarding issue 2 mentioned above, by configuring the rule configuration table and using the dynamic `getBean` method, if the monitoring rules are adjusted or expanded subsequently, the configuration of the `getBean` method can be dynamically added to meet the changing needs of the monitoring rules. For example, as the population increases, at some point in the future, the existing 15-18 digit user ID numbers may no longer meet the needs of population growth. In this case, 19-22 digit user ID numbers can be used for some newborns to overcome the problem of insufficient user ID number allocation. Therefore, the monitoring rules for monitoring the data need to be updated accordingly, that is, new monitoring rules need to be added on the basis of the existing monitoring rules. In this case, in order to avoid erroneous monitoring of the data, the new monitoring rules can be added while retaining the existing monitoring rules in the rule configuration table, and the configuration of the `getBean` method can be added accordingly. Therefore, this method only requires modifying the configuration when the monitoring rules change, improving the flexibility of the program.

[0088] Based on the same concept, embodiments of this application provide a data monitoring device, such as... Figure 5 As shown, the device includes a data table query unit 501, a data acquisition unit 502, a monitoring rule determination unit 503, and a judgment unit 504.

[0089] The data table query unit 501 is used to query the data table containing the field to be monitored; the field to be monitored is obtained through user configuration.

[0090] The monitoring data acquisition unit 502 is used to acquire the monitoring data corresponding to the monitoring field from any data table containing the monitoring field.

[0091] The monitoring rule determination unit 503 is used to obtain the monitoring rule corresponding to the field to be monitored from the preset correspondence between the field to be monitored and the monitoring rule.

[0092] The determination unit 504 is used to determine whether each piece of data to be monitored is abnormal based on the monitoring rules corresponding to the field to be monitored.

[0093] Furthermore, for this device, the determination unit 504 is specifically used to: determine whether any of the monitored data exists in the local cache; if it does not exist, detect the monitored data according to the monitoring rules corresponding to the monitored field to obtain the monitoring result; and store the monitoring result of the monitored data into the local cache.

[0094] Furthermore, for this device, the determination unit 504 is also used to: determine the hash value of the data to be monitored; store the hash value of the data to be monitored and the monitoring result of the data to be monitored in the local cache as key-value pairs; determine the hash value of the data to be monitored; and determine whether the data to be monitored exists in the local cache based on the hash value of the data to be monitored.

[0095] Furthermore, for this device, the determination unit 504 is also used to: determine whether the data to be monitored exists in the local cache by using the @Cacheable annotation method, and store the monitoring result of the data to be monitored into the local cache.

[0096] Furthermore, for this device, the field to be monitored includes different field names for the same object; the data table query unit 501 is specifically used to: for any field name, determine the table name of the data table containing the field name by querying the data table under the information_schema mode.

[0097] Furthermore, for this device, the data acquisition unit 502 is specifically used to: construct an SQL query statement based on the table name and field name of the data table using the dynamic SQL function; and obtain the data to be monitored corresponding to the field name from the data table based on the SQL query statement.

[0098] Furthermore, for this device, the monitoring rule determination unit 503 is specifically used to: obtain the monitoring rule corresponding to the field to be monitored by using the getBean method.

[0099] This application also provides a computing device, which may specifically be a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), etc. The computing device may include a central processing unit (CPU), memory, input / output devices, etc. Input devices may include a keyboard, mouse, touchscreen, etc., and output devices may include display devices, such as liquid crystal displays (LCDs) and cathode ray tubes (CRTs).

[0100] The memory may include read-only memory (ROM) and random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In the embodiments of this application, the memory may be used to store program instructions for a data monitoring method;

[0101] The processor is used to call program instructions stored in the memory and execute the data monitoring method according to the obtained program.

[0102] like Figure 6 The diagram shown is a schematic representation of a computing device provided in an embodiment of this application. The computing device includes:

[0103] The processor 601, memory 602, transceiver 603, and bus interface 604 are included; wherein the processor 601, memory 602, and transceiver 603 are connected via bus 605.

[0104] The processor 601 is used to read the program in the memory 602 and execute the above-mentioned data monitoring method;

[0105] Processor 601 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. It can also be a hardware chip. The aforementioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0106] The memory 602 is used to store one or more executable programs and can store data used by the processor 601 when performing operations.

[0107] Specifically, the program may include program code, which includes computer operation instructions. Memory 602 may include volatile memory, such as random-access memory (RAM); memory 602 may also include non-volatile memory, such as flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory 602 may also include combinations of the above types of memory.

[0108] Memory 602 stores the following elements: executable modules or data structures, or subsets thereof, or extended sets thereof:

[0109] Operation instructions: This includes various operation instructions used to perform various operations.

[0110] Operating system: includes various system programs used to implement various basic business functions and handle hardware-based tasks.

[0111] The 605 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0112] Bus interface 604 can be a wired communication interface, a wireless bus interface, or a combination thereof. The wired bus interface can be, for example, an Ethernet interface. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. The wireless bus interface can be a WLAN interface.

[0113] This application also provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a data monitoring method.

[0114] Those skilled in the art will understand that embodiments of this application can be provided as methods or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0119] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data monitoring method, characterized by, include: Query the data table containing the field to be monitored; The field to be monitored is obtained through user configuration. The fields to be monitored include different field names for the same object; For any data table containing the field to be monitored, retrieve the data to be monitored corresponding to the field to be monitored from the data table; The monitoring rule corresponding to the field to be monitored is obtained from the preset correspondence between the field to be monitored and the monitoring rule; wherein, different monitoring rules are set with different monitoring rule identifiers, and any monitoring rule and its corresponding monitoring rule identifier are pre-configured in the rule configuration table; Based on the monitoring rules corresponding to the fields to be monitored, determine whether each piece of data to be monitored is abnormal; The step of determining whether each piece of data to be monitored is abnormal based on the monitoring rules corresponding to the field to be monitored includes: For any of the monitored data, determine whether the monitored data exists in the local cache; if it does not exist, then according to the monitoring rules corresponding to the monitored field, detect the monitored data to obtain the monitoring result; and store the monitoring result of the monitored data in the local cache. The local cache is Caffeine, and the cache data structure is as follows:<key,value> The form; The step of storing the monitoring results of the data to be monitored into the local cache includes: For the data to be monitored, the hash value of the data to be monitored and the monitoring rule identifier of the corresponding monitoring rule is used as the key value, and the monitoring result obtained after the data to be monitored is processed by the monitoring rule is used as the value value. The data table containing the field to be monitored includes: For any given field name, the table name containing the field name is determined by querying the data tables under the information_schema schema, where the data tables include the most recently launched data tables.

2. The method as described in claim 1, characterized in that, Determining whether the data to be monitored exists in the local cache includes: Determine the hash value of the data to be monitored; Based on the hash value of the data to be monitored, determine whether the data to be monitored exists in the local cache.

3. The method as described in claim 1, characterized in that, The step of determining whether the data to be monitored exists in the local cache and storing the monitoring results of the data to be monitored into the local cache includes: The @Cacheable annotation method is used to determine whether the data to be monitored exists in the local cache, and to store the monitoring results of the data to be monitored in the local cache.

4. The method as described in claim 1, characterized in that, The step of obtaining the data to be monitored corresponding to the field to be monitored from the data table includes: Using the dynamic SQL function, construct an SQL query statement based on the table name and field names of the data table; Based on the SQL query statement, retrieve the data to be monitored corresponding to the field name from the data table.

5. The method as described in claim 1, characterized in that, The step of obtaining the monitoring rules corresponding to the field to be monitored includes: The monitoring rules corresponding to the field to be monitored are obtained by using the getBean method.

6. A data monitoring device, characterized in that, include: The data table query unit is used to query the data table containing the field to be monitored. The field to be monitored is obtained through user configuration. The fields to be monitored include different field names for the same object; The monitoring data acquisition unit is used to acquire the monitoring data corresponding to the monitoring field from any data table containing the monitoring field. The monitoring rule determination unit is used to obtain the monitoring rule corresponding to the field to be monitored from the preset correspondence between the field to be monitored and the monitoring rule; wherein, different monitoring rules are set with different monitoring rule identifiers, and any monitoring rule and its corresponding monitoring rule identifier are pre-configured in the rule configuration table; The determination unit is used to determine whether each piece of data to be monitored is abnormal based on the monitoring rules corresponding to the field to be monitored. The determination unit is specifically used to determine whether any of the monitored data exists in the local cache; if it does not exist, it detects the monitored data according to the monitoring rules corresponding to the monitored field to obtain the monitoring result; and stores the monitoring result of the monitored data into the local cache. The local cache is Caffeine, and the cache data structure is as follows:<key,value> The form; The determination unit is specifically used to, for the data to be monitored, take the hash value of the data to be monitored and the monitoring rule identifier of the corresponding monitoring rule as the key value, and take the monitoring result obtained after the data to be monitored is processed by the monitoring rule as the value value. The data table query unit is specifically used to determine the name of the data table containing the field name by querying the data table under the information_schema mode for any field name, wherein the data table includes the newly launched data table.

7. A computer device, characterized in that, include: Memory, used to store computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method as described in any one of claims 1-5 according to the obtained program.

8. A computer-readable storage medium, characterized in that, The storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1-5.

Images