Data verification method and platform, equipment, storage medium and program product

Abstract

The invention provides a data verification method and platform, equipment, a storage medium and a program product. The method is applied to a data verification platform. The method can comprise the following steps: acquiring to-be-verified data and a business demand for the to-be-verified data; under the condition that a preset event corresponding to the to-be-verified data is monitored, obtaining a target verification rule matched with the business demand; and verifying the to-be-verified data by using the target verification rule to obtain a verification result of the to-be-verified data. According to the technical scheme, the to-be-verified data can be verified in real time or near real time, and the timeliness of finding abnormal data is improved.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of data verification technology, and in particular, to a data verification method and platform, device, storage medium, and program product. Background Art

[0002] Data verification is crucial to ensure the long-term stability and reliable operation of the entire business system. In the current distributed microservice architecture, business data is prone to errors due to complex business, long data transmission links, and middleware data loss. Therefore, the business system needs to use data verification to ensure the integrity, validity, and accuracy of business data. However, related technologies are often fed back to technical personnel by the business side for investigation and location only after the data anomaly has actually affected the business. This process is time-consuming, resulting in poor timeliness in discovering abnormal data. Summary of the invention

[0003] The present application provides a data verification method and platform, equipment, storage medium, and program product to address deficiencies in related technologies.

[0004] According to a first aspect of one or more embodiments of the present application, a data verification method is provided, the method being applied to a data verification platform, the method comprising:

[0005] Acquire the data to be verified and the business requirements for the data to be verified;

[0006] When a preset event corresponding to the data to be verified is monitored, a target verification rule matching the business requirement is obtained;

[0007] The target verification rule is used to verify the data to be verified, and a verification result of the data to be verified is obtained.

[0008] Optionally, obtaining the data to be verified includes: subscribing to a topic corresponding to the data to be verified, and monitoring business data corresponding to the subscribed topic; when changes in the business data are monitored, obtaining the changed business data, and using the obtained data as the data to be verified.

[0009] Optionally, the preset event includes completing format conversion of the data to be verified, and the method further includes: performing format conversion on the data to be verified based on preset mapping rules deployed on the data verification platform to obtain converted data to be verified, wherein the preset mapping rules include at least one of the following: preset field mapping rules, preset dictionary mapping rules, and preset logical operation rules.

[0010] Optionally, the method further includes: detecting whether there is a verification result of the data to be verified; and if it is detected that there is no verification result, verifying the data to be verified at a predefined time based on a preset timed verification task.

[0011] Optionally, the data verification platform is deployed with a rule script engine and at least one candidate verification rule, and obtaining the target verification rule that matches the business requirement includes: searching among the at least one candidate verification rule; when the retrieval result shows that there is a candidate verification rule that matches the business requirement, determining the retrieved candidate verification rule as the target verification rule; when the retrieval result shows that all candidate verification rules cannot match the business requirement, calling the rule script engine to generate a target verification rule that matches the business requirement.

[0012] Optionally, the method also includes: when the verification result indicates that there is an abnormality in the data to be verified, calling the secondary verification interface of the target business system to verify the data to be verified, and / or calling the data correction interface of the target business system to correct the data to be verified; the target business system is the business system corresponding to the data to be verified.

[0013] Optionally, the method also includes: obtaining historical abnormal data corresponding to the target business system, and performing statistical analysis on the historical abnormal data to mine data features of the historical abnormal data, the target business system being the business system corresponding to the data to be verified; and optimizing the target verification rules based on the data features of the historical abnormal data.

[0014] According to a second aspect of one or more embodiments of the present application, a data verification platform is provided, including:

[0015] A data access module, used to obtain the data to be verified and the business requirements for the data to be verified;

[0016] The rule verification module is used to obtain a target verification rule that matches the business requirement when monitoring a preset event corresponding to the data to be verified, and use the target verification rule to verify the data to be verified to obtain a verification result of the data to be verified.

[0017] Optionally, the data verification platform also includes: a monitoring and analysis module, which is used to obtain historical abnormal data corresponding to the target business system, and perform statistical analysis on the historical abnormal data to mine data features of the historical abnormal data; optimize the target verification rules based on the data features of the historical abnormal data; the target business system is the business system corresponding to the data to be verified.

[0018] According to a third aspect of one or more embodiments of the present application, there is provided an electronic device, including:

[0019] processor;

[0020] a memory for storing processor-executable instructions;

[0021] The processor implements the method described in any one of the embodiments of the first aspect above by running the executable instructions.

[0022] According to a fourth aspect of one or more embodiments of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any embodiment of the first aspect are implemented.

[0023] According to a fifth aspect of one or more embodiments of the present application, a computer program product is provided, comprising a computer program and / or instructions, which, when executed by a processor, implement the steps of the method described in any embodiment of the first aspect above.

[0024] It can be seen from the above technical solutions that in one or more embodiments of the present application, after the data to be verified is obtained, if a preset event corresponding to the data to be verified is monitored, a target verification rule matching the business requirements is obtained to verify the data to be verified using the target verification rule. This verification method is equivalent to data verification under an event-driven architecture-once a preset event is detected, a data verification operation will be automatically triggered. In this way, real-time or near real-time data verification can be achieved to ensure that each piece of data to be verified can be verified in a timely manner, thereby helping to discover abnormal data in a timely manner and improving the timeliness of discovering abnormal data.

[0025] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] Figure 1 It is a schematic diagram of the architecture of a data verification platform provided by an exemplary embodiment.

[0028] Figure 2 It is a flowchart of a data verification method provided by an exemplary embodiment.

[0029] Figure 3 It is a flowchart of a method for creating a decision tree model provided by an exemplary embodiment.

[0030] Figure 4a It is a structural diagram of a data verification platform provided by an exemplary embodiment.

[0031] Figure 4b It is a structural schematic diagram of another data verification platform provided by an exemplary embodiment.

[0032] Figure 5 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment.

[0033] Figure 6 It is a block diagram of a data verification device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0034] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] Next, one or more embodiments of the present application are described in detail.

[0036] Figure 1 FIG. 1 is a schematic diagram of the architecture of a data verification platform provided by an exemplary embodiment. Figure 1 As shown, the data verification platform may include a server 11 and several electronic devices, such as electronic devices 12 to 14 .

[0037] The server 11 may be a physical server including an independent host, or the server 11 may be a virtual server carried by a host cluster. During operation, the server 11 may run the server-side program of the data verification platform to realize the service end of the data verification platform. The electronic devices 12-14 may include but are not limited to: mobile phones, PCs (Personal Computers), tablet devices, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), etc.

[0038] The electronic devices 12 to 14 may be any electronic devices. The electronic devices 12 to 14 and the server 11 may be directly or indirectly connected via wired communication or wireless communication, and this application does not impose any special restrictions.

[0039] based on Figure 1In the data verification platform shown, any electronic device among the electronic devices 12 to 14 can obtain the data to be verified, and report the data to be verified and the business requirements for the data to be verified to the server 11. The server 11 can monitor whether there is a preset event corresponding to the data to be verified, and if so, obtain the target verification rule that matches the received business requirements. Then, the server 11 verifies the data to be verified using the target verification rule, obtains the verification result of the data to be verified, and returns the verification result to the electronic device that reports the data to be verified, so that the electronic device can display the verification result to relevant users (such as technicians or developers, etc.).

[0040] In another embodiment, the data verification platform may only include a server. Figure 1 Taking the server 11 in the example, the server 11 obtains the data to be verified and the business requirements for the data to be verified. When the server 11 monitors the preset event corresponding to the data to be verified, it obtains the target verification rule matching the received business requirements, and uses the target verification rule to verify the data to be verified to obtain the verification result.

[0041] Figure 2 The following is a flowchart of a data verification method provided by an exemplary embodiment. The method can be applied to a data verification platform. Exemplarily, the data verification platform can be a settlement platform, a financial transaction platform, a medical health platform, a human resources management platform, etc. This application does not specifically limit the specific type and implementation form of the data verification platform. Figure 2 , the data verification process may include the following steps:

[0042] S201: Acquire data to be verified and business requirements for the data to be verified.

[0043] It should be noted that the data verification platform can be docked with multiple business systems to verify the business data of different business systems or different business scenarios. In the present embodiment, the data to be verified can be the business data of any business system or any business scenario. The present application does not specifically limit the data type, data format, source, content, acquisition method, etc. of the data to be verified. In an embodiment shown, the source of the data to be verified can be any one of MySQL, Redis, Mongodb, ElasticSearch, MQ (Message Queue), and during the operation of the data verification platform, the source of the data to be verified can be dynamically modified, so as to realize the flexible access of the data verification platform to multiple data sources. In another embodiment shown, the data verification platform and the business system can communicate based on the http protocol or multiple RPC (Remote Procedure Call, remote procedure call) protocols, so as to realize the flexible verification of multi-protocol data by the data verification platform. Exemplarily, the data to be verified can be sent to the data verification platform by the data sender (i.e., the business system corresponding to the data to be verified) by active push, or it can be actively obtained from the data sender by the data verification platform.

[0044] Generally, the business logic and characteristics of different business systems are different, so the business requirements of different business systems for business data are also different. For example, the business requirement of the social media business system is to ensure the security and compliance of user-generated content, and it cannot contain illegal or sensitive information. The business requirement of the medical business system is to ensure the high accuracy and high privacy of medical data. In this embodiment, when obtaining the data to be verified, it is necessary to obtain the business requirements for the data to be verified at the same time, so that the verification rules that meet the business requirements can be used to verify the data to be verified later, thereby ensuring the accuracy of data verification.

[0045] In one embodiment, the data verification platform can obtain the data to be verified through a data subscription service. The data verification platform can pre-subscribe to a specified topic (including the topic corresponding to the data to be verified), and then monitor the business data corresponding to the subscribed topic. Exemplarily, the binlog (binary log) of the data of the subscribed topic can be monitored to capture the changes in business data in real time through binlog. When the data verification platform monitors that the business data corresponding to the subscribed topic has changed, the changed business data is obtained, and the obtained data is used as the data to be verified. Exemplarily, a message queue can be used as a middleware to subscribe to the data to be verified in an event-driven manner based on a message queue: the business system publishes the data change event to the message queue, and the data verification platform subscribes to the specified topic or the specified queue, thereby only receiving the business data corresponding to the subscribed topic, and the received business data is used as the data to be verified. In addition, the data verification platform allows the data to be verified to be loaded into the verification platform in batches by file import, so that a large amount of data to be verified can be verified at one time, which greatly improves the verification efficiency of the data to be verified, especially in the scenario of importing a large amount of data at one time or updating data in batches regularly, which can effectively improve the verification efficiency of the data.

[0046] In this embodiment, the data to be verified is obtained through the data subscription service, so that changes in the data can be monitored efficiently and reliably, ensuring real-time verification of the changed data.

[0047] S202: When a preset event corresponding to the data to be verified is monitored, a target verification rule matching the business requirement is obtained.

[0048] This step can be implemented by an event-driven architecture. Under the event-driven architecture, the data verification platform can respond to the verification operation of the data to be verified by listening to preset events. Among them, the preset events corresponding to the data to be verified include but are not limited to: the data to be verified has been obtained, the state of the data to be verified is changed to a predefined state, and the preset trigger signal. The technicians in this field can set the preset events according to the actual needs, and this application does not limit this. When the preset event corresponding to the data to be verified is monitored, the target verification rule matching the business needs is obtained to perform the verification operation using the target verification rule. As described in the above S201 part, different business systems have different business needs for business data, so different verification rules need to be formulated according to the business needs of each business system, so that the business data that passes the verification can meet the business needs of the corresponding business system. For example, for the data to be verified of the social media business system, the target verification rules that match the business needs of the data to be verified may include: the user's interactive behavior (such as likes, comments, sharing) must be legal, the user's privacy settings must be valid, and the user's personal information must be complete and in the correct format. For the data to be verified in the medical business system, the target verification rules that match the business requirements of the data to be verified may include: the patient's personal information must be complete and in the correct format, the diagnosis results and treatment plans must be filled in by qualified doctors, and the timestamps of medical records must be legal. The above are only examples, and this application does not specifically limit the form and content of the target verification rules.

[0049] In one embodiment, the preset events corresponding to the data to be verified may include: the data to be verified completes format conversion. That is, when the data verification platform detects / listens to the completion of the format conversion of the data to be verified, the target verification rules can be obtained to verify the converted data to be verified. Exemplarily, the preset mapping rules can be deployed on the data verification platform, and then, after the data verification platform obtains the data to be verified, the format of the data to be verified is converted based on the preset mapping rules to obtain the converted data to be verified. It should be noted that the preset mapping rules can be configurable rules, that is, the developers or managers of the data verification platform can modify or adjust the preset mapping rules according to the actual verification requirements or business needs, so as to flexibly adapt to different data verification scenarios or business verification preferences.

[0050] The preset mapping rule may include a combination of one or more mapping rules shown below:

[0051] The preset mapping rule may be a preset field mapping rule. The field mapping rule defines how to map the fields in the source data to the fields in the target data. In this embodiment, the source data is the data to be verified, and the target data is the data to be verified after conversion. For example, the field name of the data to be verified is: order number orderNo. The format of the data to be verified is converted according to the preset field mapping rule, and the field name of the converted data to be verified is: bizDetailNo.

[0052] The preset mapping rule may also be a preset dictionary mapping rule. The dictionary mapping rule defines how to map a specific value in the source data to a specific value in the target data, thereby unifying data from different sources into a standard value domain. In this embodiment, the source data is the data to be verified, and the target data is the converted data to be verified. For example, the field of the data to be verified is: car series serial, and the preset dictionary mapping rules include mapping different financial interest rate policies, such as 001 corresponding to 0.01, and 007 corresponding to 0.02.

[0053] The preset mapping rules can also be preset logical operation rules, that is, the format of the data to be verified is converted through the preset logical operation rules. For example, the fields of the data to be verified are: tax-inclusive amount, tax rate. The preset logical operation rule is: tax amount = tax-inclusive amount - (tax-inclusive amount / (1+tax rate)). Then, through the preset logical operation rule, the field of the data to be verified can be converted into the tax amount. Exemplarily, the preset logical operation rules can be deployed in the logical operation rule component, and then the logical operation rule component can be integrated into the data verification platform, so that the logical operation rules contained in the logical operation rule component can be called to perform data conversion during the data verification process. This component-based deployment of logical operation rules helps to support the flexible configuration of logical operation rules according to the verification requirements of the business, and improves the flexibility and efficiency of the configuration of logical operation rules.

[0054] It should be noted that the above are only examples, and those skilled in the art can set the preset mapping rules according to actual needs, and this application does not limit this.

[0055] Since the data formats of the data to be verified are very diverse, this embodiment uses preset mapping rules to convert the format of the data to be verified, which helps to flexibly convert the data format of the data to be verified into the standard format of the data verification platform, and then verify the data to be verified in the standard format, which can further improve the verification efficiency and the accuracy of the verification results. In addition, this embodiment proposes a variety of different preset mapping rules, which can achieve flexible and highly configurable data format standardization, ensuring that format conversion can be achieved for data to be verified in any format.

[0056] In one embodiment, a rule script engine and at least one candidate verification rule may be deployed on a data verification platform. Among them, at least one candidate verification rule may be manually set or pre-generated by a rule script engine. In addition, the rule script engine is also used to load a script corresponding to the verification rule to implement verification. There are many rule script engines, such as aviator script engine and MVEL (MVFLEX Expression Language). This application does not limit the specific rule script engine.

[0057] Due to the wide variety of content of the data to be verified, it may be necessary to verify the data to be verified from multiple dimensions (such as data amount, account information, basic information, data status, etc.), and the candidate verification rules may not cover all verification dimensions. Therefore, when obtaining the target verification rules, the verification platform can first search in the candidate verification rules. If the candidate verification rules that match the business requirements of the data to be verified can be retrieved, then the retrieved candidate verification rules are determined as the target verification rules. If the candidate verification rules that match the business requirements of the data to be verified cannot be retrieved, that is, all candidate verification rules cannot match the business requirements of the data to be verified, then the rule script engine can be called to temporarily generate target verification rules that match the business requirements of the data to be verified. In addition, the rule script engine can be used to perform online debugging and real-time dynamic compilation and release of the newly generated target verification rules. The specific process of online debugging and real-time dynamic compilation and release can refer to the relevant content of script debugging and compilation in the relevant technology, which will not be elaborated here.

[0058] In this embodiment, a rule script engine is introduced into the data verification platform. Therefore, when it is impossible to obtain the target verification rules that match the business requirements of the data to be verified, the rule script engine is called to flexibly write the target verification rules. There is no need to use the traditional rule hard-coding method to write the verification rules. On the one hand, the timeliness of data verification is ensured, and on the other hand, the flexibility and practicality of the data verification platform are greatly expanded.

[0059] S203: Verify the data to be verified using the target verification rule to obtain a verification result of the data to be verified.

[0060] In the above embodiment, after the data verification platform obtains the data to be verified, if it monitors the preset event corresponding to the data to be verified, it obtains the target verification rule that matches the business needs, so as to verify the data to be verified using the target verification rule. This verification method is equivalent to data verification under the event-driven architecture-once the preset event is detected, it will automatically trigger the execution of the data verification operation. In this way, real-time or near real-time data verification can be achieved to ensure that each piece of data to be verified can be verified in time, which helps to discover abnormal data in time and improve the timeliness of discovering abnormal data.

[0061] In one embodiment, if the verification result shows that the data to be verified is abnormal, the abnormal situation can be fed back to the business system corresponding to the data to be verified (hereinafter referred to as the "target business system"). Exemplarily, the data correction interface of the target business system can be called to correct the data to be verified, so as to correct the abnormal data to be verified to normal data. Alternatively, the secondary verification interface of the target business system can be called to perform secondary verification on the data to be verified. If the result of the secondary verification shows that the data to be verified is still abnormal data, then you can choose to call the data correction interface of the target business system to correct the data to be verified, or you can choose not to correct the abnormal data to be verified, but generate corresponding alarm information for the abnormal data to be verified, so as to show the user the abnormal data situation through the alarm information. In an embodiment shown, the above-mentioned data automatic correction or secondary verification process can be implemented based on the magic-script script engine and the magic-api development framework. Specifically, the secondary review and data automatic correction logic of the business system can be flexibly implemented based on the Http interface or Spring bean calling the RPC interface.

[0062] In this embodiment, when abnormal data is verified, the business interface of the business system is called to perform a secondary review or automatic correction on the data to be verified, thereby achieving real-time alarm or automatic repair of abnormal data based on real-time and rapid positioning of the anomaly, further expanding the performance and availability of the data verification platform.

[0063] In one embodiment, it is possible to detect whether there is a verification result for the data to be verified. If not, it means that the data verification platform may not have listened to the preset event and failed to verify the data to be verified. Therefore, a scheduled verification task can be pre-set in the data verification platform, and the scheduled verification task will be started at a predefined time to verify the verification data. For example, the scheduled verification task is: automatically run a script at 2 a.m. every day to verify the accuracy of all transaction records. In this way, on the basis of triggering the verification through immediate preset events, further combining the scheduled verification task, it helps to ensure that each piece of data to be verified can be verified, thereby achieving flexible and efficient response to the verification needs of various data to be verified, while ensuring the accuracy of each piece of data to be verified and the stability of the data verification platform.

[0064] In one embodiment, the data verification platform can also perform statistical analysis on historical data to optimize the verification rules according to the analysis results. Exemplarily, considering that the business requirements of the same business system for business data are usually the same, that is, the business data corresponding to the same business system is usually verified using the same verification rules. Therefore, in order to optimize the target verification rules corresponding to the data to be verified, the data verification platform can obtain the historical abnormal data corresponding to the target business system, and the target business system is the business system corresponding to the data to be verified. Then, the obtained historical abnormal data is statistically analyzed to dig out the data characteristics and data trends of the historical abnormal data. For example, the historical abnormal data can be statistically analyzed by decision trees and random forest algorithms. Furthermore, the target verification rules are optimized according to the data characteristics and data trends of the historical abnormal data, so that the data verification platform can perform more comprehensive and accurate verification operations on the business data of the target business system based on the optimized target verification rules.

[0065] In some possible implementations, a decision tree model can be used to automatically generate data verification rules to verify historical data through these data verification rules, and an Isolation Forest algorithm can be used to detect historical data to find historical abnormal data. The isolation forest algorithm is used to detect historical data. The basic concept of this process is to build a random forest tree (i.e., an isolation tree), and then select a split point to divide the historical data into two sets: historical normal data and historical abnormal data. The decision tree model can be obtained through model training. Figure 3 FIG. 1 is a flowchart of a method for creating a decision tree model provided by an exemplary embodiment. Figure 3 As shown, the creation process may include the following steps:

[0066] S301: Collect sample data.

[0067] Sample data for at least one business scenario may be collected.

[0068] S302: Preprocess the sample data.

[0069] The sample data can be cleaned to remove missing sample data (i.e. incomplete sample data) and duplicate sample data. The cleaned sample data is labeled to classify the sample data into two categories: normal sample data and abnormal sample data. Also, feature extraction is performed on the sample data, and the extracted sample features can be used for data verification. Sample features may include, but are not limited to: basic attribute features, time attribute features, and amount attribute features of the sample data.

[0070] S303: Select a decision tree model and divide the sample data.

[0071] The decision tree model can be selected according to the characteristics of the sample data, such as determining the tree depth (i.e., the path length from the root node to the farthest leaf node), the number of sample nodes, the number of sample splits, etc., according to the characteristics of the sample data. Among them, the "minimum number of sample splits" refers to the minimum number of samples required for the internal node to be re-divided, and the "minimum number of sample leaf nodes" specifies the minimum number of samples that each leaf node must contain. By determining the above parameters of the decision tree, the decision tree model can be determined.

[0072] In addition, the sample data needs to be divided into a sample training set, a sample validation set, and a sample test set. The sample training set is used to train the decision tree model. The sample validation set is used to adjust model parameters and select the best model configuration. The sample test set is used to evaluate the performance of the decision tree model after training. For example, 70% of the sample data can be used as a sample training set, 15% of the sample data as a sample validation set, and 15% of the sample data as a sample test set. Of course, the specific distribution ratio of the sample data can be set according to the actual data volume and business needs.

[0073] S304: Perform model training on the decision tree model using the sample training set.

[0074] The training goal of a decision tree model is to enable the decision tree model to discover data patterns from the sample data contained in the sample training set, and to generate corresponding verification rules based on the discovered data patterns, so as to verify future or unseen data based on these verification rules. Usually, during the training process, the model to be trained will try to minimize a mathematical expression called an "objective function" or "loss function." For a decision tree, this usually means choosing the best split point so that each child node is as "pure" as possible, that is, the samples in each child node belong to the same category as much as possible (for classification tasks), or have similar values ​​(for regression tasks). The process of optimizing the parameters of a decision tree model is to find the split conditions that maximize the performance of the decision tree model.

[0075] The trained decision tree model can generate verification rules for verifying whether the data meets the expected standards. Specifically, in the decision tree model, the verification rule refers to the if-else logical condition learned from the sample data. As the sample data is continuously divided, a tree structure is formed, in which the top is the root node, representing the entire sample training set; the branches of the tree represent different decision paths; and finally, the leaf nodes represent the final classification results or predicted values. When moving from the root node to any leaf node along the decision tree, you are actually following a series of if-else decision conditions. Therefore, all the conditions on the path from the root node to any leaf node are combined to form a complete verification rule, which can be used to determine which category the new data should be classified into or predict its value.

[0076] S305: Use the sample validation set to adjust the parameters of the decision tree model.

[0077] The performance of the decision tree model can be evaluated using the sample data contained in the sample validation set. For example, the model performance can be evaluated by calculating performance indicators such as accuracy, precision, recall, F1 score, mean square error, etc. Then, according to the performance of the decision tree model on the sample validation set, the parameters of the decision tree model are adjusted until a set of parameter configurations that can provide the best performance is found. For the decision tree model, the adjustable model parameters include but are not limited to: the maximum depth of the tree, the minimum number of samples required for internal node re-division, the minimum number of samples for leaf nodes, and the metric for splitting quality.

[0078] S306: Use the sample test set to evaluate the performance of the decision tree model.

[0079] The sample test set is independent of the training and validation process of the decision tree model. In this way, the evaluation results can more realistically reflect the performance of the decision tree model on new data encountered in the future.

[0080] S307: Rules sorting and optimization.

[0081] New data can be used as the input of the decision tree model, and the data that does not meet expectations can be screened out based on the verification results output by the decision tree model. Then, combined with manual judgment, the decision tree model's misjudgments and missed judgments can be analyzed to achieve reverse tuning of the decision tree model.

[0082] In this way, the performance of the decision tree model is ensured through multiple optimizations so that it can generate correct data verification rules. Furthermore, the data verification platform can verify the newly accessed data based on the decision tree model obtained in the above steps to obtain correct verification results.

[0083] In this embodiment, by statistically analyzing the historical abnormal data, it is helpful to discover the data features and patterns hidden in the historical abnormal data, and then optimize the target verification rules according to the mined data features and data trends, so that the optimized target verification rules can be more comprehensive, accurate, and scientific, and more in line with the business needs of the corresponding business system. By continuously optimizing the target verification rules, the stability and reliability of the data verification platform can be effectively improved.

[0084] Figure 4a FIG. 1 is a schematic diagram of a data verification platform according to an exemplary embodiment of the present application. Figure 4aAs shown, the data verification platform 40 includes a data access module 401 and a rule verification module 402. Among them, the data access module 401 can dynamically configure the access of the data to be verified, such as obtaining / accessing the data to be verified based on MySQL binlog logs, MQ subscription messages, Redis caches, Http interfaces, RPC interfaces, etc. Then, the data conversion configuration submodule 4011 is used to convert the format of the data to be verified to convert the data to be verified into the data to be verified in a standard format (hereinafter referred to as "converted data to be verified"). Exemplarily, a preset mapping rule is deployed on the data conversion configuration submodule 4011, and the preset mapping rule can be used to convert the format of the data to be verified. Then, the data access module 401 sends the converted data to be verified to the rule verification module 402. The rule verification module 402 is driven based on the event mode, that is, when the rule verification module 402 monitors the existence of the converted data to be verified, it obtains the target verification rule from the rule writing submodule 4021, and the target verification rule matches the business requirements for the data to be verified. The rule verification module 402 verifies the converted data to be verified using the target verification rule to obtain the corresponding verification result. It should be noted that a timed verification task is also pre-set in the rule verification module 402. When the rule verification module 402 does not detect the verification result of the data to be verified, the timed verification task can be started to verify the data to be verified at the predefined time corresponding to the timed verification task.

[0085] If the verification result of the data to be verified indicates that the data to be verified is abnormal, the rule verification module 402 may send a call request to the business system call module 4022. In response to the call request, the business system call module 4022 calls the data correction interface on the business system corresponding to the data to be verified to automatically correct the data to be verified. Alternatively, in response to the call request, the business system call module 4022 calls the secondary verification interface on the business system corresponding to the data to be verified to perform secondary verification on the data to be verified. If the verification result of the secondary verification indicates that the data to be verified is still abnormal data, then the data correction interface on the business system corresponding to the data to be verified is called again to automatically correct the data to be verified.

[0086] In one embodiment, the data verification platform may further include a monitoring and analysis module 403 (eg Figure 4bAs shown). If the rule verification module 402 does not call the data correction interface on the business system corresponding to the data to be verified, the rule verification module 402 can send the abnormal data to be verified to the monitoring and analysis module 403. The monitoring and analysis module 403 can generate corresponding alarm information based on the abnormal data to be verified. In addition, the monitoring and analysis module 403 also includes a statistical analysis submodule 4031, which is used to obtain historical abnormal data corresponding to the target business system, and perform statistical analysis on the historical abnormal data to mine the data characteristics and data trends of the historical abnormal data. Among them, the target business system refers to the business system corresponding to the data to be verified. Then, the data verification platform can optimize the target verification rules based on the data characteristics and data trends of the historical abnormal data.

[0087] Figure 5 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 5 At the hardware level, the electronic device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other services. The processor 502 reads the corresponding computer program from the non-volatile memory 510 into the memory 508 and then runs it. Of course, in addition to the software implementation, this application does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0088] Corresponding to the above method embodiment, the present application also provides an embodiment of a data verification device.

[0089] Figure 6 1 is a block diagram of a data verification device according to an exemplary embodiment of the present application. The device can be applied to a data verification platform. Figure 6 The device comprises: a data acquisition unit 601, a rule acquisition unit 602 and a verification unit 603, wherein:

[0090] The data acquisition unit 601 is configured to acquire the data to be verified and the business requirements for the data to be verified;

[0091] A rule acquisition unit 602 is configured to acquire a target verification rule matching the business requirement when a preset event corresponding to the data to be verified is monitored;

[0092] The verification unit 603 is configured to verify the data to be verified by using the target verification rule to obtain a verification result of the data to be verified.

[0093] Optionally, the data acquisition unit 601 is specifically used to: subscribe to the topic corresponding to the data to be verified, and monitor the business data corresponding to the subscribed topic; when changes in the business data are monitored, obtain the changed business data, and use the obtained data as the data to be verified.

[0094] Optionally, the preset event includes that the data to be verified completes format conversion, and the device further includes:

[0095] The data conversion unit 604 is configured to perform format conversion on the data to be verified based on preset mapping rules deployed on the data verification platform to obtain converted data to be verified, wherein the preset mapping rules include at least one of the following: preset field mapping rules, preset dictionary mapping rules, and preset logical operation rules.

[0096] Optionally, the device further comprises:

[0097] The timed verification unit 605 is configured to detect whether there is a verification result of the data to be verified; if it is detected that there is no verification result, verify the data to be verified at a predefined time based on a preset timed verification task.

[0098] Optionally, the data verification platform is deployed with a rule script engine and at least one candidate verification rule, and the rule acquisition unit 602 is specifically used to: search among the at least one candidate verification rule; when the retrieval result shows that there is a candidate verification rule that matches the business requirement, determine the retrieved candidate verification rule as the target verification rule; when the retrieval result shows that all candidate verification rules cannot match the business requirement, call the rule script engine to generate a target verification rule that matches the business requirement.

[0099] Optionally, the device further comprises:

[0100] The calling unit 606 is configured to call the secondary verification interface of the target business system to verify the data to be verified when the verification result indicates that there is an abnormality in the data to be verified, and / or call the data correction interface of the target business system to correct the data to be verified; the target business system is the business system corresponding to the data to be verified.

[0101] Optionally, the device further comprises:

[0102] The statistical analysis unit 607 is configured to obtain historical abnormal data corresponding to the target business system, and perform statistical analysis on the historical abnormal data to mine data features of the historical abnormal data, wherein the target business system is the business system corresponding to the data to be verified; and optimize the target verification rules based on the data features of the historical abnormal data.

[0103] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0104] The devices or modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0105] In a typical configuration, a computer includes one or more processors, including a central processing unit (CPU) and a graphics processing unit (GPU), an input / output interface, a network interface, and a memory. The CPU is used for computing simulations, and the GPU is used for outputting high-quality 3D images.

[0106] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0107] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0108] Corresponding to the embodiments of the aforementioned method, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any embodiment of the aforementioned method are implemented.

[0109] Corresponding to the embodiments of the aforementioned method, the present application also provides a computer program product, including a computer program and / or instructions, which implement the steps of any embodiment of the aforementioned method when executed by a processor.

[0110] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A data verification method, characterized in that: The method is applied to a data verification platform, and the method comprises: Acquire the data to be verified and the business requirements for the data to be verified; When a preset event corresponding to the data to be verified is monitored, a target verification rule matching the business requirement is obtained; The target verification rule is used to verify the data to be verified, and a verification result of the data to be verified is obtained.

2. The method according to claim 1, characterized in that The preset event includes that the data to be verified completes format conversion, and the method further includes: The data to be verified is format converted based on preset mapping rules deployed on the data verification platform to obtain converted data to be verified, wherein the preset mapping rules include at least one of the following: preset field mapping rules, preset dictionary mapping rules, and preset logical operation rules.

3. The method according to claim 1, characterized in that Also includes: Detecting whether there is a verification result of the data to be verified; In the case where it is detected that the verification result does not exist, the data to be verified is verified at a predefined time based on a preset timed verification task.

4. The method according to claim 1, characterized in that: The data verification platform is deployed with a rule script engine and at least one candidate verification rule, and the step of obtaining a target verification rule that matches the business requirement includes: Retrieving from the at least one candidate validation rule; If the search result indicates that there is a candidate verification rule that matches the business requirement, determining the retrieved candidate verification rule as the target verification rule; When the search result shows that all candidate verification rules cannot match the business requirement, the rule script engine is called to generate a target verification rule that matches the business requirement.

5. The method according to claim 1, characterized in that Also includes: When the verification result indicates that there is an abnormality in the data to be verified, the secondary verification interface of the target business system is called to verify the data to be verified, and / or the data correction interface of the target business system is called to correct the data to be verified; the target business system is the business system corresponding to the data to be verified.

6. The method according to claim 1, characterized in that Also includes: Acquire historical abnormal data corresponding to a target business system, and perform statistical analysis on the historical abnormal data to mine data features of the historical abnormal data, wherein the target business system is a business system corresponding to the data to be verified; The target verification rule is optimized based on the data features of the historical abnormal data.

7. A data verification platform, characterized in that: include: A data access module, used to obtain the data to be verified and the business requirements for the data to be verified; The rule verification module is used to obtain a target verification rule that matches the business requirement when monitoring a preset event corresponding to the data to be verified, and use the target verification rule to verify the data to be verified to obtain a verification result of the data to be verified.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 6 by running the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises a computer program and / or instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images