A data checking rule recommendation method and device, and a computer device

Abstract

The specification relates to the technical field of computers, in particular to a data verification rule recommendation method and device and computer equipment. The data verification rule recommendation method comprises determining a to-be-verified data set according to a data verification rule recommendation request of to-be-verified data; determining a target field type of the to-be-verified data according to the to-be-verified data set; determining a data analysis rule according to the target field type; processing the to-be-verified data set based on the data analysis rule to obtain analysis result information; and determining a target data verification rule according to the analysis result information. By using the embodiment, the to-be-verified data set that needs to be verified is analyzed to obtain analysis result information, and then the target data verification rule is adaptively recommended according to the analysis result information, so that the accuracy of the verification result is improved while reducing resource waste.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data verification rule method, apparatus, and computer equipment. Background Technology

[0002] As data volumes increase, data verification presents new challenges. Currently, methods for verifying big data include: administrators configuring verification rules in real-time for the data to be verified. This method demands high levels of expertise from administrators and is resource-intensive. Another method involves obtaining pre-defined verification rules based on the data to be verified and then using these rules to check the data to identify problematic data. This method cannot be adapted to different data; applying the same rules to all data leads to a high error rate in the verification results.

[0003] How to reduce resource waste while adaptably configuring corresponding data verification rules for the data to be verified is a problem that urgently needs to be solved in the existing technology. Summary of the Invention

[0004] To address the problems in the prior art, embodiments of this specification provide a data verification rule recommendation method, apparatus, computer device, and storage medium. This method analyzes a dataset to be verified, obtains analysis results, and then adaptively recommends target data verification rules based on these results. This reduces resource waste while improving the accuracy of verification results.

[0005] To solve the above-mentioned technical problems, the specific technical solution in this specification is as follows:

[0006] On the one hand, the embodiments of this specification provide a data verification rule recommendation method, including,

[0007] Based on the data verification rule recommendation request for the data to be verified, determine the set of data to be verified;

[0008] Based on the data set to be verified, determine the target field type of the data set to be verified;

[0009] Based on the target field type, determine the data analysis rules;

[0010] Based on the aforementioned data analysis rules, the dataset to be verified is processed to obtain analysis results; and

[0011] Based on the analysis results, the target data verification rules are determined.

[0012] Furthermore, based on the data verification rules recommendation request for the data to be verified, the set of data to be verified further includes:

[0013] Based on the data verification rule recommendation request for the data to be verified, the target database metadata is determined; and

[0014] Based on the target database metadata, the set of data to be verified is determined from the database.

[0015] Furthermore, determining the target field type of the data set to be verified, based on the data set to be verified, further includes:

[0016] Based on the data set to be verified, determine the pre-field type of the data set to be verified;

[0017] If the type of the pre-field is determined to be character, multiple preset regular expressions are obtained;

[0018] The set of data to be checked is processed using the multiple preset regular expressions to obtain the matching rate data between the set of data to be checked and each preset regular expression;

[0019] Based on the matching rate data, determine the target field type; and

[0020] If it is determined that the pre-field type is not a character type, then the pre-field type is determined to be the target field type.

[0021] Furthermore, when the target field type is date / time, the data analysis rules include multiple preset regular expressions. The step of processing the data set to be verified based on the data analysis rules to obtain analysis results further includes:

[0022] The plurality of preset regular expressions are used to process each piece of data to be verified in the dataset to be verified, so as to determine the target data to be verified that matches each preset regular expression; and

[0023] The numerical values ​​of the target data to be checked that match each preset regular expression are determined as the analysis result information.

[0024] Furthermore, the determination of target data verification rules based on the analysis results further includes,

[0025] Sort the given values ​​according to their magnitude to obtain a target sequence of values;

[0026] From the target number of numerical values, determine the first target number of target numerical values; and

[0027] The preset regular expression corresponding to each of the target values ​​is used as the target data verification rule.

[0028] The data verification rule recommendation method also includes:

[0029] The target data verification rule is stored as the preset data verification rule associated with the data verification rule recommendation request for the data to be verified.

[0030] Furthermore, when the target field type is numeric, the data analysis rules include numeric type determination rules and sub-analysis rules corresponding to each numeric type. The step of processing the data set to be verified based on the data analysis rules to obtain analysis results further includes...

[0031] Based on the aforementioned numerical type determination rules, the data set to be verified is processed to determine the target numerical type; and

[0032] The analysis results are obtained by processing the data set to be verified using the sub-analysis rules corresponding to the target numerical type.

[0033] Furthermore, the analysis results obtained by processing the dataset to be verified using the sub-analysis rules corresponding to the target data type further include:

[0034] If the target numerical type is determined to be an enumerated numerical type, the deduplication sub-analysis rule is used to perform deduplication processing on the data set to be checked to obtain the processing result information.

[0035] If the target data type is determined to be a continuous numerical type, the outlier sub-analysis rules corresponding to the continuous numerical type are used to process the data set to be checked, and the processing result information is obtained.

[0036] On the other hand, embodiments of this specification also provide a data verification rule recommendation device, including,

[0037] The first determining unit is used to determine the set of data to be verified based on the data verification rule recommendation request for the data to be verified.

[0038] The second determining unit is used to determine the target field type of the data set to be verified based on the data set to be verified.

[0039] The third determining unit is used to determine data analysis rules based on the target field type;

[0040] The processing unit is configured to process the dataset to be verified based on the data analysis rules to obtain analysis result information; and

[0041] The fourth determining unit is used to determine the target data verification rules based on the analysis results.

[0042] On the other hand, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0043] On the other hand, embodiments of this specification also provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described method.

[0044] Using the embodiments of this specification, upon receiving a request to recommend data verification rules for data to be verified, a data set to be verified is determined from the database; based on the target field type of the data set to be verified, a data analysis rule for analyzing the data to be verified is determined; based on the data analysis rule, the data set to be verified is processed to obtain analysis result information; and then, based on the analysis result information, a target data verification rule is determined. This achieves adaptive analysis of the data set to be verified using the data analysis rule corresponding to the data to be verified, thereby determining the corresponding target data verification rule. This ensures that a target data verification rule with high accuracy can be adaptively recommended for all data sets to be verified. Therefore, the embodiments of this specification improve the accuracy of verification results while reducing resource waste. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 The diagram shown is a schematic representation of an implementation system for a data verification rule recommendation method according to an embodiment of this specification.

[0047] Figure 2 The diagram shown is a flowchart of a data verification rule recommendation method according to an embodiment of this specification;

[0048] Figure 3 The diagram shown is a flowchart of a method for determining the type of a target field according to an embodiment of this specification;

[0049] Figure 4A The diagram shown is a flowchart of a method for determining analysis result information according to an embodiment of this specification;

[0050] Figure 4B The diagram shown is a flowchart of a method for determining analysis result information according to another embodiment of this specification;

[0051] Figure 5 The diagram shown is a schematic diagram of a data verification rule recommendation method according to an embodiment of this specification.

[0052] Figure 6A The diagram shown is a structural schematic of a data verification rule recommendation device according to an embodiment of this specification.

[0053] Figure 6B The diagram shown is a structural schematic of a data verification rule recommendation device according to another embodiment of this specification.

[0054] Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of this specification.

[0055] [Explanation of Labels in the Attached Image]

[0056] 101. User terminal;

[0057] 102. Server;

[0058] 501. Data verification rule recommendation request;

[0059] 510. Pre-set data verification rule base;

[0060] 521. Data set to be verified;

[0061] 531. Multiple preset regular expressions;

[0062] 532. Rules for determining numerical data types;

[0063] 533. Rules for deduplication analysis;

[0064] 534. Outlier Sub-analysis Rules;

[0065] 541. First analysis results;

[0066] 542. First Target Data Verification Rules;

[0067] 551. Second analysis results information;

[0068] 552. Second Target Data Verification Rules;

[0069] 561. Third analysis results information;

[0070] 562. Third Target Data Verification Rules;

[0071] 6110. First Determined Unit;

[0072] 6120. Second Determined Unit;

[0073] 6130, Third Determined Unit;

[0074] 6140. Processing unit;

[0075] 6150, Fourth Determined Unit;

[0076] 6210. First Determined Unit;

[0077] 6220. Second Determined Unit;

[0078] 6230, Third Determined Unit;

[0079] 6240, Processing Unit;

[0080] 6250, Fourth Determined Unit;

[0081] 6260, Fifth Determined Unit;

[0082] 702. Computer equipment;

[0083] 704. Processing equipment;

[0084] 706. Storage resources;

[0085] 708. Drive mechanism;

[0086] 710. Input / Output Module;

[0087] 712. Input devices;

[0088] 714. Output devices;

[0089] 716. Presentation equipment;

[0090] 718. Graphical User Interface;

[0091] 720. Network interface;

[0092] 722. Communication link;

[0093] 724. Communication bus. Detailed Implementation

[0094] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0095] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0096] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0097] In the technical solutions of this specification, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information all comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0098] Figure 1The diagram illustrates an implementation system for a data verification rule recommendation method according to an embodiment of this specification. The system may include a user terminal 101 and a server 102, which communicate via a network. This network may include a Local Area Network (LAN), a Wide Area Network (WAN), the Internet, or a combination thereof, and is connected to a website, user equipment (e.g., a computing device), and a backend system. An administrator can send a data verification rule recommendation request for the data to be verified to the server 102 via the user terminal 101. This request includes information for indexing the data to be verified. Upon receiving the request, the server 102, based on the target field type of the indexed set of data to be verified, determines data analysis rules for analyzing the data. Based on these rules, it processes the data set to obtain analysis results. Then, based on these results, it determines target data verification rules and sends these rules to the user terminal 101. If any of the following verifications fails: the cross-border transaction user identifier, the incoming user identifier, or the virtual resource value, a status message indicating a failed cross-border transaction will be sent to the user terminal 101. Optionally, server 102 may be a node in a cloud computing system (not shown in the figure), or each server 102 may be a separate cloud computing system, comprising multiple computers interconnected by a network and operating as a distributed processing system.

[0099] In an optional embodiment, the user terminal 101 may include electronic devices, including but not limited to smartphones, data acquisition devices, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, and other similar electronic devices. Optionally, the operating system running on the electronic device may include, but is not limited to, Android, iOS, Linux, Windows, etc.

[0100] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided in this manual. In actual applications, it may include multiple user terminals 101, and this manual does not impose any restrictions.

[0101] like Figure 2The diagram shows a flowchart of a data verification rule recommendation method according to an embodiment of this specification. This diagram illustrates the data verification rule recommendation process, but based on conventional or non-creative labor, it may include more or fewer operational steps. The order of steps listed in the embodiment is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the method can be executed sequentially or in parallel according to the embodiment or the accompanying drawings. Specifically, as shown... Figure 2 As shown, the method may include:

[0102] S210, Based on the data verification rule recommendation request for the data to be verified, determine the set of data to be verified;

[0103] S220, Determine the target field type of the data set to be verified based on the data set to be verified;

[0104] S230, Determine the data analysis rules based on the target field type;

[0105] S240, based on data analysis rules, processes the data set to be verified to obtain analysis results information;

[0106] S250, Based on the analysis results, determine the target data verification rules.

[0107] Using the embodiments of this specification, upon receiving a request to recommend data verification rules for data to be verified, a data set to be verified is determined from the database; based on the target field type of the data set to be verified, a data analysis rule for analyzing the data to be verified is determined; based on the data analysis rule, the data set to be verified is processed to obtain analysis result information; and then, based on the analysis result information, a target data verification rule is determined. This achieves adaptive analysis of the data set to be verified using the data analysis rule corresponding to the data to be verified, thereby determining the corresponding target data verification rule. This ensures that a target data verification rule with high accuracy can be adaptively recommended for all data sets to be verified. Therefore, the embodiments of this specification improve the accuracy of verification results while reducing resource waste.

[0108] According to one embodiment of this specification, when an administrator needs to verify certain data to be verified, the first step is to determine the data verification rules for that data. The administrator sends a data verification rule recommendation request to the server via a user terminal. This data verification rule recommendation request includes any information that can be indexed to the data to be verified.

[0109] When the server receives a data verification rule recommendation request sent by the user terminal, it determines the set of data to be verified based on the data verification rule recommendation request. The set of data to be verified includes the data that needs to be verified.

[0110] For each piece of data in the dataset to be verified, determine the corresponding target field type. This target field type represents the field type of the data to be verified, such as numeric, character, or date / time type.

[0111] After determining the data field type corresponding to each piece of data to be verified, for each data field type, the number of identical data field types is counted. Based on this count, the target field type of the data set to be verified is determined. Specifically, determining the target field type based on the count of identical data field types involves identifying the field type corresponding to the largest count among multiple identical data field types. For example, if the data set to be verified includes 300 numeric data entries and 100 character data entries, then for numeric data entries, the count of identical data field types is 300, and for character data entries, the count is 100. Therefore, the target field type for the data set to be verified is determined to be 300.

[0112] Pre-configure corresponding preset data analysis rules for each data field type to obtain a field type analysis rule library. Data analysis rules represent the rules by which users analyze the data to be verified to determine its nature. For example, this data analysis rule could be an outlier identification model, which could be any model that processes a sequence of data sets to determine outlier representation information or outliers. The outlier identification model processes the data to be verified in the dataset to obtain outlier representation information for that dataset. This outlier representation information could be, for example, at least one of a maximum value and a minimum value. The analysis result information could be, for example, this outlier representation information. For example, if the outlier identification model processes the data to be verified in the dataset and obtains outlier representation information including a minimum value of 21 and a maximum value of 36, then the analysis result would be the minimum value of 21 and the maximum value of 36.

[0113] The determined target field type is matched with each data field type in the field type analysis rule base to determine the target data field type that matches the target field type. The preset data analysis rule associated with the target data field type is then used as the data analysis rule corresponding to the target field type.

[0114] By utilizing data analysis rules, each piece of data in the dataset to be verified is processed to obtain analysis results. Then, based on these results, the corresponding target data verification rules are determined.

[0115] For example, based on the analysis results, the specific target data verification rules are determined by identifying the data properties represented by the analysis results. For instance, a corresponding verification rule template is pre-configured for each data property. Based on the data properties represented by the analysis results, the corresponding verification rule template is obtained and updated to obtain the corresponding verification rule.

[0116] According to another embodiment of this specification, target database metadata is determined based on a data verification rule recommendation request for the data to be verified; and a set of data to be verified is determined from the database based on the target database metadata.

[0117] Metadata is data that describes other data. In other words, metadata is structured data used to provide information about a resource. Metadata describes information resources or data objects, and its purpose is to: identify resources; evaluate resources; track changes in resources during use; achieve simple and efficient management of large amounts of networked data; and enable effective discovery, retrieval, integrated organization, and effective management of information resources.

[0118] Database metadata includes: management attributes (e.g., creator, application system, business line, business manager, etc.), lifecycle (e.g., creation time, DDL time, last update time, version information, etc.), storage attributes (e.g., location, space size, physical size, etc.), data characteristics (e.g., data skew, average length, etc.), usage characteristics (e.g., DML, refresh rate, etc.), data structure tables / partitions (e.g., name, type, remarks, etc.), columns (e.g., name, type, length, precision, etc.), indexes (e.g., name, type, field, etc.), and constraints (e.g., type, field, etc.).

[0119] The target database metadata includes metadata about the application system, database IP address, database port, database name, table name, table field name, and field type. A connection is established with the target database using a database connection driver (JDBC driver). Then, using Structured Query Language (SQL), each table and each field in the database is traversed sequentially to obtain the seven metadata items from the entire database. A consistency match is then performed between this metadata and the target database metadata. Data corresponding to the metadata matching the target database metadata is selected as the data to be verified. This data to be verified is then aggregated to obtain the set of data to be verified.

[0120] According to another embodiment of this specification, after determining the target data verification rule based on the analysis results, it may further include, for example, storing the target data verification rule as a preset data verification rule associated with the data verification rule recommendation request for the data to be verified. Specifically, it may also be storing the target data verification rule as a preset data verification rule associated with the target database metadata included in the data verification rule recommendation request for the data to be verified.

[0121] According to another embodiment of this specification, before determining the target field type of the data set to be verified based on the data set to be verified, the method may further include matching the target database metadata recommended by the data verification rules of the data to be verified with the associated metadata associated with each preset data verification rule in the preset data verification rule library. When it is determined that there is associated metadata that matches the target database metadata, the preset data verification rule corresponding to the associated metadata is used as the target data verification rule.

[0122] Because data of the same category is often stored together, the data to be verified in the dataset indexed by the target database metadata is likely to be of the same category and have similar data properties. Therefore, when verifying the data to be verified corresponding to the metadata of a target database, data analysis can be performed only once on that dataset to determine the recommended target data verification rule. When recommending verification rules for subsequent datasets including data stored thereafter, the same target data verification rule should also be recommended, thus reducing the cost of recommending data verification rules.

[0123] Figure 3 The diagram shows a flowchart of a method for determining the type of a target field according to an embodiment of this specification. This diagram describes a process for determining the type of a target field, but based on conventional or non-creative work, it may include more or fewer operational steps. Specifically, as shown... Figure 3 As shown, the method may include:

[0124] S321, Determine the pre-field type of the data set to be verified based on the data set to be verified;

[0125] S322, Determine if the pre-field type is character;

[0126] S323, if the pre-field type is determined to be character, obtain multiple pre-defined regular expressions;

[0127] S324, using multiple preset regular expressions to process the data set to be checked, to obtain the matching rate data between the data set to be checked and each preset regular expression;

[0128] S325, Determine the target field type based on the matching rate data;

[0129] S326, if it is determined that the pre-field type is not character, then the pre-field type is determined to be the target field type.

[0130] During database construction, developers often store date and time data as character types for ease of data storage. This presents a significant challenge to the construction of data quality verification rules, especially for date and time data. To adaptively determine the corresponding target data verification rules for the dataset to be verified, the correct target field type must first be identified. Therefore, it is necessary to identify all character-type data to be verified, correcting instances where date and time data is stored as character types, and thus providing accurate target data verification rules for subsequent data verification rule recommendations. Using the embodiments in this specification, character-type data to be verified is further validated to improve the accuracy of the finally determined target field type, thereby increasing the accuracy of data verification based on the target data verification rules.

[0131] According to another embodiment of this specification, different databases use significantly different field types when storing data. The field types included in MySQL and Oracle databases are categorized into four types: numeric, character, date / time, and others. Detailed information on each database, each field type, and the corresponding database field types is shown in Table 1 below.

[0132] Table 1

[0133]

[0134] For each data to be verified in the data set to be verified, respectively determine the corresponding data type based on Table 1 above. And based on the data type corresponding to each data to be verified, determine the pre-field type corresponding to the data set to be verified. Specifically, for each field type, respectively count the number of values of the same data field type, and based on the number of values of the same data field type, determine the pre-field type of the data set to be verified.

[0135] After determining the pre-field type of the data set to be verified, determine whether the pre-field type is a character type. When it is determined that the pre-field type is a character type, obtain multiple preset regular expressions. The preset regular expressions include, for example, regular expressions in the following date formats: YYYY-MM-DD, YYYY / MM / DD, YYYY year MM month DD day, YYYY-MM, YYYY / MM, YYYY year MM month, YYYY year, YYYY-MM-DD hh:mm:ss, and YYYY / MM / DD hh:mm:ss, etc.

[0136] Use each of the multiple preset regular expressions to process the data to be verified in the data set to be verified respectively, and determine the target regular expression corresponding to each data to be verified. For each preset regular expression, count the number of values of the same target regular expression, and based on the number of values of the target regular expression and the total number of data to be verified, determine the matching rate data between the data set to be verified and each preset regular expression. Furthermore, when it is determined that the sum of the matching rate data is greater than or equal to the preset matching rate threshold, determine that the time and date type is the target field type.

[0137] For example, the preset regular expressions include YYYY-MM-DD, YYYY / MM / DD, and YYYY year MM month DD day. Use this preset regular expression to process each of the 100 data to be verified included in the data set to be verified, and determine that YYYY-MM-DD matches 60 data to be verified, YYYY / MM / DD matches 31 data to be verified, and YYYY year MM month DD day matches 0 data to be verified. Thus, determine that the matching rate data corresponding to YYYY-MM-DD is the quotient of 60 and 100, which is 0.6, the matching rate data corresponding to YYYY / MM / DD is 0.31, and the matching rate data corresponding to YYYY year MM month DD day is 0. When the preset matching rate threshold is 0.8, the sum of the matching rate data is 0.91, which is greater than 0.8, so the target field type of the data set to be verified is the time and date type.

[0138] When it is determined that the pre-field type is not a character type, determine that the pre-field type is the target field type.

[0139] According to another embodiment of this specification, when processing a set of data to be verified using a preset regular expression, processing can be performed on a predetermined number of data items randomly selected from the set of data to be verified. This predetermined number can be, for example, 100.

[0140] Figure 4A The diagram shows a flowchart of a method for determining analysis result information according to an embodiment of this specification. While this diagram describes a process for determining analysis result information, it can include more or fewer operational steps based on conventional or non-creative methods. Specifically, as shown... Figure 4A As shown, the method may include:

[0141] S4141, Process each piece of data to be checked in the set of data to be checked using multiple preset regular expressions to determine the target data to be checked that matches each preset regular expression;

[0142] S4142, determine the numerical values ​​of the target data to be checked that match each preset regular expression, which are the analysis result information.

[0143] Using the embodiments in this specification, multiple preset regular expressions are used to perform data analysis on the data to be verified, with the target field type being date and time, to determine the corresponding analysis result information, and then to determine the target data verification rules corresponding to the date and time type data to be verified.

[0144] According to another embodiment of this specification, each of a plurality of preset regular expressions is used to perform regular expression matching with each piece of data to be verified in the dataset to be verified, thereby determining the target data to be verified that matches each preset regular expression. That is, for each preset regular expression, a plurality of target data to be verified that match it will be obtained. Then, the number of target data to be verified that match each preset regular expression is counted, and the number of target data to be verified that match each preset regular expression is used as the analysis result information.

[0145] According to another embodiment of this specification, when the target field type is date and time, the target data verification rules are determined based on the analysis results, including: sorting the values ​​according to their size to obtain a target value sequence; determining the first target values ​​from the target value sequence; and using a preset regular expression corresponding to each target value as the target data verification rule.

[0146] For example, the preset regular expressions include YYYY-MM-DD, YYYY / MM / DD, and YYYY year MM month DD day. The individual values corresponding to YYYY-MM-DD are 625, the individual values corresponding to YYYY / MM / DD are 32, and the individual values corresponding to YYYY year MM month DD day are 310. Then the target numerical value sequence is {625, 310, 32}. When the previous target numerical value is 2, the target individual values are 625 and 310. Furthermore, the target data verification rule is determined to be YYYY-MM-DD and YYYY year MM month DD day.

[0147] According to another embodiment of the present specification, the data verification rule recommendation method further includes: using the target data verification rule as a preset data verification rule associated with and stored for the data verification rule recommendation request of the to-be-verified data.

[0148] After using this target verification rule as the preset data verification rule associated with the data verification rule recommendation request, it can be used to directly output this preset data verification rule as the target data verification rule to the user terminal without further data processing when the data verification rule recommendation request is received later, thereby improving efficiency and reducing resource waste.

[0149] Figure 4B The figure shows a flowchart of a method for determining analysis result information according to another embodiment of the present specification; another process for determining analysis result information is described in this figure, but based on routine or non-creative labor, it may include more or fewer operation steps. Specifically, as Figure 4B shown, the method may include:

[0150] S4241, based on the numerical type determination rule, process the to-be-verified data set to determine the target numerical type;

[0151] S4242, use the sub-analysis rule corresponding to the target numerical type to process the to-be-verified data set to obtain the analysis result information.

[0152] Using the embodiments of the present specification, use the numerical type determination rule and the sub-analysis rule corresponding to each numerical type to perform data analysis on the to-be-verified data with the target field type being numerical, determine the corresponding analysis result information, and further determine the target data verification rule corresponding to the to-be-verified data of the time and date type.

[0153] According to another embodiment of the present specification, the target numerical types include enumerated numerical types and continuous numerical types.

[0154] The rules for determining the numerical type may include, for example, a deduplication rate formula and a numerical type judgment rule. Specifically, the deduplication rate formula may include, for example, the following formula (1), and the data type judgment rule may include, the following formula (2).

[0155] r=len(set(data)) / len(data) formula (1)

[0156]

[0157] Where r represents the deduplication rate, data represents the set of data to be checked, len(.) represents the total number of data items in the set, set(.) represents the deduplication function, and a and b are positive integers. For example, a and b are 0.2 and 100 respectively.

[0158] After determining the target numerical type, the data set to be verified is processed based on the sub-analysis rules associated with that target numerical type to obtain the corresponding analysis results.

[0159] According to another embodiment of this specification, the data set to be verified is processed using sub-analysis rules corresponding to the target data type to obtain analysis result information, including: when the target numerical type is determined to be an enumerated numerical type, the deduplication sub-analysis rules are used to perform deduplication processing on the data set to be verified to obtain processing result information; when the target data type is determined to be a continuous numerical type, the outlier sub-analysis rules corresponding to the continuous numerical type are used to process the data set to be verified to obtain processing result information.

[0160] De-baric analysis rules may include, for example, the following formula (3).

[0161]

[0162] Where, date i To represent any piece of data to be verified, use `set(data)`. j The `set(data)` function represents any set of data to be verified after deduplication.

[0163] For example, if the data set to be verified is {0, 2, 0, 2, 1, 1, 1, 1, 1, 1, 0, 2, 0}, then the standard deduplicated data set to be verified is {0, 2, 1}, where date1 is 0, date2 is 2, and date3 is 1. For example, set(data)1 is {0}, set(data)2 is {0, 1}, set(data)3 is {0, 1, 2}, set(data)4 is {0, 2}, and so on.

[0164] Based on the dataset to be verified, the numerical frequency of each deduplicated dataset to be verified is calculated; and the numerical frequency of each deduplicated dataset to be verified is used as the analysis result information.

[0165] Based on the set of data to be verified, the numerical frequency of each deduplicated set of data to be verified is calculated. For example, based on the set of data to be verified, the sub-numerical frequency corresponding to each deduplicated data in each set of data to be verified is calculated, and then the numerical frequency of each set of data to be verified is determined based on the sub-numerical frequency corresponding to each deduplicated data.

[0166] For example, if the data set to be verified is {0, 2, 0, 2, 1, 1, 1, 1, 1, 1, 0, 2, 0}, and set(data)1 is {0}, the sum of the numerical frequencies of these values ​​with the deduplicated data set is 4. For set(data)2 being {0, 1}, the sub-numerical frequency corresponding to the deduplicated data 0 is 4, and the sub-numerical frequency corresponding to the deduplicated data 1 is 6. Therefore, the sum of the numerical frequencies of these values ​​with the deduplicated data set is 10. Similarly, set(data)3 being {0, 1, 2}, the sum of the numerical frequencies of these values ​​with the deduplicated data set is 13. Similarly, set(data)3 being {0, 2}, the sum of the numerical frequencies of these values ​​with the deduplicated data set is 7.

[0167] According to another embodiment of this specification, when the target field type is date / time, the target data verification rule is determined based on the analysis results. Specifically, it can be done by determining the frequency sum of the target values ​​before the target threshold value, based on the sorted frequency sum of the values. The deduplicated dataset to be verified corresponding to each target value frequency sum is then used as the target data verification rule.

[0168] For example, the data set to be checked is {0, 2, 0, 2, 1, 1, 1, 1, 1, 1, 0, 2, 0}. The sum of the frequencies of the values ​​corresponding to {0} in set(data)1 is 4. The sum of the frequencies of the values ​​corresponding to {0, 1} in set(data)2 is 10, the sum of the frequencies of the values ​​corresponding to {0, 1, 2} in set(data)3 is 13, and the sum of the frequencies of the values ​​corresponding to {0, 2} in set(data)3 is 7. The sum of the frequencies of these values ​​after sorting is {13, 10, 7, 4}. Given that the values ​​of the preceding target are among the top 3, the target data checking rules are {0, 1, 2}, {0, 1}, and {0, 2}.

[0169] Outlier analysis rules may include at least one of the following: P-quantile method, 3-sigma criterion, and box plot method. In the P-quantile method, P can be, for example, the 0.05 and 0.95 quantiles, with the value corresponding to the 0.05 quantile as the lower bound and the value corresponding to the 0.95 quantile as the upper bound. In the 3-sigma criterion, for example, the value corresponding to μ (first parameter) - 3σ (second parameter) can be used as the lower bound, and the value corresponding to μ + 3σ can be used as the upper bound. In the box plot method, for example, the value corresponding to Q1 (lower quartile) - 1.5IQR (difference between the upper and lower quartiles) can be used as the lower bound, and the value corresponding to Q3 (upper quartile) + 1.5IQR can be used as the upper bound.

[0170] When the outlier sub-analysis rules can include multiple methods such as P-quantiles, the 3-sigma criterion, and box plots, multiple upper and lower bounds will be obtained. The weight value corresponding to each outlier sub-analysis rule is obtained, and based on this weight value, the corresponding upper and lower bounds are processed to obtain the target upper and lower bounds corresponding to the dataset to be verified. These target upper and lower bounds are then used as the processing results. If the analysis results are target upper and lower bounds, these values ​​are used as the corresponding target data verification rules.

[0171] The outlier sub-analysis rule can also be a deep learning model, such as a neural network model. In this case, the outlier sub-analysis rule can be a pre-defined outlier analysis model, and the corresponding processing result information is the trained outlier analysis model. When the processing result information is the trained outlier analysis model, this trained outlier analysis model is used as the target data verification rule.

[0172] Figure 5 The diagram shown is a schematic diagram of a data verification rule recommendation method according to an embodiment of this specification.

[0173] like Figure 5 As shown, upon receiving a data verification rule recommendation request 501, the corresponding data set 521 to be verified is determined based on the data verification rule recommendation request 501. Based on the target database metadata included in the data verification rule recommendation request 501, it is determined whether an associated preset data verification rule exists in the preset data verification rule library 510. If the preset data verification rule exists, it is output to the user terminal as the target data verification rule.

[0174] If it is determined that the preset data verification rule does not exist, the target field type is determined based on the data set to be verified, 521.

[0175] If the target field type is determined to be date / time, multiple preset regular expressions 531 are used to process the data set 521 to be verified, resulting in first analysis result information 541. Based on this first analysis result information, a corresponding first target data verification rule 542 is determined. This first target data verification rule 542 is output to the user terminal, and the preset data verification rule associated with this first target data verification rule 542 as the target database metadata included in the data verification rule recommendation request 501 is stored in the preset data verification rule library 510.

[0176] If the target field type is determined to be numeric, the numeric type determination rule 532 is used to process the data set 521 to be verified to obtain the target numeric type corresponding to the data set 521 to be verified.

[0177] When the target data type is determined to be an enumerated numeric type, the deduplication sub-analysis rule 533 is used to process the data set 521 to be verified, resulting in a second analysis result information 551. Based on this second analysis result information 551, a corresponding second target data verification rule 552 is determined. This second target data verification rule 552 is output to the user terminal, and the preset data verification rule associated with this second target data verification rule 552 as the target database metadata included in the data verification rule recommendation request 501 is stored in the preset data verification rule library 510.

[0178] When the target data type is determined to be continuous numerical, outlier sub-analysis rule 534 is used to process the data set 521 to be verified, resulting in third analysis result information 561. Based on this third analysis result information 561, a corresponding third target data verification rule 562 is determined. This third target data verification rule 562 is output to the user terminal, and the preset data verification rule associated with this third target data verification rule 562, as part of the target database metadata included in the data verification rule recommendation request 501, is stored in the preset data verification rule library 510.

[0179] Figure 6A The diagram shown is a structural schematic of a data verification rule recommendation device according to an embodiment of this specification. Figure 6A As shown, including,

[0180] The first determining unit 6110 is used to determine the set of data to be verified based on the data verification rule recommendation request of the data to be verified;

[0181] The second determining unit 6120 is used to determine the target field type of the data set to be verified based on the data set to be verified.

[0182] The third determining unit 6130 is used to determine data analysis rules based on the target field type;

[0183] Processing unit 6140 is used to process the dataset to be verified based on data analysis rules to obtain analysis results; and

[0184] The fourth determining unit 6150 is used to determine the target data verification rules based on the analysis results.

[0185] Since the principle of the above-mentioned device in solving the problem is similar to that of the above-mentioned method, the implementation of the above-mentioned device can refer to the implementation of the above-mentioned method, and the repeated parts will not be described again.

[0186] Figure 6B The diagram shown is a structural schematic of a data verification rule recommendation device according to another embodiment of this specification. Figure 6B As shown, including,

[0187] The first determining unit 6210 is used to determine the set of data to be verified based on the data verification rule recommendation request of the data to be verified;

[0188] The second determining unit 6220 is used to determine the target field type of the data set to be verified based on the data set to be verified.

[0189] The third determining unit 6230 is used to determine data analysis rules based on the target field type;

[0190] The processing unit 6240 is used to process the data set to be verified based on data analysis rules to obtain analysis result information;

[0191] The fourth determining unit 6250 is used to determine the target data verification rules based on the analysis results.

[0192] The fifth determining unit 6260 is used to store the target data verification rule as a preset data verification rule associated with the data identifier corresponding to the data verification rule recommendation request of the data to be verified.

[0193] Since the principle of the above-mentioned device in solving the problem is similar to that of the above-mentioned method, the implementation of the above-mentioned device can refer to the implementation of the above-mentioned method, and the repeated parts will not be described again.

[0194] like Figure 7The diagram illustrates the structure of a computer device according to an embodiment of this specification. The apparatus described in this specification can be the computer device in this embodiment, performing the methods described above. The computer device 702 may include one or more processing devices 704, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 702 may also include any storage resource 706 for storing information of any kind, such as code, settings, data, etc. Without limitation, for example, the storage resource 706 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any storage resource can use any technology to store information. Furthermore, any storage resource can provide volatile or non-volatile retention of information. Further, any storage resource may represent a fixed or removable component of the computer device 702. In one case, when the processing device 704 executes associated instructions stored in any storage resource or combination of storage resources, the computer device 702 can perform any operation of the associated instructions. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any storage resource, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0195] Computer device 702 may also include an input / output module 710 (I / O) for receiving various inputs (via input device 712) and providing various outputs (via output device 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface (GUI) 718. In other embodiments, the input / output module 710 (I / O), input device 712, and output device 714 may be omitted, and the device may function solely as a computer device within a network. Computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0196] Communication link 722 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0197] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0198] This specification also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method.

[0199] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0201] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0202] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0203] The above specific embodiments further illustrate the purpose, technical solutions, and beneficial effects of this specification. It should be understood that the above are merely specific embodiments of this specification and are not intended to limit the scope of protection of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A method for recommending data verification rules, characterized in that, include: Based on the data verification rule recommendation request for the data to be verified, determine the set of data to be verified; Based on the data set to be verified, determine the target field type of the data set to be verified; Based on the target field type, determine the data analysis rules; Based on the data analysis rules, the data set to be verified is processed to obtain analysis results. as well as Based on the analysis results, the target data verification rules are determined; When the target field type is date / time, the data analysis rules include multiple preset regular expressions. The analysis results obtained by processing the dataset to be verified based on these data analysis rules include: The plurality of preset regular expressions are used to process each piece of data to be verified in the dataset to be verified, so as to determine the target data to be verified that matches each preset regular expression; and Determine the numerical values ​​of the target data to be checked that match each preset regular expression, which are the analysis result information; When the target field type is numeric, the data analysis rules include numeric type determination rules and sub-analysis rules corresponding to each numeric type. Based on the data analysis rules, the data set to be verified is processed to obtain the following analysis results: Based on the aforementioned numerical type determination rules, the data set to be verified is processed to determine the target numerical type; and Using the sub-analysis rules corresponding to the target numerical type, the data set to be verified is processed to obtain the analysis result information; The determination of target data verification rules based on the analysis results includes: Sort the given values ​​according to their magnitude to obtain a target sequence of values; From the target number of numerical values, determine the first target number of target numerical values; and The preset regular expression corresponding to each of the target values ​​is used as the target data verification rule; The data verification rule recommendation method also includes: The target data verification rule is used as a preset data verification rule associated with the data verification rule recommendation request for the data to be verified.

2. The method according to claim 1, characterized in that, The process of determining the set of data to be verified based on the data verification rule recommendation request includes: Based on the data verification rule recommendation request for the data to be verified, the target database metadata is determined; and Based on the target database metadata, the set of data to be verified is determined from the database.

3. The method according to claim 1, characterized in that, The step of determining the target field type of the data set to be verified includes: Based on the data set to be verified, determine the pre-field type of the data set to be verified; If the type of the pre-field is determined to be character, multiple preset regular expressions are obtained; The set of data to be checked is processed using the multiple preset regular expressions to obtain the matching rate data between the set of data to be checked and each preset regular expression; Based on the matching rate data, determine the target field type; and If it is determined that the pre-field type is not a character type, then the pre-field type is determined to be the target field type.

4. The method according to claim 1, characterized in that, The step of processing the dataset to be verified using the sub-analysis rules corresponding to the target numerical type to obtain the analysis result information includes: If the target numerical type is determined to be an enumerated numerical type, the deduplication sub-analysis rule is used to perform deduplication on the data set to be checked to obtain the analysis result information. If the target numerical value is determined to be a continuous numerical value, the outlier sub-analysis rules corresponding to the continuous numerical value are used to process the data set to be checked to obtain the analysis result information.

5. A data verification rule recommendation device, characterized in that, include: The first determining unit is used to determine the set of data to be verified based on the data verification rule recommendation request for the data to be verified. The second determining unit is used to determine the target field type of the data set to be verified based on the data set to be verified. The third determining unit is used to determine data analysis rules based on the target field type; The processing unit is used to process the set of data to be verified based on the data analysis rules to obtain analysis result information; as well as The fourth determining unit is used to determine the target data verification rules based on the analysis results. When the target field type is date / time, the data analysis rules include multiple preset regular expressions. The analysis results obtained by processing the dataset to be verified based on these data analysis rules include: The plurality of preset regular expressions are used to process each piece of data to be verified in the dataset to be verified, so as to determine the target data to be verified that matches each preset regular expression; and Determine the numerical values ​​of the target data to be checked that match each preset regular expression, which are the analysis result information; When the target field type is numeric, the data analysis rules include numeric type determination rules and sub-analysis rules corresponding to each numeric type. Based on the data analysis rules, the data set to be verified is processed to obtain the following analysis results: Based on the aforementioned numerical type determination rules, the data set to be verified is processed to determine the target numerical type; and Using the sub-analysis rules corresponding to the target numerical type, the data set to be verified is processed to obtain the analysis result information; The determination of target data verification rules based on the analysis results includes: Sort the given values ​​according to their magnitude to obtain a target sequence of values; From the target number of numerical values, determine the first target number of target numerical values; and The preset regular expression corresponding to each of the target values ​​is used as the target data verification rule; The data verification rule recommender device is also used for: The target data verification rule is used as a preset data verification rule associated with the data verification rule recommendation request for the data to be verified.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method of any one of claims 1-4.

Images