Data element evaluation method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data element evaluation method and system, electronic equipment and a storage medium, and belongs to the field of data processing. The method comprises the following steps: receiving a to-be-evaluated data element, and generating a corresponding identification code; according to the characteristics and requirements of the data elements, evaluation indexes are determined, a corresponding evaluation standard is formulated for each evaluation index, and the evaluation indexes include but not limited to integrity, accuracy, normalization, consistency and timeliness; based on the evaluation indexes and the evaluation standards, evaluating the data elements by using a pre-constructed evaluation model; and analyzing the identification code corresponding to each data element to obtain an evaluation result of each data element. According to the method, the quality of the data elements is automatically and efficiently evaluated through the evaluation model, the accuracy and efficiency of data processing are improved, the data elements are comprehensively, meticulously and quantitatively evaluated through the identification analysis technology, and support is provided for data selection, processing and application.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and specifically to a data element evaluation method, system, electronic device and storage medium. Background Art

[0002] With the rapid development of information technology, data has become an indispensable and important resource in modern society. Driven by emerging technologies such as big data, cloud computing, and artificial intelligence, the quality and value of data elements are crucial to data collection, storage, processing, analysis, and application. However, due to the diversity of data sources, the complexity of data formats, and the uneven quality of data, how to effectively evaluate the quality and value of data elements has become an urgent problem to be solved.

[0003] Data element evaluation is an important part of the data processing process. Its purpose is to evaluate the integrity, accuracy, consistency, availability and other aspects of data to determine the quality and value of data. However, traditional data element evaluation methods often rely on manual judgment and experience summary, which is not only inefficient but also easily affected by subjective factors, resulting in inaccurate evaluation results. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data element evaluation method, system, electronic device and storage medium, which are used to solve all or at least part of the technical problems existing in the above-mentioned prior art.

[0005] In order to achieve the above object, an embodiment of the present invention provides a data element evaluation method, comprising:

[0006] receiving data elements to be evaluated, and generating corresponding identification codes for each data element;

[0007] Determine the evaluation indicators based on the characteristics and requirements of the data elements, and formulate corresponding evaluation standards for each evaluation indicator, where the evaluation indicators include but are not limited to completeness, accuracy, standardization, consistency and timeliness;

[0008] Based on the evaluation index and the evaluation standard and using a pre-built evaluation model, the data element is evaluated;

[0009] Parse the identification code corresponding to each data element to obtain the evaluation result of each data element.

[0010] Optionally, receiving the data elements to be evaluated and generating a corresponding identification code for each data element includes:

[0011] According to the first category of the data element to be evaluated, construct an identification prefix of the identification code using the industry code corresponding thereto, wherein the type and length of the identification prefix are related to the first category of the data element;

[0012] constructing an identification suffix of the identification code according to the second category of the data element to be evaluated, wherein the type and length of the identification suffix are related to the second category of the data element;

[0013] The second category is a subcategory of the first category.

[0014] Optionally, determine the evaluation indicators based on the characteristics and requirements of the data elements, and formulate corresponding evaluation standards for each evaluation indicator, including:

[0015] Corresponding evaluation criteria are formulated for integrity evaluation indicators, which are characterized by checking whether the field is not empty;

[0016] The corresponding evaluation criteria are formulated for the accuracy evaluation index to represent whether the content of the verification field conforms to the preset regular expression;

[0017] Corresponding evaluation criteria are formulated for normative evaluation indicators to represent whether the verification fields comply with the mailbox rules;

[0018] The corresponding evaluation standard for consistency evaluation indicators is formulated to verify whether the current field of the same category of data elements is consistent with the historical field;

[0019] Corresponding evaluation standards are formulated for timeliness evaluation indicators, which are represented by the update frequency of verification data elements.

[0020] Optionally, based on the evaluation index, the evaluation standard and a pre-built evaluation model, the data element is evaluated, including:

[0021] Integrate the evaluation index and the evaluation standard to generate an evaluation index system;

[0022] Each data element is input into the evaluation model respectively, and the evaluation model performs the following verification process one by one:

[0023] Check whether the field is not empty. If it is not empty, it indicates that the data element is complete.

[0024] Check whether the field content conforms to the preset regular expression. If it does, it indicates that the data element is accurate.

[0025] Check whether the field complies with the mailbox rules. If it does, it indicates that the data element is standardized.

[0026] Verify that the current field of the same category of data elements is consistent with the historical field. If they are consistent, the data elements are consistent.

[0027] The update frequency of the data elements is verified. If the update frequency of the data elements meets the preset requirements, it indicates that the data elements are timely.

[0028] Optionally, the evaluation model construction process includes:

[0029] Based on the evaluation index and the evaluation criteria, determining the structure of the evaluation model and setting the qualified weight of each evaluation criterion;

[0030] The evaluation model is trained using historical data so that the evaluation model can accurately calculate the evaluation results of the data elements and obtain the target evaluation model.

[0031] Optionally, after parsing the identification code corresponding to each data element and obtaining the evaluation result of each data element, the data element evaluation method further includes:

[0032] Using the identification resolution algorithm, the data elements with identification codes are analyzed to extract the key attributes and routing information of the data elements;

[0033] Based on the routing information, determining a source address of the data element;

[0034] Based on the key attributes, the quality and value of the data element is determined.

[0035] Optionally, an identification resolution algorithm is used to resolve data elements with identification codes to extract key attributes and routing information of the data elements, including:

[0036] For data elements containing eCode characters, use the registration and parsing method of the eCode identifier:

[0037] According to the encoding rules of eCode, a separator for separating different parts is determined, and the eCode string is divided into different parts using the separator, wherein each part represents different attribute information;

[0038] Constructing a query statement or request for query or mapping according to the segmented eCode parts, and using the constructed query statement or request to access the eCode registration center and database to obtain key attributes and routing information associated with the eCode;

[0039] Receive responses from registries and databases, and parse response data to extract key attributes and routing information;

[0040] For data elements containing OID characters, use the OID identification resolution method:

[0041] The data elements containing OID characters are split according to points to obtain a series of digital nodes, where the first number represents the root node;

[0042] Start from the root node and query each node step by step downward, where each node has a unique path consisting of all nodes from the root node to the node in the OID tree;

[0043] By parsing the OID, key attribute information associated with a specific object is obtained, wherein each node in the OID tree represents a characteristic attribute.

[0044] On the other hand, the present invention also provides a data element evaluation system, comprising:

[0045] A receiving module, used for receiving the data elements to be evaluated and generating a corresponding identification code for each data element;

[0046] An evaluation standard formulation module is used to determine evaluation indicators according to the characteristics and requirements of data elements, and formulate corresponding evaluation standards for each evaluation indicator, wherein the evaluation indicators include but are not limited to completeness, accuracy, standardization, consistency and timeliness;

[0047] An evaluation module, used to evaluate the data elements based on the evaluation indicators, the evaluation standards and in combination with a pre-built evaluation model;

[0048] The parsing module is used to parse the identification code corresponding to each data element and obtain the evaluation result of each data element.

[0049] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data element evaluation method when executing the program.

[0050] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned data element evaluation method when executed by a processor.

[0051] Through the above technical solution, the evaluation model is used to realize the automatic and efficient evaluation of the quality of data elements, improve the accuracy and efficiency of data processing, and use the identification resolution technology to conduct a comprehensive, detailed and quantitative evaluation of data elements, providing strong support for the selection, processing and application of data. Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent specific implementation method section. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0053] Figure 1 It is an implementation flow chart of a data element evaluation method provided by an embodiment of the present invention;

[0054] Figure 2 It is a structural diagram of a data element evaluation system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0056] See also Figure 1 As shown, it is a flowchart of an implementation method of a data element evaluation method provided by an embodiment of the present invention, including the following execution steps:

[0057] Step 100: Receive data elements to be evaluated, and generate corresponding identification codes for each data element.

[0058] The data elements include but are not limited to people, objects, materials and industrial equipment.

[0059] Specifically, when executing step 100, the following steps may be specifically performed:

[0060] S1000: According to the first category of the data element to be evaluated, construct an identification prefix of the identification code using its corresponding industry code, wherein the type and length of the identification prefix are related to the first category of the data element.

[0061] S1001: Constructing an identification suffix of an identification code according to a second category of the data element to be evaluated, wherein a type and a length of the identification suffix are related to the second category of the data element.

[0062] By assigning a unique identification code to each data element and recording its source, attributes, format and other basic information, the uniqueness and traceability of the data element can be ensured.

[0063] S1002: The second category is a subcategory of the first category.

[0064] Step 101: Determine evaluation indicators based on the characteristics and requirements of data elements, and formulate corresponding evaluation standards for each evaluation indicator, wherein the evaluation indicators include but are not limited to completeness, accuracy, standardization, consistency and timeliness.

[0065] Specifically, when executing step 101, the following steps may be specifically performed:

[0066] S1010: Formulate corresponding evaluation criteria for the integrity evaluation index, which is characterized by checking whether the field is not empty.

[0067] S1011: A corresponding evaluation standard is formulated for the accuracy evaluation index to represent whether the content of the verification field conforms to a preset regular expression.

[0068] S1012: Corresponding evaluation standards are formulated for normative evaluation indicators to represent whether the verification fields comply with mailbox rules.

[0069] S1013: Corresponding evaluation standards are formulated for consistency evaluation indicators, which are characterized by verifying whether the current fields of data elements of the same category are consistent with the historical fields.

[0070] S1014: Formulate corresponding evaluation standards for the timeliness evaluation index, which are represented by the update frequency of the verification data elements.

[0071] Step 102: Evaluate the data elements based on the evaluation indicators, the evaluation standards and using a pre-built evaluation model.

[0072] Specifically, when executing step 102, the following steps may be specifically performed:

[0073] S1020: Integrate the evaluation index and the evaluation standard to generate an evaluation index system.

[0074] S1021: Input each data element into the evaluation model respectively, and the evaluation model performs the following verification process one by one:

[0075] Check whether the field is not empty. If it is not empty, it indicates that the data element is complete; check whether the field content conforms to the preset regular expression. If it does, it indicates that the data element is accurate; check whether the field conforms to the mailbox rules. If it does, it indicates that the data element is standardized; check whether the current field of the same category of data elements is consistent with the historical field. If they are consistent, it indicates that the data element is consistent; check the update frequency of the data element. If the update frequency of the data element meets the preset requirements, it indicates that the data element is timely.

[0076] For example, the accuracy check of a certain field content uses the regular expression: / ^(?=.*[az])(?=.*[AZ])(?=.*\d)[az / AZ\dl8.201$ / . The one that meets the regular expression is qualified, otherwise it is unqualified. The consistency check is whether a certain field of the current data asset is consistent with the previous data asset (unmodified). If it is not modified, it is qualified. The standardization check is whether a certain field complies with the mobile phone number or email rules. If it is consistent, it is qualified.

[0077] In some embodiments, the process of constructing the evaluation model includes: determining the structure of the evaluation model and setting the qualified weight of each evaluation criterion based on the evaluation indicators and the evaluation criteria; training the evaluation model with historical data so that the evaluation model accurately calculates the evaluation results of the data elements to obtain the target evaluation model.

[0078] In the evaluation implementation stage, the present invention automatically evaluates the data elements to be evaluated based on the evaluation model, effectively reducing manual participation and intervention and improving evaluation efficiency. At the same time, the evaluation process is strictly carried out in accordance with the evaluation model to ensure the objectivity and accuracy of the evaluation results. After the evaluation is completed, the evaluation results and existing problems are recorded to provide a basis for subsequent improvement and optimization.

[0079] In some embodiments, the evaluation results are analyzed in depth to explore the quality and value of the data elements. This can be done through charts, statistical analysis, etc. Feedback and suggestions are provided to the user based on the evaluation results and analysis results to help the user improve and optimize the quality and value of the data elements.

[0080] Step 103: parse the identification code corresponding to each data element to obtain the evaluation result of each data element.

[0081] Specifically, when executing step 103, the following steps may be performed:

[0082] S1030: Utilize an identification resolution algorithm to resolve data elements with identification codes, and extract key attributes and routing information of the data elements.

[0083] Specifically, when executing step S1030, there are three situations as follows:

[0084] Scenario 1: For data elements containing eCode characters, the registration and parsing method of the eCode identifier is used: according to the encoding rules of the eCode, the separator used to separate different parts is determined, and the eCode string is divided into different parts using the separator, where each part represents different attribute information; according to the divided eCode parts, a query statement or request for query or mapping is constructed, and the constructed query statement or request is used to access the eCode registration center and database to obtain the key attributes and routing information associated with the eCode; a response is received from the registration center and database, and the response data is parsed to extract the key attributes and routing information.

[0085] Scenario 2: For data elements containing OID characters, use the OID identification resolution method: split the data elements containing OID characters according to points to obtain a series of digital nodes, where the first number represents the root node; start from the root node and query each node step by step downward, where each node has a unique path consisting of all nodes from the root node to the node in the OID tree; by parsing the OID, obtain the key attribute information associated with the specific object, where each node in the OID tree represents a characteristic attribute.

[0086] Scenario 3: For data elements containing Handles, the Handle parsing method is used: based on the initial part of the Handle, the Handle server responsible for parsing the Handle is determined; a parsing request containing the complete Handle is sent to the Handle server; the Handle server returns the parsing result, including the location information pointing to the target resource; key attributes and actual routing information are extracted from the returned metadata.

[0087] S1031: Based on the routing information, determine the source address of the data element.

[0088] S1032: Based on the key attributes, determine the quality and value of the data elements.

[0089] Through the above-mentioned technical solution, the evaluation model is used to realize automatic and efficient evaluation of the quality of data elements, thereby improving the accuracy and efficiency of data processing. Moreover, the identification resolution technology is used to conduct a comprehensive, detailed and quantitative evaluation of data elements, providing strong support for data selection, processing and application.

[0090] See also Figure 2 FIG. 1 is a schematic diagram of a data element evaluation system provided by an embodiment of the present invention, comprising:

[0091] The receiving module 20 is used to receive the data elements to be evaluated and generate a corresponding identification code for each data element;

[0092] The evaluation standard formulation module 21 is used to determine the evaluation indicators according to the characteristics and requirements of the data elements, and formulate corresponding evaluation standards for each evaluation indicator, wherein the evaluation indicators include but are not limited to completeness, accuracy, standardization, consistency and timeliness;

[0093] An evaluation module 22, for evaluating data elements based on the evaluation indicators, the evaluation criteria and in combination with a pre-built evaluation model;

[0094] The parsing module 23 is used to parse the identification code corresponding to each data element to obtain the evaluation result of each data element.

[0095] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data element evaluation method described in any one of the above embodiments when executing the program.

[0096] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data element evaluation method described in any of the above embodiments.

[0097] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0102] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0103] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0105] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data element evaluation method, characterized in that: include: receiving data elements to be evaluated, and generating corresponding identification codes for each data element; Determine the evaluation indicators based on the characteristics and requirements of the data elements, and formulate corresponding evaluation standards for each evaluation indicator, where the evaluation indicators include but are not limited to completeness, accuracy, standardization, consistency and timeliness; Based on the evaluation index and the evaluation standard and using a pre-built evaluation model, the data element is evaluated; Parse the identification code corresponding to each data element to obtain the evaluation result of each data element.

2. The data element evaluation method according to claim 1, characterized in that: Receive the data elements to be evaluated, and generate a corresponding identification code for each data element, including: According to the first category of the data element to be evaluated, construct an identification prefix of the identification code using the industry code corresponding thereto, wherein the type and length of the identification prefix are related to the first category of the data element; constructing an identification suffix of the identification code according to the second category of the data element to be evaluated, wherein the type and length of the identification suffix are related to the second category of the data element; The second category is a subcategory of the first category.

3. The data element evaluation method according to claim 1, characterized in that: Determine the evaluation indicators based on the characteristics and needs of the data elements, and formulate corresponding evaluation standards for each evaluation indicator, including: Corresponding evaluation criteria are formulated for integrity evaluation indicators, which are characterized by checking whether the field is not empty; The corresponding evaluation criteria are formulated for the accuracy evaluation index to represent whether the content of the verification field conforms to the preset regular expression; Corresponding evaluation criteria are formulated for normative evaluation indicators to represent whether the verification fields comply with the mailbox rules; The corresponding evaluation standard for consistency evaluation indicators is formulated to verify whether the current field of the same category of data elements is consistent with the historical field; Corresponding evaluation standards are formulated for timeliness evaluation indicators, which are represented by the update frequency of verification data elements.

4. The data element evaluation method according to claim 3, characterized in that: Based on the evaluation indicators, the evaluation standards and in combination with the pre-built evaluation model, the data elements are evaluated, including: Integrate the evaluation index and the evaluation standard to generate an evaluation index system; Each data element is input into the evaluation model respectively, and the evaluation model performs the following verification process one by one: Check whether the field is not empty. If it is not empty, it indicates that the data element is complete. Check whether the field content conforms to the preset regular expression. If it does, it indicates that the data element is accurate. Check whether the field complies with the mailbox rules. If it does, it indicates that the data element is standardized. Verify that the current field of the same category of data elements is consistent with the historical field. If they are consistent, the data elements are consistent. The update frequency of the data elements is verified. If the update frequency of the data elements meets the preset requirements, it indicates that the data elements are timely.

5. The data element evaluation method according to any one of claims 1 to 4, characterized in that: The construction process of the evaluation model includes: Based on the evaluation index and the evaluation criteria, determining the structure of the evaluation model and setting the qualified weight of each evaluation criterion; The evaluation model is trained using historical data so that the evaluation model can accurately calculate the evaluation results of the data elements and obtain the target evaluation model.

6. The data element evaluation method according to claim 1, characterized in that: After parsing the identification code corresponding to each data element, the evaluation result of each data element is obtained, including: Using the identification resolution algorithm, the data elements with identification codes are analyzed to extract the key attributes and routing information of the data elements; Based on the routing information, determining a source address of the data element; Based on the key attributes, the quality and value of the data element is determined.

7. The data element evaluation method according to claim 6, characterized in that: Using the identification resolution algorithm, the data elements with identification codes are parsed to extract the key attributes and routing information of the data elements, including: For data elements containing eCode characters, use the registration and parsing method of the eCode identifier: According to the encoding rules of eCode, a separator for separating different parts is determined, and the eCode string is divided into different parts using the separator, wherein each part represents different attribute information; Constructing a query statement or request for query or mapping according to the segmented eCode parts, and using the constructed query statement or request to access the eCode registration center and database to obtain key attributes and routing information associated with the eCode; Receive responses from registries and databases, and parse response data to extract key attributes and routing information; For data elements containing OID characters, use the OID identification resolution method: The data elements containing OID characters are split according to points to obtain a series of digital nodes, where the first number represents the root node; Start from the root node and query each node step by step downward, where each node has a unique path consisting of all nodes from the root node to the node in the OID tree; By parsing the OID, key attribute information associated with the target object is obtained, wherein each node in the OID tree represents a characteristic attribute.

8. A data element evaluation system, characterized in that: include: A receiving module, used for receiving the data elements to be evaluated and generating a corresponding identification code for each data element; An evaluation criteria formulation module is used to determine evaluation indicators according to the characteristics and requirements of data elements, and formulate corresponding evaluation standards for each evaluation indicator, wherein the evaluation indicators include but are not limited to completeness, accuracy, standardization, consistency and timeliness; An evaluation module, used to evaluate the data elements based on the evaluation indicators, the evaluation standards and in combination with a pre-built evaluation model; The parsing module is used to parse the identification code corresponding to each data element and obtain the evaluation result of each data element.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the steps of the data element evaluation method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data element evaluation method as described in any one of claims 1 to 7 are implemented.

Images