Master data auditing method and device, electronic equipment and storage medium

Abstract

The invention provides a master data auditing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining a business data set and a master data set; the business data set comprises at least one business data item, and the main data set comprises at least one main data item; based on each business data item and each main data item, auditing each main data item by adopting a similarity measurement mode and a recommendation mechanism; the recommendation mechanism is used for recommending the main data item as a correction item of the business data item. Through a similarity measurement mode and a recommendation mechanism, the most appropriate correction item is recommended under the condition that incomplete matching exists between the business data and the master data, and the efficiency and accuracy of master data auditing are improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a master data audit method, device, electronic equipment and storage medium. Background Art

[0002] With the increasing complexity of railway business systems and the increase in data volume, data quality management is becoming increasingly important. As the core data in railway business systems, master data is widely used in multiple business processes. However, in actual business operations, business data is often inconsistent or incompletely matched with master data, resulting in a decline in data quality and affecting subsequent business processing.

[0003] Existing master data auditing methods mainly rely on the logic of exact matching, which is difficult to handle approximate matching problems caused by input errors or other reasons, making the efficiency of master data auditing low. Summary of the invention

[0004] The present invention provides a master data audit method, device, electronic equipment and storage medium to solve the problem of low efficiency of master data audit.

[0005] The present invention provides a master data audit method, comprising: Acquire a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; Based on each of the business data items and each of the master data items, each of the master data items is audited using a similarity measurement method and a recommendation mechanism; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item.

[0006] According to a master data audit method provided by the present invention, based on each of the business data items and each of the master data items, a similarity measurement method and a recommendation mechanism are used to audit each of the master data items, including: For each business data item, respectively calculating a similarity score between the business data item and each of the master data items; Based on each of the similarity scores and the recommendation mechanism, each of the master data items is audited.

[0007] According to a master data audit method provided by the present invention, auditing each of the master data items based on the similarity scores and the recommendation mechanism includes: Comparing each of the similarity scores with a preset threshold value respectively; In the case where there is a similarity score greater than the preset threshold among the similarity scores, auditing each of the master data items based on the recommendation mechanism; In the case that there is no similarity score greater than the preset threshold among the similarity scores, it is returned that there is no recommended correction item.

[0008] According to a master data audit method provided by the present invention, the audit of each of the master data items based on the recommendation mechanism includes: In the case where the number of similarity scores greater than the preset threshold is a target value, the master data items corresponding to the similarity scores greater than the preset threshold are recommended as correction items of the business data items; When the number of similarity scores greater than the preset threshold is not the target value, the master data items corresponding to the similarity scores greater than the preset threshold are averaged, and the average is recommended as a correction item for the business data item.

[0009] According to a master data audit method provided by the present invention, the method further includes: A recommended correction item list is output, wherein the correction item list includes a plurality of recommended correction items of the business data items and a plurality of non-recommended business data items.

[0010] According to a master data audit method provided by the present invention, the similarity measurement method is an edit distance method or a cosine similarity method.

[0011] The present invention also provides a master data auditing device, comprising: An acquisition module, used to acquire a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; The audit module is used to audit each of the business data items and each of the master data items by using a similarity measurement method and a recommendation mechanism; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the master data audit methods described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the master data audit method described in any one of the above methods is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the master data audit method described above is implemented.

[0015] The master data audit method, device, electronic device and storage medium provided by the present invention obtain a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; based on each of the business data items and each of the master data items, a similarity measurement method and a recommendation mechanism are used to audit each of the master data items; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item. Through the similarity measurement method and the recommendation mechanism, the most appropriate correction item can be recommended in the case of an incomplete match between the business data and the master data, thereby improving the efficiency and accuracy of the master data audit. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the master data audit method provided by the present invention.

[0018] Figure 2 This is the second flow chart of the master data audit method provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the master data auditing device provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The existing fully matching master data auditing method is prone to missing a large number of incomplete matching items, which requires an increase in the workload of manual reprocessing; it has poor flexibility; it is more suitable for simple verification and cleanup tasks, and has limited ability to handle complex data relationships. Manually detecting data problems is only half the work, and they need to be compared and corrected one by one to the correct master data items, which makes it inflexible.

[0023] Combine the following Figure 1-Figure 2 The master data audit method of the present invention is described.

[0024] Figure 1 This is one of the flow charts of the master data audit method provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 and 102.

[0025] Step 101: Acquire a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item.

[0026] It should be noted that the master data audit method provided by the present invention can be applied to the master data audit scenario in the railway field, and the executor of the method can be a master data audit device, such as an electronic device, or a control module in the master data audit device for executing the master data audit method.

[0027] Specifically, business data item (D) refers to multiple business data collected from actual business operations. Business data is usually uncleaned data and may have data quality issues, such as duplication, irregular format or errors. The types of business data include at least one of the following: vehicle data, engineering data, and electrical data; among which, vehicle data includes the running status, dispatch, and train number information of the relevant trains; engineering data includes information related to the maintenance of railway lines and infrastructure, such as track inspections and maintenance records; electrical data includes data related to railway power equipment and signal equipment, such as the status and maintenance records of signal machines or turnouts.

[0028] Master data items (M) refer to multiple core master data that have been standardized and verified. Master data is data with a stable structure and format that can be coded and uniquely identified, and is often used for consistency reference in multiple business systems. For example, master data includes station names and / or line names; among them, station names are the standard names of various stations in the national railway network, and line names are the official names of railway lines and their codes.

[0029] Business Dataset ,in is the i-th business data item, each Represents a specific business data, which may be irregular or inconsistent. The main data set M={m1,m2,…,m k},in is the jth master data item. Represents a standardized master data item that serves as a reference for business data and is used to detect and correct inconsistencies or errors in business data.

[0030] Step 102: Based on each of the business data items and each of the master data items, each of the master data items is audited using a similarity measurement method and a recommendation mechanism; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item.

[0031] Specifically, the similarity measurement method is the edit distance method or the cosine similarity method, where the edit distance (Levenshtein Distance) is defined as the minimum number of editing operations required to convert one string into another string. These operations include inserting, deleting, and replacing single characters. The similarity of two data items is evaluated by calculating the minimum number of editing operations; the edit distance method uses the formula It indicates that, and Respectively represent business data items and master data items The cosine similarity measures the similarity of two data items by comparing their vector representations and calculating their cosine angle. The recommendation mechanism is used to recommend the master data item as a correction item of the business data item, that is, to replace the master data item with the business data item.

[0032] Based on each business data item and each master data item, similarity measurement methods and recommendation mechanisms can be used to audit each master data item, that is, to discover inconsistencies between business data and master data, and recommend appropriate corrections to ensure the accuracy and consistency of business data.

[0033] The master data audit method provided by the present invention obtains a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; based on each of the business data items and each of the master data items, a similarity measurement method and a recommendation mechanism are used to audit each of the master data items; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item. Through the similarity measurement method and the recommendation mechanism, the most appropriate correction item can be recommended in the case of an incomplete match between the business data and the master data, thereby improving the efficiency and accuracy of the master data audit.

[0034] Optionally, a specific implementation of step 102 includes: For each business data item, a similarity score between the business data item and each of the master data items is calculated respectively; and based on each of the similarity scores and the recommendation mechanism, each of the master data items is audited.

[0035] Specifically, for each business data item , using edit distance or cosine similarity to calculate business data items and each master data item Based on the similarity scores and recommendation mechanism, each master data item can be further audited.

[0036] Optionally, auditing each of the master data items based on the similarity scores and the recommendation mechanism includes: Compare each of the similarity scores with a preset threshold value; if there is a similarity score greater than the preset threshold value among the similarity scores, audit each of the master data items based on the recommendation mechanism; if there is no similarity score greater than the preset threshold value among the similarity scores, return no recommended correction item.

[0037] Specifically, the preset threshold value θ is a preset threshold value, for example, θ=0.8. Each similarity score is compared with the preset threshold value to determine whether each similarity score is greater than the preset threshold value. In the case where there is a similarity score greater than the preset threshold value among the similarity scores, that is, there is a similarity score exceeding the preset threshold value, based on the recommendation mechanism, each master data item can be audited. In the case where there is no similarity score greater than the preset threshold value among the similarity scores, that is, there is no similarity score that meets the preset threshold standard, and no recommended correction item is returned.

[0038] Optionally, auditing each of the master data items based on the recommendation mechanism includes: When the number of similarity scores greater than the preset threshold is the target value, the master data items corresponding to the similarity scores greater than the preset threshold are recommended as correction items for the business data items; when the number of similarities greater than the preset threshold is not the target value, the average of the master data items corresponding to the similarity scores greater than the preset threshold is calculated, and the average is recommended as the correction item for the business data item.

[0039] Specifically, the target value is 1. When the number of similarity scores greater than the preset threshold is the target value, that is, the number of similarity scores greater than the preset threshold is 1, the master data items corresponding to the similarity scores greater than the preset threshold are Recommended as business data item For example, suppose the business data item in the business data set is For "Beijing Station", the only master data item in the master data set The similarity score between the two is calculated by similarity measurement. , the preset threshold θ=0.8, the similarity score 0.85 is greater than the preset threshold 0.8, and the number of similarity scores greater than 0.8 is 1, the master data item Recommended as The correction item.

[0040] In the case where the number of similarity scores greater than the preset threshold is not the target value, that is, the number of similarity scores greater than the preset threshold is multiple, the master data items corresponding to the similarity scores greater than the preset threshold are Calculate the average value and recommend it as a business data item For example, suppose the business data item in the business data set is The master data item in the master data set is "Shanghai Hongqiao", and the multiple master data items in the master data set are "Shanghai Station", "Shanghai Hongqiao Station" and "Hongqiao Station". The similarity measurement method is used to calculate the similarity between each master data item "Shanghai Station", "Shanghai Hongqiao Station" and "Hongqiao Station" and the business data item The similarity score between "Shanghai Hongqiao", for example, the similarity score between "Shanghai Station" and "Shanghai Hongqiao" is 0.3, the similarity score between "Shanghai Hongqiao Station" and "Shanghai Hongqiao" is 0.4, and the similarity score between "Hongqiao Station" and "Shanghai Hongqiao" is 0.3. The preset threshold is 0.25, then the number of similarity scores greater than the preset threshold 0.25 is 3. The average value of each master data item "Shanghai Station", "Shanghai Hongqiao Station" and "Hongqiao Station" is calculated, and the average value is recommended as the correction item for the business data item "Shanghai Hongqiao".

[0041] Optionally, the method further comprises: A recommended correction item list is output, wherein the correction item list includes a plurality of recommended correction items of the business data items and a plurality of non-recommended business data items.

[0042] Specifically, the final output is the recommended correction item list R={r1,r2,…,r n}, each Corresponding business data items Recommended fixes, Recommended master data items Or unrecommended business data items When the most suitable master data item is matched by similarity measurement , and when the similarity score exceeds the preset threshold, It is recommended to replace the business data item Master data items , that is, it is recommended to Replace with To ensure data accuracy and consistency. If no master data item can be found by similarity measurement Satisfy the requirement that the similarity score exceeds the preset threshold, Will return the original business data item , which means that no suitable correction item was found, so the business data item Modification is not recommended.

[0043] Figure 2 This is the second flow chart of the master data audit method provided by the present invention. Figure 2 As shown, the method includes steps 201 to 208.

[0044] Step 201: Acquire a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item.

[0045] Step 202: For each business data item, respectively calculate the similarity score between the business data item and each master data item.

[0046] Step 203: Determine whether each similarity score is greater than a preset threshold. If there is a similarity score greater than the preset threshold among the similarity scores, go to step 204; if there is no similarity score greater than the preset threshold among the similarity scores, go to step 207.

[0047] Step 204: Determine whether the number of similarity scores greater than the preset threshold is the target value. If the number of similarity scores greater than the preset threshold is the target value, go to step 205; if the number of similarity scores greater than the preset threshold is not the target value, go to step 206.

[0048] Step 205: recommend the master data item corresponding to the similarity score greater than the preset threshold as a correction item of the business data item.

[0049] Step 206: Calculate the average value of each master data item corresponding to the similarity score greater than the preset threshold, and recommend the average value as a correction item of the business data item.

[0050] Step 207: Return that there is no recommended correction item.

[0051] Step 208: Output a recommended correction item list, where the correction item list includes a plurality of recommended correction items for the business data items and a plurality of non-recommended business data items.

[0052] The master data audit method provided by the present invention is used to perform efficient and intelligent audit and correction of business data, and can more flexibly handle data audit problems that are not fully matched and more in line with real business scenarios. By introducing a similarity measurement method, it is possible to intelligently handle incomplete matching situations, improve the comprehensiveness of the audit, and be applicable to various types of business data and master data. Different similarity measurement methods can be selected according to actual needs, and the recommended correction items can be flexibly controlled by adjusting the preset threshold of the similarity score to meet the needs of different business scenarios; by introducing an intelligent recommendation mechanism, after completing the problem items found by the data audit, the correction items can be automatically recommended through calculation, reducing manual workload, thereby improving the accuracy and efficiency of data audits.

[0053] The master data auditing device provided by the present invention is described below. The master data auditing device described below and the master data auditing method described above can be referenced to each other.

[0054] Figure 3 Schematic diagram of the structure of the master data audit device provided by the present invention. Figure 3 As shown, the master data audit device 300 includes: an acquisition module 301 and an audit module 302; wherein,

[0055] The acquisition module 301 is used to acquire a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; The audit module 302 is used to audit each of the business data items and each of the master data items by using a similarity measurement method and a recommendation mechanism; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item.

[0056] The master data auditing device provided by the present invention obtains a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; based on each of the business data items and each of the master data items, a similarity measurement method and a recommendation mechanism are used to audit each of the master data items; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item. Through the similarity measurement method and the recommendation mechanism, the most appropriate correction item can be recommended in the case of an incomplete match between the business data and the master data, thereby improving the efficiency and accuracy of the master data audit.

[0057] Optionally, the audit module 302 is specifically used to: For each business data item, respectively calculating a similarity score between the business data item and each of the master data items; Based on each of the similarity scores and the recommendation mechanism, each of the master data items is audited.

[0058] Optionally, the audit module 302 is further used to: Comparing each of the similarity scores with a preset threshold value respectively; In the case where there is a similarity score greater than the preset threshold among the similarity scores, auditing each of the master data items based on the recommendation mechanism; In the case that there is no similarity score greater than the preset threshold among the similarity scores, it is returned that there is no recommended correction item.

[0059] Optionally, the audit module 302 is further used to: In the case where the number of similarity scores greater than the preset threshold is a target value, the master data items corresponding to the similarity scores greater than the preset threshold are recommended as correction items of the business data items; When the number of similarity scores greater than the preset threshold is not the target value, the master data items corresponding to the similarity scores greater than the preset threshold are averaged, and the average is recommended as a correction item for the business data item.

[0060] Optionally, the master data auditing device 300 further includes: The output module is used to output a recommended correction item list, wherein the correction item list includes correction items of a plurality of recommended business data items and a plurality of non-recommended business data items.

[0061] Optionally, the similarity measurement method is an edit distance method or a cosine similarity method.

[0062] Figure 4 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 4 As shown, the electronic device 400 may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the master data audit method, which includes: obtaining a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; based on each of the business data items and each of the master data items, a similarity measurement method and a recommendation mechanism are used to audit each of the master data items; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item.

[0063] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0064] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the master data audit method provided by the above methods, and the method includes: obtaining a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; based on each of the business data items and each of the master data items, a similarity measurement method and a recommendation mechanism are used to audit each of the master data items; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item.

[0065] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the master data audit method provided by the above-mentioned methods, the method comprising: obtaining a business data set and a master data set; the business data set comprises at least one business data item, and the master data set comprises at least one master data item; based on each of the business data items and each of the master data items, a similarity measurement method and a recommendation mechanism are used to audit each of the master data items; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item.

[0066] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0067] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A master data audit method, characterized in that: include: Acquire a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; Based on each of the business data items and each of the master data items, each of the master data items is audited using a similarity measurement method and a recommendation mechanism; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item.

2. The master data audit method according to claim 1, characterized in that: Based on each of the business data items and each of the master data items, a similarity measurement method and a recommendation mechanism are used to audit each of the master data items, including: For each business data item, respectively calculating a similarity score between the business data item and each of the master data items; Based on each of the similarity scores and the recommendation mechanism, each of the master data items is audited.

3. The master data audit method according to claim 2, characterized in that: The auditing of each of the master data items based on the similarity scores and the recommendation mechanism includes: Comparing each of the similarity scores with a preset threshold value respectively; In the case where there is a similarity score greater than the preset threshold among the similarity scores, auditing each of the master data items based on the recommendation mechanism; In the case that there is no similarity score greater than the preset threshold among the similarity scores, it is returned that there is no recommended correction item.

4. The master data audit method according to claim 3, characterized in that: The auditing of each of the master data items based on the recommendation mechanism includes: In the case where the number of similarity scores greater than the preset threshold is a target value, the master data items corresponding to the similarity scores greater than the preset threshold are recommended as correction items of the business data items; When the number of similarity scores greater than the preset threshold is not the target value, the master data items corresponding to the similarity scores greater than the preset threshold are averaged, and the average is recommended as a correction item for the business data item.

5. The master data audit method according to any one of claims 1 to 4, characterized in that: The method further comprises: A recommended correction item list is output, wherein the correction item list includes a plurality of recommended correction items of the business data items and a plurality of non-recommended business data items.

6. The master data audit method according to any one of claims 1 to 4, characterized in that: The similarity measurement method is an edit distance method or a cosine similarity method.

7. A master data auditing device, characterized in that: include: An acquisition module, used to acquire a business data set and a master data set; the business data set includes at least one business data item, and the master data set includes at least one master data item; The audit module is used to audit each of the business data items and each of the master data items by using a similarity measurement method and a recommendation mechanism; the recommendation mechanism is used to recommend the master data item as a correction item of the business data item.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the master data audit method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the master data audit method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the master data audit method according to any one of claims 1 to 6 is implemented.

Images