A method and system for detecting quality of dynamic arrangement of industrial data

Abstract

The present invention discloses a method and system for detecting the quality of dynamic arrangement of industrial data, which relates to the technical field of data governance and quality detection, including formulating industrial data standards, performing metadata inspection of dropped tags, collecting data of various cities into a collection library according to different data types and integration methods; configuring data quality models and quality inspection rules, automatically generating a mapping relationship between data standards and data models, configuring and executing quality inspection plans, and performing data quality audits; diverting audit data, extracting correct data into a standard library, and marking data labels to form data assets, and returning data to the source of data to form a complete closed loop of data governance. The method described in the present invention improves the efficiency and accuracy of data quality inspections and reduces the workload and errors of manual inspections by performing data quality audits; forms closed-loop management through data reflux, establishes a continuously improved data governance mechanism, and ensures the long-term stability and optimization of data quality.

Description

Technical Field

[0001] The present invention relates to the technical field of data governance and quality detection, and specifically to a method and system for detecting the quality of dynamic arrangement of industrial data. Background Art

[0002] With the rapid development of the industrial Internet, the collection and processing technology of industrial data has also made significant progress. Industrial data involves multiple departments, multiple systems and multiple industries, including manufacturing, energy, transportation and other fields. These data are generated by sensors, monitoring systems and other data collection equipment, covering equipment status, production processes, environmental monitoring and other aspects. In order to effectively manage and utilize these huge amounts of data, data standardization and governance technologies have emerged. Existing data governance technologies mainly rely on architectures such as data warehouses, data lakes and data middle platforms. By establishing unified data standards and specifications, data cleaning, integration and analysis are carried out to support corporate operational decisions and intelligent manufacturing. However, with the continuous growth of data scale and the complexity of data structure, traditional data governance technologies face huge challenges and require more efficient and intelligent solutions.

[0003] Although existing data governance technologies have achieved certain results in processing industrial data, there are still many shortcomings. The formulation of data standards lacks uniformity and flexibility, and the data standards of different systems and departments are inconsistent, resulting in poor data integration and interoperability. Existing data quality inspection methods mainly rely on static rules and manual review, which cannot automatically adapt to the ever-changing data environment. The audit efficiency is low and prone to missed detections and false detections. In addition, the data collection and processing process is complex and lacks dynamic orchestration capabilities, making it difficult to cope with diverse data types and integration methods. Traditional data governance processes are usually linear, lack closed-loop management, and cannot achieve real-time data reflux and continuous optimization. Especially in data quality auditing, existing technologies mainly rely on predefined rules and models, lack adaptive capabilities and intelligent analysis methods, and find it difficult to cope with large-scale, complex and changeable industrial data environments. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problems solved by the present invention are: the existing data governance methods have inconsistent standards, low data quality inspection efficiency, and cannot dynamically adapt to the data environment, as well as how to achieve a complete closed loop of data governance.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a method for dynamic orchestration quality detection of industrial data, including formulating industrial data standards, performing metadata checks, and collecting data from various cities into a collection library according to different data types and integration methods; configuring data quality models and quality inspection rules, automatically generating a mapping relationship between data standards and data models, configuring and executing quality inspection plans, and performing data quality audits; diverting audit data, extracting correct data into a standard library, and marking data labels to form data assets, and returning data to the data source to form a complete closed loop of data governance.

[0007] As a preferred solution of the industrial data dynamic arrangement quality detection method described in the present invention, wherein: the formulation of industrial data standards includes establishing four types of data standards and root libraries of technology, business, management and quality according to business classification, and establishing a mapping relationship between data standards and metadata through an intelligent mapping function; when a standard is recommended for different metadata, if two or more metadata with the same Chinese name appear, matching is performed, and the matching rules include matching metadata with the same Chinese name, and the English name selection includes selecting the English name of the first metadata in the matched metadata as the unified English name, and the exclusion condition includes if there is an English name with the same name but does not contain any root. If the situation is not met, no match is performed; when matching Chinese roots and synonyms, the metadata is preliminarily screened by matching the Chinese roots and synonyms of the metadata. When the number of matched metadata is greater than or equal to 2, the selected metadata is considered to be matched. For the matched metadata, the English name directly uses the English combination of the roots, and the roots are separated by underscores; when matching English roots and English abbreviations, the metadata is preliminarily screened by matching the English roots and abbreviations of the metadata. If the number of matched metadata is greater than or equal to 2, the selected metadata is considered to be matched. For the matched metadata, the Chinese name directly uses the Chinese combination of the roots, and the roots are not separated.

[0008] As a preferred solution of the industrial data dynamic arrangement quality detection method described in the present invention, the formulation of industrial data standards also includes analyzing the metadata of the top 10 comprehensive criticality, taking the metadata with the highest criticality for matching, and calculating the criticality as:

[0009] ,

[0010] in, represents the criticality evaluation function, represents the matching degree calculation function, Represents the function for calculating the number of matching root words. Represents the association calculation function; the matching calculation function is used to measure the similarity between two data objects, expressed as:

[0011] ,

[0012] in, The number of root characters that match Chinese. is the number of Chinese characters, The number of root characters that match English. is the number of English characters. After the data standard is established, the potential mark metadata is checked and evaluated based on the number of matching root words and the degree of relevance to verify the quality of the data. The function for calculating the number of matching root words is expressed as:

[0013] ,

[0014] in, and are the minimum and maximum matching quantities for Chinese characters, and are the minimum and maximum matching numbers for English, respectively. The correlation calculation function is expressed as:

[0015] ,

[0016] in, is the correlation number, and are the minimum and maximum association numbers, respectively.

[0017] As a preferred solution of the industrial data dynamic arrangement quality detection method described in the present invention, wherein: the data quality audit includes establishing a data quality model and a data quality rule base according to data characteristics, and automatically generating quality inspection rules from data standards, the quality inspection rules include null value check, value range check, specification check, repeatability check, timeliness check, reference value check, and logic check quality audit rules, the quality inspection plan includes freely combining the quality audit rules, and setting the data period, data label and operation strategy for the execution of the plan; the data to be quality inspected is input into a pre-trained classifier model, each classifier model classifies the input data, and uses a cross-validation method to evaluate the accuracy and recall of different classifiers on the data set, and the average of the accuracy and recall of different classifiers is used as the health status assessment score of the training data.

[0018] As a preferred solution of the method for detecting the quality of dynamic arrangement of industrial data described in the present invention, the data quality audit also includes constructing a data set containing noise labels. , expressed as:

[0019] ,

[0020] in, Indicates Industrial data characteristics of samples, Indicates The noisy labels of samples, Represents the total number of samples; for noisy data Make a prediction and get the noise prediction probability of each sample, expressed as:

[0021] ,

[0022] in, Represents a given model Next, Samples Noise label The predicted probability of is the conditional probability, which means that in the model Next, given a sample Time Noise Label Based on the predicted probability, calculate the noise label and the true label The joint probability distribution of is expressed as: ,

[0023] in, represents the confidence joint matrix, represents the noise label and the true label The joint confidence of represents the Dirac delta function, when The value is 1 when , otherwise it is 0. Indicates True labels;

[0024] The label combinations are counted and normalized to obtain the joint probability distribution, which is expressed as:

[0025] ,

[0026] in, represents the normalized joint probability matrix, Represents the sum of confidences of all true labels in the confidence union matrix, which is used for normalization;

[0027] Using Cleanlab's pruning function, based on the confidence union matrix and normalized count data Perform data pruning, remove noise data, obtain clean data, calculate the confidence of each sample, and perform data pruning based on the confidence, expressed as:

[0028] ,

[0029] in, Indicates The confidence level of the samples, Indicates that all samples have noise labels The sum of the predicted probabilities under , sets the pruning threshold, removes samples below the threshold according to the confidence level, and obtains clean data, which is expressed as:

[0030] ,

[0031] in, represents a clean data set, Indicates the set pruning threshold.

[0032] As a preferred solution of the industrial data dynamic orchestration quality detection method described in the present invention, the formation of data assets includes diverting audit data, archiving abnormal data into an error data table, triggering a data quality rectification process, generating a data quality sorting work order, extracting correct data into a standard library, and marking data tags to form data assets.

[0033] As a preferred solution of the industrial data dynamic orchestration quality inspection method described in the present invention, the formation of a complete closed loop of data governance includes creating an error data archive and a quality inspection report, analyzing the quality problems in the report, separating each data from the problem data set, forming different problem data work orders according to the data source, pushing them to the data source system, tracking the repair of source data quality problems, and after the repair is completed, entering the collection library from the front-end machine again through data aggregation to perform repeated data quality management.

[0034] Another object of the present invention is to provide an industrial data dynamic orchestration quality inspection system, which can automatically generate a mapping relationship between data standards and data models by configuring data quality models and quality inspection rules, configure and execute quality inspection plans, and perform data quality audits, thereby solving the problem of low accuracy in current industrial data quality inspections.

[0035] As a preferred solution of the industrial data dynamic orchestration quality detection system described in the present invention, it includes: a data standardization module, a quality audit module, and a data diversion module; the data standardization module is used to formulate industrial data standards, perform metadata checks, and collect data from various cities into a collection library according to different data types and integration methods; the quality audit module is used to configure data quality models and quality inspection rules, automatically generate a mapping relationship between data standards and data models, configure and execute quality inspection plans, and perform data quality audits; the data diversion module is used to divert audit data, extract correct data to the standard library, and mark data labels to form data assets, and the data flows back to the data source to form a complete closed loop of data governance.

[0036] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a method for detecting the quality of dynamic arrangement of industrial data.

[0037] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for detecting the quality of dynamic arrangement of industrial data.

[0038] Beneficial effects of the present invention: The industrial data dynamic arrangement quality detection method provided by the present invention formulates industrial data standards and performs metadata checks. According to different data types and integration methods, data from various cities are collected into a collection library to ensure the uniformity and consistency of data standards, effectively integrate and summarize data, form a complete data resource library, ensure the standardization and consistency of data, improve the efficiency and accuracy of data collection, ensure that the data has undergone preliminary standardization processing before entering the collection library, and reduce the problems of format and type inconsistencies that may occur during the data integration process; by configuring data quality models and quality inspection rules, the mapping relationship between data standards and data models is automatically generated, the quality inspection plan is configured and executed, and data quality audits are performed to realize automated inspection and evaluation of data quality, ensure the quality consistency of data in different processing links, and conduct comprehensive quality inspections on data. Auditing discovers and marks quality problems in data, improves the efficiency and accuracy of data quality inspection, and reduces the workload and errors of manual inspection; by automatically generating quality inspection rules and mapping relationships, it realizes the rapid configuration and execution of quality inspection plans, and improves the automation level of data governance; by diverting audit data, extracting correct data into the standard library, and marking data tags to form data assets, the data flows back to the data source, forming a complete closed loop of data governance, and realizing refined management and efficient use of data. Through data tagging, it realizes rapid retrieval and classification of data, improves data utilization and management efficiency, ensures data quality and consistency in the data governance process, forms closed-loop management through data reflux, establishes a continuously improved data governance mechanism, and ensures long-term stability and optimization of data quality. The present invention achieves better results in accuracy, efficiency and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0040] Figure 1This is an overall flow chart of a method for detecting the quality of dynamic arrangement of industrial data provided by the first embodiment of the present invention.

[0041] Figure 2 This is an overall flow chart of an industrial data dynamic arrangement quality detection system provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0043] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a method for detecting the quality of dynamic arrangement of industrial data, comprising:

[0044] S1: Establish industrial data standards, conduct metadata checks, and collect data from various cities into a collection library according to different data types and integration methods.

[0045] Furthermore, the formulation of industrial data standards includes establishing four types of data standards and root word libraries, namely technology, business, management and quality, according to business classification, and establishing a mapping relationship between data standards and metadata through the intelligent mapping function; when a standard is recommended for different metadata, if two or more metadata with the same Chinese name appear, matching is performed, and the matching rules include matching metadata with the same Chinese name, and the English name selection includes selecting the English name of the first metadata as the unified English name in the matched metadata, and the exclusion condition includes not matching if the English name is the same but does not contain any root word; when performing root word matching When matching Chinese and synonyms, the metadata is preliminarily screened by matching the Chinese roots and synonyms of the metadata. When the number of matched metadata is greater than or equal to 2, the selected metadata is considered to be matched. For the matched metadata, the English name directly uses the English combination of roots, and the roots are separated by underscores; when matching the English of roots and English abbreviations, the metadata is preliminarily screened by matching the English roots and abbreviations of the metadata. If the number of matched metadata is greater than or equal to 2, the selected metadata is considered to be matched. For the matched metadata, the Chinese name directly uses the Chinese combination of roots, and the roots are not separated.

[0046] It should be noted that the formulation of industrial data standards also includes analyzing the top 10 metadata with the highest comprehensive criticality, matching the metadata with the highest criticality, and calculating the criticality as:

[0047] ,

[0048] in, represents the criticality evaluation function, represents the matching degree calculation function, Represents the function for calculating the number of matching root words. Represents the association calculation function; the matching calculation function is used to measure the similarity between two data objects, expressed as:

[0049] ,

[0050] in, The number of root characters that match Chinese. is the number of Chinese characters, The number of root characters that match English. is the number of English characters. After the data standard is established, the potential mark metadata is checked and evaluated based on the number of matching root words and the degree of relevance to verify the quality of the data. The function for calculating the number of matching root words is expressed as:

[0051] ,

[0052] in, and are the minimum and maximum matching quantities for Chinese characters, and are the minimum and maximum matching numbers for English, respectively. The correlation calculation function is expressed as:

[0053] ,

[0054] in, is the correlation number, and are the minimum and maximum association numbers, respectively.

[0055] It should also be noted that by formulating unified data standards, the problem of inconsistent data standards is solved, and the standardization and consistency of data are ensured. The data standards in the existing technology are not unified, which makes data integration difficult and the data quality is uneven. By formulating industrial data standards and conducting metadata checks, this problem can be effectively solved, the efficiency of data collection and integration is improved, the standardization and consistency of data are ensured, and a solid foundation is provided for subsequent data processing and analysis. By formulating industrial data standards, the uniformity and compatibility of various types of data in different cities and different systems are ensured, which not only improves the efficiency of data integration, but also reduces the complexity of data format conversion. According to the requirements of hierarchical and classified database construction for data use purposes, resources are planned in a unified manner. By organizing and mining data resources with unified standards and standardized processes, urban infrastructure data, monitoring data, and regulatory data are integrated. Relying on platform tools and components, data modeling tools are used to build a data foundation for the urban infrastructure safety operation big data resource center including collection libraries, theme libraries, and special libraries, providing data support for comprehensive display, data services, leadership decision-making, etc. Data standardization and unified management enhance data sharing and interoperability between different departments and systems, and promote the efficient use of data resources.

[0056] S2: Configure data quality models and quality inspection rules, automatically generate mapping relationships between data standards and data models, configure and execute quality inspection plans, and conduct data quality audits.

[0057] Furthermore, data quality auditing includes establishing a data quality model and a data quality rule base based on data characteristics, and automatically generating quality inspection rules from data standards. The quality inspection rules include quality auditing rules for null value check, value range check, specification check, repeatability check, timeliness check, reference value check, and logic check. The quality inspection plan includes freely combining quality auditing rules, and setting the data period, data label, and operation strategy for plan execution; the data to be quality inspected is input into a pre-trained classifier model, and each classifier model classifies the input data. The accuracy and recall of different classifiers on the data set are evaluated using cross-validation, and the average of the accuracy and recall of different classifiers is used as the health status assessment score of the training data.

[0058] It should be noted that data quality auditing also includes constructing a dataset containing noisy labels. , expressed as:

[0059] ,

[0060] in, Indicates Industrial data characteristics of samples, Indicates The noisy labels of samples, Represents the total number of samples; for noisy data Make a prediction and get the noise prediction probability of each sample, expressed as:

[0061] ,

[0062] in, Represents a given model Next, Samples Noise label The predicted probability of is the conditional probability, which means that in the model Next, given a sample Time Noise Label Based on the predicted probability, calculate the noise label and the true label The joint probability distribution of is expressed as: ,

[0063] in, represents the confidence joint matrix, represents the noise label and the true label The joint confidence of represents the Dirac delta function, when The value is 1 when , otherwise it is 0. Indicates True labels;

[0064] The label combinations are counted and normalized to obtain the joint probability distribution, which is expressed as:

[0065] ,

[0066] in, represents the normalized joint probability matrix, Represents the sum of confidences of all true labels in the confidence union matrix, which is used for normalization;

[0067] Using Cleanlab's pruning function, based on the confidence union matrix and normalized count data Perform data pruning, remove noise data, obtain clean data, calculate the confidence of each sample, and perform data pruning based on the confidence, expressed as:

[0068] ,

[0069] in, Indicates The confidence level of the samples, Indicates that all samples have noise labels The sum of the predicted probabilities under , sets the pruning threshold, removes samples below the threshold according to the confidence level, and obtains clean data, which is expressed as:

[0070] ,

[0071] in, represents a clean data set, Indicates the set pruning threshold.

[0072] It should also be noted that by configuring data quality models and quality inspection rules, data quality inspection is automated and intelligent; in the prior art, data quality inspection mainly relies on manual work, which is inefficient and error-prone. By automatically generating a mapping relationship between data standards and data models, and configuring and executing quality inspection plans, the efficiency and accuracy of data quality inspection can be greatly improved, manual intervention can be reduced, the efficiency and accuracy of data quality inspection can be improved, the consistency and reliability of data quality can be ensured, and through the execution of automated quality inspection rules, real-time monitoring and feedback of data quality can be achieved, and data problems can be discovered and resolved in a timely manner.

[0073] S3: Divert audit data, extract correct data to the standard library, and mark data tags to form data assets. The data flows back to the data source to form a complete closed loop of data governance.

[0074] Furthermore, the formation of data assets includes diverting audit data, archiving abnormal data into error data tables, triggering data quality rectification processes, generating data quality sorting work orders, extracting correct data into standard libraries, and marking data tags to form data assets.

[0075] It should be noted that forming a complete closed loop of data governance includes creating an error data archive and quality inspection reports, analyzing the quality issues in the reports, separating each data set from the problem data set, forming different problem data work orders based on the data source, pushing them to the data source system, tracking the repair of source data quality issues, and once the repair is completed, once again entering the collection library from the front-end machine through data aggregation to perform duplicate data quality management.

[0076] It should also be noted that through data diversion and data return, a complete closed loop of data governance is achieved, ensuring the continuous improvement and optimization of data quality. In the existing technology, data governance lacks closed-loop management and cannot achieve continuous improvement of data quality. By diverting audit data, extracting correct data to the standard library and marking data tags to form data assets, and returning data to the data source, this problem can be effectively solved, improving the efficiency and effectiveness of data governance, ensuring the continuous improvement and optimization of data quality, forming a complete closed loop of data governance, and further enhancing the utilization value and reliability of data. The closed-loop management mechanism ensures the continuous tracking and resolution of data quality issues, forming a continuous improvement mechanism for data quality, and improving the utilization value of data through data assetization and labeling, providing high-quality data support for the company's data-driven decision-making.

[0077] Example 2, an embodiment of the present invention, provides a method for detecting the quality of dynamic arrangement of industrial data. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0078] First, four types of data standards and root word libraries are established according to business classification, namely technology, business, management and quality. The mapping relationship between data standards and metadata is established through the intelligent mapping function. When a standard is recommended for different metadata, if two or more metadata with the same Chinese name appear, they are matched. The matching rules include matching metadata with the same Chinese name. The English name selection includes selecting the English name of the first metadata as the unified English name in the matched metadata. The exclusion condition includes not matching if the English name is the same but does not contain any root word. When matching root Chinese and synonyms, the metadata is matched by matching the Chinese root word and synonyms of the metadata. A preliminary screening is performed. When the number of matched metadata is greater than or equal to 2, the selected metadata is considered to be matched. For the matched metadata, the English name directly uses the English combination of roots, and the roots are separated by underscores. When matching the English roots and English abbreviations, the metadata is initially screened by matching the English roots and abbreviations of the metadata. If the number of matched metadata is greater than or equal to 2, the selected metadata is considered to be matched. For the matched metadata, the Chinese name directly uses the Chinese combination of roots, and the roots are not separated. After the data standards are formulated, the potential metadata is checked, and the quality of the data is verified based on the number of matched roots and the degree of relevance. According to the data characteristics, the metadata is constructed. Establish a data quality model and a data quality rule base, and automatically generate quality inspection rules from data standards. The quality inspection rules include quality audit rules for null value check, value range check, specification check, repeatability check, timeliness check, reference value check, and logic check. The quality inspection plan includes free combination of quality audit rules, and sets the data period, data label and operation strategy for the execution of the plan. The data to be quality inspected is input into a pre-trained classifier model. Each classifier model classifies the input data. The cross-validation method is used to evaluate the accuracy and recall of different classifiers on the data set. The average of the accuracy and recall of different classifiers is used as the health status assessment score of the training data, and the audit data is classified. Flow, abnormal data is archived to the error data table, and the data quality rectification process is triggered, a data quality sorting work order is generated, the correct data is extracted to the standard library, and the data tags are marked to form data assets. The data flows back to the data source to form a complete closed loop of data governance. The formation of a complete closed loop of data governance includes creating an error data archive library and a quality inspection report, analyzing the quality problems in the report, separating various data from the problem data set, forming different problem data work orders according to the data source, pushing them to the data source system, tracking the repair of source data quality problems, and once the repair is completed, it enters the collection library from the front-end machine through data aggregation again to repeat data quality management. Refer to Table 1 to record and analyze the experimental data.

[0079] Table 1 Experimental data record table

[0080] ,

[0081] The table records the standard compliance rate of each city's data. The data shows that by formulating industrial data standards and conducting metadata checks, the standard compliance rate of each city's data has reached more than 85%. Among them, the data standard compliance rate of city C is the highest, reaching 98%. This result shows that the method of the present invention can improve the standardization of data and ensure the consistency and standardization of data. The health status assessment score is obtained by comprehensively calculating the accuracy and recall rate of different classifiers. The data shows that the health status assessment score of each city's data is above 0.82, with the highest reaching 0.95. This shows that by configuring data quality models and quality inspection rules, automatically generating a mapping relationship between data standards and data models, and configuring and executing quality inspection plans, the accuracy and efficiency of data quality audits can be effectively improved. The "Data Label" column in the table shows that the data of all test subjects have been standardized to form data assets. This shows that by diverting the audit data, extracting the correct data into the standard library and marking the data labels, the data is finally returned to the data source, forming a complete closed loop of data governance. The realization of the data governance closed loop ensures the continuous improvement and optimization of data quality.

[0082] Example 3, reference Figure 2 , as an embodiment of the present invention, provides an industrial data dynamic orchestration quality detection system, including a data standardization module, a quality audit module, and a data diversion module.

[0083] The data standardization module is used to formulate industrial data standards, conduct metadata checks, and collect data from various cities into a collection library according to different data types and integration methods; the quality audit module is used to configure data quality models and quality inspection rules, automatically generate mapping relationships between data standards and data models, configure and execute quality inspection plans, and conduct data quality audits; the data diversion module is used to divert audit data, extract correct data into the standard library, and mark data labels to form data assets, and the data flows back to the data source to form a complete closed loop of data governance.

[0084] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0085] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0086] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk case (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0087] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logical function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0088] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for detecting the quality of dynamic arrangement of industrial data, characterized in that: include: Formulate industrial data standards, conduct metadata checks, and collect data from various cities into a collection database according to different data types and integration methods; Configure data quality models and quality inspection rules, automatically generate mapping relationships between data standards and data models, configure and execute quality inspection plans, and conduct data quality audits; Audit data is diverted, correct data is extracted to the standard library, and data tags are marked to form data assets. Data flows back to the data source to form a complete closed loop of data governance. The formulation of industrial data standards includes establishing four types of data standards and root word libraries, namely technology, business, management and quality, according to business classification, and establishing a mapping relationship between data standards and metadata through intelligent mapping function; When a standard is recommended for different metadata, if two or more metadata with the same Chinese name appear, matching is performed. The matching rule includes matching metadata with the same Chinese name, and the English name selection is included in the matched metadata. The English name of the first metadata is selected as the unified English name. The exclusion condition includes not matching if there is a situation where the English name is the same but does not contain any root word; When matching Chinese root words and synonyms, the metadata is preliminarily screened by matching the Chinese root words and synonyms of the metadata. When the number of matched metadata is greater than or equal to 2, the selected metadata is considered to be matched. For the matched metadata, the English name directly uses the English combination of the root words, and the root words are separated by underscores; When matching English roots and English abbreviations, the metadata is initially screened by matching the English roots and abbreviations of the metadata. If the number of matched metadata is greater than or equal to 2, the selected metadata is considered to be matched. For the matched metadata, the Chinese name directly uses the Chinese combination of the roots, and the roots are not separated; The formulation of industrial data standards also includes analyzing the top 10 metadata with the highest comprehensive criticality, matching the metadata with the highest criticality, and calculating the criticality as: Key(x)=0.4·Sim)x,y)+0.25·RootCount(x)+0.35·Rel(x,y) Among them, Key(x) represents the key degree evaluation function, Sim(x,y) represents the matching degree calculation function, RootCount(x) represents the matching root number calculation function, and Rel(x,y) represents the relevance calculation function; The matching calculation function is used to measure the similarity between two data objects, expressed as: Among them, C match is the number of Chinese root characters, C total is the number of Chinese characters, E match E is the number of English root characters. total is the number of English characters; After the data standards are established, the potential metadata is checked and evaluated based on the number of matching root words and the degree of relevance to verify the quality of the data. The function for calculating the number of matching root words is expressed as: Among them, C min and C max are the minimum and maximum matching numbers of Chinese characters, E min and E max are the minimum and maximum matching numbers for English, respectively. The correlation calculation function is expressed as: Among them, R is the correlation number, R min and R max are the minimum and maximum association numbers, respectively; Data quality auditing includes establishing a data quality model and a data quality rule base based on data characteristics, and automatically generating quality inspection rules from data standards. Quality inspection rules include null value inspection, value range inspection, specification inspection, repeatability inspection, timeliness inspection, reference value inspection, and logic inspection quality auditing rules. The quality inspection plan includes freely combining quality auditing rules and setting the data period, data label, and operation strategy for the execution of the plan. The data to be quality inspected is input into the pre-trained classifier model. Each classifier model classifies the input data. The accuracy and recall rates of different classifiers on the data set are evaluated using cross-validation. The average of the accuracy and recall rates of different classifiers is used as the health status assessment score of the training data.

2. The method for detecting the quality of dynamic arrangement of industrial data according to claim 1, characterized in that: The data quality audit also includes constructing a data set X containing noise labels, which is expressed as: Among them, x i represents the industrial data characteristics of the i-th sample, represents the noise label of the i-th sample, and n represents the total number of samples; Predict the noise data X and get the noise prediction probability of each sample, expressed as: in, Indicates that under a given model θ, the i-th sample x i Noise label The predicted probability of is the conditional probability, indicating that under the model θ, given the sample x i Time Noise Label The probability of Based on the predicted probability, calculate the noise label and the true label y * The joint probability distribution of is expressed as: in, represents the confidence joint matrix, represents the noise label and the true label y * The joint confidence of , δ represents the Dirac delta function, when The value is 1 when y is j represents the jth true label; The label combinations are counted and normalized to obtain the joint probability distribution, which is expressed as: in, represents the normalized joint probability matrix, Represents the sum of confidences of all true labels in the confidence union matrix, which is used for normalization; Using Cleanlab's pruning function, based on the confidence union matrix and normalized count data Perform data pruning, remove noise data, obtain clean data, calculate the confidence of each sample, and perform data pruning based on the confidence, expressed as: in, represents the confidence of the i-th sample, Indicates that all samples have noise labels The sum of the predicted probabilities under , sets the pruning threshold, removes samples below the threshold according to the confidence level, and obtains clean data, which is expressed as: Among them, X clean represents the clean data set, and τ represents the set pruning threshold.

3. The method for detecting the quality of dynamic arrangement of industrial data according to claim 2, characterized in that: The formation of data assets includes diverting audit data, archiving abnormal data into an error data table, triggering a data quality rectification process, generating a data quality sorting work order, extracting correct data into a standard library, and marking data tags to form data assets.

4. The method for detecting the quality of dynamic arrangement of industrial data according to claim 3, characterized in that: The formation of a complete closed loop of data governance includes creating an error data archive and a quality inspection report, analyzing the quality issues in the report, separating each data item from the problem data set, forming different problem data work orders based on the data source, pushing them to the data source system, tracking the repair of source data quality issues, and once the repair is completed, once again entering the collection library from the front-end machine through data aggregation to perform duplicate data quality management.

5. A system using the industrial data dynamic arrangement quality detection method according to any one of claims 1 to 4, characterized in that: Including data standardization module, quality audit module, and data diversion module; The data standardization module is used to formulate industrial data standards, perform metadata checks, and collect data from various cities into a collection library according to different data types and integration methods; The quality audit module is used to configure data quality models and quality inspection rules, automatically generate data standards and data model mapping relationships, configure and execute quality inspection plans, and conduct data quality audits; The data diversion module is used to divert audit data, extract correct data to the standard library, and mark data tags to form data assets. The data flows back to the data source to form a complete closed loop of data governance.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method for detecting the quality of dynamic arrangement of industrial data according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting the quality of dynamic arrangement of industrial data according to any one of claims 1 to 4 are implemented.

Images