A dynamic integration and governance method for multi-source heterogeneous data

Abstract

The present invention relates to the field of industrial data processing technology, and in particular to a dynamic integration and management method for multi-source heterogeneous data, comprising: step one, classifying information according to data type to obtain multiple time series sets; step two, establishing an information coordinate system for the fault data information type, and determining the position parameters of the fault point based on the modulus and angle value of the vector; step three, marking the data value of the influencing point on the information coordinate axis for the information data affecting the data, and determining the unit length of this information coordinate system for the fault data based on multiple data values, and marking it as a coordinate comparison coefficient; step four, directly determining the data position of the fault data or the influencing data according to the coordinate comparison coefficient between the fault data and the influencing data; the present invention can accurately determine the position of the abnormal data corresponding to the fault or influencing data based on the pre-established coordinate comparison coefficient, thereby greatly improving the efficiency and accuracy of abnormal data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial data processing, and in particular to a method for dynamically integrating and managing multi-source heterogeneous data. Background Art

[0002] Under the wave of development of intelligent manufacturing, multi-source heterogeneous data such as equipment data, manufacturing execution system (MES) data, and logistics system data in the production process contain huge value and are the key foundation for achieving production process optimization, quality control, and resource scheduling.

[0003] Existing technology CN119759883A discloses a "dual data" middle-end data governance method for process industries, including: 1) Defining data standards: 1.1) Building a data hierarchy; 1.2) Configuring data encoding rules; 1.3) Defining data entities; 1.4) Determining data tags. 2) Data collection and integration: 2.1) Formulating data collection tasks; 2.2) Data cleaning; 2.3) Data quality verification; 2.4) Data integration. 3) Data security: 3.1) Data classification and grading; 3.2) Formulating security policies; 3.3) Running security tasks. 4) Data application: Providing data services to external systems through API interfaces;

[0004] However, in the process of industrial digitalization, the multi-source heterogeneous data generated by the production system, such as equipment operation, manufacturing execution, and logistics scheduling, are huge in scale and complex in type. Traditional multi-source heterogeneous data integration and governance methods mostly use static rule processing, which is difficult to adapt to the dynamic changes in the production environment. The correlation analysis between data in different regions lacks systematicity and cannot quickly locate the root cause of the fault and its derivative effects. If all data are traversed in sequence to monitor fault data, on the one hand, fault data will be missed, and on the other hand, it will restrict the improvement of production efficiency and quality control capabilities. Summary of the Invention

[0005] The purpose of the present invention is to solve the problems in the background technology and to propose a dynamic integration and management method for multi-source heterogeneous data.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for dynamically integrating and managing multi-source heterogeneous data, the method specifically comprising the following steps:

[0008] Step 1: Collect historical multi-source heterogeneous data from industrial production and classify the information according to data type to obtain multiple time series sets;

[0009] Step 2: Identify the fault data in the time series set and the impact data corresponding to the fault data in the remaining time series, establish an information coordinate system for the fault data information type, and determine the location parameters of the fault point based on the vector modulus and angle value;

[0010] Step 3: Establish a blank information coordinate system, locate the fault point in this information coordinate system according to its location parameters, obtain the impact point, and then obtain the impact data corresponding to the fault data. According to the information data of the impact data, mark the data value of the impact point on the information coordinate axis. Based on multiple data values, determine the unit length of this information coordinate system for the fault data and mark it as the coordinate comparison coefficient;

[0011] Step 4: Identify abnormal data in the industrial production process and calculate the location parameters of the abnormal data. Then, based on the coordinate comparison coefficient between the fault data and the influencing data, directly determine the data location of the fault data or the influencing data, and transmit them to the corresponding staff for information processing.

[0012] As a further solution of the present invention, a method for obtaining a time series set includes:

[0013] Collect historical multi-source heterogeneous data from industrial production and cleanse the data. The multi-source heterogeneous data includes equipment data, manufacturing data, quality data, and transportation data in the industrial production process.

[0014] After data cleaning, the multi-source heterogeneous data are first classified according to data type to obtain multiple data sets. One data set corresponds to one information type, including equipment data, production data, quality data and transportation data. The data in each data set are then arranged in chronological order to obtain a time series set.

[0015] As a further solution of the present invention, data cleaning refers to identifying structurally abnormal data in multi-source heterogeneous data and deleting the structurally abnormal data, wherein data cleaning includes removing missing values, correcting erroneous data, unifying data formats and removing duplicate data. Missing values, erroneous data and duplicate data are structurally abnormal data.

[0016] As a further solution of the present invention, a method for determining the location parameters of a fault point includes:

[0017] S1: Obtain all time series sets, and establish corresponding information coordinate systems based on the information type corresponding to each time series set. The information coordinate system is a two-dimensional plane coordinate system, and in the information coordinate system, time is set as the horizontal coordinate, and the information type is set as the vertical coordinate;

[0018] Randomly select a time series set and mark it as the target set. At the same time, randomly select a time series set from the remaining time series sets and mark it as the associated set. Identify all the fault data in the target set and the location of the fault data in the corresponding information coordinate system. At the same time, mark the corresponding point of the fault data in the information coordinate system as the fault point.

[0019] S2: Identify the impact data corresponding to the fault data in the associated set in the target set. Fault data refers to data directly generated by a fault during the operation of a system or device, and impact data refers to abnormal data indirectly generated by data problems in the fault data. At the same time, the information type corresponding to the time series set containing the impact data is marked as the associated information of the fault data information type, that is, the information type of the impact data is the associated information of the fault data information type;

[0020] S3: Identify the location coordinates of the fault point on the information coordinate system of the target set and mark them as Pi (Xi, Yi), where i represents different fault points. Then, use the vector algorithm to mark the location coordinates of the origin O on the information coordinate system as (0, 0). = (Xi, Yi), then use the formula Get vector The module Di, in which ;

[0021] Reuse formula Get the angle value of the fault point Pi , where arctan (*) is the inverse tangent function;

[0022] Then the vector modulus Di and angle value of each fault point Pi are Integrate into combinations and mark them as position parameters (Di, ).

[0023] As a further solution of the present invention, a method for obtaining the coordinate comparison coefficient includes:

[0024] SS1: Identify the fault data corresponding to each fault point and obtain the impact data corresponding to each fault data. Use the information coordinate system of the associated set as the analysis object and mark this coordinate system as the designated coordinate system. In this case, the designated coordinate system is a blank coordinate system.

[0025] SS2: According to the location parameters of the fault point (Di, ), according to the position parameters of the fault point (Di, ) Locate the impact point Xj;

[0026] Obtain the impact data and the time when the impact data occurs, and mark them as the information data of the impact point Xj (Tj, Ej), where Tj represents the time when the impact data occurs, Ej represents the value of the impact data, and j represents different impact points;

[0027] According to the information data (Tj, Ej) of the influencing points, the data values ​​of the X-axis and Y-axis are marked respectively in the specified coordinate system. After the information data of all the influencing points are marked, the unit lengths of the X-axis and Y-axis are adjusted multiple times based on the actual marked data values ​​on the X-axis and Y-axis using mathematical tools. Finally, the unit lengths of the X-axis and the Y-axis of the associated set information coordinate system are determined, and the unit lengths of the X-axis and the Y-axis are marked as coordinate comparison coefficients.

[0028] As a further solution of the present invention, when the fault data and the impact data occur synchronously, the time axes of the information coordinate system of the associated set and the information coordinate system of the target set are the same unit length, that is, any time scale position on the information coordinate system of the associated set is the same as the corresponding time scale position on the information coordinate system of the target set.

[0029] As a further solution of the present invention, a method for determining the location of fault data or influencing data based on abnormal data includes:

[0030] Real-time monitoring of industrial production data. When abnormal data is detected in the industrial production data, the information type corresponding to the abnormal data is first identified, and whether the abnormal data is fault data or impact data is distinguished.

[0031] If the abnormal data is fault data, the abnormal data is marked in the corresponding information coordinate system, and the position parameter of the abnormal data is calculated based on the marked position;

[0032] Obtaining association information of the abnormal data information type, and based on the coordinate comparison coefficient of the association information to the information type of the fault data and the position parameter of the abnormal data, directly determining the data information of the affected data corresponding to the abnormal data according to the position parameter of the abnormal data in the information coordinate system of the association information, wherein the data information includes the data value of the affected data and the time when the data appeared;

[0033] The abnormal data and the corresponding impact data are then transmitted to the device display terminal of the corresponding staff respectively, and the staff confirms and processes the abnormal data and the impact data.

[0034] As a further solution of the present invention, if the abnormal data belongs to the impact data, according to the information type of the abnormal data, the information type of the corresponding fault data when the information type is used as the associated information is identified, and the information type corresponding to the fault data is marked as abnormal information;

[0035] Based on the coordinate comparison coefficient of the abnormal data to the abnormal information, the position parameters of the abnormal data in the information coordinate system established based on the coordinate comparison coefficient are obtained, and the corresponding position data in the information coordinate system of the abnormal information is identified according to the position parameters, and the position data is marked as data to be verified;

[0036] The data to be checked is transmitted to the device display terminal of the corresponding staff, who then verify the data to be checked and determine the fault data;

[0037] After the fault data is determined, according to the above method, the associated information of the fault data information is obtained, and based on the location parameters of the fault data, the data information affecting the data in the associated information is identified in turn, and the data information affecting the data is transmitted to the equipment display terminal of the corresponding staff respectively, and the staff confirms and processes the information affecting the data.

[0038] Compared with the existing technology, the advantages of the present invention are:

[0039] The present invention establishes an information coordinate system based on fault data and impact data and determines the coordinate comparison coefficient, thereby realizing in-depth correlation and quantitative analysis of fault data and impact data, and then quickly locating the fault and its diffusion range; in abnormal data processing, based on the pre-established coordinate comparison coefficient, the position of the abnormal data corresponding to the fault or impact data can be accurately determined, which greatly improves the efficiency and accuracy of abnormal data processing; this method effectively solves the problem of dynamic correlation analysis of multi-source heterogeneous data, realizes intelligent integrated management of data, improves the fault warning and abnormal response capabilities of industrial production, reduces the cost of manual intervention, and provides reliable data support for production optimization and resource scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the method flow structure of the present invention. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0042] Reference Figure 1 A dynamic integration and management method for multi-source heterogeneous data includes the following steps:

[0043] Step 1: Collect historical multi-source heterogeneous data from industrial production and clean the historical multi-source heterogeneous data;

[0044] Among them, multi-source heterogeneous data includes equipment data, manufacturing data, quality data, and transportation data in the industrial production process. Data cleaning refers to identifying structural abnormal data in multi-source heterogeneous data and deleting structural abnormal data. Furthermore, data cleaning includes removing missing values, correcting erroneous data, unifying data formats, and removing duplicate data. Missing values, erroneous data, and duplicate data are structural abnormal data.

[0045] After data cleaning, the multi-source heterogeneous data is first classified according to data type to obtain multiple multi-source heterogeneous data sets, where one multi-source heterogeneous data set corresponds to one multi-source heterogeneous information type, including equipment data, production data, quality data, and transportation data. Then, the data in each multi-source heterogeneous data set is arranged in chronological order to obtain a time series set.

[0046] Step 2: Obtain all time series sets and establish corresponding information coordinate systems based on the information type corresponding to each time series set. The information coordinate system is a two-dimensional plane coordinate system, and in the information coordinate system, time is set as the horizontal coordinate and the information type is set as the vertical coordinate. Based on the information coordinate system, determine the location parameters of the fault data. The specific method for determining the location parameters includes:

[0047] S1: Randomly select a time series set and mark it as the target set. At the same time, randomly select a time series set from the remaining time series sets and mark it as the associated set. Identify all the fault data in the target set and the location of the fault data in the corresponding information coordinate system. At the same time, mark the corresponding point of the fault data in the information coordinate system as the fault point.

[0048] S2: Identify the impact data corresponding to the fault data in the associated set in the target set. Fault data refers to data directly generated by a fault during the operation of a system or device, and impact data refers to abnormal data indirectly generated by data problems in the fault data. At the same time, the information type corresponding to the time series set containing the impact data is marked as the associated information of the fault data information type, that is, the information type of the impact data is the associated information of the fault data information type;

[0049] S3: Identify the location coordinates of the fault point on the information coordinate system of the target set and mark them as Pi (Xi, Yi), where i represents different fault points. Then, use the vector algorithm to mark the location coordinates of the origin O on the information coordinate system as (0, 0). = (Xi, Yi), then use the formula Get vector The module Di, in which ;

[0050] Reuse formula Get the angle value of the fault point Pi , where arctan (*) is the inverse tangent function;

[0051] Then the vector modulus Di and angle value of each fault point Pi are Integrate into combinations and mark them as position parameters (Di, );

[0052] Step 3: Based on the location parameters of the fault point Pi, analyze the information coordinate system of the associated set and determine the coordinate comparison coefficient of the key set information coordinate system. The specific method for determining the coordinate comparison coefficient includes:

[0053] SS1: Identify the fault data corresponding to each fault point and obtain the impact data corresponding to each fault data. Use the information coordinate system of the associated set as the analysis object and mark this coordinate system as the designated coordinate system. At this time, the designated coordinate system is a blank coordinate system, that is, there is no unit length on the designated coordinate system.

[0054] SS2: According to the location parameters of the fault point (Di, ), according to the position parameters of the fault point (Di, ) Locate the impact point Xj;

[0055] Obtain the impact data and the time when the impact data occurs, and mark them as the information data of the impact point Xj (Tj, Ej), where Tj represents the time when the impact data occurs, Ej represents the value of the impact data, and j represents different impact points;

[0056] Then, data values ​​are marked on the X-axis and Y-axis in the specified coordinate system according to the information data (Tj, Ej) of the influencing points. After the information data of all influencing points are marked, the unit lengths of the X-axis and Y-axis are adjusted multiple times based on the actual marked data values ​​on the X-axis and Y-axis using mathematical tools. Finally, the unit lengths of the X-axis and the Y-axis of the associated set information coordinate system are determined. At the same time, the unit lengths of the X-axis and the Y-axis are marked as coordinate comparison coefficients. The mathematical tool used in this embodiment is MATLAB.

[0057] It should be further explained that when the fault data and the impact data occur synchronously, the time axis (i.e., the X-axis) of the information coordinate system of the associated set and the information coordinate system of the target set have the same unit length. That is, the position of any time scale on the information coordinate system of the associated set is the same as the position of the corresponding time scale on the information coordinate system of the target set. The specific value of the unit length on the time axis of the target set is determined by those skilled in the art based on big data experience.

[0058] Then, the remaining time series sets except the target set are marked as associated sets in turn, and processed according to the above method to obtain the coordinate comparison coefficient of each associated set with respect to the target set. At the same time, the time series sets are set as the target sets in turn, and the remaining time series sets are used as associated sets in turn. The fault data in the target set and the impact data in the associated sets are identified. Based on the fault data and the impact data, the coordinate comparison coefficients between the time series sets are determined respectively.

[0059] Step 4: Monitor industrial production data in real time. When abnormal data is detected in the industrial production data, first identify the information type corresponding to the abnormal data and distinguish whether the abnormal data is fault data or impact data;

[0060] If the abnormal data belongs to fault data, the abnormal data is marked in the corresponding information coordinate system, and based on the marked position, the position parameter of the abnormal data is calculated. Then, based on the information type of the abnormal data, the associated information of the information type of the abnormal data is obtained. Based on the coordinate comparison coefficient of the associated information for the information type of the fault data and the position parameter of the abnormal data, the data information of the impact data corresponding to the abnormal data is directly determined according to the position parameter of the abnormal data in the information coordinate system of the associated information, wherein the data information includes the data value of the impact data and the data occurrence time. Then, the abnormal data and the corresponding impact data are respectively transmitted to the device display terminal of the corresponding staff, and the staff confirms and processes the information of the abnormal data and the impact data;

[0061] If the abnormal data belongs to the influencing data, the information type of the abnormal data is identified as the information type of the corresponding fault data when the information type is used as the associated information according to the information type of the abnormal data, and the information type of the corresponding fault data is marked as abnormal information. At the same time, based on the information coordinate system of the abnormal information, the coordinate comparison coefficient of the abnormal data to the abnormal information is first obtained, and the position parameters of the abnormal data on the information coordinate system established based on the coordinate comparison coefficient are obtained. At the same time, according to the position parameters, the corresponding position data is identified in the information coordinate system of the abnormal information, and the position data is marked as data to be verified. After that, the data to be verified is transmitted to the device display terminal of the corresponding staff respectively, and the staff verifies the information of the data to be verified, and then determines the fault data. When the fault data is identified, according to the above method, the associated information of the fault data information is obtained, and based on the position parameters of the fault data, the data information of the influencing data in the associated information is identified in turn, and the data information of the influencing data is transmitted to the device display terminal of the corresponding staff respectively, and the staff confirms and processes the information of the influencing data.

[0062] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A dynamic integration and management method for multi-source heterogeneous data, characterized by: The method specifically comprises the following steps: Step 1: Collect historical multi-source heterogeneous data from industrial production and classify the information according to data type to obtain multiple time series sets; Step 2: Identify the fault data in the time series set and identify the impact data corresponding to the fault data in the remaining time series. The impact data refers to the abnormal data indirectly generated by the data problem of the fault data. Establish an information coordinate system for the fault data information type and determine the location parameters of the fault point based on the vector modulus and angle value. Step 3: Establish a blank information coordinate system, locate the fault point in this information coordinate system according to its location parameters, obtain the impact point, and then obtain the impact data corresponding to the fault data. According to the information data of the impact data, mark the data value of the impact point on the information coordinate axis. Based on multiple data values, determine the unit length of this information coordinate system for the fault data and mark it as the coordinate comparison coefficient; Step 4: Identify abnormal data in the industrial production process and calculate the location parameters of the abnormal data. Then, based on the coordinate comparison coefficient between the fault data and the influencing data, directly determine the data location of the fault data or the influencing data, and transmit them to the corresponding staff for information processing.

2. A dynamic integration and management method for multi-source heterogeneous data according to claim 1, characterized in that: Methods for obtaining time series collections include: Collect historical multi-source heterogeneous data from industrial production and cleanse the data. The multi-source heterogeneous data includes equipment data, manufacturing data, quality data, and transportation data in the industrial production process. The cleaned multi-source heterogeneous data is classified to obtain multiple multi-source heterogeneous data sets. One multi-source heterogeneous data set corresponds to one multi-source heterogeneous information type, which includes equipment data, production data, quality data, and transportation data. The data in each multi-source heterogeneous data set is then arranged in chronological order to obtain a time series set.

3. The method for dynamic integration and management of multi-source heterogeneous data according to claim 2 is characterized in that: Data cleaning refers to identifying structurally abnormal data in multi-source heterogeneous data and deleting the structurally abnormal data. Data cleaning includes removing missing values, correcting erroneous data, unifying data formats, and removing duplicate data. Missing values, erroneous data, and duplicate data are structurally abnormal data.

4. The method for dynamic integration and management of multi-source heterogeneous data according to claim 1 is characterized in that: Methods for determining the location parameters of the fault point include: S1: Obtain all time series sets, and establish corresponding information coordinate systems based on the information type corresponding to each time series set. The information coordinate system is a two-dimensional plane coordinate system, and in the information coordinate system, time is set as the horizontal coordinate, and the information type is set as the vertical coordinate; Randomly select a time series set and mark it as the target set. At the same time, randomly select a time series set from the remaining time series sets and mark it as the associated set. Identify all the fault data in the target set and the location of the fault data in the corresponding information coordinate system. At the same time, mark the corresponding point of the fault data in the information coordinate system as the fault point. S2: Identify the impact data corresponding to the fault data in the target set in the associated set. Fault data refers to data directly generated by a fault during the operation of a system or device. At the same time, mark the information type corresponding to the time series set containing the impact data as associated information of the fault data type. S3: Identify the location coordinates of the fault point on the information coordinate system of the target set and mark them as Pi (Xi, Yi), where i represents different fault points. Then, use the vector algorithm to mark the location coordinates of the origin O on the information coordinate system as (0, 0). = (Xi, Yi), then use the formula Get vector The module Di, in which ; Reuse formula Get the angle value of the fault point Pi , where arctan (*) is the inverse tangent function; Then the vector modulus Di and angle value of each fault point Pi are Integrate into combinations and mark them as position parameters (Di, ).

5. The method for dynamic integration and management of multi-source heterogeneous data according to claim 1 is characterized in that: Methods for obtaining coordinate comparison coefficients include: SS1: Identify the fault data corresponding to each fault point and obtain the impact data corresponding to each fault data. Use the information coordinate system of the associated set as the analysis object and mark this coordinate system as the designated coordinate system. In this case, the designated coordinate system is a blank coordinate system. SS2: According to the location parameters of the fault point (Di, ), according to the position parameters of the fault point (Di, ) Locate the impact point Xj; Obtain the impact data and the time when the impact data occurs, and mark them as the information data of the impact point Xj (Tj, Ej), where Tj represents the time when the impact data occurs, Ej represents the value of the impact data, and j represents different impact points; According to the information data (Tj, Ej) of the influencing points, the data values ​​of the X-axis and Y-axis are marked respectively in the specified coordinate system. After the information data of all the influencing points are marked, the unit lengths of the X-axis and Y-axis are adjusted multiple times based on the actual marked data values ​​on the X-axis and Y-axis using mathematical tools. Finally, the unit lengths of the X-axis and the Y-axis of the associated set information coordinate system are determined, and the unit lengths of the X-axis and the Y-axis are marked as coordinate comparison coefficients.

6. A dynamic integration and management method for multi-source heterogeneous data according to claim 5, characterized in that: When the fault data and the impact data occur synchronously, the time axes of the information coordinate system of the associated set and the information coordinate system of the target set have the same unit length.

7. The method for dynamic integration and management of multi-source heterogeneous data according to claim 1 is characterized in that: Methods for determining the location of faulty data or data affecting data based on abnormal data include: Real-time monitoring of industrial production data. When abnormal data is detected in the industrial production data, the information type corresponding to the abnormal data is first identified, and whether the abnormal data is fault data or impact data is distinguished. If the abnormal data is fault data, the abnormal data is marked in the corresponding information coordinate system, and the position parameter of the abnormal data is calculated based on the marked position; Obtaining association information of the abnormal data information type, and based on the coordinate comparison coefficient of the association information to the information type of the fault data and the position parameter of the abnormal data, directly determining the data information of the affected data corresponding to the abnormal data according to the position parameter of the abnormal data in the information coordinate system of the association information, wherein the data information includes the data value of the affected data and the time when the data appeared; The abnormal data and the corresponding impact data are then transmitted to the device display terminal of the corresponding staff respectively, and the staff confirms and processes the abnormal data and the impact data.

8. The method for dynamic integration and management of multi-source heterogeneous data according to claim 7 is characterized in that: If the abnormal data belongs to the impact data, according to the information type of the abnormal data, identify the information type of the corresponding fault data when the information type is used as the associated information, and mark the corresponding fault data information type as abnormal information; Based on the coordinate comparison coefficient of the abnormal data to the abnormal information, the position parameters of the abnormal data in the information coordinate system established based on the coordinate comparison coefficient are obtained, and the corresponding position data in the information coordinate system of the abnormal information is identified according to the position parameters, and the position data is marked as data to be verified; The data to be checked is transmitted to the device display terminal of the corresponding staff, who then verify the data to be checked and determine the fault data; After the fault data is determined, according to the above method, the associated information of the fault data information is obtained, and based on the location parameters of the fault data, the data information affecting the data in the associated information is identified in turn, and the data information affecting the data is transmitted to the equipment display terminal of the corresponding staff respectively, and the staff confirms and processes the information affecting the data.

Images