Multimodal Data Fusion Method and System Based on Identification Resolution

Abstract

The present invention relates to the technical field of data fusion and matching, and particularly to a multi-modal data fusion method and system based on identification and resolution. The method includes inputting real-time operation data and operation images into a trained neural network to obtain device state types. The training process of the neural network is as follows: performing anomaly recognition on each operation data of historical fault events to determine the data anomaly coefficient; determining the image anomaly coefficient according to the change in the abnormal area in the operation images from each perspective of historical fault events; based on the data anomaly coefficient and the image anomaly coefficient, matching the operation data and the operation images to obtain an abnormal fusion group, and determining the hysteresis factor of the abnormal fusion group; training the neural network according to the abnormal fusion group and the corresponding hysteresis factor. By associating and fusing each operation data and operation image, and training the neural network with the fused data, the present invention can identify faults with higher accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion and matching, and specifically relates to a multi-modal data fusion method and system based on identification resolution. Background Art

[0002] In industrial production, through identification resolution, the operation data, image data, and text data of large-scale equipment in a factory can be fused to achieve comprehensive monitoring of equipment status and fault prediction. Among them, operation data such as temperature and pressure, image data such as equipment appearance detection, and text data such as equipment manuals. Among them, operation data can show whether the equipment is working within normal parameters, image data may capture problems visible to the naked eye, etc. By combining various types of data, the equipment status can be analyzed more comprehensively, faults can be predicted, maintenance plans can be optimized, and even some processes can be automated.

[0003] When existing methods fuse data of different modalities for fault identification, they often directly fuse the original data at the data acquisition stage. For example, the equipment operation data and image data are directly spliced together and combined into an input vector or matrix. However, such fusion is only superficial and shallow fusion. For fault identification, it is difficult to make full use of the advantages of multi-modal data and unable to model cross-modal semantic or spatial associations. Summary of the Invention

[0004] In order to solve the technical problem of simple splicing of data and inability to model cross-modal semantic or spatial associations, the purpose of the present invention is to provide a multi-modal data fusion method and system based on identification resolution, and the specific technical solutions adopted are as follows:

[0005] In the first aspect, an embodiment of the present invention provides a multi-modal data fusion method based on identification resolution, and the method includes:

[0006] Obtain the operation data and operation images during the operation of equipment on the production line during industrial production;

[0007] Input the real-time operation data and operation images into a trained neural network to obtain the equipment status type;

[0008] The training process of the neural network is as follows: For each type of operation data of historical fault events, perform anomaly recognition to determine the data anomaly coefficient of each type of operation data; according to the change in the abnormal area in adjacent frames of operation images from each perspective of historical fault events, determine the image anomaly coefficient of the operation images from each perspective; based on the data anomaly coefficient and the image anomaly coefficient, perform one-to-one matching on the operation data and the operation images to obtain an abnormal fusion group; according to the matching situation of the abnormal rates of the operation data and the operation images in the abnormal fusion group, determine the hysteresis factor of the abnormal fusion group; and train the neural network according to the abnormal fusion group and the corresponding hysteresis factor.

[0009] Further, the step of performing anomaly recognition on each type of operation data of historical fault events to determine the data anomaly coefficient of each type of operation data includes:

[0010] Whenever a historical fault event occurs, perform anomaly recognition on each type of operation data, and take the number of times each type of operation data shows anomalies under all historical fault occurrences as the data anomaly coefficient of each type of operation data.

[0011] Further, the step of performing anomaly recognition on each type of operation data whenever a historical fault event occurs includes:

[0012] Use the Z-score algorithm to determine whether the operation data shows anomalies.

[0013] Further, the step of determining the image anomaly coefficient of the operation images from each perspective according to the change in the abnormal area in adjacent frames of operation images from each perspective of historical fault events includes:

[0014] Whenever a historical fault event occurs, for the operation images from each perspective, use the frame difference method to obtain the difference area between the operation image and the previous frame of the operation image, and determine the area ratio of the difference area in the operation image, denoted as the abnormal area ratio; determine whether the abnormal area ratio of the operation image from the current perspective in the historical fault event is increasing;

[0015] Take the number of times the abnormal area ratio of the operation images from each perspective shows an increase in all historical fault events as the image anomaly coefficient of the operation images from each perspective.

[0016] Further, the step of determining whether the abnormal area ratio of the operation image from the current perspective in the historical fault event is increasing includes:

[0017] Taking the sequence value corresponding to the running image as the abscissa and the proportion of the abnormal area of the running image as the ordinate, coordinate points in a two-dimensional rectangular coordinate system are constructed; all the coordinate points are used as the input of the PCA algorithm to obtain multiple projection directions and the projection values corresponding to each projection direction; the projection direction corresponding to the maximum projection value is denoted as the main projection direction, and the arctangent value of the ratio of the ordinate to the abscissa of the coordinate points corresponding to the main projection direction; when the arctangent value is greater than 0, it is determined that the proportion of the abnormal area of the running image in the current perspective in the currently analyzed historical fault event is increasing.

[0018] Further, based on the data anomaly coefficient and the image anomaly coefficient, one-to-one matching is performed on the running data and the running image to obtain an anomaly fusion group, including:

[0019] The running data and the running image are respectively used as the nodes on the left and right sides in the KM matching algorithm, and the data anomaly coefficient of the running data and the image anomaly coefficient of the running image are respectively used as the corresponding node values, and the absolute value of the difference between the node values is used as the edge value between the nodes. Based on the minimum matching principle, one-to-one matching is performed on the running data and the running image to obtain multiple groups of fusion groups; among them, each group of fusion groups includes a piece of running data and a running image.

[0020] Obtain the frequency of each piece of running data in the fusion group and the frequency of the running image in the fusion group.

[0021] Take the minimum value of the frequencies of the running data and the running image in the fusion group as the frequency of the fusion group, retain the fusion groups whose frequency of the fusion group is greater than the preset frequency threshold and the edge value is less than the preset edge value threshold, and record the running data with the retained corresponding relationship and the corresponding running image as the anomaly fusion group.

[0022] Further, determining the hysteresis factor of the anomaly fusion group according to the matching situation of the anomaly rates of the running data and the running image in the anomaly fusion group includes:

[0023] According to the difference between the data to be analyzed at the current moment and the data at the previous moment in the running data in the anomaly fusion group, determine the data anomaly rate corresponding to the current moment to be analyzed, and obtain the sequence of data anomaly rates of the running data in the historical fault event.

[0024] Take the proportion of the area of the difference image between the data to be analyzed at the current moment and the previous moment of the running image in the anomaly fusion group as the image anomaly rate corresponding to the current moment to be analyzed, and obtain the sequence of image anomaly rates of the running image in the historical fault event.

[0025] Match the data anomaly rate sequence and the image anomaly rate sequence to obtain the hysteresis value between the two sequences as the hysteresis factor of the anomaly fusion group.

[0026] Further, determining the data anomaly rate corresponding to the moment to be analyzed according to the difference between the data at the moment to be analyzed and the data at the previous moment in the abnormal fusion group includes:

[0027] Taking the absolute value of the difference between the data at the moment to be analyzed and the data at the previous moment in the abnormal fusion group as the numerator, taking the maximum value of the data at the moment to be analyzed and the data at the previous moment in the abnormal fusion group as the denominator, and taking the ratio of the numerator to the denominator as the data anomaly rate corresponding to the moment to be analyzed.

[0028] Further, training the neural network according to the abnormal fusion group and the corresponding hysteresis factor includes:

[0029] Taking the abnormal fusion group and the corresponding hysteresis factor as the input of the neural network, and taking the corresponding fault annotation as the output target of the neural network to train the neural network.

[0030] In a second aspect, a multi-modal data fusion system based on identity resolution is provided. The system includes the following modules:

[0031] A device data acquisition module, configured to acquire the operation data and operation images when the devices on the production line are operating during industrial production;

[0032] A device state determination module, configured to input the real-time operation data and operation images into the trained neural network to obtain the device state type;

[0033] The training process of the neural network is as follows: performing anomaly recognition on each type of operation data of historical fault events to determine the data anomaly coefficient of each type of operation data; determining the image anomaly coefficient of the operation images in adjacent frames from each perspective of historical fault events according to the change in the abnormal area; based on the data anomaly coefficient and the image anomaly coefficient, performing one-to-one matching on the operation data and the operation images to obtain an abnormal fusion group; determining the hysteresis factor of the abnormal fusion group according to the matching situation of the anomaly rates of the operation data and the operation images in the abnormal fusion group; and training the neural network according to the abnormal fusion group and the corresponding hysteresis factor.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor. An executable code is stored in the memory. When the processor executes the executable code, the methods in all possible implementation embodiments of the first aspect are implemented.

[0035] In a fourth aspect, an embodiment of the present invention provides a computer program product, which includes: computer program code. When the computer program code runs on a computer, the computer is caused to execute the methods in the first aspect or any possible implementation manner of the first aspect.

[0036] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the embodiments of all possible implementations in the first aspect.

[0037] The embodiments of the present invention at least have the following beneficial effects:

[0038] The present invention realizes inputting real-time operation data and operation images into a trained neural network to obtain the device state type. Among them, the neural network is trained using the operation data and operation data of historical fault events. Since when a fault event occurs, not all data is abnormal. It may be that some operation data and operation images are abnormal, while other parts of the data are still within the normal range. Therefore, the present invention matches the operation data and operation images according to the abnormal coefficients of the operation data and operation images to obtain an abnormal fusion group. When a fault occurs, the data belonging to the same abnormal fusion group can indicate the occurrence of a certain fault. The present invention associates the fault event with the operation data and operation images through analysis to obtain fusion data. The neural network that can perform fault diagnosis obtained by training with the fused data can identify faults with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0040] Figure 1 It is a flowchart of a method for multi-modal data fusion based on identifier resolution provided by an embodiment of the present invention;

[0041] Figure 2 It is a flowchart of the steps of the training process of a neural network provided by an embodiment of the present invention;

[0042] Figure 3 It is a schematic structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following, in conjunction with the drawings and preferred embodiments, details the specific implementation manners, structures, features and effects of a multi-modal data fusion method and system based on identifier resolution proposed according to the present invention.

[0044] In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0045] Among them, in the description of the embodiments of the present invention, unless otherwise specified, " / " means "or". For example, A / B can represent A or B. The "and / or" in the text is just a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present invention, "a plurality of" means two or more than two.

[0046] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this invention belongs.

[0048] The embodiments of the present invention will be described below with reference to the accompanying drawings. As those of ordinary skill in the art know, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0049] The specific solutions of a multi-modal data fusion method and system based on identifier resolution provided by the present invention will be specifically described below with reference to the accompanying drawings.

[0050] Please refer to Figure 1 , which shows a flowchart of the steps of a multi-modal data fusion method based on identifier resolution provided by an embodiment of the present invention. The method includes the following steps:

[0051] Step S100, obtain the operation data and operation images when the equipment on the production line is running during industrial production.

[0052] Temperature sensors, pressure sensors, ammeters, voltmeters, and power sensors are set on the production line to collect temperature data, pressure data, current data, voltage data, and power data respectively. The temperature data, pressure data, current data, voltage data, and power data when the equipment on the production line is running during industrial production are collectively referred to as operation data, that is, the operation data includes multiple types of data; and cameras with different perspectives are installed at different angles on the production line to collect the operation images of the equipment. Among them, the operation data can also include rotational speed.

[0053] Step S200: Input the real-time operation data and operation images into the trained neural network to obtain the device status type.

[0054] In the traditional method for data fusion of operation data and operation images, the obtained data is simply vectorized and then spliced together to form a spliced vector. Such fusion is only superficial and shallow, and cannot fully utilize the advantages of multi-modal data. Based on this, in the present invention, through fault analysis, although the operation data and operation images of faults belong to different types of data, since both can represent fault information, then through the common representation and the order of representation of the two types of data for faults, the respective identification data of the two types of data are obtained, and then the fault identification data is formed and participates in the training of the neural network. This neural network is also the network for realizing fault identification.

[0055] It is realized to input the real-time operation data and operation images during the operation of equipment on the production line during industrial production into the trained neural network. Through forward propagation calculation, the result value is output. According to the result value, the device status type is judged. The output result value is usually a probability distribution, indicating the possibility of the device being in different fault events. Specifically, according to the result value to judge the device status type: if the probability corresponding to the output fault event is greater than the preset judgment threshold, it is judged that the device has an abnormality, and the device status type is judged to be abnormal; otherwise, the device status type is judged to be normal. In an embodiment of the present invention, the preset judgment threshold can be set to the mode of the label values corresponding to the operation data when historical fault events of the same type occur.

[0056] During the process of training the neural network, optimization algorithms such as the optimization algorithm based on the first-order gradient (Adam optimization algorithm, Adam) and the stochastic gradient descent algorithm (Stochastic Gradient Descent, SGD) can be used to adjust the weights and biases of the neural network to minimize the loss function of the network. At the same time, methods such as cross-validation can be used to evaluate the performance of the network to avoid overfitting.

[0057] As a preferred embodiment of the present invention, the training process of the neural network includes steps S210 to S240. Please refer to Figure 2 as shown Figure 2 is the flowchart of the steps of the training process of the neural network:

[0058] Step S210: Perform anomaly identification on each type of operation data of historical fault events to determine the data anomaly coefficient of each type of operation data.

[0059] Record the operation data and operation images separately when a fault event occurs in history. The obtained operation data and operation images are as follows: Obtain the operation data of each of multiple historical fault events and the operation images of the device from different perspectives. For example, fault event 1 includes: operation data 1: {current data, voltage data, power data, temperature data, pressure data}, operation image 1: {operation image from the first perspective, operation image from the second perspective, operation image from the third perspective...}; fault event 2 includes: operation data 2: {current data, voltage data, power data, temperature data, pressure data}, operation image 2: {operation image from the first perspective, operation image from the second perspective, operation image from the third perspective...}.

[0060] Since when a fault event occurs, not all data are abnormal. For example: When a high-voltage motor in a factory has a high-noise abnormality, vibration data may be abnormal. When the vibration data is abnormal, the vibration sensor detects that the vibration frequency and amplitude at the non-shaft extension end change significantly, especially within a certain specific frequency range, and the vibration energy increases significantly. While other data is normal: Other operation data such as the current, voltage, and speed of the motor are all within the normal range and there are no abnormal fluctuations.

[0061] Therefore, the present invention first forms a fusion group of data with similar abnormal frequencies in a fault, that is, when a fault occurs, these data often become abnormal simultaneously, which can indicate the occurrence of a certain fault.

[0062] Whenever a historical fault event occurs, perform abnormality identification on each type of operation data. Take the number of times each type of operation data is abnormal under all historical fault occurrences as the data abnormality coefficient of each type of operation data. Among them, use the Z-score algorithm to determine whether the operation data is abnormal. It can be understood that, that is, for each type of operation data, use the Z-score algorithm to obtain whether the operation data is abnormal data, and then obtain whether each type of operation data in each historical fault event is abnormal data. For the operation data that is abnormal data, add 1 to the size of the data abnormality coefficient corresponding to the operation data. By judging and analyzing all historical fault events, the data abnormality coefficient of each type of operation data can be obtained.

[0063] Step S220, determine the image abnormality coefficient of the operation image from each perspective according to the change in the abnormal area in adjacent frames of the operation image from each perspective of the historical fault event.

[0064] Whenever a historical fault event occurs, for the running images from each perspective, the difference region between the current running image and the previous frame of the running image is obtained through the frame difference method, and the proportion of the area of the difference region in the running image is determined, which is denoted as the abnormal area proportion; it is judged whether the abnormal area proportion of the running image from the current perspective in the historical fault event is increasing; the number of times the abnormal area proportion of the running images from each perspective increases in all historical fault events is used as the image abnormality coefficient of the running images from each perspective.

[0065] It can be understood that, that is, for the running images from each perspective, through the frame difference method, the difference region is calculated, and the proportion of the area of the difference region between adjacent running images in the running image is obtained, and then the abnormal area proportion sequence of the difference region is obtained. If this sequence is an increasing sequence, it indicates that there is an abnormality, and at the same time, the area of the abnormal region gradually increases. For example, in the running image, when the engine overheats, the area of the high-temperature region increases. The abnormal region shows an increasing area, and the running image from this perspective is an abnormal image. If in a certain fault data, the running image from this perspective is abnormal data, then the value of the image abnormality coefficient of the running image from this perspective is incremented by 1. It should be noted that the running image is a thermal imaging image; if all the running images from the current perspective are abnormal images, then the value of the image abnormality coefficient of the running images from this perspective is equal to the number of the running images from this perspective.

[0066] As a preferred embodiment of the present invention, the method for judging whether the abnormal area proportion of the running image from the current perspective in the historical fault event is increasing is as follows:

[0067] Taking the sequence value corresponding to the running image as the abscissa and the abnormal area proportion of the running image as the ordinate, coordinate points in a two-dimensional rectangular coordinate system are constructed; all the coordinate points are used as the input of the PCA algorithm to obtain multiple projection directions and the projection values corresponding to each projection direction; the projection direction corresponding to the maximum projection value is denoted as the main projection direction, and the arctangent value of the ratio of the ordinate to the abscissa of the coordinate point corresponding to the main projection direction; when the arctangent value is greater than 0, it is determined that the abnormal area proportion of the running image from the current perspective in the currently analyzed historical fault event is increasing.

[0068] It can be understood that, that is, an abnormal area ratio sequence is constructed according to the abnormal area ratio of the running image. The ordinal value of each value in the abnormal area ratio sequence is used as the abscissa, and each value in the sequence is used as the ordinate to obtain multiple coordinate points. All the coordinate points are used as the input of the PCA algorithm to obtain multiple projection directions and the projection values corresponding to each projection direction. Each projection direction is a two-dimensional vector. The projection direction corresponding to the maximum projection value is denoted as the main projection direction. The arctangent value s of the ratio of the ordinate to the abscissa of the two-dimensional vector corresponding to the main projection direction is calculated. If s is greater than 0, the abnormal area shows an increasing trend, and the running image from this perspective is abnormal data. If in a certain fault data, the running image from this perspective is abnormal data, then the magnitude of the image abnormality coefficient of the running image from this perspective is incremented by 1.

[0069] The abnormality coefficients of each type of running data and running image can be obtained through calculation.

[0070] Step S230, based on the data abnormality coefficient and the image abnormality coefficient, perform one-to-one matching on the running data and the running image to obtain an abnormal fusion group.

[0071] If a certain type of running data is abnormal data in more faults, then the frequency of this running data being abnormal is higher. Similarly, if the running image from a certain perspective is abnormal data in more faults, then the frequency of this running image being abnormal is higher. If the frequency of a certain running data is higher and the frequency of a certain running image is also higher, and the two frequencies are close, then when a fault is likely to occur, these two data have problems simultaneously. Furthermore, if these two data have problems simultaneously, then a certain fault is likely to have occurred. At this time, the running data and the running image can probably represent the same fault. Therefore, these two types of data are used as a fusion group, which is helpful for subsequent data fusion.

[0072] In the embodiment of the present invention, the existing KM matching algorithm is adopted to perform one-to-one matching on the operation data and the operation image. The existing KM matching is to calculate the one-to-one matching between the left nodes and the right nodes. In the present invention, the operation data and the operation image are respectively used as the nodes on the left and right sides in the KM matching algorithm, and the data anomaly coefficient of the operation data and the image anomaly coefficient of the operation image are respectively used as the corresponding node values. The absolute value of the difference between the node values is used as the edge value between the nodes. Based on the minimum matching principle, one-to-one matching is performed on the operation data and the operation image to obtain multiple fusion groups; where each fusion group includes one operation data and one operation image. More specifically: in the present invention, the operation data is used as the node on the left side in the KM matching algorithm, and the operation image is used as the node on the right side in the KM matching algorithm; the data anomaly coefficient of each operation data is used as the left node value, and the image anomaly coefficient of each operation image is used as the right node value. Each node on the left is connected to each node on the right by an edge, and the edge value is the absolute value of the difference between the corresponding node values. Following the minimum matching principle, through KM matching, a one-to-one matching relationship between the left nodes and the right nodes is obtained, that is, a one-to-one relationship between each operation data and each operation image is obtained.

[0073] Obtain the frequency of each operation data in the fusion group and the frequency of the operation image in the fusion group. It should be noted that the frequency of the operation data in the fusion group is the frequency of the operation data in all fusion groups. Since the fusion group reflects the abnormal situation, the frequency of the operation data in the fusion group can also be understood as the frequency of the operation data belonging to the abnormal data.

[0074] Take the minimum value of the frequencies of the operation data and the operation image in the fusion group as the frequency of the fusion group. Retain the fusion groups whose frequency of the fusion group is greater than the preset frequency threshold and the edge value is less than the preset edge value threshold. Record the operation data and the corresponding operation image with the retained corresponding relationship as the abnormal fusion group. In the embodiment of the present invention, the value of the preset frequency threshold is 0.5, and the value of the preset edge value threshold is 0.1. In other embodiments, these two thresholds can also be determined by the implementer according to the actual situation. It can be understood that that is, for each one-to-one relationship, take the minimum value of the frequencies of the operation data and the corresponding operation image as the frequency of the corresponding relationship. Retain the corresponding relationship whose frequency of the corresponding relationship is greater than 0.5 and the edge value is less than 0.1. Record the operation data and the corresponding operation image with the retained corresponding relationship as the abnormal fusion group, and then multiple abnormal fusion groups are obtained. The occurrence frequencies of the operation data and the operation image in the same abnormal fusion group are relatively large and the frequencies are close. It is very likely to represent a kind of fault, that is, when a certain fault occurs, these two kinds of data show large anomalies, that is, when a certain fault of the device occurs, the data in a certain abnormal fusion group all show large anomalies.

[0075] Step S240: Determine the hysteresis factor of the abnormal fusion group according to the matching condition of the abnormal rates of the operation data and operation images in the abnormal fusion group; train the neural network according to the abnormal fusion group and the corresponding hysteresis factor.

[0076] Align the operation data and operation images in the abnormal fusion group in terms of time points, and use the later start data point in the two types of data as the starting point of the abnormal fusion group. For example, if the start time of the first type of data in the abnormal fusion group is 10:10 and the start time of the other type of data is 10:12, then the later start data point here is 10:12.

[0077] For each abnormal fusion group, obtain the acquisition intervals of the two types of data therein. For the data with a smaller acquisition interval, obtain the sampled data of this type of data through the larger acquisition interval. The sampled data and the data with a larger acquisition interval form the prepared abnormal fusion group. The time starting point of the prepared abnormal fusion group and the sampling time points of each data can be corresponding one by one, which is convenient for subsequent calculations. Subsequent calculations are all carried out on the basis of the prepared abnormal fusion group. For the convenience of description, it is still called the abnormal fusion group later.

[0078] One type of data in the abnormal fusion group is operation data and the other type is operation image. For the device, usually the operation data is abnormal first, and then the abnormal feedback is generated on the device. For example, when the vibration of the motor is abnormal, first the data of the vibration sensor is abnormal, and then the change of the high-temperature area on the thermal infrared image of the motor occurs.

[0079] For the operation data and operation images in the same abnormal fusion group, due to different degrees of manifestation of the abnormality, although each has generated an abnormality and the degrees of abnormality are different, the degrees of change of the abnormality are similar. When the device failure is greater, the abnormal changes of the two types of data are also greater.

[0080] First, determine the data abnormal rate corresponding to the moment to be analyzed according to the difference between the data at the moment to be analyzed and the data at the previous moment in the operation data of the abnormal fusion group, and obtain the sequence of data abnormal rates of the operation data in the historical fault events. The sequence of data abnormal rates is a sequence composed of the data abnormal rates corresponding to the operation data at the moments when individual fault events occur.

[0081] Take the absolute value of the difference between the data at the moment to be analyzed and the data at the previous moment in the operation data of the abnormal fusion group as the numerator, and take the maximum value of the data at the moment to be analyzed and the data at the previous moment in the operation data of the abnormal fusion group as the denominator, and take the ratio of the numerator and the denominator as the data abnormal rate corresponding to the moment to be analyzed.

[0082] In some embodiments, the calculation formula of the data abnormal rate y is: ; where a is the data at the previous moment of the operation data in the abnormal fusion group; b is the data at the moment to be analyzed of the operation data in the abnormal fusion group; max is the function to take the maximum value. Furthermore, multiple data anomaly rates can be calculated, and then a data anomaly rate sequence of the operation data is formed.

[0083] Determine the image anomaly rate corresponding to the moment to be analyzed based on the proportion of the area of the difference image between the moment to be analyzed and the previous moment of the operation image in the abnormal fusion group, and obtain the image anomaly rate sequence of the operation images in the historical fault events.

[0084] For the operation images in each abnormal fusion group, obtain the difference image between the previous frame and the next frame through the frame difference method. Denote the number of non-zero pixel points in the difference image as s, and denote the number of pixels in each frame image as N. Calculate the image anomaly rate Y, and the calculation formula for the image anomaly rate is Y = s / N. Furthermore, multiple image anomaly rates can be calculated, and then an anomaly rate sequence of the operation images is formed.

[0085] Match the data anomaly rate sequence and the image anomaly rate sequence to obtain the hysteresis value between the two sequences as the hysteresis factor of the abnormal fusion group. As a preferred embodiment of the present invention, perform DTW matching on the data anomaly rate sequence and the image anomaly rate sequence to obtain all the matching relationships.

[0086] The matching relationships include one-to-one matching, one-to-many matching, and many-to-one matching. For one-to-one matching, calculate the absolute value of the difference in the abscissa as the hysteresis value; for one-to-many matching, calculate the absolute value of the difference in the abscissa of each one-to-one matching, and take the average value of the multiple absolute values of the differences as the hysteresis value. The same method is used to obtain the hysteresis value for many-to-one matching.

[0087] Take the average value of all the hysteresis values as the hysteresis factor of the operation data and the operation images in this abnormal fusion group. The hysteresis factor represents the temporal connection in the fault manifestation of two types of data of the same fault.

[0088] Multiple hysteresis factors can be obtained through calculation to form a hysteresis factor sequence. Use the historical fault events, the operation images and operation data in the abnormal data corresponding to the historical fault events, and the corresponding hysteresis factor sequence as sample data, and label the samples according to the actual operation status of the device, such as normal status, abnormal status, etc.

[0089] An embodiment of the present invention provides a multi-modal data fusion system based on identity resolution. The system includes:

[0090] A device data acquisition module, which is used to acquire the operation data and operation images during the operation of the devices on the production line during industrial production;

[0091] A device status determination module, configured to input real-time operation data and operation images into a trained neural network to obtain a device status type;

[0092] The training process of the neural network is as follows: perform anomaly recognition on each piece of operation data of historical fault events to determine the data anomaly coefficient of each piece of operation data; determine the image anomaly coefficient of the operation images in adjacent frames from each perspective of historical fault events according to the change in the anomaly area; based on the data anomaly coefficient and the image anomaly coefficient, perform one-to-one matching on the operation data and the operation images to obtain an anomaly fusion group; determine the hysteresis factor of the anomaly fusion group according to the matching situation of the anomaly rates of the operation data and the operation images in the anomaly fusion group; train the neural network according to the anomaly fusion group and the corresponding hysteresis factor.

[0093] Optionally, the transmission medium may be a wired link, such as but not limited to, coaxial cable, optical fiber, digital subscriber line, etc., or a wireless link, such as but not limited to, Wireless Fidelity (WIFI), Bluetooth, mobile device network, etc.

[0094] It should be noted that: for the device provided in the above embodiment, only the above-mentioned division of each functional module is used for illustration. In actual application, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above.

[0095] Figure 3 It is a schematic structural diagram of a computer device provided by an embodiment of the present invention. Exemplarily, as Figure 3 shown, the computer device 300 includes: a memory 310, a processor 320, and a computer program 330 stored in the memory 310 and running on the processor 320. When the processor 320 executes the computer program 330, the computer device can execute any of the above-described multi-modal data fusion methods based on identity resolution.

[0096] In addition, an embodiment of the present invention also protects a device, which may include a memory and a processor. The memory stores executable program code, and the processor is configured to call and execute the executable program code to execute the multi-modal data fusion method based on identity resolution provided by the embodiment of the present invention.

[0097] In the embodiments of the present invention, the device can be divided into functional modules according to the above method examples. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0098] In the case of dividing each module according to each function, the device can also include a signal uploading module, a determination module, an adjustment module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be cited in the function description of the corresponding functional module, and will not be elaborated here.

[0099] It should be understood that the device provided by the embodiments of the present invention is used to execute the above multi-modal data fusion method based on identity resolution, so the same effect as the above implementation method can be achieved.

[0100] In the case of adopting an integrated unit, the device can include a processing module and a storage module. Among them, when the device is applied to a device, the processing module can be used to control and manage the actions of the device. The storage module can be used to support the device to execute mutual program codes, etc. Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the present invention. The processor can also be a combination that realizes computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory.

[0101] In addition, the device provided by the embodiments of the present invention can specifically be a chip, a component or a module. The chip can include a connected processor and a memory; among them, the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute the multi-modal data fusion method based on identity resolution provided by the above embodiments.

[0102] The embodiments of the present invention also provide a computer-readable storage medium, in which computer program code is stored. When the computer program code runs on a computer, the computer is enabled to execute the above relevant method steps to implement the multi-modal data fusion method based on identity resolution provided by the above embodiments.

[0103] The embodiments of the present invention also provide a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above relevant steps to implement the multi-modal data fusion method based on identity resolution provided by the above embodiments.

[0104] Among them, the device, computer-readable storage medium, computer program product or chip provided by the embodiments of the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here. Through the description of the above embodiments, those skilled in the art can understand that for the convenience and conciseness of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways.

[0105] The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0106] It should also be noted that in this article, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. Without further limitation, the element defined by the statement "including one..." does not exclude the existence of other identical elements in the process, method, article or terminal device including the said element.

[0107] It should be noted that the above sequence of the embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0108] Each embodiment in this specification is described in a progressive manner. The same or similar parts between each embodiment can be referred to each other. The key point of each embodiment is to illustrate the differences from other embodiments.

[0109] The above content is only a specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention.

Claims

1. A multimodal data fusion method based on identity resolution, characterized in that: The method comprises the following steps: Obtaining operation data and operation images of equipment on the production line during industrial production; Input the real-time operation data and operation images into the trained neural network to obtain the equipment status type; The training process of the neural network is as follows: performing abnormal identification on each type of operation data of the historical fault event and determining the data abnormality coefficient of each type of operation data; determining the image abnormality coefficient of the operation image at each viewing angle according to the abnormal area change in the operation image of adjacent frames at each viewing angle of the historical fault event; performing one-to-one matching of the operation data and the operation image based on the data abnormality coefficient and the image abnormality coefficient to obtain an abnormal fusion group; determining the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rate of the operation data and the operation image in the abnormal fusion group; and training the neural network according to the abnormal fusion group and the corresponding hysteresis factor; Among them, the method for obtaining the abnormal fusion group is: taking the operation data and the operation image as the nodes on the left and right sides of the KM matching algorithm respectively, and taking the data anomaly coefficient of the operation data and the image anomaly coefficient of the operation image as the corresponding node values ​​respectively, taking the absolute value of the difference between the node values ​​as the edge value between the nodes, and performing one-to-one matching on the operation data and the operation image based on the minimum matching principle to obtain multiple fusion groups; wherein each fusion group includes an operation data and an operation image; obtaining the frequency of each operation data in the fusion group and the frequency of the operation image in the fusion group; taking the minimum value of the frequency of the operation data and the operation image in the fusion group as the frequency of the fusion group, retaining the fusion group whose frequency is greater than the preset frequency threshold and whose edge value is less than the preset edge value threshold, and recording the operation data and the corresponding operation image with the retained corresponding relationship as the abnormal fusion group; Among them, the method for obtaining the hysteresis factor of the abnormal fusion group is: according to the difference between the time to be analyzed and the data at the previous time of the operating data in the abnormal fusion group, the data anomaly rate corresponding to the time to be analyzed is determined, and the data anomaly rate sequence of the operating data in the historical fault events is obtained; the area ratio of the difference image between the time to be analyzed and the previous time of the operating image in the abnormal fusion group is used to determine the image anomaly rate corresponding to the time to be analyzed, and the image anomaly rate sequence of the operating images in the historical fault events is obtained; the data anomaly rate sequence and the image anomaly rate sequence are matched to obtain the hysteresis value between the two sequences as the hysteresis factor of the abnormal fusion group.

2. The multimodal data fusion method based on identification resolution according to claim 1 is characterized in that: The abnormality identification of each type of operation data of the historical fault event and determination of the data abnormality coefficient of each type of operation data includes: Whenever a historical fault event occurs, anomaly identification is performed on each type of operating data, and the number of times each type of operating data appears abnormal under all historical faults is used as the data anomaly coefficient of each type of operating data.

3. The multimodal data fusion method based on identification resolution according to claim 2 is characterized in that: Whenever a historical fault event occurs, abnormal identification is performed for each type of operation data, including: Use the Z-score algorithm to determine whether the running data is abnormal.

4. The multimodal data fusion method based on identification resolution according to claim 1 is characterized in that: The step of determining the image abnormality coefficient of the running image at each viewing angle according to the abnormal area change in the running images of adjacent frames at each viewing angle of the historical fault event includes: Whenever a historical fault event occurs, for the running image at each viewing angle, the frame difference method is used to obtain the difference area between the running image and the previous frame running image, and the area ratio of the difference area in the running image is determined, which is recorded as the abnormal area ratio; it is determined whether the abnormal area ratio of the running image at the current viewing angle in the historical fault event increases; The number of times the abnormal area ratio of the running image at each viewing angle increases in all historical fault events is taken as the image abnormality coefficient of the running image at each viewing angle.

5. The multimodal data fusion method based on identification resolution according to claim 4 is characterized in that: The determining whether the abnormal area ratio of the running image at the current viewing angle in the historical fault event increases progressively includes: The order value corresponding to the running image is used as the horizontal coordinate, and the abnormal area ratio of the running image is used as the vertical coordinate to construct the coordinate points in the two-dimensional rectangular coordinate system; all the coordinate points are used as the input of the PCA algorithm to obtain multiple projection directions and the projection values ​​corresponding to each projection direction; the projection direction corresponding to the maximum projection value is recorded as the main projection direction, and the inverse tangent value of the ratio of the vertical coordinate to the horizontal coordinate of the coordinate point corresponding to the main projection direction is taken; when the inverse tangent value is greater than 0, it is determined that the abnormal area ratio of the running image under the current perspective in the historical fault event currently analyzed is increasing.

6. The multimodal data fusion method based on identification resolution according to claim 1 is characterized in that: The step of determining the data anomaly rate corresponding to the time to be analyzed based on the difference between the data at the time to be analyzed and the data at the previous time in the abnormal fusion group includes: The absolute value of the difference between the time to be analyzed and the data at the previous time in the abnormal fusion group is used as the numerator, the maximum value of the time to be analyzed and the data at the previous time in the abnormal fusion group is used as the denominator, and the ratio between the numerator and the denominator is used as the data anomaly rate corresponding to the time to be analyzed.

7. The multimodal data fusion method based on identification resolution according to claim 1 is characterized in that: The training of the neural network according to the abnormal fusion group and the corresponding hysteresis factor includes: The abnormal fusion group and the corresponding hysteresis factor are used as the input of the neural network, and the corresponding fault label is used as the output target of the neural network to train the neural network.

8. A multimodal data fusion system based on identity resolution, characterized in that: The system includes the following modules: The equipment data acquisition module is used to obtain the operation data and operation images of the equipment on the production line during industrial production; The equipment status determination module is used to input the real-time operation data and operation images into the trained neural network to obtain the equipment status type; The training process of the neural network is as follows: performing abnormal identification on each type of operation data of the historical fault event and determining the data abnormality coefficient of each type of operation data; determining the image abnormality coefficient of the operation image at each viewing angle according to the abnormal area change in the operation images of adjacent frames at each viewing angle of the historical fault event; performing one-to-one matching on the operation data and the operation image based on the data abnormality coefficient and the image abnormality coefficient to obtain an abnormal fusion group; Determining the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rates of the operation data and the operation images in the abnormal fusion group; Training a neural network according to the abnormal fusion group and the corresponding hysteresis factor; Among them, the method for obtaining the abnormal fusion group is: taking the operation data and the operation image as the nodes on the left and right sides of the KM matching algorithm respectively, and taking the data anomaly coefficient of the operation data and the image anomaly coefficient of the operation image as the corresponding node values ​​respectively, taking the absolute value of the difference between the node values ​​as the edge value between the nodes, and performing one-to-one matching on the operation data and the operation image based on the minimum matching principle to obtain multiple fusion groups; wherein each fusion group includes an operation data and an operation image; obtaining the frequency of each operation data in the fusion group and the frequency of the operation image in the fusion group; taking the minimum value of the frequency of the operation data and the operation image in the fusion group as the frequency of the fusion group, retaining the fusion group whose frequency is greater than the preset frequency threshold and whose edge value is less than the preset edge value threshold, and recording the operation data and the corresponding operation image with the retained corresponding relationship as the abnormal fusion group; Among them, the method for obtaining the hysteresis factor of the abnormal fusion group is: according to the difference between the time to be analyzed and the data at the previous time of the operating data in the abnormal fusion group, the data anomaly rate corresponding to the time to be analyzed is determined, and the data anomaly rate sequence of the operating data in the historical fault events is obtained; the area ratio of the difference image between the time to be analyzed and the previous time of the operating image in the abnormal fusion group is used to determine the image anomaly rate corresponding to the time to be analyzed, and the image anomaly rate sequence of the operating images in the historical fault events is obtained; the data anomaly rate sequence and the image anomaly rate sequence are matched to obtain the hysteresis value between the two sequences as the hysteresis factor of the abnormal fusion group.

Images