Multi-modal data fusion method and system based on identification analysis

Abstract

The invention relates to the technical field of data fusion matching, in particular to a multi-modal data fusion method and system based on identification analysis. The method comprises the following steps: inputting real-time operation data and operation images into a trained neural network to obtain an equipment state type; the training process of the neural network comprises the following steps: performing anomaly identification on each type of operation data of a historical fault event, and determining a data anomaly coefficient; determining an image anomaly coefficient according to the anomaly area change in the operation image under each view angle of the historical fault event; based on the data anomaly coefficient and the image anomaly coefficient, matching the operation data and the operation image to obtain an anomaly fusion group, and determining a hysteresis factor of the anomaly fusion group; and training a neural network according to the abnormal fusion group and the corresponding hysteresis factor. According to the method, each kind of operation data and operation image are fused and associated, and the neural network is trained through the fused data, so that the fault can be identified with higher precision.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion and matching, and in particular to a multimodal data fusion method and system based on identity resolution. Background Art

[0002] In industrial production, through identity resolution, the operation data, image data and text data of large and medium-sized equipment in the factory can be integrated to achieve comprehensive monitoring of equipment status and fault prediction. Among them, operation data such as temperature and pressure, image data such as equipment appearance inspection, and text data such as equipment manuals. Among them, operation data can show whether the equipment is working within normal parameters, and image data may capture problems visible to the naked eye. Combining multiple types of data can more comprehensively analyze equipment status, predict faults, optimize maintenance plans, and even automate certain processes.

[0003] When existing methods fuse data of different modalities for fault identification, they often directly fuse the original data during the data acquisition stage. For example, the equipment operation data and image data are directly spliced ​​together and merged into an input vector or matrix. However, such fusion is only superficial and shallow. For fault identification, it is difficult to fully utilize the advantages of multimodal data and it is impossible to model cross-modal semantic or spatial associations. Summary of the invention

[0004] In order to solve the technical problem that data can be simply spliced ​​and cross-modal semantic or spatial associations cannot be modeled, the purpose of the present invention is to provide a multimodal data fusion method and system based on identity resolution. The technical solution adopted is as follows: In a first aspect, an embodiment of the present invention provides a multimodal data fusion method based on identity resolution, the method comprising: Obtaining operation data and operation images of equipment on the production line during industrial production; Input the real-time operation data and operation images into the trained neural network to obtain the equipment status type; The training process of the neural network is as follows: performing anomaly identification on each type of operation data of historical fault events and determining the data anomaly coefficient of each type of operation data; determining the image anomaly coefficient of the operation image at each perspective according to the change of abnormal area in the operation images of adjacent frames at each perspective of the historical fault events; performing one-to-one matching of the operation data and the operation image based on the data anomaly coefficient and the image anomaly coefficient to obtain an abnormal fusion group; determining the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rates of the operation data and the operation image in the abnormal fusion group; and training the neural network according to the abnormal fusion group and the corresponding hysteresis factor.

[0005] Furthermore, the abnormality identification of each type of operation data of the historical fault event and determination of the data abnormality coefficient of each type of operation data includes: Whenever a historical fault event occurs, anomaly identification is performed on each type of operating data, and the number of times each type of operating data appears abnormal under all historical faults is used as the data anomaly coefficient of each type of operating data.

[0006] Furthermore, whenever a historical fault event occurs, abnormality identification is performed on each type of operation data, including: Use the Z-score algorithm to determine whether the running data is abnormal.

[0007] Further, the method of determining the image abnormality coefficient of the running image at each viewing angle according to the abnormal area change in the running image of adjacent frames at each viewing angle of the historical fault event includes: Whenever a historical fault event occurs, for the running image at each viewing angle, the frame difference method is used to obtain the difference area between the running image and the previous frame running image, and the area ratio of the difference area in the running image is determined, which is recorded as the abnormal area ratio; it is determined whether the abnormal area ratio of the running image at the current viewing angle in the historical fault event increases; The number of times the abnormal area ratio of the running image at each viewing angle increases in all historical fault events is taken as the image abnormality coefficient of the running image at each viewing angle.

[0008] Further, the determining whether the abnormal area ratio of the running image at the current viewing angle in the historical fault event increases progressively includes: The order value corresponding to the running image is used as the horizontal coordinate, and the abnormal area ratio of the running image is used as the vertical coordinate to construct the coordinate points in the two-dimensional rectangular coordinate system; all the coordinate points are used as the input of the PCA algorithm to obtain multiple projection directions and the projection values ​​corresponding to each projection direction; the projection direction corresponding to the maximum projection value is recorded as the main projection direction, and the inverse tangent value of the ratio of the vertical coordinate to the horizontal coordinate of the coordinate point corresponding to the main projection direction is taken; when the inverse tangent value is greater than 0, it is determined that the abnormal area ratio of the running image under the current perspective in the historical fault event currently analyzed is increasing.

[0009] Furthermore, the one-to-one matching of the operation data and the operation image is performed based on the data anomaly coefficient and the image anomaly coefficient to obtain an anomaly fusion group, including: The operation data and the operation image are respectively used as the nodes on the left and right sides of the KM matching algorithm, and the data anomaly coefficient of the operation data and the image anomaly coefficient of the operation image are respectively used as the corresponding node values, and the absolute value of the difference between the node values ​​is used as the edge value between the nodes. Based on the minimum matching principle, the operation data and the operation image are matched one-to-one to obtain multiple fusion groups; each fusion group includes an operation data and an operation image; Obtaining the frequency of each running data being in the fusion group and the frequency of the running image being in the fusion group; The minimum value of the frequency of the operating data and the operating image in the fusion group is taken as the frequency of the fusion group, and the fusion group whose frequency is greater than the preset frequency threshold and whose edge value is less than the preset edge value threshold is retained, and the operating data and the corresponding operating image with retained corresponding relationship are recorded as abnormal fusion group.

[0010] Furthermore, the determining of the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rates of the operation data and the operation image in the abnormal fusion group includes: According to the difference between the operating data at the time to be analyzed and the data at the previous time in the abnormal fusion group, the data abnormality rate corresponding to the time to be analyzed is determined, and the data abnormality rate sequence of the operating data in the historical fault event is obtained; The image abnormality rate corresponding to the time to be analyzed is determined by taking the area ratio of the difference image between the time to be analyzed and the previous time of the running image in the abnormal fusion group, and the image abnormality rate sequence of the running image in the historical fault event is obtained; The data abnormality rate sequence and the image abnormality rate sequence are matched to obtain a hysteresis value between the two sequences as a hysteresis factor of the abnormal fusion group.

[0011] Further, the determining of the data anomaly rate corresponding to the time to be analyzed according to the difference between the data at the time to be analyzed and the data at the previous time in the running data in the anomaly fusion group includes: The absolute value of the difference between the time to be analyzed and the data at the previous time in the abnormal fusion group is used as the numerator, the maximum value of the time to be analyzed and the data at the previous time in the abnormal fusion group is used as the denominator, and the ratio between the numerator and the denominator is used as the data anomaly rate corresponding to the time to be analyzed.

[0012] Further, the training of the neural network according to the abnormal fusion group and the corresponding hysteresis factor includes: The abnormal fusion group and the corresponding hysteresis factor are used as the input of the neural network, and the corresponding fault label is used as the output target of the neural network to train the neural network.

[0013] In a second aspect, a multimodal data fusion system based on identity resolution is provided, the system comprising the following modules: The equipment data acquisition module is used to obtain the operation data and operation images of the equipment on the production line during industrial production; The equipment status determination module is used to input the real-time operation data and operation images into the trained neural network to obtain the equipment status type; The training process of the neural network is as follows: performing anomaly identification on each type of operation data of historical fault events and determining the data anomaly coefficient of each type of operation data; determining the image anomaly coefficient of the operation image at each perspective according to the change of abnormal area in the operation images of adjacent frames at each perspective of the historical fault events; performing one-to-one matching of the operation data and the operation image based on the data anomaly coefficient and the image anomaly coefficient to obtain an abnormal fusion group; determining the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rates of the operation data and the operation image in the abnormal fusion group; and training the neural network according to the abnormal fusion group and the corresponding hysteresis factor.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, various possible embodiments of the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer program product, which includes: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

[0016] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute various possible implementations of the first aspect.

[0017] The embodiments of the present invention have at least the following beneficial effects: The present invention realizes inputting real-time operation data and operation images into a trained neural network to obtain the equipment status type. Among them, the operation data of historical fault events and operation data are used to train the neural network. Because when a fault event occurs, not all data are abnormal, it may be that part of the operation data and operation images are abnormal, while other parts of the data are still within the normal range. Therefore, the present invention matches the operation data and the operation image according to the abnormal coefficients of the operation data and the operation image to obtain an abnormal fusion group. When a fault occurs, the data belonging to the same abnormal fusion group can indicate the occurrence of a certain fault. The present invention associates the fault event with the operation data and the operation image through analysis to obtain fused data. The neural network that can perform fault diagnosis obtained by training the fused data can identify the fault with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A method flow chart of a multimodal data fusion method based on identity resolution provided by an embodiment of the present invention; Figure 2 A flowchart of the steps of a neural network training process provided by one embodiment of the present invention; Figure 3 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following, in combination with the accompanying drawings and preferred embodiments, a multimodal data fusion method and system based on identity resolution proposed in accordance with the present invention, its specific implementation method, structure, characteristics and effects are described in detail as follows.

[0021] In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0022] Among them, in the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" refers to two or more than two.

[0023] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0024] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0025] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0026] The following is a detailed description of a multimodal data fusion method and system based on identity resolution provided by the present invention in conjunction with the accompanying drawings.

[0027] See also Figure 1 , which shows a flowchart of a multimodal data fusion method based on identity resolution provided by an embodiment of the present invention, the method comprising the following steps: Step S100, obtaining operation data and operation images of equipment on a production line during industrial production.

[0028] Temperature sensors, pressure sensors, ammeters, voltmeters and power sensors are installed on the production line to collect temperature data, pressure data, current data, voltage data and power data respectively. The temperature data, pressure data, current data, voltage data and power data when the equipment on the production line is running during industrial production are collectively referred to as operation data, that is, the operation data includes multiple data; and cameras with different viewing angles are installed at different angles of the production line to collect the operation images of the equipment. The operation data can also include the rotation speed.

[0029] Step S200, input the real-time operation data and operation image into the trained neural network to obtain the device status type.

[0030] The present invention is aimed at the traditional method in which the data fusion of the operation data and the operation image is simply vectorized and spliced ​​together to form a spliced ​​vector. Such fusion is only superficial and shallow fusion, and cannot give full play to the advantages of multimodal data. Based on this, the present invention analyzes the fault. Although the operation data and the operation image of the fault belong to different types of data, they can both represent the fault information. Then, through the common representation and representation order of the two types of data, the identification data of the two types of data are obtained, and then the fault identification data is formed to participate in the training of the neural network. The neural network is also a network for realizing fault identification.

[0031] The real-time operation data and operation images of the equipment on the production line during industrial production are input into the trained neural network, and after forward propagation calculation, the result value is output, and the equipment status type is judged according to the result value. The output result value is usually a probability distribution, which indicates the possibility of the equipment being in different fault events. The equipment status type is judged according to the result value: if the probability corresponding to the output fault event is greater than the preset judgment threshold, it is judged that the equipment has an abnormality and the equipment status type is abnormal; otherwise, the equipment status type is judged to be normal. In one embodiment of the present invention, the preset judgment threshold can be set to the mode of the label values ​​corresponding to the operation data when the same type of historical fault events occur.

[0032] In the process of training a neural network, optimization algorithms such as the first-order gradient-based optimization algorithm (Adam optimization algorithm, Adam) and the stochastic gradient descent algorithm (SGD) can be used to adjust the weights and biases of the neural network to minimize the network's loss function. At the same time, the performance of the network can be evaluated through methods such as cross-validation to avoid overfitting.

[0033] As a preferred embodiment of the present invention, the training process of the neural network includes steps S210 to S240, see Figure 2 As shown, Figure 2 Flowchart of the steps in the training process of a neural network: Step S210: performing abnormality identification on each type of operation data of the historical fault event, and determining a data abnormality coefficient of each type of operation data.

[0034] The operation data and operation images when the fault events in the history occur are recorded separately, and the obtained operation data and operation images are as follows: each operation data of multiple historical fault events and the operation images of the equipment under different perspectives are obtained. For example, fault event 1 includes: operation data 1: {current data, voltage data, power data, temperature data, pressure data}, operation image 1: {operation image under the first perspective, operation image under the second perspective, operation image under the third perspective...}; fault event 2 includes: operation data 2: {current data, voltage data, power data, temperature data, pressure data}, operation image 2: {operation image under the first perspective, operation image under the second perspective, operation image under the third perspective...}.

[0035] When a fault event occurs, not all data are abnormal. For example, when a high-voltage motor in a factory has abnormal high noise, abnormal vibration data may occur. When abnormal vibration data occurs, the vibration sensor detects a significant change in the vibration frequency and amplitude of the non-shaft extension end, especially in a certain frequency range, the vibration energy increases significantly. Other data are normal: the motor's current, voltage, speed and other operating data are within the normal range, without abnormal fluctuations.

[0036] Therefore, the present invention first forms a fusion group with data having similar frequencies of abnormalities in the fault, that is, when a fault occurs, these data often have abnormalities at the same time, which can identify the occurrence of a certain fault.

[0037] Whenever a historical fault event occurs, an abnormality identification is performed for each type of operation data, and the number of abnormalities of each type of operation data under all historical faults is used as the data abnormality coefficient of each type of operation data, wherein the Z-score algorithm is used to determine whether the operation data is abnormal. It can be understood that for each type of operation data, the Z-score algorithm is used to determine whether the operation data is abnormal data, and then whether each type of operation data in each historical fault event is abnormal data is obtained. If the operation data is abnormal data, the size of the data abnormality coefficient corresponding to the operation data is +1, and all historical fault events are judged and analyzed to obtain the data abnormality coefficient of each type of operation data.

[0038] Step S220, determining the image abnormality coefficient of the running image at each viewing angle according to the abnormal area change in the running images of adjacent frames at each viewing angle of the historical fault event.

[0039] Whenever a historical fault event occurs, for the running image at each viewing angle, the frame difference method is used to obtain the difference area between the running image and the previous frame running image, and the area ratio of the difference area in the running image is determined, which is recorded as the abnormal area ratio; it is judged whether the abnormal area ratio of the running image at the current viewing angle in the historical fault event increases; the number of times the abnormal area ratio of the running image at each viewing angle in all historical fault events increases is taken as the image abnormality coefficient of the running image at each viewing angle.

[0040] It can be understood that, for the running image at each viewing angle, the frame difference method is used to calculate the difference area, and the area ratio of the difference area of ​​the adjacent running images in the running image is obtained, and then the abnormal area ratio sequence of the difference area is obtained. If the sequence is an increasing sequence, it means that there is an abnormality, and the area of ​​the abnormal area gradually increases. For example; in the running image, when the engine is overheated, the area of ​​the high temperature area increases. The abnormal area shows an increasing area, and the running image at this viewing angle is an abnormal image. If in a certain fault data, the running image at this viewing angle is abnormal data, then the size of the image abnormality coefficient of the running image at this viewing angle is +1. It should be noted that the running image is a thermal imaging image; if all running images at the current viewing angle are abnormal images, then the size of the image abnormality coefficient of the running image at this viewing angle is equal to the number of running images at this viewing angle.

[0041] As a preferred embodiment of the present invention, a method for determining whether the abnormal area ratio of the running image at the current viewing angle in the historical fault event is increasing is as follows: The order value corresponding to the running image is used as the horizontal coordinate, and the abnormal area ratio of the running image is used as the vertical coordinate to construct the coordinate points in the two-dimensional rectangular coordinate system; all the coordinate points are used as the input of the PCA algorithm to obtain multiple projection directions and the projection values ​​corresponding to each projection direction; the projection direction corresponding to the maximum projection value is recorded as the main projection direction, and the inverse tangent value of the ratio of the vertical coordinate to the horizontal coordinate of the coordinate point corresponding to the main projection direction is taken; when the inverse tangent value is greater than 0, it is determined that the abnormal area ratio of the running image under the current perspective in the historical fault event currently analyzed is increasing.

[0042] It can be understood that, according to the abnormal area ratio of the running image, an abnormal area ratio sequence is constructed, and the order value of each value in the abnormal area ratio sequence is used as the horizontal coordinate, and each value in the sequence is used as the vertical coordinate to obtain multiple coordinate points, and all coordinate points are used as the input of the PCA algorithm to obtain multiple projection directions and projection values ​​corresponding to each projection direction. Each projection direction is a two-dimensional vector, and the projection direction corresponding to the maximum projection value is recorded as the main projection direction. The inverse tangent value s of the ratio of the vertical coordinate to the horizontal coordinate of the two-dimensional vector corresponding to the main projection direction, if s is greater than 0, the abnormal area presents an increasing area, and the running image under this perspective is abnormal data. If the running image under this perspective is abnormal data in a certain fault data, the size of the image abnormality coefficient of the running image under this perspective is +1.

[0043] The abnormal coefficient of each operating data and operating image can be obtained by calculation.

[0044] Step S230: Based on the data anomaly coefficient and the image anomaly coefficient, one-to-one matching is performed on the operation data and the operation image to obtain an anomaly fusion group.

[0045] If a certain operation data is abnormal data in more and more faults, the frequency of abnormal operation data will be greater. Similarly, if the operation image under a certain perspective is abnormal data in more and more faults, the frequency of abnormal operation image will be greater. If the frequency of a certain operation data is greater and the frequency of a certain operation image is greater at the same time, the two frequencies are similar, and it is highly likely that when a fault occurs, these two data will have problems at the same time. If these two data have problems at the same time, it is highly likely that a certain fault has occurred. At this time, the operation data and the operation image are likely to represent the same fault, and the two data are used as a fusion group, which is helpful for subsequent data fusion.

[0046] In an embodiment of the present invention, an existing KM matching algorithm is used to perform one-to-one matching of operation data and operation image. The existing KM matching is to calculate the one-to-one matching of the left node and the right node. In the present invention, the operation data and the operation image are respectively used as the nodes on the left and right sides of the KM matching algorithm, and the data anomaly coefficient of the operation data and the image anomaly coefficient of the operation image are respectively used as the corresponding node values, and the absolute value of the difference between the node values ​​is used as the edge value between the nodes. Based on the minimum matching principle, the operation data and the operation image are matched one-to-one to obtain multiple groups of fusion groups; wherein each group of fusion groups includes an operation data and an operation image. More specifically: in the present invention, the operation data is used as the node on the left side of the KM matching algorithm, and the operation image is used as the node on the right side of the KM matching algorithm; the data anomaly coefficient of each operation data is used as the left node value, and the image anomaly coefficient of each operation image is used as the right node value, each node on the left is connected to each node on the right by an edge, and the edge value is the absolute value of the difference between the corresponding node values. Following the minimum matching principle, through KM matching, a one-to-one matching relationship between the left node and the right node is obtained, that is, a one-to-one relationship between each operation data and each operation image is obtained.

[0047] The frequency of each operating data being in the fusion group and the frequency of the operating image being in the fusion group are obtained. It should be noted that the frequency of the operating data being in the fusion group is the frequency of the operating data appearing in all fusion groups. Since the fusion group reflects abnormal situations, the frequency of the operating data being in the fusion group here can also be understood as the frequency of the operating data being abnormal data.

[0048] The minimum value of the frequency of the operation data and the operation image in the fusion group is taken as the frequency of the fusion group, and the fusion group whose frequency is greater than the preset frequency threshold and whose margin value is less than the preset margin threshold is retained, and the operation data and the corresponding operation image with the corresponding relationship retained are recorded as abnormal fusion groups. In the embodiment of the present invention, the value of the preset frequency threshold is 0.5, and the value of the preset margin threshold is 0.1. In other embodiments, the two thresholds can also be determined by the implementer according to the actual situation. It can be understood that, that is, the minimum value of the frequency of the operation data and the corresponding operation image in each one-to-one relationship is taken as the frequency of the corresponding relationship, and the corresponding relationship with the frequency greater than 0.5 and the margin value less than 0.1 is retained, and the operation data and the corresponding operation image with the corresponding relationship retained are recorded as abnormal fusion groups, thereby obtaining multiple abnormal fusion groups. The operation data and the operation image in the same abnormal fusion group have a large frequency of occurrence and a similar frequency, which can represent a fault with a high probability, that is, when a certain fault occurs, these two data have a large abnormality, that is, when a certain fault of the equipment occurs, the data in a certain abnormal fusion group all show a large abnormality.

[0049] Step S240, determining the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rates of the operation data and the operation image in the abnormal fusion group; and training the neural network according to the abnormal fusion group and the corresponding hysteresis factor.

[0050] The running data and running images in the abnormal fusion group are time-aligned, and the later starting data point of the two data is used as the starting point of the abnormal fusion group. For example, the starting time of the first data in the abnormal fusion group is: 10:10, and the starting time of the other data is: 10:12, then the later starting data point here is: 10:12.

[0051] For each abnormal fusion group, the collection intervals of two types of data are obtained, and the data with a smaller collection interval is obtained through a larger collection interval to obtain the sampled data of the data, and the sampled data and the data with a larger sampling interval form the prepared abnormal fusion group. The time starting point of the prepared abnormal fusion group and the sampling time point of each data can be matched one by one, which is convenient for subsequent calculations. Subsequent calculations are all calculated on the basis of the prepared abnormal fusion group. For the convenience of description, it is still called the abnormal fusion group.

[0052] One type of data in the abnormal fusion group is the operating data, and the other type of data is the operating image. For equipment, the operating data is often abnormal first, and the resulting abnormality is then fed back to the equipment. For example, when the vibration of a motor is abnormal, the first abnormality is the vibration sensor data, and then the high-temperature area on the thermal infrared image of the motor changes.

[0053] For the operation data and operation images in the same abnormal fusion group, due to the different degrees of abnormal expression, although each of them has abnormalities with different degrees of abnormality, the degree of abnormal change is similar. When the equipment failure is greater, the abnormal change of the two data is also greater.

[0054] First, according to the difference between the time to be analyzed and the previous time of the running data in the abnormal fusion group, the data anomaly rate corresponding to the time to be analyzed is determined, and the data anomaly rate sequence of the running data in the historical fault event is obtained. Among them, the data anomaly rate sequence is a sequence composed of the data anomaly rate corresponding to the running data at the time when a single fault event occurs.

[0055] The absolute value of the difference between the time to be analyzed and the data at the previous time in the abnormal fusion group is used as the numerator, the maximum value of the time to be analyzed and the data at the previous time in the abnormal fusion group is used as the denominator, and the ratio between the numerator and the denominator is used as the data anomaly rate corresponding to the time to be analyzed.

[0056] In some embodiments, the calculation formula of the data anomaly rate y is: ; Where a is the data of the previous moment of the running data in the abnormal fusion group; b is the data of the running data in the abnormal fusion group at the time to be analyzed; max is the maximum value function. Then, multiple data anomaly rates can be calculated to form a data anomaly rate sequence of the running data.

[0057] The image anomaly rate corresponding to the time to be analyzed is determined by taking the area ratio of the difference image between the time to be analyzed and the previous time of the running image in the abnormal fusion group, and the image anomaly rate sequence of the running images in the historical fault events is obtained.

[0058] For each running image in the abnormal fusion group, the difference image between the previous frame and the next frame is obtained by the frame difference method. The number of non-zero pixels in the difference image is recorded as s, and the number of pixels in each frame image is recorded as N. The image abnormality rate Y is calculated. The calculation formula of the image abnormality rate is Y=s / N. Then, the abnormality rates of multiple images can be calculated to form the abnormality rate sequence of the running image.

[0059] The data anomaly rate sequence and the image anomaly rate sequence are matched to obtain the hysteresis value between the two sequences as the hysteresis factor of the anomaly fusion group. As a preferred embodiment of the present invention, the data anomaly rate sequence and the image anomaly rate sequence are DTW matched to obtain all matching relationships.

[0060] The matching relationship includes one-to-one matching, one-to-many matching and many-to-one matching. For one-to-one matching, the absolute value of the difference in the horizontal coordinates is calculated as the hysteresis value; for one-to-many matching, the absolute value of the difference in the horizontal coordinates of each one-to-one matching is calculated, and the average of the multiple absolute values ​​of the difference is used as the hysteresis value. The hysteresis value is also obtained for many-to-one matching.

[0061] The mean of all hysteresis values ​​is used as the hysteresis factor of the running data and running images in the abnormal fusion group. The hysteresis factor represents the time connection between two data of the same fault in terms of fault manifestation.

[0062] Through calculation, multiple hysteresis factors can be obtained to form a hysteresis factor sequence. The historical fault events, the operation images and operation data in the abnormal data corresponding to the historical fault events, and the corresponding hysteresis factor sequence are used as sample data, and the samples are labeled according to the actual operation status of the equipment, such as normal state, abnormal state, etc.

[0063] The embodiment of the present invention provides a multimodal data fusion system based on identity resolution, the system comprising: The equipment data acquisition module is used to obtain the operation data and operation images of the equipment on the production line during industrial production; The equipment status determination module is used to input the real-time operation data and operation images into the trained neural network to obtain the equipment status type; The training process of the neural network is as follows: performing anomaly identification on each type of operation data of historical fault events and determining the data anomaly coefficient of each type of operation data; determining the image anomaly coefficient of the operation image at each perspective according to the change of abnormal area in the operation images of adjacent frames at each perspective of the historical fault events; performing one-to-one matching of the operation data and the operation image based on the data anomaly coefficient and the image anomaly coefficient to obtain an abnormal fusion group; determining the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rates of the operation data and the operation image in the abnormal fusion group; and training the neural network according to the abnormal fusion group and the corresponding hysteresis factor.

[0064] Optionally, the transmission medium may be a wired link, such as but not limited to coaxial cable, optical fiber, and digital subscriber line, or a wireless link, such as but not limited to Wireless Fidelity (WIFI), Bluetooth, and mobile device network.

[0065] It should be noted that the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above.

[0066] Figure 3 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 3 As shown, the computer device 300 includes: a memory 310, a processor 320, and a computer program 330 stored in the memory 310 and running on the processor 320, wherein when the processor 320 executes the computer program 330, the computer device can execute any of the multimodal data fusion methods based on identity resolution introduced above.

[0067] In addition, an embodiment of the present invention also protects a device, which may include a memory and a processor, wherein the memory stores an executable program code, and the processor is used to call and execute the executable program code to execute the multimodal data fusion method based on identity resolution provided by an embodiment of the present invention.

[0068] The embodiment of the present invention can divide the functional modules of the device according to the above method example. For example, each functional module can be corresponded, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0069] In the case of dividing each module according to each function, the device may also include a signal uploading module, a determining module, an adjusting module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0070] It should be understood that the device provided in the embodiment of the present invention is used to execute the above-mentioned multimodal data fusion method based on identity resolution, and thus can achieve the same effect as the above-mentioned implementation method.

[0071] In the case of an integrated unit, the device may include a processing module and a storage module. When the device is applied to a device, the processing module may be used to control and manage the actions of the device. The storage module may be used to support the device to execute mutual program codes, etc. The processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. The processor may also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0072] In addition, the device provided by the embodiment of the present invention may specifically be a chip, a component or a module, and the chip may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute the multimodal data fusion method based on identity resolution provided in the above embodiment.

[0073] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned related method steps to implement the multimodal data fusion method based on identity resolution provided in the above embodiment.

[0074] An embodiment of the present invention also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the multimodal data fusion method based on identity resolution provided in the above embodiment.

[0075] Among them, the device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Through the description of the above implementation methods, technicians in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways.

[0076] The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. There may be other divisions in actual implementation. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0077] It should also be noted that, in this article, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0078] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0079] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0080] The above contents are only specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention.

Claims

1. A multimodal data fusion method based on identity resolution, characterized in that: The method comprises the following steps: Obtaining operation data and operation images of equipment on the production line during industrial production; Input the real-time operation data and operation images into the trained neural network to obtain the equipment status type; The training process of the neural network is as follows: performing anomaly identification on each type of operation data of historical fault events and determining the data anomaly coefficient of each type of operation data; determining the image anomaly coefficient of the operation image at each perspective according to the change of abnormal area in the operation images of adjacent frames at each perspective of the historical fault events; performing one-to-one matching of the operation data and the operation image based on the data anomaly coefficient and the image anomaly coefficient to obtain an abnormal fusion group; determining the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rates of the operation data and the operation image in the abnormal fusion group; and training the neural network according to the abnormal fusion group and the corresponding hysteresis factor.

2. The multimodal data fusion method based on identification resolution according to claim 1 is characterized in that: The abnormality identification of each type of operation data of the historical fault event and determination of the data abnormality coefficient of each type of operation data includes: Whenever a historical fault event occurs, anomaly identification is performed on each type of operating data, and the number of times each type of operating data appears abnormal under all historical faults is used as the data anomaly coefficient of each type of operating data.

3. The multimodal data fusion method based on identification resolution according to claim 2 is characterized in that: Whenever a historical fault event occurs, abnormal identification is performed for each type of operation data, including: Use the Z-score algorithm to determine whether the running data is abnormal.

4. The multimodal data fusion method based on identification resolution according to claim 1 is characterized in that: The step of determining the image abnormality coefficient of the running image at each viewing angle according to the abnormal area change in the running images of adjacent frames at each viewing angle of the historical fault event includes: Whenever a historical fault event occurs, for the running image at each viewing angle, the frame difference method is used to obtain the difference area between the running image and the previous frame running image, and the area ratio of the difference area in the running image is determined, which is recorded as the abnormal area ratio; it is determined whether the abnormal area ratio of the running image at the current viewing angle in the historical fault event increases; The number of times the abnormal area ratio of the running image at each viewing angle increases in all historical fault events is taken as the image abnormality coefficient of the running image at each viewing angle.

5. The multimodal data fusion method based on identification resolution according to claim 4 is characterized in that: The determining whether the abnormal area ratio of the running image at the current viewing angle in the historical fault event increases progressively includes: The order value corresponding to the running image is used as the horizontal coordinate, and the abnormal area ratio of the running image is used as the vertical coordinate to construct the coordinate points in the two-dimensional rectangular coordinate system; all the coordinate points are used as the input of the PCA algorithm to obtain multiple projection directions and the projection values ​​corresponding to each projection direction; the projection direction corresponding to the maximum projection value is recorded as the main projection direction, and the inverse tangent value of the ratio of the vertical coordinate to the horizontal coordinate of the coordinate point corresponding to the main projection direction is taken; when the inverse tangent value is greater than 0, it is determined that the abnormal area ratio of the running image under the current perspective in the historical fault event currently analyzed is increasing.

6. The multimodal data fusion method based on identification resolution according to claim 1 is characterized in that: The one-to-one matching of the operation data and the operation image is performed based on the data anomaly coefficient and the image anomaly coefficient to obtain an anomaly fusion group, including: The operation data and the operation image are respectively used as the nodes on the left and right sides of the KM matching algorithm, and the data anomaly coefficient of the operation data and the image anomaly coefficient of the operation image are respectively used as the corresponding node values, and the absolute value of the difference between the node values ​​is used as the edge value between the nodes. Based on the minimum matching principle, the operation data and the operation image are matched one-to-one to obtain multiple fusion groups; each fusion group includes an operation data and an operation image; Obtaining the frequency of each running data being in the fusion group and the frequency of the running image being in the fusion group; The minimum value of the frequency of the operating data and the operating image in the fusion group is taken as the frequency of the fusion group, and the fusion group whose frequency is greater than the preset frequency threshold and whose edge value is less than the preset edge value threshold is retained, and the operating data and the corresponding operating image with retained corresponding relationship are recorded as abnormal fusion group.

7. The multimodal data fusion method based on identification resolution according to claim 1 is characterized in that: Determining the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rates of the operation data and the operation image in the abnormal fusion group includes: According to the difference between the operating data at the time to be analyzed and the data at the previous time in the abnormal fusion group, the data abnormality rate corresponding to the time to be analyzed is determined, and the data abnormality rate sequence of the operating data in the historical fault event is obtained; The image abnormality rate corresponding to the time to be analyzed is determined by taking the area ratio of the difference image between the time to be analyzed and the previous time of the running image in the abnormal fusion group, and the image abnormality rate sequence of the running image in the historical fault event is obtained; The data abnormality rate sequence and the image abnormality rate sequence are matched to obtain a hysteresis value between the two sequences as a hysteresis factor of the abnormal fusion group.

8. The multimodal data fusion method based on identification resolution according to claim 7 is characterized in that: The step of determining the data anomaly rate corresponding to the time to be analyzed based on the difference between the data at the time to be analyzed and the data at the previous time in the abnormal fusion group includes: The absolute value of the difference between the time to be analyzed and the data at the previous time in the abnormal fusion group is used as the numerator, the maximum value of the time to be analyzed and the data at the previous time in the abnormal fusion group is used as the denominator, and the ratio between the numerator and the denominator is used as the data anomaly rate corresponding to the time to be analyzed.

9. The multimodal data fusion method based on identification resolution according to claim 1 is characterized in that: The training of the neural network according to the abnormal fusion group and the corresponding hysteresis factor includes: The abnormal fusion group and the corresponding hysteresis factor are used as the input of the neural network, and the corresponding fault label is used as the output target of the neural network to train the neural network.

10. A multimodal data fusion system based on identity resolution, characterized in that: The system includes the following modules: The equipment data acquisition module is used to obtain the operation data and operation images of the equipment on the production line during industrial production; The equipment status determination module is used to input the real-time operation data and operation images into the trained neural network to obtain the equipment status type; The training process of the neural network is as follows: performing abnormal identification on each type of operation data of the historical fault event and determining the data abnormality coefficient of each type of operation data; determining the image abnormality coefficient of the operation image at each viewing angle according to the abnormal area change in the operation images of adjacent frames at each viewing angle of the historical fault event; performing one-to-one matching on the operation data and the operation image based on the data abnormality coefficient and the image abnormality coefficient to obtain an abnormal fusion group; Determining the hysteresis factor of the abnormal fusion group according to the matching of the abnormality rates of the operation data and the operation images in the abnormal fusion group; The neural network is trained according to the abnormal fusion group and the corresponding hysteresis factor.

Images