A multi-modal fusion method based on attention mechanism and adversarial neural network

Abstract

The application provides a multimodal fusion method based on an attention mechanism and an adversarial neural network, and comprises the following steps: a modal anomaly detection and compensation algorithm is designed to solve information loss and disorder; different feature extractors are used for preliminary feature extraction according to different data source types to obtain initial semantic information of different data; shared and private information extraction is performed; according to the complementary and redundant characteristics of different modes, a multimodal adversarial neural network algorithm is designed to extract complementary information and redundant information in each mode from the preliminary features, remove the redundant information and reduce the calculation cost of subsequent algorithms; and a multimodal attention network is designed according to the characteristics that the importance of different modes is different for different tasks, and the weight of different modes is learned by adding expert experience knowledge. The application can extract and process the redundant information and complementary information in the different modal data without stopping.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning and intelligent manufacturing, and relates to a multi-modal fusion method based on an attention mechanism and an adversarial neural network. TECHNICAL BACKGROUND

[0002] At present, human society is in the era of big data, and a large amount of data has been collected in various industries, but the processing speed and capacity are far from keeping up, resulting in a large amount of storage space and data waste. At the same time, with the improvement of hardware manufacturing level, more and more different types of sensors appear, and the types of data that can be collected become more and more diversified. These data obtained through a certain type of sensor, such as images obtained by a camera, time series data obtained by a force torque sensor and a voltage sensor, are all considered as a modality. The advantage of using multi-modal data for learning is that it is closer to the operation mode of human brain and there is information complementation between modalities. At present, multi-modal fusion methods include matrix-based methods, ordinary neural network-based methods, generative model-based methods and attention mechanism-based methods. The matrix-based method has the disadvantage of parameter explosion when the extracted feature dimension is too large, and too many parameters will cause insufficient computer computing power and make it impossible to operate. When deep learning appeared, it solved the problem of parameter explosion in the previous matrix-based method to some extent. The ordinary neural network-based method has a lot of redundant information, and the model is too general, which leads to general performance for specific tasks. The generative model-based method is mainly based on adversarial neural network, and the disadvantage of adversarial neural network is that it is affected by the training stability and the number of input modalities.

[0003] The existing literature uses the GAN method to extract shared information (Adversarial Multimodal Representation Learning for Click-Through Rate Prediction Xiang Li, Chao Wang, Jiwei Tan, Xiaoyi Zeng, Dan Ou, Bo Zheng Alibaba Group, Hangzhou & Beijing, China), but it does not consider the instability of GAN network training, nor does it consider the extraction of private information, which may contain shared information, and does not give a solution. This method makes improvements in these two aspects, one of which is to use the WGAN-based method to extract redundant information, and the other is to design a constraint function to make the extracted private information contain shared information to the greatest extent. SUMMARY

[0004] The application provides a multi-modal fusion method based on an attention mechanism and an adversarial neural network.

[0005] The application is implemented by at least one of the following technical solutions.

[0006] A multi-modal fusion method based on an attention mechanism and an adversarial neural network comprises the following steps:

[0007] S1, using data collected by a camera and a sensor as a data set, inputting original data in the data set into a modal anomaly detection module, using a reconstruction error for anomaly detection, training an automatic encoder for each modal by using normal data, and performing anomaly detection by using a difference between data generated by the automatic encoder and the original data; when the original data is abnormal, compensating abnormal data by using a compensation module;

[0008] S2, inputting data in step S1 into different feature extraction networks to preliminarily extract features, wherein different data types are selected to use different feature extraction methods;

[0009] S3, inputting different preliminarily extracted features output in step S2 into a multi-modal adversarial neural network to extract shared information and private information;

[0010] S4, learning a weight of each modal by using the private information extracted in step S3 and the preliminarily extracted features, and obtaining final multi-modal fusion features.

[0011] Further, the automatic encoder comprises three hidden layers using a sigmoid activation function, an input layer, and an output layer using a sigmoid activation function, the number of nodes of the input and output layers is set to be consistent, the number of nodes of the hidden layers is adjusted according to different modes, and an L2 norm is used for a loss function.

[0012]

[0013] wherein x i is input modal data, is modal data reconstructed by a neural network; if the value of Loss exceeds a set threshold value, it is considered that the input modal is abnormal, and the abnormal modal is removed from the data set.

[0014] Further, the compensation module adopts a generation method based on a conditional adversarial generation network, uses the temporal correlation and redundancy between different modalities, processes the removed modalities using the remaining normal modal data that has not been removed, flattens the remaining normal modal data as a label y to guide the generator to generate removed modal data that matches the current remaining modal time, and inputs the removed abnormal modal data and the label into the generator to generate normal removed data using the abnormal data and the data of the remaining normal modal. The loss function optimized in the training stage of the compensation module is as follows,

[0015]

[0016] wherein z is the removed abnormal modal data, x is the normal data of the modal that needs to be compensated, D is a discriminator neural network in the adversarial neural network, G is a generator neural network in the adversarial neural network, and V(D, G) is a loss function value, is the mean value of all x satisfying the data distribution, and p data (x) is the data distribution, and D(x|y) is an output value of the discriminator neural network input by corresponding x under the condition of y, i.e., the label y, is the mean value of all z satisfying the z distribution, D(G(z|y)) is a value generated by inputting the value generated by inputting corresponding z under the condition of y into the generator neural network into the discriminator, and G(z|y) is the value generated by inputting corresponding z under the condition of y into the generator neural network; the label y is obtained by splicing the remaining normal modal after removing the removed modal.

[0017] Further, the compensation module includes a discriminator and a generator. In the training stage, an abnormality is added to the normal modal, and then the normal modal is spliced as a condition y. The abnormal modal and the condition y are input into the generator to generate a modal. The discriminator is used to determine whether the generated modal is normal or generated. Through the confrontation between the discriminator and the generator, the generated modal is infinitely close to the normal modal.

[0018] Further, for image data types, a time convolutional neural network (TCN) is selected to extract features.

[0019] Further, for one-dimensional sensor sequences, a long short-term memory (LSTM) neural network in a recurrent neural network is selected to extract features. Normal parameters are spliced into a one-dimensional vector, and a three-layer neural network is used for preliminary feature extraction.

[0020] Further, an autoencoder (AE) is used to extract preliminary features. The output of the middle layer of the autoencoder is selected as the preliminary extracted features of different sensors.

[0021] Further, the multi-modal adversarial neural network comprises a shared information extraction generator, a discriminator, a private information extraction autoencoder corresponding to the modal, the shared information extraction generator and the discriminator are trained in turn, finally the generator is taken as a projection matrix to project different modal distributions onto the same common distribution, so as to extract the shared information of different modal preliminary features, that is, the redundant information, and the shared information is arithmetically averaged to obtain a shared information c ave , and reduce the redundant information

[0022] The shared information is extracted first and then the private information is extracted, and the private information is extracted based on the principle of conditional probability through the following loss function:

[0023]

[0024] Wherein, represents the neural network of the modal m, c m represents the shared information of the modal m, Loss pure represents the private information purity loss function, x m represents the input of the modal m, p(x m , c m ) represents the joint distribution of x m and c m , represents x m obtained under the condition that c m is known.

[0025] Further, the step S3 comprises:

[0026] The different sensor preliminary extraction features output in the step S2 are input into the multi-modal adversarial neural network to extract the redundant information;

[0027] The multi-modal adversarial neural network is improved based on the Wasserstein distance adversarial neural network WGAN to form n-WGAN, and the n-WGAN uses the Wasserstein distance to measure the distance between two distributions, and the measurement formula is as follows

[0028]

[0029] Wherein, P r , P g represent the distribution r and the distribution g, W(P r , P g ) represents the Wasserstein distance of the distribution r and g, represents the average value of all x in the distribution r, represents the average value of all x in the distribution g, w represents all parameters in the neural network, f w (x) represents that the input x passes through the neural network f wThe output value after that, x represents the value that satisfies the distribution P. i The input, w:|f w | L ≤K represents the function f w The Lipschitz constant does not exceed the threshold K, where L represents the Lipschitz constant;

[0030] First use neural network f w (x) Perform a nonlinear transformation on n distributions to obtain a new distribution. Then minimize the sum of the Wasserstein distances between each pair of the transformed new distributions so that the transformed distributions overlap. When this goal is achieved, the RGB camera data, depth camera data, and pressure sensor sequence after neural network transformation satisfy the same distribution, which is considered to be the distribution of redundant information. According to the optimization goal of n-WGAN, the loss function of the multimodal adversarial neural network generator is Equation (4), and the loss function of the discriminator is Equation (5):

[0031]

[0032]

[0033] Loss n-gen The loss function of the n-WGAN method generator is represented by the loss function. n-dis This represents the loss function of the discriminator in the n-WGAN method, where n represents the number of input modes and k represents the distribution. Represents the distribution P i f is the average of all x, where x represents the input following distribution i. w (x) represents the input x passing through the neural network f. w The output value after that;

[0034] Further, step S4 includes: extracting private information s m With shared information c ave The fusion process is performed, preserving the multimodal features after fusion; the private information s extracted in step S3 is then utilized. m Learning the weights of each modality to reduce spatial dimensionality involves the following steps:

[0035] First, using the private features s extracted in step S3 m The attention distribution 'a' in the learning attention mechanism is as follows:

[0036]

[0037] in This represents the attention distribution value of the i-th sample in the m-th modality. W represents the private feature of the m-th modality of the i-th sample.m and b m are learnable parameters, formula (7) is deformed, and formula (8) is obtained:

[0038] b m = U m q (8)

[0039] wherein U m is a learnable parameter, and q is a query vector;

[0040] the private information s m obtained in step S3 is weighted and summed using the learned attention distribution a

[0041]

[0042] wherein M represents the number of modalities, and represents the inner product of two vectors;

[0043] Finally, the weighted and fused private information r i f is spliced with the arithmetic mean shared information to obtain the final multi-modal fusion feature r i fuse :

[0044] r i fuse = [r i f , c ave ] (10)

[0045] wherein c ave represents the mean of the extracted shared information.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] The n-WGAN method proposed in the present application removes the redundancy between different multi-source data to the greatest extent while ensuring the stability of neural network training, and proposes a compensation method to solve the compensation of sensor abnormalities in unknown situations. The method can automatically process any kind of information loss and disorder caused by data loss, data error and data inconsistency in an industrial intelligent industrial system without stopping the system. The method uses an adversarial neural network that can handle more than two modalities and solves the instability of training. The method uses multi-modal data for fusion to obtain more environmental information to guide the operation of the system, so that the system has better performance. At the same time, the method realizes the extraction of multi-modal redundant and complementary information, ensures the integrity of the information, obtains multi-modal fusion information with as low redundancy as possible, reduces the search state space of subsequent calculation, greatly reduces the calculation cost and the function can better converge to the optimal result. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 Figure is an embodiment of a multi-modal fusion method based on an attention mechanism and an adversarial neural network;

[0049] Figure 2 Figure is an embodiment of a multi-modal fusion method based on an attention mechanism and an adversarial neural network;

[0050] Figure 3 Figure is an embodiment of a modal anomaly detection and compensation method;

[0051] Figure 4 Figure is an embodiment of a modal compensation generator training process;

[0052] Figure 5 Figure is an embodiment of a redundant information removal schematic diagram;

[0053] Figure 6 Figure is an embodiment of an n-WGAN generator loss function change schematic diagram;

[0054] Figure 7 Figure is an embodiment of an n-WGAN discriminator loss function change schematic diagram;

[0055] Figure 8 Figure is an embodiment of a task training phase performance schematic diagram. DETAILED DESCRIPTION

[0056] The application will be further described in detail below in conjunction with embodiments, but the embodiments of the application are not limited to this embodiment.

[0057] The application comprises the following steps:

[0058] The multi-modal fusion method based on an attention mechanism and an adversarial neural network can automatically process any kind of information loss and disorder caused by data loss, data error, and data inconsistency in an industrial intelligent device system without stopping, maximally extracts redundant information and complementary information in different modal data after processing, realizes multi-modal fusion, reduces information common redundancy, obtains more environmental information, and improves system performance and efficiency.

[0059] As shown in Figure Figure 1 A multi-modal fusion method based on an attention mechanism and an adversarial neural network mainly comprises the following steps:

[0060] S1, original data processing. The original data mainly includes data error, data loss, and inaccurate data in an industrial intelligent device system, and a modal anomaly detection and compensation algorithm is designed to process the original data of information loss and disorder problems.

[0061] Assuming that the current industrial intelligent system has an RGB camera, an infrared camera, a pressure sensor, etc., the data collected by the camera and the sensor are taken as a data set, and the original data of the data set is input into the modal anomaly detection module. The modal anomaly detection module includes two parts: anomaly detection and anomaly data compensation. The reconstruction error is used for anomaly detection, and an autoencoder is trained for each modality (in this embodiment, the modalities include RGB images, depth images, and pressure sensor sequences) using normal data. The modal anomaly data processing module is divided into two parts: anomaly detection and compensation. The original data of the current industrial intelligent system is input into the modal anomaly detection module. When it is determined that the modalities are abnormal, the compensation module is entered, otherwise the data is directly input to the next step. The modality refers to the data of different sensors, and the data of each sensor is referred to as a modality.

[0062] The autoencoder is set to use a sigmoid activation function for three hidden layers, an input layer, and an output layer using a sigmoid activation function. The number of nodes in the input and output layers is set to be consistent, and the number of nodes in the hidden layer is adjusted according to different modalities. The loss function uses L2 norm,

[0063]

[0064] where x i is the input modal data, is the reconstructed modal data by the autoencoder neural network. During training, the loss is minimized to make the reconstructed modal data as similar to x i as possible, so that the neural network learns the distribution of the original modal data. When the data is incorrect or inaccurate, the value of the loss will exceed the set threshold value, at which point it is considered that the input modal has occurred. The abnormal modalities are removed from the data set, but not discarded. The threshold value is selected by taking the optimal AUROC area (area under the receiver operating characteristic curve).

[0065] When the original data is incorrect, inaccurate or missing, the removed modal data is compensated by the compensation module. The modal compensation adopts a generation method based on conditional generative adversarial network, which uses the temporal correlation and redundancy between different modalities, and uses the remaining normal modal data (which ensures that at most one modality is abnormal) to process the removed modal data: The remaining normal modal data is flattened and used as a label y to guide the generator to generate removed modal data that matches the current remaining modal time. The abnormal modal data is input into the generator together with the label, because the abnormal data still has normal information, and then the normal removed data is generated using the abnormal data and the remaining normal modal data.

[0066] The compensation module is composed of input data: label, abnormal modal data, discriminator and generator. The loss function optimized in the training stage of the compensation module is as follows,

[0067]

[0068] Wherein, z is the removed abnormal modal data, x is the normal data of the modal that needs to be compensated, D is the discriminator neural network in the original adversarial neural network GAN, G is the generator neural network in the original adversarial neural network GAN, V(D, G) is the loss function value, is the mean value of all x satisfying the data distribution, p data (x) is the data distribution, D(x|y) is the output value of the corresponding x input to the discriminator neural network under the condition of y, is the mean value of all z satisfying the z distribution, D(G(z|y)) is the value generated by inputting the value generated by inputting the corresponding z into the generator neural network under the condition of y into the discriminator, G(z|y) is the value generated by inputting the corresponding z into the generator neural network under the condition of y, y is the label, which is spliced from the remaining normal modal after removing the removed modal.

[0069] As Figure 4 shown, the training stage artificially adds an abnormality to the normal modal 1, and then splices the normal modal as our condition y, such as modal 2 to n in the figure, and then inputs the abnormal modal and the condition y (the condition y is the label y) into the generator, so that the generator generates a modal, and then uses the discriminator to judge whether the generated modal is normal or generated. Through the confrontation between the discriminator and the generator, the generated modal is infinitely close to the normal modal.

[0070] As an embodiment, assuming that the RGB camera data is found to be abnormal, the RGB data is removed from the data set, and the compensation module is used to compensate the RGB camera data. The modal compensation adopts a generation method based on conditional adversarial generation network, utilizes the characteristics that information exists between different modalities at the same time, and uses the remaining modal normal data (which ensures that at most only one modal is abnormal) to process the removed modal. The flattened sequence of the remaining modal normal data is used as the label y, as shown in formula (2), to guide the generator to generate the missing modal matching the current remaining modal:

[0071]

[0072] Wherein, respectively, are infrared camera data and pressure sensor data, and i represents the i-th sample. The abnormal RGB camera data is input into the generator together with the label y, and the new normal RGB camera data corresponding to other modalities at the same time is generated based on the abnormal modal data.

[0073] S2, modal information preliminary extraction: input the data processed in step S1 into different feature extraction networks, and select different feature extraction methods according to different data types.

[0074] As a preferred embodiment, the auto-encoder architecture includes two parts of an encoder and a decoder, and the encoder and the decoder are opposite structures. The encoder can be any kind of neural network. For image data type, if it is unfolded into a one-dimensional vector, the dimension of the obtained vector will be too high, and a slight increase in the number of neural network layers will cause too many parameters in the network, and the spatial information in the image will also be lost. The convolutional neural network can extract "particular information" (information related to the convolution kernel) from the original information by convolution. Since the RGB and infrared sensors are picture sequences, there is a time sequence between the pictures, so a time convolutional neural network (TCN) is used, which can extract information between picture sequences. The sequence of pressure sensors, voltage sensors, etc. is a one-dimensional vector, so the long short-term memory neural network (LSTM) in the recurrent neural network can be directly selected for extraction.

[0075] The present application uses a reconstruction network architecture auto-encoder (AutoEncoder) for preliminary feature extraction, that is, the output of the neural network should be as consistent as possible with the input. The purpose of learning is to reduce their distance, and through this purpose "supervision", the neural network extracts excellent abstract low-dimensional features. When the reconstruction network architecture auto-encoder is learned, the present application selects the output of the middle layer of the reconstruction network as the preliminary extraction feature of different sensor inputs. The feature has the characteristics of low dimension and high information density, and is used as the input of the subsequent step. The reconstruction network loss function is:

[0076]

[0077] wherein X i is the original input, is the reconstructed output. When the reconstruction network is learned, the present application selects the output of the middle layer of the reconstruction network as the preliminary extraction feature of different sensor inputs. The feature has the characteristics of low dimension and high information density, and is used as the input of the subsequent step.

[0078] S3, shared and private information extraction

[0079] Because the information contained in different sensors must have the same part and different parts, in order to reduce the optimal solution search space and improve the system efficiency and performance, the present application proposes a method for minimizing the redundant information in different modalities, such as Figure 5The different sensor preliminary extraction features output in step S2 are first input to the designed multi-modal adversarial neural network to extract redundant information. The multi-modal adversarial neural network is improved based on the Wasserstein distance adversarial neural network (WGAN) to form an n-WGAN. The n-WGAN uses the Wasserstein distance to replace the JS divergence and the KL divergence to measure the distance between two distributions, and the measurement formula is as follows,

[0080]

[0081] where P r , P g represent the distributions r and g, W(P r , P g ) represents the Wasserstein distance of the distributions r and g, represents the average of all x in the distribution r, represents the average of all x in the distribution g, w represents all parameters in the generator neural network, f w represents the generator neural network, x represents the input satisfying the distribution P i , w: |f w | L ≤ K represents that the Lipschitz constant of the function f w is not more than the threshold K, and L represents the Lipschitz constant.

[0082] The original WGAN can only measure two distributions, and let one of the distributions be as close to the other distribution as possible. When the Wasserstein distance is 0, it means that the two distributions coincide. On this basis, the new n-WGAN designed in the present application can extract the coincident part of the sample distribution of the n distributions (i.e. the RGB camera data, the depth camera data and the pressure sensor sequence) through the Wasserstein distance. The present application first uses a convolutional neural network f w (x) to perform a nonlinear transformation on the n distributions to obtain new distributions, and then minimizes the sum of the Wasserstein distances between the transformed new distributions two by two, so that the transformed distributions coincide. When this goal is reached, the samples transformed by the convolutional neural network satisfy the same distribution, and this distribution is considered to be the distribution of redundant information. According to the optimization goal of the above n-WGAN, the loss function of the neural network generator trained is formula (5), and the loss function of the discriminator is formula (6):

[0083]

[0084]

[0085] where Loss n－gen represents the loss function of the generator of the n-WGAN method, and Loss n-disThis represents the loss function of the discriminator in the n-WGAN method, where n represents the number of input modes and k represents the distribution. Represents the distribution P i f is the average of all x, where x represents the input following distribution i. w (x) represents the input x processed by the convolutional neural network f. w The output value after that;

[0086] Multimodal adversarial neural networks (MANs) consist of a shared information extraction generator, a discriminator, and a private information extraction autoencoder. The shared information extraction generator and discriminator are trained alternately, and the generator is ultimately used as a projection matrix to project different modal distributions onto a common distribution, thereby extracting shared information, i.e., redundant information, from the initial features of different modalities. During the training of the projection matrix, the loss function value changes, such as... Figure 6 , Figure 7 As shown, the training phase exhibits good stability. Since it involves shared information, an arithmetic average is performed on the information to obtain a set of shared information c. ave To reduce redundant information, this invention designs a private information extraction autoencoder corresponding to the number of modalities to extract private information from different modalities. To avoid low purity of the extracted private information and overlap with shared information, this invention first extracts shared information, and then, based on the principle of conditional probability, designs the following loss function to extract private information:

[0087]

[0088] Loss pure Let x represent the private information purity loss function. m p(x) represents the input of mode m. m c m ) represents x m and c m The joint distribution Indicates that when c is known m x obtained under the conditions m , c represents a three-layer fully connected neural network with mode m containing θ (θ represents the parameters in the neural network). m This represents the shared information of mode m. This invention trains the highest purity private information, independent of the shared information, by minimizing the design loss function, i.e., minimizing the mutual information between shared and private information. Finally, after extracting the private information, this invention verifies whether the shared information has been completely extracted by performing correlation analysis on the private information of different modes.

[0089] S4. Utilize the private information s extracted in step S3 mand the extracted preliminary feature learning the weight of each modality, to obtain the final multi-modal fusion feature.

[0090] When step S3 obtains private information s m and shared information c ave After that, in order to further compress the search space and reduce the dimension of the environmental state space vector, the extracted private information s m and shared information c ave are fused to retain a fused multi-modal feature. Since the importance of different sensors for different tasks is different, the attention mechanism can automatically select the features with higher importance from the input and reduce the weight of the data with low relevance or noise to the minimum. Therefore, the application utilizes the characteristics of the attention network, uses the private information s m extracted in step S3 and the preliminary feature extracted by the backbone network to learn the weight of each modality, thereby reducing the spatial dimension.

[0091] First, the private feature s m learning the attention distribution a in the attention mechanism, as follows:

[0092]

[0093] wherein represents the attention distribution value of the mth modality of the ith sample, represents the private feature of the mth modality of the ith sample, W, b and U are learnable parameters, and q is a query vector, which is initialized as a vector of all ones and updated to

[0094] Then, the private information s m obtained in step S3 is weighted and summed using the learned attention distribution a,

[0095]

[0096] wherein M represents the number of modalities, and represents the inner product of two vectors; finally, the weighted and fused private information r i f is spliced with the arithmetic average shared information to obtain the final multi-modal fusion feature r i fuse :

[0097] r i fuse = [r i f , c ave ] (10) wherein c aveThe mean value of the extracted shared information is represented. i fuse It has the characteristics of low dimension, high information density and less redundant information, and is input into a policy network as an environment vector in subsequent reinforcement learning to guide the monitoring device to make a response.

[0098] The present application is verified on the Gentle Push dataset, which collects planar pushing data, simulates a 7-DOF Panda Franka robot pushing a circular ice puck with an end effector, and sets the experimental task as estimating the two-dimensional position of an unknown object on a table. The task evaluation index is the Euclidean distance difference between the predicted position and the actual position. In the training stage, the distance between the predicted position and the actual position of the present application is changed as Figure 8 After training, the difference between the predicted position and the actual position in the test set is 1.18123.

[0099] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The preferred embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and utilize the present application. The present application is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-modal fusion method based on an attention mechanism and an adversarial neural network, characterized in that, Comprise the following steps: S1, using the data collected by the camera, sensor as a data set, input the original data in the data set to the modal anomaly detection module, the detection module uses the reconstruction error for anomaly detection, trains an automatic encoder for each modality using normal data, and performs anomaly detection using the difference between the data generated by the automatic encoder and the original data; when the original data is abnormal, use the compensation module to compensate the abnormal data; S2, input the data of step S1 to different feature extraction networks to preliminarily extract features, wherein different data types select different feature extraction methods; S3, input the different preliminary extracted features output in step S2 into a multi-modal adversarial neural network to extract shared information and private information, comprising: Input the different sensor preliminary extracted features output in step S2 into a multi-modal adversarial neural network to extract redundant information; The multi-modal adversarial neural network is improved based on the Wasserstein distance adversarial neural network WGAN to form n-WGAN, and the n-WGAN uses the Wasserstein distance to measure the distance between two distributions, and the measurement formula is as follows where P r , P g denote the distributions r and g, W(P r , P g ) denotes the Wasserstein distance of the distributions r and g, denotes the mean of all x in the distribution r, denotes the mean of all x in the distribution g, w denotes all parameters in the neural network, f w (x) denotes the output value of the input x after passing through the neural network f w , x denotes the input satisfying the distribution P i , w: |f w | L ≤ k denotes that the Lipschitz constant of the function f w is not more than the threshold K, L denotes the Lipschitz constant; First, the neural network f w (x) Nonlinear transformation is performed on n distributions to obtain new distributions, and then the sum of the Wasserstein distances between the transformed new distributions is minimized, and the transformed distributions are coincided. When this goal is reached, the RGB camera data, depth camera data and pressure sensor sequence after the neural network transformation satisfy the same distribution, which is considered as the distribution of redundant information. According to the optimization goal of n-WGAN, the loss function of the multi-modal adversarial neural network generator is formula (4), and the loss function of the discriminator is formula (5): where Loss n-gen represents the loss function of the n-WGAN method generator, Loss n-dis represents the loss function of the n-WGAN method discriminator, n represents the number of input modalities, k represents the distribution, represents the average of all x of the distribution P k , x represents an input satisfying the distribution i, f w (x) represents the output value of the input x after passing through the neural network f w . S4, use the private information extracted in step S3 and the extracted preliminary features to learn the weight of each modality to obtain the final multi-modal fusion features.

2. The multi-modal fusion method based on attention mechanism and adversarial neural network according to claim 1, characterized in that, The automatic encoder includes three hidden layers using sigmoid activation function, input layer, and output layer using sigmoid activation function, the number of nodes of the input and output layers is set to be consistent, the number of nodes of the hidden layer is adjusted according to different modalities, and the loss function uses L2 norm, wherein x i is the input modal data, is the reconstructed modal data by the neural network; if the value of Loss exceeds the set threshold, it is considered that the input modal is abnormal, and the abnormal modal is removed from the data set.

3. The multi-modal fusion method based on attention mechanism and adversarial neural network according to claim 2, characterized in that, The compensation module adopts a generation method based on conditional adversarial generative network, uses the temporal correlation and redundancy between different modalities, uses the remaining normal modality data that has not been removed to process the removed modality, flattens the remaining normal modality data as a label y to guide the generator to generate removed modality data that matches the current remaining modality time, and inputs the removed abnormal modality data and the label into the generator, generates normal removed data using the abnormal data and the data of the remaining normal modality, and the loss function optimized in the training stage of the compensation module is as follows: wherein z is the removed abnormal modality data, x is the normal data of the modality that needs to be compensated, D is a discriminator neural network in the adversarial neural network, G is a generator neural network in the adversarial neural network, V(D, G) is a loss function value, is the mean value of all x satisfying the data distribution, p data (x) is the data distribution, D(x|y) is an output value of the discriminator neural network when the corresponding x is input to the discriminator neural network under the condition of y, i.e., the label y, is the mean value of all z satisfying the z distribution, D(G(z|y)) is a value generated by the discriminator when the value generated by the generator neural network after the corresponding z is input to the generator neural network under the condition of y is input to the discriminator, G(z|y) is the value generated by the generator neural network after the corresponding z is input to the generator neural network under the condition of y; the label y is spliced from the remaining normal modalities after the removal of the removed modality.

4. The multi-modal fusion method based on attention mechanism and adversarial neural network according to claim 2, characterized in that, The compensation module includes a discriminator and a generator, and in the training stage, the normal modality is added with an anomaly, then the normal modality is spliced as a condition y, the abnormal modality and the condition y are input into the generator, the generator generates a modality, and then the discriminator is used to judge whether the generated modality is normal or generated, through the confrontation between the discriminator and the generator, the generated modality is infinitely close to the normal modality. 5.The method of claim 1, wherein, For image data type, time convolutional neural network TCN is selected to extract features. 6.The method of claim 1, wherein, For one-dimensional sensor sequence, long short-term memory neural network LSTM in recurrent neural network is selected to extract features, and ordinary parameters are spliced into one-dimensional vector, and then three-layer neural network is used for preliminary feature extraction.

7. The multi-modal fusion method based on attention mechanism and adversarial neural network according to claim 1, characterized in that, Use the reconstruction network architecture automatic encoder AutoEncoder to preliminarily extract features, and select the output of the middle layer of the reconstruction network architecture automatic encoder as the preliminary extracted features of different sensors. 8.The method of claim 1, wherein, The multi-modal adversarial neural network comprises a shared information extraction generator, a discriminator, a private information extraction auto-encoder corresponding to a mode, the shared information extraction generator and the discriminator are alternately trained, finally the generator is taken as a projection matrix, different modal distributions are projected onto the same common distribution, and the shared information, i.e. redundant information, of different modal preliminary features is extracted, the shared information is arithmetically averaged, and one copy of the shared information c ave is obtained, and the redundant information is reduced. First extract shared information and then extract private information, based on the principle of conditional probability, through the following loss function to extract private information: wherein, represents a neural network for modality m, c m represents shared information for modality m, Loss pure represents a private information purity loss function, x m represents an input for modality m, p(x m , c m ) represents a joint distribution of x m and c m , represents x m taken given c m .

9. The multi-modal fusion method based on attention mechanism and adversarial neural network according to any one of claims 1-8, characterized in that, The step S4 includes extracting the private information s m with the shared information c ave fusing and retaining the fused multi-modal feature; using the private information s extracted in step S3 m learning the weight of each modality to reduce the spatial dimension, including the following steps: First, the private feature s extracted by step S3 is used m The attention distribution a in the learning attention mechanism is learned as follows: wherein denotes the attention distribution value of the m-th modality of the i-th sample, denotes the private feature of the m-th modality of the i-th sample, W m and b m are learnable parameters, and the equation (7) is transformed as follows: b m = U m q (8) where U m is a learnable parameter, q is a query vector; Using the learned attention distribution a on the private information s resulting from step S3 m Perform a weighted sum: where M represents the number of modes, and represents the inner product of two vectors. Finally, the weighted fused private information is spliced with the arithmetic average shared information to obtain the final multi-modal fusion feature where c ave represents the mean of the extracted shared information.

Images