Data repair method, device, equipment and medium for multimodal data

Abstract

The present invention discloses a data repair method, device, equipment and medium for multimodal data, including: extracting data features in multimodal data to obtain an initial feature vector; performing vector alignment processing on the initial feature vector to obtain a target feature vector; using a dynamically weighted multi-head self-attention module to identify the target feature vector and determine the defective data position; splicing the target feature vector to obtain a feature matrix; using a generative adversarial network to locate and repair the feature matrix based on the defective data position to obtain repaired data. The present invention can improve the accuracy of data repair.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data repair method, device, equipment and medium for multimodal data. Background Art

[0002] In the data age where artificial intelligence is developing rapidly, the development of multimodal data has become a major trend now and in the future. By integrating data from different modalities such as images, audio, and text, and conducting collaborative training on these data, artificial intelligence can understand and predict more accurately. However, in practical problems, multimodal data often has various problems, such as data confusion, incompleteness, and incompleteness, which will affect the accuracy and speed of the model. If missing values ​​appear when genomic information is integrated into the electronic medical record system, errors will occur in the sequencing data of the sequencing company, resulting in some patients not having corresponding clinical information. For example, in multimodal emotion recognition, different types of data detection instruments and equipment have different standards, which will lead to heterogeneity problems within the data, resulting in the machine recognizing the wrong emotion during human-computer interaction. For example, in the application of multimodal dialogue intelligent voice customer service, there are missing values ​​in the multimodal data. As the dialogue progresses, the answers of the intelligent voice customer service will not match the questions. When collecting multimodal data in biometric technology, problems with the data will lead to errors in recognition, and then safety accidents and even various crises will occur. For example, when face recognition is used to unlock a mobile phone, if the face image of another person is mistakenly recognized, there will be a risk of various data on the mobile phone being exposed. If the fingerprint lock is mistakenly recognized, it will cause property loss or even personal safety. If the iris is mistakenly recognized in the confidential document room, there will be a risk of confidentiality leakage.

[0003] In the process of realizing the present invention, the inventors found that the data types are diverse, such as audio, text, images, etc., and the existing multimodal data repair methods are mostly based on kernel matrix and non-negative matrix decomposition. Most of these methods are only to learn a common clustering structure, and do not make full use of the existing data information to infer the missing data. If there is a problem with the multimodal data, it will cause the model to make wrong predictions, and the consequences will be disastrous. For example, the Chinese patent application with publication number CN109670580A discloses a data repair method based on time series and the Chinese patent application with publication number CN110597799A discloses a method, system and device for automatically filling missing values ​​of time series data. Its industrial sensor data is usually unable to accurately detect equally spaced time series data due to problems such as sensing units and signal transmission, and usually fluctuates around the preset detection time. Therefore, it is impossible to use the time series data model to predict non-equally spaced continuous data with high monitoring frequency. For example, Chinese patent application with publication number CN115019510A discloses a traffic data repair method based on a dynamic adaptive generative adversarial network, and Chinese patent application with publication number CN112185104A discloses a traffic big data repair method based on an adversarial autoencoder. The original data matrix is ​​processed by data normalization, the location of abnormal data is determined, and alignment is performed for preprocessing. However, the training of historical data takes a lot of time and the data cannot be effectively classified.

[0004] In summary, traditional methods have problems of insufficient accuracy or low efficiency and cannot meet the needs of today's artificial intelligence. Summary of the invention

[0005] Embodiments of the present invention provide a data repair method, apparatus, computer equipment and storage medium for multimodal data to improve data repair accuracy.

[0006] In order to solve the above technical problems, the present application provides a data repair method for multimodal data, including:

[0007] Extract data features from multimodal data and obtain initial feature vectors;

[0008] Performing vector alignment processing on the initial feature vector to obtain a target feature vector;

[0009] Using a dynamically weighted multi-head self-attention module to identify the target feature vector and determine the defect data location;

[0010] Concatenating the target feature vectors to obtain a feature matrix;

[0011] A generative adversarial network is used to locate and repair the feature matrix based on the position of the defect data to obtain repair data.

[0012] Optionally, performing vector alignment processing on the initial feature vector to obtain a target feature vector includes:

[0013] Traverse the initial eigenvector and calculate the maximum value of the lateral dimension of the eigenvectors of all modes in the initial eigenvector , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to perform left and right padding processing until the horizontal dimensions reach the maximum value until;

[0014] Traverse the initial eigenvector and calculate the maximum value of the longitudinal dimension of the eigenvectors of all modes in the initial eigenvector , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to perform up and down filling processing until the vertical dimensions reach the maximum value Until now, the target feature vector is obtained.

[0015] Optionally, the adopting a dynamically weighted multi-head self-attention module to identify the target feature vector and determine the defect data location includes:

[0016] Add position encoding to the initial multi-head self-attention module, where the position encoding is a vector, each vector corresponds to a position in the input feature;

[0017] Adding the position code to the multimodal data feature vector and concatenating it with the target feature vector corresponding to the position code to obtain an enhanced feature vector;

[0018] A multi-head self-attention module is used to capture the relationship between the features of each modality in the enhanced feature vector, and the attention weight is calculated to obtain the weight matrix of the feature importance between each modality;

[0019] Missing data locations are determined based on the weight matrix.

[0020] Optionally, the step of concatenating the target feature vectors to obtain a feature matrix includes:

[0021] Using the weight matrix, weighting the target feature vector to obtain a weighted feature vector;

[0022] The weighted feature vectors are concatenated to obtain a multi-module feature matrix as the feature matrix.

[0023] Optionally, the multi-head self-attention module is used to capture the relationship between the features of each modality in the enhanced feature vector, and the attention weight is calculated to obtain a weight matrix of the feature importance between each modality, including:

[0024] The importance coefficient of data features between modes is calculated using the following formula:

[0025] ;

[0026] Among them, RELU is the attention function and K is the feature With features The number of attention heads, k is the attention number, is the complete modal data feature Modal data features with missing features The correlation coefficient of ;

[0027] The weight matrix is ​​generated based on the importance coefficients.

[0028] In order to solve the above technical problems, the embodiment of the present application further provides a data repair device for multimodal data, comprising:

[0029] A feature extraction module is used to extract data features from multimodal data and obtain an initial feature vector;

[0030] A vector alignment module is used to perform vector alignment processing on the initial feature vector to obtain a target feature vector;

[0031] A defect location module, used to identify the target feature vector using a dynamically weighted multi-head self-attention module to determine the defect data location;

[0032] A feature concatenation module is used to concatenate the target feature vectors to obtain a feature matrix;

[0033] The data repair module is used to use a generative adversarial network to locate and repair the feature matrix based on the position of the defective data to obtain repair data.

[0034] Optionally, the vector alignment module includes:

[0035] The first filling unit is used to traverse the initial eigenvector and calculate the maximum value of the lateral dimension of the eigenvectors of all modes in the initial eigenvector. , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to perform left and right padding processing until the horizontal dimensions reach the maximum value until;

[0036] The second filling unit is used to traverse the initial feature vector and calculate the maximum value of the longitudinal dimension of the feature vectors of all modes in the initial feature vector. , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to perform up and down filling processing until the vertical dimensions reach the maximum value Until now, the target feature vector is obtained.

[0037] Optionally, the defect location module includes:

[0038] An adding unit for adding position encodings to the initial multi-head self-attention module, where the position encodings are vectors, each of which corresponds to a position in the input feature;

[0039] A first concatenation unit, configured to add the position code to a multimodal data feature vector and concatenate it with a target feature vector corresponding to the position code to obtain an enhanced feature vector;

[0040] A computing unit, used to capture the relationship between the features of each modality in the enhanced feature vector using a multi-head self-attention module, and calculate the attention weight to obtain a weight matrix of the feature importance between each modality;

[0041] A positioning unit is used to determine the missing data position based on the weight matrix.

[0042] Optionally, the feature stitching module includes:

[0043] A weighting unit, used to weight the target feature vector using the weight matrix to obtain a weighted feature vector;

[0044] The second concatenation unit is used to concatenate the weighted feature vectors to obtain a multi-module feature matrix as the feature matrix.

[0045] Optionally, the computing unit includes:

[0046] The calculation subunit is used to calculate the importance coefficient of the data features between modes using the following formula:

[0047] ;

[0048] Among them, RELU is the attention function and K is the feature With features The number of attention heads, k is the attention number, is the complete modal data feature Modal data features with missing features The correlation coefficient of ;

[0049] A generating subunit is used to generate the weight matrix based on the importance coefficient.

[0050] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data repair method for the above multimodal data when executing the computer program.

[0051] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data repair method for multimodal data are implemented.

[0052] The data repair method, device, computer equipment and storage medium for multimodal data provided by the embodiments of the present invention extract data features from multimodal data to obtain an initial feature vector; perform vector alignment processing on the initial feature vector to obtain a target feature vector; use a dynamically weighted multi-head self-attention module to identify the target feature vector and determine the location of defective data; splice the target feature vector to obtain a feature matrix; use a generative adversarial network to locate and repair the feature matrix based on the location of defective data to obtain repaired data. The present application can achieve high-quality, fine-grained and accurate multimodal data repair by using multimodal data features to locate and repair missing modal features, thereby improving the accuracy of data repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0054] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0055] Figure 2 is a flow chart of an embodiment of a data repair method for multimodal data of the present application;

[0056] Figure 3 is an example diagram of a Mel-scale filter bank of the present application;

[0057] Figure 4 is a structural schematic diagram of an embodiment of a data repair device for multimodal data according to the present application;

[0058] Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0060] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] See also Figure 1 ,like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0063] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc.

[0064] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0065] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0066] It should be noted that the data repair method for multimodal data provided in the embodiment of the present application is executed by a server, and accordingly, the data repair device for multimodal data is arranged in the server.

[0067] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 101, 102, 103 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0068] See also Figure 2 , Figure 2 A data repair method for multimodal data provided by an embodiment of the present invention is shown. The method is applied in Figure 1 The server in is used as an example to illustrate the details as follows:

[0069] S201: Extract data features from multimodal data to obtain an initial feature vector.

[0070] Specifically, multimodal data includes but is not limited to text, pictures, audio, video, etc., and data feature extraction in multimodal data mainly considers the extraction of text, picture and audio features.

[0071] For audio feature extraction, first, perform short-time Fourier transform on the input audio signal x(t) and the source signal x(t):

[0072] ;

[0073] Among them, w(t) is the window function, ω is the angular frequency, and t is time. For discrete signals, the above formula can be expressed as:

[0074] ;

[0075] Where n is time, is a sequence of window functions, τ represents the offset of the window function w(t) in the time domain, j represents the imaginary unit, ω represents the angular frequency in the frequency domain, and m represents the number of frames in the short-time Fourier transform. After the above transformation, the audio signal obtains the amplitude and phase information at different frequencies, but because the human ear mainly senses the amplitude information and is not sensitive to the phase information, it is converted into a spectrogram for analysis.

[0076] According to the definition of power spectrum, the relationship between short-time power spectrum and short-time Fourier transform can be expressed as:

[0077] ;

[0078] in, Represents the complex conjugate operation. The spectrogram can be regarded as a two-dimensional signal graph obtained by stacking each frame of the short-time Fourier transform along another dimension. It contains the full frequency domain information of the audio signal, so it needs to be aligned for compression.

[0079] Convert the 'HZ' frequency scale to the Mel frequency scale. The mapping relationship between the two is:

[0080] ;

[0081] ;

[0082] Mel frequency gradually slows down as f increases, and in the Mel frequency domain, the human ear's perception of pitch becomes a linear relationship, that is, the pitch change that the human ear can feel is the same as the Mel frequency change trend of the audio signal itself. Therefore, a Mel scale filter bank is designed to reduce the dimension of the spectrogram into a Mel spectrum. A Mel scale filter bank It consists of multiple triangular filters of equal height, defined as:

[0083] ;

[0084] in, , i represents the i-th filter in the Mel-scale filter bank, is the boundary point of each filter, whose position depends on the sampling frequency and point N in the DFT, and are the upper and lower boundary frequencies of the Mel-scale filter bank, respectively. M is the total number of filters in the filter bank. For details, please refer to Figure 3 , Figure 3 FIG. 4 is a schematic diagram of a Mel-scale filter bank according to this embodiment.

[0085] Taking the logarithm of the Mel spectrum obtained above, and then performing a discrete cosine transform (DCT) again, we can get the Mel frequency cepstral coefficient (MFCC), which is expressed as:

[0086] ;

[0087] in, is the logarithmic energy of the output of the i-th filter in the Mel-scale filter bank, M and J (J≤M) are the number of filters and the number of cepstral coefficients, respectively. That is, the Mel frequency cepstral coefficient is the audio feature vector .

[0088] For text feature extraction, the input is a text sequence , Represents the jth word in the sequence, m is the length of the text sequence, i represents the i-th sample, and the text is divided into Chinese and English text. Because Chinese is composed of characters, it does not need to go through the text preprocessing method of word fragments, so the following is the text processing method for English.

[0089] First, split each word into two-byte codes and keep the split parts in the word list to make the word list more concise and the word semantics clearer. Then encode the word fragments and each word after processing into a one-hot vector according to its index in this table. Only one element is 1 and the rest are all 0. Assume that the input word , the vector representation of each word in the one-hot encoding is , different words have different unique heat vectors, which can be used to distinguish different words. Use one-dimensional wide convolution and fixed convolution kernel , the width k of W is a variable, indicating the size of the sliding window. The k word vectors falling into W are represented as matrices in turn , Perform convolution operation on the word vector in the convolution kernel to obtain the features of the current window:

[0090] ;

[0091] in is the convolution operation, and b is the bias.

[0092] The feature representation obtained by convolution of n convolution kernels is: ,in is the i-th convolution check The features obtained by convolution are then subjected to maximum pooling, which means that only the maximum feature of each dimension within the pooling window is obtained. , , where l is the maximum pooling window size, and finally the text feature vector is obtained .

[0093] For image feature extraction, input image pixels And the color feature vector set of the color image, and perform LBG clustering on the color feature vector set of the color image to generate a color information dictionary . The color vector X is encoded using the local soft assignment encoding method, so the color word The soft color label value for feature X is:

[0094] ;

[0095] Among them, the distance Defined as:

[0096] ;

[0097] Here Indicates that according to the distance The determined feature X in the set C The nearest neighbor. β is the smoothing coefficient that controls the softness of the distribution. Finally, the color label feature vector value of the color vector X is:

[0098] ;

[0099] Set the image pixel The color feature value is , P neighborhood points evenly distributed on a circle with radius R (The corresponding color values ​​are expressed as ), pixels Its neighboring points The distance between color features The calculation formula is:

[0100] ;

[0101] Secondly, select the smallest one from the P distance values and set the corresponding bit value in the binary code value to 1, and the rest bit is set to 0, i.e.

[0102] ;

[0103] Image pixels The LSC encoding value of is the decimal number corresponding to the P-bit binary code, that is

[0104] ;

[0105] And the LSC color feature vector encoding has the invariance of light intensity, that is,

[0106] ;

[0107] The joint statistics of SCL and LSC are obtained in the form of a two-dimensional histogram:

[0108] ;

[0109] in is the Dirichlet function, (m,n) is the coordinate value of any pixel in the image, is the image dimension, , is the number of color words in the codebook, is the number of local color texture patterns. The size of the two-dimensional histogram H is , that is, in H, each row corresponds to a color word , each column corresponds to an LSC code value t, so the value of each element in the two-dimensional histogram It represents the sum of the weights of the color quantization in the image within interval s and the LSC mode encoding value t.

[0110] Convert the two-dimensional histogram into a one-dimensional feature vector. First, extract the rows in the two-dimensional histogram to form a one-dimensional sub-histogram, and then splice all the sub-histograms together to obtain the final one-dimensional feature vector representing the color image. .

[0111] S202: Perform vector alignment processing on the initial feature vector to obtain a target feature vector.

[0112] In a specific optional implementation manner, in step S202, the initial feature vector is subjected to vector alignment processing to obtain a target feature vector including:

[0113] Traverse the initial eigenvector and calculate the maximum value of the horizontal dimension of the eigenvectors of all modes in the initial eigenvector , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to fill in the left and right until the horizontal dimensions reach the maximum value until;

[0114] Traverse the initial eigenvector and calculate the maximum value of the longitudinal dimension of the eigenvectors of all modes in the initial eigenvector , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to fill up and down until the vertical dimensions reach the maximum value Until then, the target feature vector is obtained.

[0115] Specifically, the multimodal data feature vectors are aligned vertically as follows: the multimodal data feature vectors obtained by traversing step S201 are Calculate the maximum value of the horizontal dimension of all eigenvectors , calculate the mean of each modality's own features, copy the mean and fill it on both sides, and fill it on the left and right sides respectively until the multimodal data feature vector The horizontal dimensions of Until now.

[0116] The horizontal dimension refers to the number of columns in the data matrix, that is, the number of features. For example, in a data set with N samples and M features, the horizontal dimension is M, which means that there are M features in the data set.

[0117] The central dimension refers to the number of rows in the data matrix, that is, the number of samples. For example, in the above dataset with N samples and M features, the central dimension is N, which means there are N samples in the dataset.

[0118] The multimodal data feature vector is aligned horizontally as follows: the multimodal data feature vector obtained by traversing step S201 Calculate the maximum value of the vertical dimension of all eigenvectors ,in is the vertical dimension of the feature vector of modal data i, calculate the mean of the vertical dimension of each modal data feature vector, copy the feature vector of the mean position and fill it on both sides, and fill it on the top and bottom respectively until the multi-modal data feature vector The vertical dimension reaches dimension. Thus, the multimodal data feature vector is realized. Alignment of the vertical dimension.

[0119] S203: Use a dynamically weighted multi-head self-attention module to identify the target feature vector and determine the defect data location.

[0120] In a specific optional implementation manner, in step S203, a dynamically weighted multi-head self-attention module is used to identify the target feature vector, and determining the defect data location includes:

[0121] Add position encoding to the initial multi-head self-attention module. The position encoding is a vector, and each vector corresponds to a position in the input feature.

[0122] The position code is added to the multimodal data feature vector and concatenated with the target feature vector corresponding to the position code to obtain an enhanced feature vector;

[0123] A multi-head self-attention module is used to capture the relationship between the features of each modality in the enhanced feature vector, and the attention weight is calculated to obtain the weight matrix of the feature importance between each modality;

[0124] Determine the missing data locations based on the weight matrix.

[0125] Specifically, since subsequent operations will cause changes in feature positions, it is necessary to generate information about the relative or absolute positions of markers in the features and add them to the multimodal data features. In the example, sine and cosine sampling values ​​of different frequencies are used as position encoding information. The position encoding is a set of vectors whose dimensions are the same as the input feature vectors and are recorded as , each vector corresponds to a position in the input feature, that is:

[0126] ;

[0127] Where d is the dimension of the input feature vector, i is the row of the feature vector, and j is the column of the feature vector.

[0128] Furthermore, the feature vector and position code of each position are concatenated to obtain the enhanced multimodal data feature vector , , .Right now:

[0129] ;

[0130] Where i=1,2,3, is the multimodal data feature vector after alignment in step 2, is the position encoding vector.

[0131] Furthermore, a multi-head self-attention module is used to calculate the attention weights for the entire sample to capture and learn the relationship between features of the modal data and locate the missing positions to better repair the missing values.

[0132] The importance of features is evaluated through the attention network, and a weight matrix between multimodal data features is generated , which is composed of the importance coefficient of the data features between modes composition, The calculation process is:

[0133] ;

[0134] Among them, RELU is the attention function and K is the feature With features The number of attention heads, k is the attention number or the attention index value, is the complete modal data feature Modal data features with missing features The correlation coefficient of The input data is the modal data feature vector obtained in step 3.2 ,During the calculation, in order to make the importance not affected by the position encoding, the ,position part of the feature vector is filled with zero value mask. is a learnable parameter matrix, which is applied to all modal data features. The attention index value k is used to generate a vector storing the attention index values ​​of all data features. , denoted as the complete feature attention index vector.

[0135] Generate a weight matrix of the importance of each modality’s own data features , which is composed of the importance coefficients between the data features of each modality composition, The calculation process is:

[0136] ;

[0137] in , is the i,jth feature of this mode.

[0138] According to the importance coefficient between multimodal data features To determine the missing data, set a threshold θ, and regard the position where the importance coefficient is lower than the threshold as the missing data position. , if other model features and The importance coefficients of When , it represents the characteristic is a missing feature, where The modal data features with missing features Data features with another modality The correlation coefficient, The modal data features with missing features With another modality data feature The correlation coefficient is, then the attention index value k of the feature vector is recorded in the missing feature attention index vector That is, there are t features missing in total, and the missing feature attention index vector and the complete feature attention index vector will be used to generate an adversarial network to generate features corresponding to the missing data feature positions of the modality with missing data features.

[0139] In a specific optional implementation, a multi-head self-attention module is used to capture the relationship between the features of each modality in the enhanced feature vector, and the attention weight is calculated to obtain a weight matrix of the feature importance between each modality, including:

[0140] The importance coefficient of data features between modes is calculated using the following formula:

[0141] ;

[0142] Among them, RELU is the attention function and K is the feature With features The number of attention heads, k is the attention number, is the complete modal data feature Modal data features with missing features The correlation coefficient of ;

[0143] Generate a weight matrix based on the importance coefficients.

[0144] S204: concatenate the target feature vectors to obtain a feature matrix.

[0145] Specifically, according to the importance coefficient between multimodal data features To determine the missing data, set a threshold θ, and regard the position where the importance coefficient is lower than the threshold as the missing data position. , if other model features and The importance coefficients of When , it represents the characteristic is a missing feature, where The modal data features with missing features Data features with another modality The correlation coefficient, The modal data features with missing features With another modality data feature The correlation coefficient is, then the attention index value k of the feature vector is recorded in the missing feature attention index vector That is, there are t features missing in total, and the missing feature attention index vector and the complete feature attention index vector will be used to generate an adversarial network to generate features corresponding to the missing data feature positions of the modality with missing data features.

[0146] In a specific optional implementation manner, in step S204, the target feature vectors are concatenated to obtain a feature matrix including:

[0147] The target feature vector is weighted by using a weight matrix to obtain a weighted feature vector;

[0148] The weighted feature vectors are concatenated to obtain a multi-module feature matrix as the feature matrix.

[0149] Specifically, in order to make full use of multimodal information and emphasize the importance of different modal features, the importance matrix of the modality obtained in step S203 is The multimodal data feature matrix is ​​weighted. The specific process is as follows:

[0150] ;

[0151] L is the concatenated feature matrix, is the j-th feature of mode i.

[0152] Furthermore, the weighted feature vector Spliced ​​into a multimodal data feature matrix Y, its dimension is .

[0153] S205: Using a generative adversarial network, the feature matrix is ​​located and repaired based on the defect data position to obtain repair data.

[0154] First, the generator is constructed. Its principle is to use the complete modal data features to restore the features of the specific positions of the missing modal data. Therefore, when the generator is trained, the input is the multimodal data feature matrix Y after the importance matrix of the modality itself is concatenated, as well as the complete feature attention index vector and the missing feature attention index vector. The multimodal data feature matrix Y and the complete feature attention index vector and the missing feature attention index vector are used to jointly generate the features of the corresponding positions of the missing modal data. The encoder and decoder are located in the generator. The encoder consists of L convolution blocks, each of which consists of a 3×3 convolution kernel with a step size of 2, a BN layer and a ReLU activation function. The decoder consists of L deconvolution layers. The deconvolution layer operates oppositely to the convolution layer, so the input dimension and output dimension of the network can be maintained consistent under specific settings. Finally, the modal features obtained through the nonlinear mapping of the encoder and decoder are the same as the original modal features. The entire generation process is described as follows:

[0155] ;

[0156] in, is the generated features of G, deconv() represents the deconvolution operation, and the multimodal data feature matrix Y and the complete feature attention index vector and the missing feature attention index vector are used to jointly generate the missing features, which can make the generated modal features more realistic and efficient. That is to say, the multimodal data feature matrix Y and the complete feature attention index vector and the missing feature attention index vector are used to repair the features at the specified position of a modality, which makes the result more realistic and reliable.

[0157] The loss function of the generator is:

[0158] ;

[0159] in and are the real samples of the first modality and the real samples of the second modality, is the real sample of the missing modality, O is the complete feature attention index vector, and M is the missing feature attention index vector.

[0160] With the pseudo features generated by the generator, the next step is to construct a discriminator to distinguish between real features and pseudo features. Through continuous training, a dynamic balance is finally achieved, so that the discriminator cannot distinguish the authenticity of the generated features, making the features generated by the generator more realistic. With real characteristics As well as the complete feature attention index vector O and the missing feature attention index vector M, and as well as , Respectively Channel connection is performed to generate one positive sample pair and three negative sample pairs respectively. These four sample pairs and the complete feature attention index vector and the missing feature attention index vector are put into the discriminator network to obtain the final discrimination result. The whole discrimination process is as follows:

[0161] ;

[0162] where y is the real feature (in Represents the true features of the vth mode, v=1,2,3) or the features generated by the generator , The result of the discriminator is The probability value of being considered a real feature.

[0163] The adversarial loss function is:

[0164] ;

[0165] The generator and the discriminator are trained adversarially until the network is balanced.

[0166] Using Generative Adversarial Networks for Positioning Repair:

[0167] According to the generator and discriminator in the above steps, the repaired features are obtained:

[0168] ;

[0169] in, is the feature obtained after the final repair. Then find the corresponding position of the missing feature according to the attention index vector for repair.

[0170] In this embodiment, data features in multimodal data are extracted to obtain an initial feature vector; vector alignment processing is performed on the initial feature vector to obtain a target feature vector; a dynamically weighted multi-head self-attention module is used to identify the target feature vector and determine the defective data location; the target feature vector is spliced ​​to obtain a feature matrix; a generative adversarial network is used to locate and repair the feature matrix based on the defective data location to obtain repaired data. By using multimodal data features to locate and repair missing modal features, high-quality, fine-grained, and accurate multimodal data repair can be achieved, thereby improving the accuracy of data repair.

[0171] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0172] Figure 4 The principle block diagram of the multimodal data data repair device corresponding to the multimodal data repair method of the above embodiment is shown. Figure 4 As shown, the data repair device for multimodal data includes a feature extraction module 31, a vector alignment module 32, a defect location module 33, a feature splicing module 34 and a data repair module 35. The functional modules are described in detail as follows:

[0173] A feature extraction module 31 is used to extract data features from multimodal data and obtain an initial feature vector;

[0174] A vector alignment module 32 is used to perform vector alignment processing on the initial feature vector to obtain a target feature vector;

[0175] The defect location module 33 is used to identify the target feature vector using a dynamically weighted multi-head self-attention module to determine the defect data location;

[0176] A feature concatenation module 34 is used to concatenate target feature vectors to obtain a feature matrix;

[0177] The data repair module 35 is used to use a generative adversarial network to locate and repair the feature matrix based on the defect data position to obtain repair data.

[0178] Optionally, the vector alignment module 32 includes:

[0179] The first filling unit is used to traverse the initial eigenvector and calculate the maximum value of the lateral dimension of the eigenvectors of all modes in the initial eigenvector. , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to fill in the left and right until the horizontal dimensions reach the maximum value until;

[0180] The second filling unit is used to traverse the initial eigenvector and calculate the maximum value of the longitudinal dimension of the eigenvectors of all modes in the initial eigenvector. , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to fill up and down until the vertical dimensions reach the maximum value Until then, the target feature vector is obtained.

[0181] Optionally, the defect location module 33 includes:

[0182] Add a unit to add position encoding to the initial multi-head self-attention module. The position encoding is a vector, and each vector corresponds to a position in the input feature.

[0183] A first concatenation unit is used to add the position code to the multimodal data feature vector and concatenate it with the target feature vector corresponding to the position code to obtain an enhanced feature vector;

[0184] A computing unit, used to capture the relationship between the features of each modality in the enhanced feature vector using a multi-head self-attention module, and calculate the attention weight to obtain a weight matrix of the feature importance between each modality;

[0185] The location unit is used to determine the missing data location based on the weight matrix.

[0186] Optionally, the feature stitching module 34 includes:

[0187] A weighting unit, used for weighting the target feature vector by using a weight matrix to obtain a weighted feature vector;

[0188] The second concatenation unit is used to concatenate the weighted feature vectors to obtain a multi-module feature matrix as a feature matrix.

[0189] Optionally, the computing unit includes:

[0190] The calculation subunit is used to calculate the importance coefficient of the data features between modes using the following formula:

[0191] ;

[0192] Among them, RELU is the attention function and K is the feature With features The number of attention heads, k is the attention number, is the complete modal data feature Modal data features with missing features The correlation coefficient of ;

[0193] A generating subunit is used to generate a weight matrix based on the importance coefficients.

[0194] The specific definition of the data repair device for multimodal data can be found in the definition of the data repair method for multimodal data above, which will not be repeated here. Each module in the above-mentioned data repair device for multimodal data can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0195] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device in this embodiment.

[0196] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0197] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0198] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code for data repair of multimodal data, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0199] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code for data repair of multimodal data.

[0200] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0201] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the data repair method for multimodal data as described above.

[0202] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0203] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some technical features in the alignment. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data repair method for multimodal data, characterized in that: The data repair method of the multimodal data comprises: Extract data features from multimodal data and obtain initial feature vectors; Performing vector alignment processing on the initial feature vector to obtain a target feature vector; Using a dynamically weighted multi-head self-attention module to identify the target feature vector and determine the defect data location; Concatenating the target feature vectors to obtain a feature matrix; Using a generative adversarial network, the feature matrix is ​​located and repaired based on the defect data position to obtain repair data; The step of extracting data features from multimodal data to obtain an initial feature vector includes: Perform short-time Fourier transform on the input audio signal x(t): Among them, w(t) is the window function, ω is the angular frequency, and t is the time. For discrete signals, the above formula can be expressed as: Where n is time, w(nm) is the window function sequence, τ represents the offset of the window function w(t) in the time domain, j represents the imaginary unit, ω represents the angular frequency in the frequency domain, and m represents the number of frames in the short-time Fourier transform; According to the definition of power spectrum, the spectrogram formed by the relationship between short-time power spectrum and short-time Fourier transform is expressed as: SP x (m,ω)=X(m,ω)·X * (m,ω)=|X(m,ω)| 2 Where * represents the complex conjugate operation; The spectrogram is converted into a Mel spectrum by reducing the dimension, and the Mel spectrum is logarithmized, and a discrete cosine transform is performed to obtain a Mel frequency cepstrum coefficient as an audio feature vector; The performing vector alignment processing on the initial feature vector to obtain the target feature vector comprises: Traverse the initial eigenvector and calculate the maximum value d of the lateral dimension of the eigenvectors of all modes in the initial eigenvector max , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to perform left and right padding processing until the horizontal dimensions reach the maximum value d max until; Traverse the initial eigenvector and calculate the maximum value r of the longitudinal dimension of the eigenvectors of all modes in the initial eigenvector max , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to perform up and down filling processing until the vertical dimension reaches the maximum value r max Until the target feature vector is obtained; The method of using a dynamically weighted multi-head self-attention module to identify the target feature vector and determine the defect data location includes: Add position encoding to the initial multi-head self-attention module, where the position encoding is a vector, each vector corresponds to a position in the input feature; Adding the position code to the multimodal data feature vector and concatenating it with the target feature vector corresponding to the position code to obtain an enhanced feature vector; A multi-head self-attention module is used to capture the relationship between the features of each modality in the enhanced feature vector, and the attention weight is calculated to obtain the weight matrix of the feature importance between each modality; Determining missing data locations based on the weight matrix; Among them, the multi-head self-attention module is used to capture the relationship between the features of each modality in the enhanced feature vector, and the attention weight is calculated to obtain the weight matrix of the feature importance between each modality, including: The importance coefficient of data features between modes is calculated using the following formula: Among them, RELU is the attention function, K is the feature u i With feature u j The number of attention heads, k is the attention number, e is ij is the complete modal data feature u i The modal data features u with missing features j The correlation coefficient, and e ij ≥0; The weight matrix is ​​generated based on the importance coefficients.

2. The method for repairing multimodal data according to claim 1, characterized in that: The target feature vectors are concatenated to obtain a feature matrix including: Using the weight matrix, weighting the target feature vector to obtain a weighted feature vector; The weighted feature vectors are concatenated to obtain a multi-module feature matrix as the feature matrix.

3. A data repair device for multimodal data, characterized in that: The method for repairing multimodal data according to claim 1 or 2, wherein the device for repairing multimodal data comprises: A feature extraction module is used to extract data features from multimodal data and obtain an initial feature vector; A vector alignment module is used to perform vector alignment processing on the initial feature vector to obtain a target feature vector; A defect location module, used to identify the target feature vector using a dynamically weighted multi-head self-attention module to determine the defect data location; A feature concatenation module is used to concatenate the target feature vectors to obtain a feature matrix; The data repair module is used to use a generative adversarial network to locate and repair the feature matrix based on the position of the defective data to obtain repair data.

4. The data repair device for multimodal data according to claim 3, characterized in that: The vector alignment module comprises: The first filling unit is used to traverse the initial eigenvector and calculate the maximum value d of the lateral dimension of the eigenvectors of all modes in the initial eigenvector. max , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to perform left and right padding processing until the horizontal dimensions reach the maximum value d max until; The second filling unit is used to traverse the initial eigenvector and calculate the maximum value r of the longitudinal dimension of the eigenvectors of all modes in the initial eigenvector. max , and calculate the mean of the eigenvectors corresponding to each mode, and use the mean to perform up and down filling processing until the vertical dimension reaches the maximum value r max Until now, the target feature vector is obtained.

5. The data repair device for multimodal data according to claim 3, characterized in that: The defect location module includes: An adding unit for adding position encodings to the initial multi-head self-attention module, where the position encodings are vectors, each of which corresponds to a position in the input feature; A first concatenation unit, configured to add the position code to a multimodal data feature vector and concatenate it with a target feature vector corresponding to the position code to obtain an enhanced feature vector; A computing unit, used to capture the relationship between the features of each modality in the enhanced feature vector using a multi-head self-attention module, and calculate the attention weight to obtain a weight matrix of the feature importance between each modality; A positioning unit is used to determine the missing data position based on the weight matrix.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data repair method for multimodal data as claimed in claim 1 or 2 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data repair method for multimodal data as claimed in claim 1 or 2 is implemented.

Images