Drawing data extraction method based on multi-modal features

Abstract

The invention discloses a drawing data extraction method based on multi-modal features, which comprises the following steps: S1, multi-modal data input and preprocessing: receiving multi-modal data, and preprocessing the multi-modal data; s2, multi-modal feature extraction: carrying out feature extraction on the data of each modal, and respectively generating a 256-dimensional feature vector of each modal; s3, feature fusion and alignment: performing multi-channel fusion and feature alignment on features generated by different modal feature generators; s4, training a drawing data detection model: carrying out model training by using the marked normal sample data and a multi-mode auto-encoder model; and S5, drawing data extraction: inputting new multi-modal data into a plurality of feature generators to generate multi-modal features, carrying out fusion to generate a C * 256 feature matrix, inputting the C * 256 feature matrix into the trained multi-modal auto-encoder, and carrying out data extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of data extraction, and in particular to a drawing data extraction method based on multimodal features. Background Art

[0002] In the current data-driven environment, multimodal drawing data such as images and texts are widely present. In order to improve the conversion of drawing data into structured storage data, it is often necessary to extract multimodal drawing data so as to convert and store the dimensions and icon data in the drawings. Since each modality of data has unique characteristics and structures, traditional drawing data extraction methods have certain difficulties in processing these multi-source information and are difficult to meet actual usage needs. Summary of the invention

[0003] The purpose of the present invention is to provide a drawing data extraction method based on multimodal features, which can comprehensively consider information from different modes, is suitable for processing various multimodal data, can effectively improve data extraction efficiency, and has a good use effect.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A drawing data extraction method based on multimodal features comprises the following steps:

[0006] S1. Multimodal data input and preprocessing: receiving multimodal data and preprocessing the multimodal data;

[0007] S2, multi-modal feature extraction: feature extraction is performed on the data of each modality to generate a 256-dimensional feature vector for each modality;

[0008] The specific process of step S2 is:

[0009] S21. Use an image feature generator to extract image features from image data. The input data is a color high-definition RGB image with a resolution of 1920*1080. The image is input to the image feature generator, and the output is a 256-dimensional image feature vector. One image is one modality, and R images are input to R image feature generators to generate R 256-dimensional image feature vectors.

[0010] S22. For the text data, extract words of a fixed length, encode the words with word vectors, input the word vector codes into a text feature generator, and output a 256-dimensional text feature vector; if the text is too long, divide it into R text segments, each of which is a mode, and input them into R text feature generators to generate R 256-dimensional text feature vectors;

[0011] S23. For a column of discrete signal data, count the number of discrete categories, assume that the number of categories is K, then perform onehot encoding, and encode into a K-dimensional feature vector; if there are L columns of discrete signal data, construct L multi-dimensional feature vectors respectively, and the dimensions are related to the number of categories in each column; the L multi-dimensional vectors are input to the discrete data feature generator, and a 256-dimensional discrete feature vector is output; if the number of discrete signal data columns is too large, group them, each group is a mode, each group is input to the discrete data feature generator to generate a 256-dimensional feature, and R groups are input to R discrete data feature generators to generate R 256-dimensional feature vectors;

[0012] S24. For a certain column of continuous signal data, first normalize it to data between 0 and 1, and the maximum and minimum value method is used for normalization; if there are L columns of continuous signal data, then input the L continuous signal data into the continuous data feature extractor, and output a 256-dimensional continuous feature vector; if the number of continuous signal data columns is too large, then group them, each group is a mode, each group is input into the continuous data feature generator to generate a 256-dimensional feature, and R groups are input into R continuous data feature generators to generate R 256-dimensional feature vectors;

[0013] S3, feature fusion and alignment: perform multi-channel fusion and feature alignment on the features generated by different modal feature generators; different modalities generate 256-dimensional feature vectors, which is equivalent to feature alignment; assuming there are C modalities in total, the feature map of C channels formed after multi-modal fusion is a C*256 feature matrix;

[0014] S4. Drawing data detection model training: Use the labeled normal sample data and the multimodal autoencoder model to perform model training;

[0015] The specific process of step S4 is:

[0016] S41. After preprocessing the normal samples, C different modal data are input into C different feature generators to obtain C 256-dimensional feature vectors: F i =FeatureGenerator i (Preprocessed(NormalData i )),i=1,2,...,C, where FeatureGenerator i It is a multimodal feature generator, Preprocessed() is a preprocessing function, NormalData i is the normal sample data of the ith mode, F iis the 256-dimensional feature vector of the i-th mode;

[0017] S42, concatenate the C 256-dimensional feature vectors to obtain a C*256 feature matrix: I = Concatenate(F1, F2, ..., F C ), where I is the C*256 feature matrix formed by concatenating C feature vectors, and Concatenate() is the concatenation function;

[0018] S43, inputting the feature matrix I into a multimodal autoencoder for reconstruction, and obtaining a reconstructed feature matrix I′: I′=MultiModalAutoencoder(I), wherein MultiModalAutoencoder is a multimodal autoencoder;

[0019] S44, training the multimodal autoencoder using reconstruction error as a loss function, the loss function is expressed as follows: Loss = ReconstructionError(I, I′), where ReconstructionError() represents the reconstruction error function;

[0020] S45, when the reconstruction error converges, the training is stopped, and the multimodal autoencoder completes the training process;

[0021] S5. Drawing data extraction: Input a new multimodal data into multiple feature generators to generate multimodal features, fuse them to generate a C*256 feature matrix, and then input the C*256 feature matrix into the trained multimodal autoencoder for data extraction.

[0022] Preferably, the multimodal data in step S1 comes from different sensors, cameras or data acquisition systems; the multimodal data includes image data, text data, discrete signal data and continuous signal data.

[0023] Preferably, the preprocessing process of the multimodal data in step S1 is: denoising, deblurring and scaling alignment of image data; and removing spaces, symbols and truncating alignment of text data.

[0024] Preferably, the specific process of step S45 is: define the reconstruction error in the training process as Loss, set the stopping condition as the convergence of the reconstruction error, that is, define a threshold ∈, when |Loss is satisfied t -Loss t-1 When ∣<∈, the reconstruction error is considered to converge, and the multimodal autoencoder training process stops. t Represents the reconstruction error when the training iteration is t, Loss t-1 represents the reconstruction error of the previous round of t.

[0025] Preferably, the specific process of step S5 is:

[0026] S51. Input a new multimodal data into multiple feature generators to generate multimodal features, and obtain C 256-dimensional feature vectors: F i new =FeatureGenerator i (NewMultimodalData i ), i = 1, 2, ..., C, where NewMultimodalData i It is the preprocessed data of multimodal data, FeatureGenerator i is a multimodal feature generator, F i new is the 256-dimensional feature vector generated by the new data on the i-th mode;

[0027] S52, concatenate C 256-dimensional feature vectors to obtain a C*256 feature matrix: new =Concatenate(F1 new ,F2 new ,...,F C new ), where I new It is a C*256 feature matrix formed by concatenating the C feature vectors of the new data. Concatenate() is the concatenation function.

[0028] S53, input the C*256 feature matrix of the new data into the trained multimodal autoencoder to perform anomaly detection and obtain the reconstructed feature matrix: I' new =MultiModalAutoencoder(I new ), where MultiModalAutoencoder is the multimodal autoencoder trained above, I' new is the reconstructed feature matrix of the new data;

[0029] S54, by calculating the reconstruction error, setting a threshold, when the reconstruction error is greater than the set threshold, it is determined to be an abnormal sample; the calculation formula of the reconstruction error is: Loss new =ReconstructionError(I new ,I' new ), where Loss new is the reconstruction error of the new data, and the error is calculated using the least squares method;

[0030] S55, Data Extraction: If Loss new>θ, then the new test data is judged as invalid data; where θ is the set reconstruction error threshold. When the reconstruction error of the new data is less than this threshold, the data is extracted and saved.

[0031] After adopting the above technical scheme, the present invention has the following beneficial effects: the multimodal fusion adopted by the present invention can comprehensively consider information from different modes such as image data, text data, discrete signal data and continuous signal data, and perform feature generation, feature alignment and feature fusion, thereby improving the extraction efficiency of multimodal drawing data, and the feature extraction technology adopted improves the accurate identification of abnormal behavior, and is suitable for processing various multimodal drawing data, including images, texts, discrete signals and continuous signals, etc., has good versatility, can effectively improve the drawing data extraction accuracy, and has a good use effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the present invention;

[0033] Figure 2 It is a structural schematic diagram of the image feature generator of the present invention;

[0034] Figure 3 It is a structural schematic diagram of a text feature generator of the present invention;

[0035] Figure 4 It is a structural schematic diagram of a discrete data feature generator of the present invention;

[0036] Figure 5 It is a structural schematic diagram of a continuous data feature generator of the present invention;

[0037] Figure 6 It is a structural schematic diagram of the multimodal feature fusion and alignment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0039] like Figures 1 to 6 As shown, a drawing data extraction method based on multimodal features includes the following steps:

[0040] S1. Multimodal data input and preprocessing: receiving multimodal data and preprocessing the multimodal data;

[0041] The multimodal data in step S1 comes from different sensors, cameras or data acquisition systems; the multimodal data includes image data, text data, discrete signal data and continuous signal data;

[0042] The preprocessing process of the multimodal data in step S1 is: denoising, deblurring and scaling alignment of image data; removing spaces, symbols and truncating alignment of text data;

[0043] S2, multi-modal feature extraction: feature extraction is performed on the data of each modality to generate a 256-dimensional feature vector for each modality;

[0044] The specific process of step S2 is:

[0045] S21. Use an image feature generator to extract image features from image data. The input data is a color high-definition RGB image with a resolution of 1920*1080. The image is input to the image feature generator, and the output is a 256-dimensional image feature vector. One image is one modality, and R images are input to R image feature generators to generate R 256-dimensional image feature vectors.

[0046] The image feature generator in step S21 uses the ResNet residual backbone network, which is described as follows:

[0047] (A) Input image data: X∈R 1920*1080*3 , where 1920*1080 is the image resolution and 3 represents the RGB channels of the color image;

[0048] (B) ResNet backbone network processes image data: H = ResNet (X), where H represents the feature map after ResNet processing;

[0049] (C) Extract image features: F = FeatureExtractor(H), where F is a 256-dimensional image feature vector;

[0050] (D) Generation of R modality image feature vectors: For R images, each image is passed through the ResNet backbone network to obtain a 256-dimensional image feature vector, which is expressed as: F i =FeatureExtractor(ResNet(X i )), i = 1, 2, ..., R, where F i is the 256-dimensional feature vector of the i-th image. In this way, for the image modality data, R 256-dimensional image feature vectors can be obtained;

[0051] S22. For the text data, extract words of a fixed length, encode the words with word vectors, input the word vector codes into a text feature generator, and output a 256-dimensional text feature vector; if the text is too long, divide it into R text segments, each of which is a mode, and input them into R text feature generators to generate R 256-dimensional text feature vectors;

[0052] The text feature generator in step S22 uses the Transform backbone network, which is described as follows:

[0053] (A). Extract words of fixed length: Text i =SegmentWords(OriginalText i ), where OriginalText i Represents the original text of the i-th text mode, Text i It is a fixed-length word after truncation;

[0054] (B) Word vector encoding: V i =WordEmbedding(Text i ), where V i Represents the word vector encoding of the i-th text modality;

[0055] (C) Text feature generator processes word vector data: F i =Transform(V i ), where F i is the 256-dimensional text feature vector of the i-th text modality;

[0056] (D) Generation of R text modality feature vectors: For R text modalities, each text is passed through the text feature generator to obtain a 256-dimensional text feature vector, which is expressed as follows:

[0057] F i =Transform(WordEmbedding(SegmentWords(OriginalText i ))), i = 1, 2, ..., R,

[0058] Among them, F i is the 256-dimensional feature vector of the i-th text modality. In this way, for the data of the text modality, R 256-dimensional text feature vectors can be obtained.

[0059] S23. For a column of discrete signal data, count the number of discrete categories, assume that the number of categories is K, then perform onehot encoding, and encode into a K-dimensional feature vector; if there are L columns of discrete signal data, construct L multi-dimensional feature vectors respectively, and the dimensions are related to the number of categories in each column; the L multi-dimensional vectors are input to the discrete data feature generator, and a 256-dimensional discrete feature vector is output; if the number of discrete signal data columns is too large, group them, each group is a mode, each group is input to the discrete data feature generator to generate a 256-dimensional feature, and R groups are input to R discrete data feature generators to generate R 256-dimensional feature vectors;

[0060] In step S23, the discrete data feature generator uses an LSTM backbone network, which is specifically described as follows:

[0061] (A) Count the number of discrete categories and perform one-hot encoding: OneHot i =OneHotEncoding(DiscreteData i ), where DiscreteData i Indicates the discrete signal data in the i-th column, OneHot i is the one-hot encoding of the number of categories in the column, which is K;

[0062] (B) Constructing multi-dimensional feature vector: V i =Concatenate(OneHot1,OneHot2,...,OneHot L ), where V i is the multi-dimensional feature vector of the i-th discrete signal data, obtained by concatenating L columns of one-hot encoding;

[0063] (C) Discrete data feature generator processes multi-dimensional feature vectors: F i =LSTMTransform(V i ) Among them, F i is the 256-dimensional discrete feature vector of the i-th column of discrete signal data;

[0064] (D) Generation of R discrete data feature vectors: For R groups of discrete data, each group of data is passed through the LSTM backbone network to obtain a 256-dimensional discrete feature vector, which is expressed as: F i =LSTMTransform(Concatenate(OneHot1,OneHot2,...,OneHot L )), i = 1, 2, ..., R, where F iis the 256-dimensional feature vector of the i-th group of discrete data. In this way, for the mode of discrete data, R 256-dimensional feature vectors can be obtained.

[0065] S24. For a certain column of continuous signal data, first normalize it to data between 0 and 1, and the maximum and minimum value method is used for normalization; if there are L columns of continuous signal data, then input the L continuous signal data into the continuous data feature extractor, and output a 256-dimensional continuous feature vector; if the number of continuous signal data columns is too large, then group them, each group is a mode, each group is input into the continuous data feature generator to generate a 256-dimensional feature, and R groups are input into R continuous data feature generators to generate R 256-dimensional feature vectors;

[0066] The continuous data feature extractor in step S24 uses the ResNet backbone network, which is specifically described as follows:

[0067] (A) Data normalized to between 0 and 1:

[0068] Normalized i =max(ContinuousData i )-min(ContinuousData i ) /

[0069] ContinuousData i -min(ContinuousData i ), where ContinuousData i Represents the continuous signal data of the i-th column, Normalized i The data is normalized by the maximum and minimum method;

[0070] (B) Input L columns of continuous data into the continuous data feature extractor: V i =Concatenate(Normalized1,Normalized2,...,Normalized L ), where V i is the multi-dimensional feature vector of the i-th continuous signal data, obtained by connecting L columns of normalized data;

[0071] (C) Continuous data feature generator processes multi-dimensional feature vectors: F i =ResNetTransform(V i ) Among them, F i is the 256-dimensional continuous feature vector of the continuous signal data in the i-th column;

[0072] (D) Generation of R continuous data feature vectors: For R groups of continuous data, each group of data is passed through the ResNet backbone network to obtain a 256-dimensional continuous feature vector, which is expressed as: F i =ResNetTransform(Concatenate(Normalized1,Normalized2,...,Normalized L )), i = 1, 2, ..., R, where F i is the 256-dimensional feature vector of the i-th group of continuous data. In this way, for the mode of continuous data, R 256-dimensional feature vectors can be obtained.

[0073] S3, feature fusion and alignment: perform multi-channel fusion and feature alignment on the features generated by different modal feature generators; different modalities generate 256-dimensional feature vectors, which is equivalent to feature alignment; assuming there are C modalities in total, the feature map of C channels formed after multi-modal fusion is a C*256 feature matrix;

[0074] S4. Drawing data detection model training: Use the labeled normal sample data and the multimodal autoencoder model to perform model training;

[0075] The specific process of step S4 is:

[0076] S41. After preprocessing the normal samples, C different modal data are input into C different feature generators to obtain C 256-dimensional feature vectors: F i =FeatureGenerator i (Preprocessed(NormalData i )),i=1,2,...,C, where FeatureGenerator i It is a multimodal feature generator, Preprocessed() is a preprocessing function, NormalData i is the normal sample data of the ith mode, F i is the 256-dimensional feature vector of the i-th mode;

[0077] S42, concatenate the C 256-dimensional feature vectors to obtain a C*256 feature matrix: I = Concatenate(F1, F2, ..., F C ), where I is the C*256 feature matrix formed by concatenating C feature vectors, and Concatenate() is the concatenation function;

[0078] S43, inputting the feature matrix I into a multimodal autoencoder for reconstruction, and obtaining a reconstructed feature matrix I′: I′=MultiModalAutoencoder(I), wherein MultiModalAutoencoder is a multimodal autoencoder;

[0079] S44, training the multimodal autoencoder using reconstruction error as a loss function, the loss function is expressed as follows: Loss = ReconstructionError(I, I′), where ReconstructionError() represents the reconstruction error function;

[0080] S45, when the reconstruction error converges, the training is stopped, and the multimodal autoencoder completes the training process;

[0081] The specific process of step S45 is: define the reconstruction error in the training process as Loss, set the stopping condition as the convergence of the reconstruction error, that is, define a threshold ∈, when |Loss is satisfied t -Loss t-1 When ∣<∈, the reconstruction error is considered to converge, and the multimodal autoencoder training process stops. t Represents the reconstruction error when the training iteration is t, Loss t-1 represents the reconstruction error of the previous round of t;

[0082] S5, drawing data extraction: a new multimodal data is input into multiple feature generators to generate multimodal features, which are fused to generate a C*256 feature matrix, and then the C*256 feature matrix is ​​input into the trained multimodal autoencoder for data extraction;

[0083] The specific process of step S5 is:

[0084] S51. Input a new multimodal data into multiple feature generators to generate multimodal features, and obtain C 256-dimensional feature vectors: i=1,2,...,C,where NewMultimodalData i It is the preprocessed data of multimodal data, FeatureGenerator i is a multimodal feature generator, is the 256-dimensional feature vector generated by the new data on the i-th mode;

[0085] S52. Concatenate C 256-dimensional feature vectors to obtain a C*256 feature matrix: Among them, I newIt is a C*256 feature matrix formed by concatenating the C feature vectors of the new data. Concatenate() is the concatenation function.

[0086] S53, input the C*256 feature matrix of the new data into the trained multimodal autoencoder to perform anomaly detection and obtain the reconstructed feature matrix: I' new =MultiModalAutoencoder(I new ), where MultiModalAutoencoder is the multimodal autoencoder trained above, I' new is the reconstructed feature matrix of the new data;

[0087] S54, by calculating the reconstruction error, setting a threshold, when the reconstruction error is greater than the set threshold, it is determined to be an abnormal sample; the calculation formula of the reconstruction error is: Loss new =ReconstructionError(I new ,I' new ), where Loss new is the reconstruction error of the new data, and the error is calculated using the least squares method;

[0088] S55, Data Extraction: If Loss new >θ, then the new test data is judged as invalid data; where θ is the set reconstruction error threshold. When the reconstruction error of the new data is less than this threshold, the data is extracted and saved.

[0089] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A drawing data extraction method based on multimodal features, characterized in that: The following steps are involved: S1. Multimodal data input and preprocessing: receiving multimodal data and preprocessing the multimodal data; S2, multi-modal feature extraction: feature extraction is performed on the data of each modality to generate a 256-dimensional feature vector for each modality; The specific process of step S2 is: S21. Use an image feature generator to extract image features from image data. The input data is a color high-definition RGB image with a resolution of 1920*1080. The image is input to the image feature generator, and the output is a 256-dimensional image feature vector. One image is one modality, and R images are input to R image feature generators to generate R 256-dimensional image feature vectors. S22. For the text data, extract words of a fixed length, encode the words with word vectors, input the word vector codes into a text feature generator, and output a 256-dimensional text feature vector; if the text is too long, divide it into R text segments, each of which is a mode, and input them into R text feature generators to generate R 256-dimensional text feature vectors; S23. For a column of discrete signal data, count the number of discrete categories, assume that the number of categories is K, then perform onehot encoding, and encode into a K-dimensional feature vector; if there are L columns of discrete signal data, construct L multi-dimensional feature vectors respectively, and the dimensions are related to the number of categories in each column; the L multi-dimensional vectors are input to the discrete data feature generator, and a 256-dimensional discrete feature vector is output; if the number of discrete signal data columns is too large, group them, each group is a mode, each group is input to the discrete data feature generator to generate a 256-dimensional feature, and R groups are input to R discrete data feature generators to generate R 256-dimensional feature vectors; S24. For a certain column of continuous signal data, first normalize it to data between 0 and 1, and the maximum and minimum value method is used for normalization; if there are L columns of continuous signal data, then input the L continuous signal data into the continuous data feature extractor, and output a 256-dimensional continuous feature vector; if the number of continuous signal data columns is too large, then group them, each group is a mode, each group is input into the continuous data feature generator to generate a 256-dimensional feature, and R groups are input into R continuous data feature generators to generate R 256-dimensional feature vectors; S3, feature fusion and alignment: perform multi-channel fusion and feature alignment on the features generated by different modal feature generators; different modalities generate 256-dimensional feature vectors, which is equivalent to feature alignment; assuming there are C modalities in total, the feature map of C channels formed after multi-modal fusion is a C*256 feature matrix; S4. Drawing data detection model training: Use the labeled normal sample data and the multimodal autoencoder model to perform model training; The specific process of step S4 is: S41. After preprocessing the normal samples, C different modal data are input into C different feature generators to obtain C 256-dimensional feature vectors: F i =FeatureGenerator i (Preprocessed(NormalData i )),i=1,2,...,C, where FeatureGenerator i It is a multimodal feature generator, Preprocessed() is a preprocessing function, NormalData i is the normal sample data of the ith mode, F i is the 256-dimensional feature vector of the i-th mode; S42, concatenate the C 256-dimensional feature vectors to obtain a C*256 feature matrix: I = Concatenate(F1, F2, ..., F C ), where I is the C*256 feature matrix formed by concatenating C feature vectors, and Concatenate() is the concatenation function; S43, inputting the feature matrix I into a multimodal autoencoder for reconstruction, and obtaining a reconstructed feature matrix I′: I′=MultiModalAutoencoder(I), wherein MultiModalAutoencoder is a multimodal autoencoder; S44, training the multimodal autoencoder using reconstruction error as a loss function, the loss function is expressed as follows: Loss = ReconstructionError(I, I′), where ReconstructionError() represents the reconstruction error function; S45, when the reconstruction error converges, the training is stopped, and the multimodal autoencoder completes the training process; S5. Drawing data extraction: Input a new multimodal data into multiple feature generators to generate multimodal features, fuse them to generate a C*256 feature matrix, and then input the C*256 feature matrix into the trained multimodal autoencoder for data extraction.

2. The method for extracting drawing data based on multimodal features according to claim 1, characterized in that: The multimodal data in step S1 comes from different sensors, cameras or data acquisition systems; the multimodal data includes image data, text data, discrete signal data and continuous signal data.

3. A drawing data extraction method based on multimodal features as claimed in claim 1, characterized in that: The preprocessing process of the multimodal data in step S1 is: denoising, deblurring and scaling alignment of image data; and removing spaces, symbols and truncating alignment of text data.

4. A drawing data extraction method based on multimodal features as claimed in claim 1, characterized in that: The specific process of step S45 is: define the reconstruction error in the training process as Loss, set the stopping condition as the convergence of the reconstruction error, that is, define a threshold ∈, when |Loss is satisfied t -Loss t-1 When ∣<∈, the reconstruction error is considered to converge, and the multimodal autoencoder training process stops. t Represents the reconstruction error when the training iteration is t, Loss t-1 represents the reconstruction error of the previous round of t.

5. The method for extracting drawing data based on multimodal features according to claim 1, characterized in that: The specific process of step S5 is: S51. Input a new multimodal data into multiple feature generators to generate multimodal features, and obtain C 256-dimensional feature vectors: i=1,2,...,C,where NewMultimodalData i It is the preprocessed data of multimodal data, FeatureGenerator i is a multimodal feature generator, is the 256-dimensional feature vector generated by the new data on the i-th mode; S52. Concatenate C 256-dimensional feature vectors to obtain a C*256 feature matrix: Among them, I new It is a C*256 feature matrix formed by concatenating the C feature vectors of the new data. Concatenate() is the concatenation function. S53, input the C*256 feature matrix of the new data into the trained multimodal autoencoder to perform anomaly detection and obtain the reconstructed feature matrix: I' new =MultiModalAutoencoder(I new ), where MultiModalAutoencoder is the multimodal autoencoder trained above, I' new is the reconstructed feature matrix of the new data; S54, by calculating the reconstruction error, setting a threshold, when the reconstruction error is greater than the set threshold, it is determined to be an abnormal sample; the calculation formula of the reconstruction error is: Loss new =ReconstructionError(I new ,I' new ), where Loss new is the reconstruction error of the new data, and the error is calculated using the least squares method; S55, Data Extraction: If Loss new >θ, then the new test data is judged as invalid data; where θ is the set reconstruction error threshold. When the reconstruction error of the new data is less than this threshold, the data is extracted and saved.

Images