Heterogeneous data feature extraction method based on multi-modal information fusion

Abstract

This invention discloses a heterogeneous data feature extraction method based on multimodal information fusion. The implementation steps are as follows: A dual-stream architecture is constructed using a text encoder and an image encoder. Different encoders are built for feature extraction from different modalities. When fusing heterogeneous data features, the local correspondences between different modalities are fully considered, and an attention mechanism is used to fuse the heterogeneous data. This solves the problems of chaotic encoder model parameters, complex training processes, and incomplete fusion of heterogeneous data feature information in existing technologies. This invention enables better processing of heterogeneous data and improves the accuracy of heterogeneous data feature extraction. It also improves the completeness of information fusion, ensuring the effectiveness of the extracted heterogeneous information feature fusion.

Description

Technical Field

[0001] This invention belongs to the field of physical technology, and further relates to a method for extracting heterogeneous data features based on multimodal information fusion in the field of data processing technology. This invention comprehensively considers the impact of heterogeneous data such as text and images in the model input data on data vectorization representation, fully extracts heterogeneous data features through multimodal technology, and generates data representation vectors with rich semantic information by using an attention mechanism. Background Technology

[0002] With the rapid development of deep learning technologies in recent years, various deep learning models can effectively eliminate redundant features and accurately extract target features from input data such as images and text. However, much input data exists in heterogeneous forms. If a unimodal method is used to extract heterogeneous data, it will cause information loss, resulting in a discrepancy between the semantic information contained in the heterogeneous data representation vector and the semantic information contained in the original data.

[0003] Data encoders are constructed for text and image data respectively. The text encoder is pre-trained by predicting the current word based on the text context. The image encoder constructs a decoder for each image encoding vector and pre-trains using an encoder-decoder method. After obtaining the encoding vectors of heterogeneous data from the encoders of different modalities, a multimodal cross-fusion method based on an attention mechanism is used to fuse the features of the heterogeneous data, thereby obtaining data representation vectors containing heterogeneous information.

[0004] In their paper "Exploring deep multimodal fusion of text and photo for hate speech classification" (Proceedings of the third workshop on abusive language online, 11-18, 2019), Yang F et al. disclosed a social network speech classification method based on modal fusion. First, they obtained user-posted speech from social networks, including text and images. Then, they used an MLP to encode the text into text vectors, and ResNet to extract features from the images. Finally, they performed a concatenation operation on the text vectors and image matrices to obtain a multimodal vector representation of the speech data. However, this method still has shortcomings. When fusing the representation vectors of heterogeneous data, directly performing the concatenation operation fails to establish connections between the heterogeneous data and does not consider the local correspondences between them.

[0005] Zhang Lanlan et al. disclosed a multimodal data fusion method in their patent "Text Processing System and Method Based on Multimodal Data Fusion" (Patent Application No.: 202211133670.1, Publication No.: CN115203380A). This method first obtains retrieval information through a retrieval request receiving module, then uses a convolutional neural network to encode the text content contained in the retrieval information into corresponding semantic feature vectors. Next, the same convolutional neural network encoding module used in text encoding is used to encode the image content contained in the retrieval information, obtaining corresponding image encoded data. Finally, a multimodal representation matrix is ​​calculated as the result of multimodal data fusion and applied to downstream tasks. However, this method still has a drawback: using the same data encoding module for different modalities leads to the need to forcibly adapt the pre-trained model weights to another model feature, resulting in chaotic model parameters. Summary of the Invention

[0006] First, the purpose of this invention is to address the shortcomings of the existing technologies by proposing a heterogeneous data feature extraction method based on multimodal information fusion. This method solves the problems in multimodal technologies where single-stream architectures use a single encoder to extract features from different modalities simultaneously, leading to chaotic encoder model parameters and complex training processes. It also addresses the incomplete fusion of heterogeneous data feature information caused by directly using the concatenate global fusion method in heterogeneous data fusion.

[0007] The basic idea behind this invention is to address the problem of parameter confusion when using a single data encoding module for extracting data from different modalities. This invention comprehensively considers the heterogeneous data from different modalities contained in the model input data and employs a dual-stream structure consisting of a text encoder and an image encoder to extract features from both text and image data. This ensures that the encoders from different modalities do not interfere with each other. Since the local feature correspondences in heterogeneous data from different modalities are not considered during multimodal feature fusion, leading to incomplete information fusion, this invention uses an attention mechanism to fuse heterogeneous features, ensuring that the multimodal representation vector output by the model fully contains the semantic information of the input data. A text encoder and an image encoder are constructed separately, and the encoders are pre-trained using subject data corresponding to the respective modalities. Feature extraction is performed using the corresponding encoders for different modalities, thus solving the problem of chaotic model parameters caused by using the same encoder for feature extraction from different modalities. A recurrent neural network (RNN) is used as the student knowledge state generation network. The network is trained using student answer records, and an initial personalized question bank is generated from the trained network. Features are extracted from the questions in this personalized question bank using an encoder, and then fused using an attention mechanism to generate multimodal vectors that represent both text and image features, thus addressing the issue of incomplete multimodal feature fusion. Finally, similarity is calculated among the multimodal vectors, and the k most similar questions are used to form the student's personalized exam.

[0008] The implementation steps of this invention are as follows:

[0009] Step 1, Generate the training set:

[0010] Step 1.1: Collect all exam answer records of at least 100 students in one semester for a subject exam; organize the answers of each student's exam paper for that subject into the format (q-id, kc-id, ans), as a sample data of one question in that subject exam. Using the same method, obtain the answer records of all students for each question in the exam paper for that subject; where q-id represents the id-th question answered by the q-th student, kc-id represents the knowledge point kc contained in the id-th question, and ans is the label of the id-th question, which takes the value 0 or 1, ans=0 indicates that the answer to the question is correct, and ans=1 indicates that the answer to the question is incorrect;

[0011] Step 1.2: Using the same method as in Step 1.1, randomly select 20,000 sample data from the 30,000 answer records of each subject's exam to form a student answer record training set;

[0012] Step 2, generate pre-training sets for the text encoder and image encoder:

[0013] Step 2.1 Using web crawling technology, crawl a total of 40,000 test questions of the same subject as in Step 1.1, and obtain the test text, test image, test knowledge points, and test score information for each test question; after removing test questions containing the same text and those without test image, the remaining 32,000 unique test questions containing both text and image modal data are used to form a student exam question bank;

[0014] Step 2.2: Organize the text of each question in the question bank into a formatted text and use it as a text sample data for that question; randomly select 10,000 text sample data from the question bank to form the text pre-training set for the text encoder;

[0015] Step 2.3: Organize the image of each question in the question bank into an img format, which serves as an image sample data for that question; randomly select 10,000 image sample data from the question bank to form the image pre-training set for the image encoder;

[0016] Step 3, Configure the encoder:

[0017] Step 3.1: Use the BERT pre-trained model as the text encoder, set the word embedding vector dimension of the text encoder to 768, and set the maximum length of the input text to 80.

[0018] Step 3.2: Use the Transformer model as the image encoder, set the hidden state dimension of the image encoder to 512, and set the number of attention heads of the multi-head attention module to 6.

[0019] Step 4, Pre-trained encoder:

[0020] Step 4.1: Input the pre-trained text set into the text encoder, randomly select 15% of the words in the input text and replace them with [mask] tags, output the predicted words with [mask] tags, use the cross-entropy loss function to calculate the loss value between the predicted words output by the text encoder and the real words, and iteratively update all the parameters in the text encoder through the gradient descent method until the loss value no longer changes, and obtain the pre-trained text encoder.

[0021] Step 4.2: Input the image pre-training set into the image encoder, outputting an image feature vector that can represent the image features. Use principal component analysis to restore the image feature vector into an image. Calculate the loss value between the real image and the restored image using the cross-entropy loss function. Iteratively update all parameters of the image encoder using gradient descent until the loss value no longer changes, thus obtaining the pre-trained image encoder.

[0022] Step 5, train the student knowledge state generation network:

[0023] Step 5.1: Use the recurrent neural network as the student knowledge state generation network;

[0024] Step 5.2: Input the student answer record training set into the student knowledge state generation network, output the predicted label of each sample in the training, use the cross-entropy loss function to calculate the loss value between the predicted label and the true label of the training sample, and iteratively update all parameters in the student knowledge state generation network through gradient descent until the loss function converges, and obtain the trained student knowledge state generation network.

[0025] Step 6: Generate a personalized question bank for each student.

[0026] Input all the questions from the student exam question bank in step 2.1 into the trained student knowledge state generation network, and add the questions corresponding to the input samples with a network prediction label of 0 to the student personalized question bank.

[0027] Step 7: Extract features from multimodal test data containing heterogeneous data:

[0028] Step 7.1: Input the question texts of all test questions from the student's personalized question bank into the pre-trained text file.

[0029] The encoder outputs the text vector of each question.

[0030] Step 7.2: Input the image charts of all questions from the student's personalized question bank into the pre-trained image database.

[0031] The encoder outputs the question image vector for each test question;

[0032] Step 8, Fuse heterogeneous features:

[0033] The text vector of each test question and the vector of the attached image are both input into the attention mechanism for feature fusion, and the output is a multimodal representation vector that integrates text features and image features.

[0034] Step 9: Generate personalized test papers for students:

[0035] The similarity between pairwise multimodal vectors is obtained by calculating the cosine of the angle between the vectors. The questions corresponding to the multimodal vectors with the maximum similarity are selected to form personalized test papers for students.

[0036] Compared with existing technologies, the present invention has the following advantages:

[0037] Secondly, because the present invention uses a dual-stream architecture consisting of a text encoder and an image encoder to construct different encoders for the extraction of data features from different modalities, it overcomes the shortcomings of the single-stream architecture in existing multimodal technologies, which uses a single encoder to extract features from different modalities simultaneously, resulting in chaotic encoder model parameters and complex training processes. This allows the present invention to better handle heterogeneous data and improve the accuracy of heterogeneous data feature extraction.

[0038] Third, because this invention fully considers the local correspondences between different modalities when fusing heterogeneous data features, it uses an attention mechanism to fuse heterogeneous data. This overcomes the shortcomings of existing technologies that directly use global fusion methods such as concatenation in heterogeneous data fusion, resulting in incomplete fusion of heterogeneous data feature information. This invention improves the completeness of information fusion and ensures the effectiveness of the extracted heterogeneous information feature fusion. Attached Figure Description

[0039] Figure 1 A flowchart of the present invention. Detailed Implementation

[0040] The following is in conjunction with the appendix Figure 1 The present invention will be further described in conjunction with the embodiments.

[0041] Step 1: Generate the training set.

[0042] An embodiment of this invention collects the chemistry exam answer records of all students from 100 students across 15 chemistry exams within a semester from a junior high school chemistry exam system. The answers from each student's exam are organized into a (q-id, kc-id, ans) format as a sample of the chemistry question, resulting in the complete chemistry exam answer records for all students on that exam paper. Here, q-id represents the id-th chemistry question answered by the q-th student, kc-id represents the knowledge point kc contained in the id-th chemistry question, and ans is the label for the id-th question, with a value of 0 or 1. ans=0 indicates a correct answer, and ans=1 indicates an incorrect answer. Using the same method, 20,000 sample data points are randomly selected from 30,000 answer records from each of the 100 students taking 15 chemistry exams within a semester to form a student knowledge state network training set.

[0043] Step 2: Generate pre-training sets for the text encoder and image encoder.

[0044] A total of 40,000 junior high school chemistry test questions were crawled from various educational websites using web scraping technology. The text, image, knowledge points, and score information for each question were obtained. After removing questions with duplicate text and those without example images, 32,000 unique chemistry questions containing both text and image modalities were obtained, forming a student exam question bank. Each chemistry question in the question bank was formatted as a (text, img) sample data. Here, text represents the text of a chemistry question, and img represents the corresponding image in a chemistry question. Using the same method, 10,000 sample data were selected from the question bank as the pre-training set for the text encoder and image encoder.

[0045] Step 3, set up the encoder.

[0046] Step 3.1: Use the BERT pre-trained model as the text encoder, set the word embedding vector dimension of the text encoder to 768, and set the maximum length of the input text to 80.

[0047] Step 3.2: Use the Transformer model as the image encoder, set the hidden state dimension of the image encoder to 512, and set the number of attention heads of the multi-head attention module to 6.

[0048] Step 4, pre-train the encoder.

[0049] This invention uses the existing BERT pre-trained model as a text encoder, setting the word embedding vector dimension to 768 and the maximum text length to 80. The chemistry test text data from step 2 is used as input to the text encoder. The machine selects 15% of the words in the input text and replaces them with [mask] tags, outputting the predicted words with [mask] tags. Using the cross-entropy loss function, the loss value between the predicted words and the real words is calculated. Gradient descent is used to iteratively update all parameters in the text encoder until the loss value no longer changes, resulting in a pre-trained text encoder. This invention also uses the existing Transformer model as an image encoder, setting the number of attention heads in the multi-head attention module to 6 and the hidden state dimension to 512. The chemistry test image data from step 2 is used as input to the image encoder. The image data input to the image encoder will be processed by the encoder to extract features, and output an image feature vector that can represent the image features. Using principal component analysis, the covariance matrix of the image feature vector is calculated after averaging all elements of the image feature vector. Eigenvalue decomposition is performed on the covariance matrix to obtain the eigenvector. The image is then reconstructed using the eigenvector. The cross-entropy loss function is used to calculate the loss value between the real image and the reconstructed image. The gradient descent method is used to iteratively update all parameters of the image encoder until the loss value no longer changes, thus obtaining the pre-trained image encoder.

[0050] The formula for the cross-entropy loss function is as follows:

[0051] ;

[0052] in, Let represent the loss value between the true label and the predicted label, and y represent the true label. This represents the logarithmic operation to the base 2. This indicates the predicted label.

[0053] The loss between the predicted words and the true words output by the text encoder is calculated using the following formula:

[0054] ;

[0055] in, This represents the loss value between the predicted word output by the text encoder and the real word. This represents the total number of data points in the i-th training sample group. This represents the summation operation. Represents the cross-entropy function. This indicates that the text encoder processed the data in the i-th group. Predicted words for tag replacement Indicates that in the i-th group of data... The real words replaced by tags.

[0056] The loss value between the real image and the reconstructed image is calculated using the following formula:

[0057] ;

[0058] in, This represents the loss value between the real image and the image whose feature vectors are restored using principal component analysis. This represents the total number of data points in the j-th training sample group. This represents the summation operation. Represents the cross-entropy function. This represents the image obtained after reconstructing the feature vectors generated by the encoder from the j-th group of data using principal component analysis. This represents the real image in the i-th set of data.

[0059] The principal component analysis method refers to averaging all elements in the image feature vector, calculating the covariance matrix of the image feature vector, performing eigenvalue decomposition on the covariance matrix to obtain eigenvectors, and using these eigenvectors to reconstruct the image.

[0060] Step 5: Train the student knowledge state generation network.

[0061] This paper uses a recurrent neural network (RNN) as the student knowledge state generation network. Student answer records are input into the network for training. The model output, after processing, is a scalar representing the probability of a student answering a question correctly. A scalar value less than 0.5 indicates the student cannot answer correctly, while a value greater than 0.5 indicates the student can answer correctly. The cross-entropy loss function is used to calculate the loss between the predicted and true labels of the training samples. Gradient descent is then used to iteratively update all parameters in the student knowledge state generation network until the loss function converges, resulting in the trained student knowledge state generation network.

[0062] The network output results will be generated according to the following formula. and students' answers in the training data Network parameters are updated using cross-entropy loss:

[0063] ;

[0064] in, This represents the loss value between the predicted results of the training samples and the true labels. The predicted label value represents the student's knowledge state generated by the network. This represents the logarithmic function with base 2. This represents the true label of the training sample.

[0065] Step 6: Generate a personalized question bank for each student.

[0066] Input all the questions from the student exam question bank in step 2 into the trained student knowledge state generation network, and add the questions corresponding to the input samples with a network prediction label of 0 to the student personalized question bank.

[0067] Step 7: Extract features from multimodal test data containing heterogeneous data.

[0068] Input the question text of all questions in the student's personalized question bank into a pre-trained text encoder, and output the question text vector of each question; input the question image of all questions in the student's personalized question bank into a pre-trained image encoder, and output the question image vector of each question.

[0069] Step 8: Fuse heterogeneous features.

[0070] Attention operations are performed on the question text vector and the question image vector for each question. The attention score between the input question text vector and the question image vector is calculated. The attention score is multiplied by the input vector to perform vector fusion, which yields a multimodal representation vector that integrates image features and text features.

[0071] The attention mechanism refers to vector fusion by calculating the attention score between input vectors and multiplying the attention score with the input vector.

[0072] Step 9: Generate personalized test papers for students.

[0073] The similarity between pairwise multimodal vectors is obtained by calculating the cosine of the angle between the vectors. The questions corresponding to the multimodal vectors with the maximum similarity are selected to form personalized test papers for students.

[0074] Calculate the cosine of the angle between two vectors using the following formula:

[0075] ;

[0076] in, The cosine of the angle between the text vector of the test question and the vector of the attached image of the test question, where x represents the text vector of the test question and y represents the vector of the attached image of the test question. This indicates the modulo operation.

Claims

1. A heterogeneous feature extraction method based on multimodal information fusion, characterized in that, A dual-stream structure consisting of a text encoder and an image encoder is used to extract features from text data and image data respectively. An attention mechanism is then used to fuse these heterogeneous features. The specific steps of this heterogeneous feature extraction method are as follows: Step 1, Generate the training set: Step 1.1: Collect all exam answer records of at least 100 students in one semester for a subject exam; organize the answers of each student's exam paper for that subject into the format (q-id, kc-id, ans), as a sample data of one question in that subject exam. Using the same method, obtain the answer records of all students for each question in the exam paper for that subject; where q-id represents the id-th question answered by the q-th student, kc-id represents the knowledge point kc contained in the id-th question, and ans is the label of the id-th question, which takes the value 0 or 1, ans=0 indicates that the answer to the question is correct, and ans=1 indicates that the answer to the question is incorrect; Step 1.2: Using the same method as in Step 1.1, randomly select 20,000 sample data from the 30,000 answer records of each subject's exam to form a student answer record training set; Step 2, generate pre-training sets for the text encoder and image encoder: Step 2.1 Using web crawling technology, crawl a total of 40,000 test questions of the same subject as in Step 1.1, and obtain the test text, test image, test knowledge points, and test score information for each test question; after removing test questions containing the same text and those without test image, the remaining 32,000 unique test questions containing both text and image modal data are used to form a student exam question bank; Step 2.2: Organize the text of each question in the question bank into a formatted text and use it as a text sample data for that question; randomly select 10,000 text sample data from the question bank to form the text pre-training set for the text encoder; Step 2.3: Organize the image of each question in the question bank into an img format, which serves as an image sample data for that question; randomly select 10,000 image sample data from the question bank to form the image pre-training set for the image encoder; Step 3, Configure the encoder: Step 3.1: Use the BERT pre-trained model as the text encoder, set the word embedding vector dimension of the text encoder to 768, and set the maximum length of the input text to 80. Step 3.2: Use the Transformer model as the image encoder, set the hidden state dimension of the image encoder to 512, and set the number of attention heads of the multi-head attention module to 6. Step 4, Pre-trained encoder: Step 4.1: Input the pre-trained text set into the text encoder, randomly select 15% of the words in the input text and replace them with [mask] tags, output the predicted words with [mask] tags, use the cross-entropy loss function to calculate the loss value between the predicted words output by the text encoder and the real words, and iteratively update all the parameters in the text encoder through the gradient descent method until the loss value no longer changes, and obtain the pre-trained text encoder. Step 4.2: Input the image pre-training set into the image encoder, outputting an image feature vector that can represent the image features. Use principal component analysis to restore the image feature vector into an image. Calculate the loss value between the real image and the restored image using the cross-entropy loss function. Iteratively update all parameters of the image encoder using gradient descent until the loss value no longer changes, thus obtaining the pre-trained image encoder. Step 5, train the student knowledge state generation network: Step 5.1: Use the recurrent neural network as the student knowledge state generation network; Step 5.2: Input the student answer record training set into the student knowledge state generation network, output the predicted label of each sample in the training, use the cross-entropy loss function to calculate the loss value between the predicted label and the true label of the training sample, and iteratively update all parameters in the student knowledge state generation network through gradient descent until the loss function converges, and obtain the trained student knowledge state generation network. Step 6: Generate a personalized question bank for each student. Input all the questions from the student exam question bank in step 2.1 into the trained student knowledge state generation network, and add the questions corresponding to the input samples with a network prediction label of 0 to the student personalized question bank. Step 7: Extract features from multimodal test data containing heterogeneous data: Step 7.1: Input the question texts of all test questions from the student's personalized question bank into the pre-trained text file. The encoder outputs the text vector of each question. Step 7.2: Input the image charts of all questions from the student's personalized question bank into the pre-trained image database. The encoder outputs the question image vector for each test question; Step 8, Fuse heterogeneous features: The text vector of each test question and the vector of the attached image are both input into the attention mechanism for feature fusion, and the output is a multimodal representation vector that integrates text features and image features. The attention mechanism refers to the process of calculating attention scores between input vectors and then multiplying the attention scores with the input vectors to perform vector fusion. Step 9: Generate personalized test papers for students: The similarity between pairwise multimodal vectors is obtained by calculating the cosine of the angle between the vectors. The questions corresponding to the multimodal vectors with the maximum similarity are selected to form personalized test papers for students.

2. The heterogeneous feature extraction method based on multimodal information fusion according to claim 1, characterized in that, The formula for the cross-entropy loss function mentioned in step 4 is as follows: ； in, Let represent the loss value between the true label and the predicted label, and y represent the true label. This represents the logarithmic operation to the base 2. This indicates the predicted label.

3. The heterogeneous feature extraction method based on multimodal information fusion according to claim 1, characterized in that, The loss value between the predicted word and the real word output by the text encoder in step 4.1 is obtained by the following formula: ； in, This represents the loss value between the predicted word output by the text encoder and the real word. This represents the total number of data points in the i-th group of the text pre-training set. This represents the summation operation. Represents the cross-entropy function. This indicates that the text encoder processed the data in the i-th group. Predicted words for tag replacement Indicates that in the i-th group of data... The real words replaced by tags.

4. The heterogeneous feature extraction method based on multimodal information fusion according to claim 1, characterized in that, The principal component analysis method described in step 4.2 refers to averaging all elements in the image feature vector, calculating the covariance matrix of the image feature vector, performing eigenvalue decomposition on the covariance matrix to obtain the eigenvector, and using the eigenvector to reconstruct the image.

5. The heterogeneous feature extraction method based on multimodal information fusion according to claim 1, characterized in that, The loss value between the real image and the reconstructed image, as described in step 4.2, is obtained by the following formula: ； in, This represents the loss value between the real image and the image whose feature vectors are restored using principal component analysis. This represents the total number of data points in the j-th training sample group. This represents the summation operation. Represents the cross-entropy function. This represents the image obtained after reconstructing the feature vectors generated by the encoder from the j-th group of data using principal component analysis. This represents the real image in the i-th data set.

6. The heterogeneous feature extraction method based on multimodal information fusion according to claim 1, characterized in that, The loss value between the predicted label and the true label of the training sample in step 5 is obtained by the following formula: ； in, This represents the loss value between the predicted results of the training samples and the true labels. The predicted label value represents the student's knowledge state generated by the network. This represents the logarithmic function with base 2. This represents the true label of the training sample.

7. The heterogeneous feature extraction method based on multimodal information fusion according to claim 1, characterized in that, The cosine value of the included angle mentioned in step 9 is obtained by the following formula: ； in, The cosine of the angle between the text vector of the test question and the vector of the attached image of the test question, where x represents the text vector of the test question and y represents the vector of the attached image of the test question. This indicates the modulo operation.

Images