A multimodal test question knowledge point annotation method, system, device and storage medium

Abstract

The present invention discloses a multimodal test question knowledge point annotation method based on image-text fusion, a system, a device and a storage medium, and belongs to the field of natural language processing technology. The present invention extracts features from test question data through a self-supervised assisted dual-stream test question feature extraction network to obtain a text feature vector and an image feature vector of the test question; the text feature vector and the image feature vector are input into a dual-stage cross-modal alignment network for alignment to obtain an aligned text feature vector and an aligned image feature vector; the aligned text feature vector and the aligned image feature vector are input into a difference perception fusion network to obtain a fusion vector; the fusion vector, the text feature vector and the image feature vector are weightedly calculated to obtain a test question feature vector; the test question feature vector is input into a knowledge point annotation network to obtain a knowledge point label and annotate the question. The present invention can improve the accuracy and efficiency of knowledge point annotation, and can also provide more accurate and personalized support for teaching services.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a multimodal test question knowledge point annotation method based on image-text fusion, a system, a device and a storage medium. Background Art

[0002] With the rise of online education platforms like MOOC, Leagle, and Coursera, high-quality educational resources are constantly emerging, driving the development of data-driven smart education and making it an inevitable choice in the digital age. By efficiently organizing and managing these educational resources, learners can be provided with a rich, diverse, and highly personalized learning experience.

[0003] In the field of intelligent education, there are several key technical tasks that are crucial for improving learner experience and learning outcomes. These tasks include: 1) Similar Question Tasks: Given a test question, similar question analysis tools can find similar questions to help students consolidate knowledge points. This task analyzes the content and characteristics of the test question and uses machine learning algorithms to find similar questions, thereby improving students' learning efficiency and comprehension. 2) Knowledge Tracking Tasks: By analyzing a learner's past answers, we can discover their knowledge mastery and predict their next answer. This task uses a knowledge tracking model to analyze learners' learning progress and knowledge mastery, providing data support for developing personalized learning paths. 3) Resource Recommendation Tasks: To improve learning efficiency, learners need access to educational resources that match their needs and interests. Therefore, an intelligent educational resource recommendation system is needed to effectively filter and recommend appropriate resources. This task collects learners' learning history, interests, preferences, and behavior data, and uses recommendation algorithms (such as collaborative filtering) to generate personalized resource recommendations, thereby improving learner satisfaction and learning outcomes.

[0004] The realization of these tasks is based on knowledge point annotation. However, traditional knowledge point annotation mainly relies on experienced teachers or domain experts to complete it manually. It is difficult to obtain high-quality knowledge point annotation in this way. It is subject to the subjective ideas of the annotators and is time-consuming and labor-intensive. With the deepening of the process of educational informatization, educational resources are becoming increasingly abundant, and traditional test question annotation methods can no longer meet actual needs. Therefore, it has become urgent to develop a novel automatic annotation system to improve the efficiency and accuracy of knowledge point annotation. The knowledge point annotation task is of vital importance for grasping students' knowledge structure and learning path. Before performing knowledge point annotation, educational resources need to be comprehensively represented. Images, as an indispensable element in textbooks and exercises, together with text, constitute the rich connotation of educational resources. Therefore, the test question knowledge point annotation method based on image and text fusion has become a core research topic in the field of intelligent education.

[0005] Although existing research has proposed numerous pre-training models that perform knowledge point annotation in downstream tasks by pre-training on large-scale question corpora, these models are mostly limited to the field of mathematics and often only consider text information. At the same time, there are also end-to-end supervised models specifically for knowledge point annotation tasks, but they mainly focus on text-based machine learning models. Although existing methods have been tried to be applied to heterogeneous test question resources, they are still too focused on formula representation based on Latex format, or only perform knowledge point annotation on teaching images. In the field of vision, commonly used feature extractors such as convolutional neural networks such as VGG and Resnet are all trained based on natural images, which makes it difficult to effectively capture the special features of images in the education field. In order to solve the problems existing in existing technologies, effective solutions need to be proposed. Summary of the Invention

[0006] In order to overcome the problems existing in the current test question knowledge point annotation methods in terms of image and text alignment and fusion, the present invention provides a multimodal test question knowledge point annotation method, system, device and storage medium based on image and text fusion, aiming to achieve accurate and efficient knowledge point annotation by deeply integrating the multimodal features of text and images.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The present invention provides a method for labeling knowledge points of multimodal test questions, comprising:

[0009] The test data is subjected to feature extraction through a self-supervised dual-stream test feature extraction network to obtain the text feature vector and image feature vector of the test question;

[0010] Input the text feature vector and the image feature vector into the two-stage cross-modal alignment network for alignment to obtain the aligned text feature vector and the aligned image feature vector;

[0011] Input the aligned text feature vector and the aligned image feature vector into the difference-aware fusion network to obtain a fusion vector;

[0012] Perform weighted calculation on the fusion vector, text feature vector and image feature vector to obtain the test question feature vector;

[0013] The test question feature vector is input into the knowledge point annotation network to obtain the knowledge point label and annotate the question.

[0014] The self-supervised assisted dual-stream test question feature extraction network includes an information extraction layer, a text feature extraction layer, and an image feature extraction layer. The test question data is subjected to feature extraction through the self-supervised assisted dual-stream test question feature extraction network to obtain a text feature vector and an image feature vector of the test question, including:

[0015] Input the test data into the information extraction layer to obtain text information and image information;

[0016] Input text information into the text feature extraction layer to obtain text feature vector;

[0017] The image information is input into the image feature extraction layer to obtain the image feature vector.

[0018] The dual-stage cross-modal alignment network includes a unified layer and a shared layer. The text feature vector and the image feature vector are input into the dual-stage cross-modal alignment network for alignment to obtain an aligned text feature vector and an aligned image feature vector, including:

[0019] Input the text feature vector and the image feature vector into the unification layer to unify the dimensions and obtain the unified text feature vector and image feature vector;

[0020] The unified text feature vector and image feature vector are input into the shared layer to obtain the aligned text feature vector and aligned image feature vector.

[0021] The step of inputting the aligned text feature vector and the aligned image feature vector into a difference perception fusion network to obtain a fusion vector includes:

[0022] The aligned text feature vector and the aligned image feature vector are input into the difference-aware fusion network. Through cross-attention, the aligned text feature vector and the aligned image feature vector are semantically interacted across modalities, and the aligned image features are used to guide the attention distribution of the aligned text features. After the aligned text feature vector and the aligned image feature vector are fused, a fusion vector is obtained.

[0023] The step of performing weighted calculation on the fusion vector, the text feature vector, and the image feature vector to obtain the test question feature vector includes:

[0024] The fusion vector, text feature vector and image feature vector are weightedly calculated based on KL divergence to obtain the test question feature vector.

[0025] The step of inputting the test question feature vector into the knowledge point annotation network to obtain the knowledge point label and annotate the question includes:

[0026] The test question feature vector is input into the knowledge point labeling network, the test question feature vector is mapped to the knowledge point label set, the probability distribution of the knowledge points in the knowledge point set is calculated using the softmax layer, the knowledge point label with the highest probability is selected, and the knowledge points of the question are labeled.

[0027] A method for labeling knowledge points in a multimodal test question, the method further comprising:

[0028] Obtain sample test question data and preset knowledge point labels corresponding to the sample test question data;

[0029] Input the sample test question data into the preset self-supervised assisted dual-stream test question feature extraction network for feature extraction to obtain the text feature vector and image feature vector of the sample test question data;

[0030] Input the text feature vector and image feature vector of the sample test question data into a preset two-stage cross-modal alignment network for alignment, thereby obtaining an aligned text feature vector and an aligned image feature vector of the sample test question data;

[0031] Inputting the aligned text feature vector and the aligned image feature vector of the sample test question data into a preset difference perception fusion network to obtain a fusion vector of the sample test question data;

[0032] Perform weighted calculation on the fusion vector, text feature vector and image feature vector of the sample test question data to obtain the test question feature vector of the sample test question data;

[0033] Input the test question feature vector of the sample test question data into the preset knowledge point labeling network to obtain the knowledge point label of the sample test question data;

[0034] Determine the total loss based on the preset knowledge point labels corresponding to the sample test question data and the knowledge point labels of the sample test question data;

[0035] According to the total loss, the preset self-supervised assisted dual-stream test question feature extraction network, the preset dual-stage cross-modal alignment network, the preset difference perception fusion network and the preset knowledge point labeling network are trained to obtain the self-supervised assisted dual-stream test question feature extraction network, the dual-stage cross-modal alignment network, the difference perception fusion network and the knowledge point labeling network.

[0036] The present invention also provides a multimodal test question knowledge point annotation system, comprising:

[0037] The test question feature acquisition module is used to extract features from the test question data through a self-supervised dual-stream test question feature extraction network to obtain the text feature vector and image feature vector of the test question;

[0038] A feature alignment module is used to input the text feature vector and the image feature vector into a two-stage cross-modal alignment network for alignment, thereby obtaining an aligned text feature vector and an aligned image feature vector.

[0039] A feature fusion module is used to input the aligned text feature vector and the aligned image feature vector into the difference perception fusion network to obtain a fusion vector;

[0040] The feature weighting module is used to perform weighted calculation on the fusion vector, text feature vector and image feature vector to obtain the test question feature vector;

[0041] The knowledge point annotation module is used to input the test question feature vector into the knowledge point annotation network, obtain the knowledge point label, and annotate the question.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multimodal test question knowledge point annotation method are implemented.

[0043] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned multimodal test question knowledge point annotation method.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The method of the present invention effectively extracts respective feature vectors from the text and image of the test questions through a self-supervised auxiliary dual-stream test question feature extraction network; through a dual-stage cross-modal alignment network, the alignment problem between text and image features is further solved, so that the features of two different modalities can be effectively compared and associated in the same space; the difference perception fusion network realizes the deep fusion of text and image features, generates a fusion vector containing two modal information, thereby improving the accuracy of knowledge point annotation; by weighted calculation of the fusion vector and the single modality, the feature vector of the test question is obtained, which can fully reflect the content and information of the test question; the knowledge point annotation network uses this feature vector to automatically annotate knowledge points for the test questions, greatly improving annotation efficiency. The present invention solves the alignment and fusion problems between test question images and texts in different subjects, and realizes the automation of knowledge point annotation, which can provide support for teaching services such as downstream task teaching assistance, student cognitive ability modeling, adaptive test question recommendation, similar question judgment, and intelligent test paper compilation. These downstream tasks can provide more accurate and personalized teaching services based on the annotated knowledge points, thereby improving students' learning effects and teachers' teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 Schematic diagram of the process of the multimodal test question knowledge point labeling method of the present invention;

[0048] Figure 2This is a flow chart of the method for labeling knowledge points in a multimodal test question according to the present invention;

[0049] Figure 3 Schematic diagram of the self-supervised assisted dual-stream test question feature extraction network of the present invention;

[0050] Figure 4 Schematic diagram of the dual-stage cross-modal alignment network of the present invention;

[0051] Figure 5 Schematic diagram of the cross attention mechanism in the difference perception fusion network of the present invention;

[0052] Figure 6 This is a picture of a test question in the Own-PHY2 dataset in the embodiment;

[0053] Figure 7 : The knowledge point distribution diagrams of the four subject data sets in the embodiment are shown in FIG. 1 , (a) is the knowledge point distribution diagram of the ScienceQA-BIO data set, (b) is the knowledge point distribution diagram of the ScienceQA-GEO data set, (c) is the knowledge point distribution diagram of the Own-PHY1 data set, and (d) is the knowledge point distribution diagram of the Own-PHY2 data set.

[0054] Figure 8 This is a bar chart comparing the accuracy of ablation experiments in the embodiment;

[0055] Figure 9 This is a schematic diagram of the structure of a multimodal test question knowledge point annotation system according to a preferred embodiment of the present invention;

[0056] Figure 10 This is a schematic diagram of the structure of an electronic device according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0057] To facilitate understanding of the features and effects of the present invention by those skilled in the art, the following provides a general description and definition of the terms and expressions used in the specification and claims. Unless otherwise indicated, all technical and scientific terms used herein have the ordinary meanings as understood by those skilled in the art regarding the present invention. In the event of conflict, the definitions in this specification shall prevail.

[0058] The theories or mechanisms described and disclosed herein, whether correct or incorrect, should not limit the scope of the present invention in any way, that is, the present invention can be implemented without being limited by any specific theory or mechanism.

[0059] All features, such as values, amounts, contents, and concentrations, described herein as numerical ranges or percentage ranges are provided for simplicity and convenience only. Accordingly, the description of numerical ranges or percentage ranges should be considered to include and specifically disclose all possible subranges and individual values within the range (including integers and fractions).

[0060] In this document, unless otherwise specified, “include,” “including,” “contains,” “has” or similar terms cover the meanings of “consisting of” and “mainly consisting of,” for example, “A includes a” covers the meanings of “A includes a and other” and “A only includes a.”

[0061] In this document, for the sake of brevity, not all possible combinations of the various technical features in each embodiment or example are described. Therefore, as long as there are no contradictions in the combination of these technical features, the various technical features in each embodiment or example can be combined in any way, and all possible combinations should be considered to be within the scope of this specification.

[0062] like Figure 1 and Figure 2 As shown, the present invention provides a multimodal test question knowledge point annotation method based on image-text fusion, comprising:

[0063] The test data is subjected to feature extraction through a self-supervised dual-stream test feature extraction network to obtain the text feature vector and image feature vector of the test question;

[0064] Input the text feature vector and the image feature vector into the two-stage cross-modal alignment network for alignment to obtain the aligned text feature vector and the aligned image feature vector;

[0065] Input the aligned text feature vector and the aligned image feature vector into the difference-aware fusion network to obtain a fusion vector;

[0066] Perform weighted calculation on the fusion vector, text feature vector and image feature vector to obtain the test question feature vector;

[0067] The test question feature vector is input into the knowledge point annotation network to obtain the knowledge point label and annotate the question.

[0068] like Figure 3 As shown, S1, extracts features from the test data through a self-supervised assisted dual-stream test feature extraction network to obtain the text feature vector and image feature vector of the test question, specifically:

[0069] The self-supervised assisted dual-stream test question feature extraction network is obtained by training a preset self-supervised assisted dual-stream test question feature extraction network, and the preset self-supervised assisted dual-stream test question feature extraction network includes an information extraction layer, a text feature extraction layer and an image feature extraction layer.

[0070] S101, input the test question data into the information extraction layer to obtain text information and image information;

[0071] S102, inputting the text information into a text feature extraction layer to obtain a text feature vector;

[0072] Set the text message to ,in is a word, N is the maximum length of the sequence;

[0073] Get feature vector through semantic network Bert H ,in , the eigenvector H It is the output of the last layer of the semantic network Bert, and the size of N is 768.

[0074] The feature vector H Input convolutional layer TextCNN, convolutional layer TextCNN uses convolution kernels of different sizes to compare feature vectors H Perform sliding window convolution to obtain the key features of the sentence. The TextCNN convolution kernel size is set to 1, 2, 4, and 8 to obtain the text feature vector, which is expressed as ;

[0075] S103, inputting the image information into the image feature extraction layer to obtain an image feature vector;

[0076] Set the image information as image, and divide the image information image into Pixels, normalized;

[0077] Through the image encoder, the image feature vector is obtained, which is expressed as ,in .

[0078] In some embodiments, the image encoder is a Resnet34 image encoder

[0079] like Figure 4 As shown in S2, the text feature vector and the image feature vector are input into the two-stage cross-modal alignment network for alignment to obtain the aligned text feature vector and the aligned image feature vector, specifically;

[0080] Taking into account the differences in data from different modalities, a two-stage cross-modal alignment network is used to fine-grainedly balance the semantic gap between different modalities.

[0081] The two-stage cross-modal alignment network is obtained by training a preset two-stage cross-modal alignment network, where the preset two-stage cross-modal alignment network includes a unified layer and a shared layer.

[0082] S201: Input the text feature vector and the image feature vector into a unification layer for dimension unification to obtain the unified text feature vector and image feature vector, which are expressed as follows:

[0083]

[0084]

[0085] in, , is the text feature vector after unification of dimensions, is the image feature vector after unification of dimensions.

[0086] The text feature vector and the image feature vector are aligned by mapping them into a shared semantic space through a shared layer in a two-stage cross-modal alignment network, which consists of a multi-layer perceptron.

[0087] )

[0088] )

[0089] in, For the shared layer, ( (x)), To align text feature vectors, To align image feature vectors, and , Indicates the dimension after alignment.

[0090] like Figure 5 As shown, S3 inputs the aligned text feature vector and the aligned image feature vector into the difference perception fusion network to obtain the fusion vector, which is specifically:

[0091] The difference perception fusion network is obtained by training a preset difference perception fusion network.

[0092] The aligned text feature vector and the aligned image feature vector are input into the difference perception fusion network. The aligned text feature vector and the aligned image feature vector are subjected to cross-modal semantic interaction through cross-modal semantic interaction. The aligned image features are used to guide the attention distribution of the aligned text features through cross-modal semantic interaction. After the aligned text feature vector and the aligned image feature vector are fused, the fusion vector is obtained, which is expressed as m .

[0093] The aligned text feature vector representation is regarded as Query (Q), and the aligned image feature vector is regarded as Key (K) and Value (V) of multi-head cross attention. The cross-modal correlation and mapping relationship between image and text modalities is learned. Let , , then the cross attention calculation process is expressed as follows:

[0094]

[0095] in, is the learned projection matrix, Indicates the dimension after alignment, Dimensions for queries, key-value pairs.

[0096] S4, perform weighted calculation on the fusion vector, text feature vector and image feature vector to obtain the test question feature vector, specifically:

[0097] The fusion vector, text feature vector and image feature vector are weighted based on KL divergence to obtain the test question feature vector, including:

[0098] Since different modalities play different roles in knowledge point annotation tasks, KL divergence is introduced into multimodal data to consider the differences between different modalities. , the single-modal variational posterior distribution can be expressed as:

[0099]

[0100]

[0101] in, is the mean, is the variance, is the formula for text modal variational distribution, is a latent variable, is the observation data, is the formula for the modal variational distribution of the image, is a latent variable, For observation data.

[0102] Considering the distribution of the entire data set, we have:

[0103]

[0104]

[0105] Then, the average KL divergence between the different modal unimodal distributions can be calculated for each sample The difference score is calculated as follows:

[0106]

[0107]

[0108]

[0109] Where, Indicates the calculation of KL divergence, sigmoid is the normalization function. Assume that the two probability distributions are A and B, and under the premise of being set as continuous random variables, their corresponding probability density functions are and If you use To approximate , then the calculation formula of KL divergence is as follows:

[0110]

[0111] In order to obtain the test question feature vector, the fusion vector, text feature vector and image feature vector are adaptively concatenated, controlled by the modal difference score. The formula is as follows:

[0112]

[0113] in, is the feature vector of the test question of the i-th sample, is the cross-modal difference score of the i-th sample, is the text feature vector of the i-th sample, is the image feature vector of the i-th sample, is the fusion vector of the i-th sample

[0114] S5: Input the question feature vector into the knowledge point annotation network to obtain the knowledge point label and annotate the question. Specifically:

[0115] The knowledge point annotation network is obtained by training the preset knowledge point annotation network.

[0116] Input the test question feature vector into the knowledge point annotation network, map the test question feature vector to the knowledge point label set, and calculate the use The probability distribution of knowledge points in the layer knowledge point set, select the knowledge point label with the highest probability, mark the knowledge points of the question, and label the knowledge point label. , which is expressed as follows:

[0117]

[0118] in, is the knowledge point label, is the test question feature vector, and b is the bias term.

[0119] In some embodiments, the above-mentioned dual-stream test question feature extraction network, dual-stage cross-modal alignment network, difference perception fusion network, and knowledge point annotation network can be obtained by training a preset self-supervised assisted dual-stream test question feature extraction network, a preset dual-stage cross-modal alignment network, a preset difference perception fusion network, and a preset knowledge point annotation network.

[0120] The following is the network training method, specifically:

[0121] Obtain sample test question data and preset knowledge point labels corresponding to the sample test question data;

[0122] Input the sample test question data into the preset self-supervised assisted dual-stream test question feature extraction network for feature extraction to obtain the text feature vector and image feature vector of the sample test question data;

[0123] Input the text feature vector and image feature vector of the sample test question data into a preset two-stage cross-modal alignment network for alignment, thereby obtaining an aligned text feature vector and an aligned image feature vector of the sample test question data;

[0124] Inputting the aligned text feature vector and the aligned image feature vector of the sample test question data into a preset difference perception fusion network to obtain a fusion vector of the sample test question data;

[0125] Perform weighted calculation on the fusion vector, text feature vector and image feature vector of the sample test question data to obtain the test question feature vector of the sample test question data;

[0126] Input the test question feature vector of the sample test question data into the preset knowledge point labeling network to obtain the knowledge point label of the sample test question data;

[0127] Determine the total loss based on the preset knowledge point labels corresponding to the sample test question data and the knowledge point labels of the sample test question data;

[0128] According to the total loss, the preset self-supervised assisted dual-stream test question feature extraction network, the preset dual-stage cross-modal alignment network, the preset difference perception fusion network and the preset knowledge point labeling network are trained to obtain the self-supervised assisted dual-stream test question feature extraction network, the dual-stage cross-modal alignment network, the difference perception fusion network and the knowledge point labeling network.

[0129] The total loss is expressed as follows:

[0130]

[0131] Among them, β, γ, and δ are hyperparameters. For puzzle loss, is the cross entropy loss, is the cosine loss, is the cross-modal contrastive learning loss.

[0132] The puzzle loss is designed to optimize the image encoder of the self-supervised two-stream test feature extraction network, enabling the image encoder to better learn image features, especially highly structured black and white line image features. The puzzle loss is determined as follows:

[0133] Split the image into Block, each block is recorded as , by The image encoder is trained by randomly shuffling and reassembling the puzzle pieces to learn the feature map of the image and its correct spatial arrangement. puzzle pieces, a total of There are possible permutations. The present invention selects the first 100 permutations with the largest Hamming distance for prediction. Therefore, puzzle prediction can be regarded as a 100-classification problem. The image encoder is optimized by minimizing the cross entropy loss. The puzzle loss is defined as:

[0134]

[0135] in, is the position of the predicted block.

[0136] Cosine loss is used to characterize the alignment relationship between images and text. It measures the similarity between images and text through cosine distance and optimizes the first-stage alignment of the two-stage cross-modal alignment network. If the image and text come from the same test question, they are positively correlated. If they come from different test questions, they are negatively correlated. Cosine loss is defined as:

[0137]

[0138]

[0139] in, To align text feature vectors, is the aligned image feature vector.

[0140] The paper introduces inter-modal contrastive learning and further learns the semantic similarity constraints of different modalities, with the goal of bringing text and images from the same test question closer together and pushing images and text from different test questions further apart. The paper optimizes the second-stage alignment of the two-stage cross-modal alignment network through the cross-modal contrastive learning loss function. A question combination is randomly defined, and the image of the same question as the question text is considered as a positive sample. The image is used as the negative sample pair of the test text, and the cross-modal contrastive learning loss function is as follows:

[0141]

[0142] in, () is the calculated cosine similarity, is the temperature coefficient, used to adjust contrast loss.

[0143] Knowledge point labeling is a multi-classification task, and the cross entropy loss function is used to optimize the knowledge point labeling network:

[0144]

[0145] Among them, c represents the number of knowledge point categories, y represents the preset knowledge point label, The label of the knowledge point.

[0146] In some embodiments, the method of training a preset self-supervised assisted dual-stream question feature extraction network, a preset dual-stage cross-modal alignment network, and a preset difference perception fusion network based on the total loss to obtain a dual-stream question feature extraction network, a dual-stage cross-modal alignment network, a difference perception fusion network, and a knowledge point annotation network includes:

[0147] According to the total loss, determine the puzzle prediction loss, cosine loss, cross-modal contrastive learning loss, and cross entropy loss;

[0148] Based on the puzzle prediction loss, cosine loss, cross-modal contrastive learning loss, and cross-entropy loss, the network parameters of the preset self-supervised assisted two-stream test feature extraction network, the preset two-stage cross-modal alignment network, the preset difference perception fusion network, and the preset point recognition and annotation network are updated;

[0149] The updated preset self-supervised assisted dual-stream test question feature extraction network, the preset two-stage cross-modal alignment network, the preset difference perception fusion network and the preset knowledge point annotation network are iteratively trained until the convergence conditions are reached. The obtained dual-stream test question feature extraction network, two-stage cross-modal alignment network, difference perception fusion network and knowledge point annotation network are used for knowledge point annotation.

[0150] In some embodiments, in order to obtain a network with high accuracy, training is performed using a small-batch stochastic gradient descent (SGD) method.

[0151] Example

[0152] In order to verify the wide application of the invention in different disciplines, this embodiment uses four data sets to implement this embodiment. Due to the lack of publicly available specialized data sets, this embodiment uses a multimodal scientific question-answering data set ScienceQA, which provides a wealth of scientific questions and their answers. In order to be closer to the research objectives of this embodiment, the biology and geography subjects in ScienceQA were selected, and the focus was on those test questions containing graphic information, with category and skill as knowledge points. At the same time, in order to adapt to the Chinese environment, two Chinese physics data sets PHY1 and PHY2 of inconsistent scales were constructed by crawling test questions on educational websites and preprocessing the data, including deduplication and removal of outliers. These two data sets provide data support for this embodiment that is more in line with actual application scenarios. The statistical information of the data set is shown in Table 1. If it is a multiple-choice question, the text input includes the question stem and all options. If it is a fill-in-the-blank question, it is the question stem plus the answer. Figure 7 The distribution of knowledge points in four subject datasets is shown.

[0153] Table 1 Dataset statistics

[0154]

[0155] In order to explain this embodiment more clearly, a test question is randomly selected from the Own-PHY2 data set for understanding and analysis of this embodiment, and this embodiment is further described in detail with reference to the accompanying drawings. The test question is "As shown in the figure, a beam of electrons flows in the positive direction along the z-axis. What is the direction of the magnetic field at point A on the y-axis in the figure? A. +x direction B. -x direction C. +y direction D. -x direction" and the picture information is available at Figure 6 The first-level knowledge point of this test question is "electromagnetism" and the second-level knowledge point is "several common magnetic fields". In this embodiment, before inputting the self-supervised assisted dual-stream test question feature extraction network, the text information is spliced into "As shown in the figure, a beam of electrons flows in the positive direction along the z-axis, then the direction of the magnetic field at point A on the y-axis in the figure is (). +x direction -x direction +y direction -x direction".

[0156] This example uses the multimodal test question knowledge point annotation method provided above to perform knowledge point annotation. This example is implemented using the PyTorch framework, using HuggingFace's Bert-Base-Cased and Bert-Base-Chinese as the text encoders for the Chinese and English datasets, respectively, and Resnet34 as the image encoder.

[0157] In this embodiment, the convolution kernel sizes of TextCNN are set to 1, 2, 4, and 8. The temperature coefficient of contrastive learning is 0.1. During the training process, alignment is trained first and then fusion is trained in each epoch. To prevent overfitting, dropout is set to 0.2. Adam is used as the optimizer. The number of heads of the multi-head attention mechanism is set to 8. When comparing with other baselines, the training set and test set are kept consistent by setting a random seed. The server memory used in this embodiment is 100GB, the CPU is a 20-core Intel(R) Xeon(R) Platinum 8457C, and the GPU is L20 (48GB).

[0158] In order to illustrate the effectiveness and advantages of this embodiment, this embodiment is compared with existing methods. Since there is little research on this task, some general multimodal frameworks and multimodal methods in other fields are selected for comparison. Reference [1]: SpotFake: A Multi-modal Framework for Fake News Detection, this model uses Bert as a text feature extractor and a pre-trained VGG19 as a visual feature extractor, and then simply splices the two modalities for multimodal fake news detection tasks. Reference [2]: Multimodal Fusion with Co-Attention Networks for Fake News Detection, this method first uses three different sub-models to extract features from the spatial domain, frequency domain, and text, and then performs feature fusion through multiple cross-attention layers for multimodal fake news detection modality. Reference [3]: Cross-modal Ambiguity Learning for Multimodal Fake News Detection. This method maps multimodal features to a shared space after extracting text and image features using a feature encoder. From the perspective of information theory, it proposes an ambiguity-aware method to adaptively aggregate unimodal features and cross-modal correlations for multimodal fake news detection. Reference [4]: Multimodal Representation Learning by Alternating Unimodal Adaptation. This model is a general multimodal fusion framework. By converting the traditional joint multimodal learning process into an alternating unimodal learning process, the interference between modalities is minimized. The prediction function of each modality includes a modality-specific encoder and a shared head for all modalities. The shared head introduces a gradient modification mechanism to encourage the orthogonalization of gradient directions between modalities. This embodiment was experimented on 4 subject datasets and compared with the above four references using 7 evaluation indicators. The results are shown in Table 2. The experimental results show that this embodiment performs best on most indicators of the 4 datasets, which also proves that the present invention is interpretable and feasible.

[0159] Table 2 Comparative experimental results of this embodiment and the baseline

[0160]

[0161] In ScienceQA-BIO, this embodiment achieved an accuracy of 0.99 in the knowledge point labeling task, and the accuracy and weighted indicators performed best, but the macro precision and macro F1 value were lower than the MCAN algorithm. Although the sample texts are relatively short, they capture and summarize the key knowledge points in a highly refined manner, so they have a higher accuracy in the knowledge point labeling task. The macro value is lower than the weighted value as a whole because the labels of the dataset are distributed in a long-tail manner, which means that this embodiment performs better in categories with a large number of samples and performs poorly in categories with a small number of samples. In ScienceQA-GEO, almost all indicators are optimal, but the macro precision and weighted precision are the best, which means that this embodiment still has certain limitations when the knowledge points are unevenly distributed, but it also has certain competitiveness. In Own-PHY1, this embodiment performed best in all 7 indicators. In Own-PHY2, except for the macro precision, the other 6 indicators performed best. Spotfake's performance in the four datasets was 20.5%, 5.03%, 22.85%, and 9.41% lower than that of this embodiment, respectively. This suggests that simply concatenating the outputs of encoders from different modalities is not an effective multimodal fusion method. MCAN's accuracy was 0.78% and 3.58% lower than that of this embodiment in ScienceQA-BIO and ScienceQA-GEO, respectively, but 14.65% and 12.59% lower in Own-PHY1 and Own-PHY2. This discrepancy suggests that frequency-domain feature extraction is suitable for natural images but has limited effectiveness in black and white lines.

[0162] In general, this embodiment performs well on most metrics across most datasets, which not only demonstrates the effectiveness of the present invention but also highlights its wide applicability across datasets from different fields.

[0163] In order to verify the effectiveness of the present invention, this embodiment conducted an ablation experiment, and deleted the key components of the system module of the present invention respectively to design three model variants: (1) w / o jsw: The image encoder is not trained using the puzzle prediction task of the self-supervised dual-stream question feature extraction network. (2) w / o cl: The two-stage cross-modal alignment network only performs the first-stage alignment task and does not perform the second-stage alignment task. (3) w / o both: The puzzle prediction task of the self-supervised dual-stream question feature extraction network and the second-stage alignment task of the two-stage cross-modal alignment network are removed at the same time. The accuracy comparison of the ablation experiments on different datasets is as follows: Figure 8As shown. Analysis shows that after deleting the self-supervised auxiliary task in the two-stream feature extraction network and the second-stage alignment task in the two-stage cross-modal alignment network, the accuracy of knowledge point classification in the four data sets is 1.75%, 1.56%, 2.73%, and 0.52% lower than the method respectively, which shows that the two networks can improve the effectiveness of multimodal knowledge point labeling tasks. When the jigsaw prediction task and the contrastive learning alignment task are removed, it can be seen that the accuracy of the two networks in ScienceQA-BIO is the same. In ScienceQA-GEO, the result of removing the contrastive learning task is better than the result of removing the jigsaw prediction task. In Own-PHY1, the result of removing the jigsaw prediction task is better than the result of removing the contrastive learning task. In Own-PHY2, removing any task is lower than the accuracy of this embodiment, indicating that the two networks perform differently in different data sets and there will be certain limitations, so a balance coefficient is needed to balance the relationship between the two networks.

[0164] like Figure 9 As shown, another object of the present invention is to provide a multimodal test question knowledge point annotation system, comprising:

[0165] The test question feature acquisition module is used to extract features from the test question data through a self-supervised dual-stream test question feature extraction network to obtain the text feature vector and image feature vector of the test question;

[0166] A feature alignment module is used to input the text feature vector and the image feature vector into a two-stage cross-modal alignment network for alignment, thereby obtaining an aligned text feature vector and an aligned image feature vector.

[0167] A feature fusion module is used to input the aligned text feature vector and the aligned image feature vector into the difference perception fusion network to obtain a fusion vector;

[0168] The feature weighting module is used to perform weighted calculation on the fusion vector, text feature vector and image feature vector to obtain the test question feature vector;

[0169] The knowledge point annotation module is used to input the test question feature vector into the knowledge point annotation network, obtain the knowledge point label, and annotate the question.

[0170] like Figure 10 As shown, the third object of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multimodal test question knowledge point labeling method when executing the computer program.

[0171] The test data is subjected to feature extraction through a self-supervised dual-stream test feature extraction network to obtain the text feature vector and image feature vector of the test question;

[0172] Input the text feature vector and the image feature vector into the two-stage cross-modal alignment network for alignment to obtain the aligned text feature vector and the aligned image feature vector;

[0173] Input the aligned text feature vector and the aligned image feature vector into the difference-aware fusion network to obtain a fusion vector;

[0174] Perform weighted calculation on the fusion vector, text feature vector and image feature vector to obtain the test question feature vector;

[0175] The test question feature vector is input into the knowledge point annotation network to obtain the knowledge point label and annotate the question.

[0176] The fourth object of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal test question knowledge point labeling method are implemented.

[0177] The test data is subjected to feature extraction through a self-supervised dual-stream test feature extraction network to obtain the text feature vector and image feature vector of the test question;

[0178] Input the text feature vector and the image feature vector into the two-stage cross-modal alignment network for alignment to obtain the aligned text feature vector and the aligned image feature vector;

[0179] Input the aligned text feature vector and the aligned image feature vector into the difference-aware fusion network to obtain a fusion vector;

[0180] Perform weighted calculation on the fusion vector, text feature vector and image feature vector to obtain the test question feature vector;

[0181] The test question feature vector is input into the knowledge point annotation network to obtain the knowledge point label and annotate the question.

[0182] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0184] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for labeling knowledge points in multimodal test questions, characterized in that: include: The test data is subjected to feature extraction through a self-supervised dual-stream test feature extraction network to obtain the text feature vector and image feature vector of the test question; Input the text feature vector and the image feature vector into the two-stage cross-modal alignment network for alignment to obtain the aligned text feature vector and the aligned image feature vector; Input the aligned text feature vector and the aligned image feature vector into the difference-aware fusion network to obtain a fusion vector; The fusion vector, text feature vector and image feature vector are weighted and calculated to obtain the test question feature vector, specifically: The fusion vector, text feature vector, and image feature vector are weighted based on KL divergence to obtain the test feature vector, including: For test questions , the variational posterior distribution of a single modality is expressed as: in, is the mean, is the variance, is the formula for text modal variational distribution, is a latent variable, is the observation data, is the formula for the modal variational distribution of the image, is a latent variable, is the observation data; Based on the distribution of the entire data set, we have: Each sample is calculated by the average KL divergence between the different modal unimodal distributions The difference score is calculated as follows: Where, is the cross-modal difference score of the i-th sample, Represents the calculation of KL divergence, sigmoid is the normalization function; Controlled by the difference score of the modalities, the fusion vector, text feature vector, and image feature vector are adaptively concatenated to obtain the test feature vector. The formula is as follows: in, is the feature vector of the test question of the i-th sample, is the text feature vector of the i-th sample, is the image feature vector of the i-th sample, is the fusion vector of the i-th sample; The test question feature vector is input into the knowledge point annotation network to obtain the knowledge point label and annotate the question.

2. A multimodal test question knowledge point annotation method according to claim 1, characterized in that: The self-supervised assisted dual-stream test question feature extraction network includes an information extraction layer, a text feature extraction layer, and an image feature extraction layer. The test question data is subjected to feature extraction through the self-supervised assisted dual-stream test question feature extraction network to obtain a text feature vector and an image feature vector of the test question, including: Input the test data into the information extraction layer to obtain text information and image information; Input text information into the text feature extraction layer to obtain text feature vector; The image information is input into the image feature extraction layer to obtain the image feature vector.

3. A multimodal test question knowledge point annotation method according to claim 1, characterized in that: The dual-stage cross-modal alignment network includes a unified layer and a shared layer. The text feature vector and the image feature vector are input into the dual-stage cross-modal alignment network for alignment to obtain an aligned text feature vector and an aligned image feature vector, including: Input the text feature vector and the image feature vector into the unification layer to unify the dimensions and obtain the unified text feature vector and image feature vector; The unified text feature vector and image feature vector are input into the shared layer to obtain the aligned text feature vector and aligned image feature vector.

4. A multimodal test question knowledge point annotation method according to claim 1, characterized in that: The step of inputting the aligned text feature vector and the aligned image feature vector into a difference perception fusion network to obtain a fusion vector includes: The aligned text feature vector and the aligned image feature vector are input into the difference-aware fusion network. Through cross-attention, the aligned text feature vector and the aligned image feature vector are semantically interacted across modalities, and the aligned image features are used to guide the attention distribution of the aligned text features. After the aligned text feature vector and the aligned image feature vector are fused, a fusion vector is obtained.

5. A multimodal test question knowledge point annotation method according to claim 1, characterized in that: The step of inputting the test question feature vector into the knowledge point annotation network to obtain the knowledge point label and annotate the question includes: The test question feature vector is input into the knowledge point labeling network, the test question feature vector is mapped to the knowledge point label set, the probability distribution of the knowledge points in the knowledge point set is calculated using the softmax layer, the knowledge point label with the highest probability is selected, and the knowledge points of the question are labeled.

6. A multimodal test question knowledge point annotation method according to claim 1, characterized in that: The method further comprises: Obtain sample test question data and preset knowledge point labels corresponding to the sample test question data; Input the sample test question data into the preset self-supervised assisted dual-stream test question feature extraction network for feature extraction to obtain the text feature vector and image feature vector of the sample test question data; Input the text feature vector and image feature vector of the sample test question data into a preset two-stage cross-modal alignment network for alignment, thereby obtaining an aligned text feature vector and an aligned image feature vector of the sample test question data; Inputting the aligned text feature vector and the aligned image feature vector of the sample test question data into a preset difference perception fusion network to obtain a fusion vector of the sample test question data; Perform weighted calculation on the fusion vector, text feature vector and image feature vector of the sample test question data to obtain the test question feature vector of the sample test question data; Input the test question feature vector of the sample test question data into the preset knowledge point labeling network to obtain the knowledge point label of the sample test question data; Determine the total loss based on the preset knowledge point labels corresponding to the sample test question data and the knowledge point labels of the sample test question data; According to the total loss, the preset self-supervised assisted dual-stream test question feature extraction network, the preset dual-stage cross-modal alignment network, the preset difference perception fusion network and the preset knowledge point labeling network are trained to obtain the self-supervised assisted dual-stream test question feature extraction network, the dual-stage cross-modal alignment network, the difference perception fusion network and the knowledge point labeling network.

7. A multimodal test question knowledge point annotation system, characterized by: include: The test question feature acquisition module is used to extract features from the test question data through a self-supervised dual-stream test question feature extraction network to obtain the text feature vector and image feature vector of the test question; A feature alignment module is used to input the text feature vector and the image feature vector into a two-stage cross-modal alignment network for alignment, thereby obtaining an aligned text feature vector and an aligned image feature vector. A feature fusion module is used to input the aligned text feature vector and the aligned image feature vector into the difference perception fusion network to obtain a fusion vector; The feature weighting module is used to perform weighted calculation on the fusion vector, text feature vector and image feature vector to obtain the test question feature vector, specifically: The fusion vector, text feature vector, and image feature vector are weighted based on KL divergence to obtain the test feature vector, including: For test questions , the variational posterior distribution of a single modality is expressed as: in, is the mean, is the variance, is the formula for text modal variational distribution, is a latent variable, is the observation data, is the formula for the modal variational distribution of the image, is a latent variable, is the observation data; Based on the distribution of the entire data set, we have: Each sample is calculated by the average KL divergence between the different modal unimodal distributions The difference score is calculated as follows: Where, is the cross-modal difference score of the i-th sample, Represents the calculation of KL divergence, sigmoid is the normalization function; Controlled by the difference score of the modalities, the fusion vector, text feature vector, and image feature vector are adaptively concatenated to obtain the test feature vector. The formula is as follows: in, is the feature vector of the test question of the i-th sample, is the text feature vector of the i-th sample, is the image feature vector of the i-th sample, is the fusion vector of the i-th sample; The knowledge point annotation module is used to input the test question feature vector into the knowledge point annotation network, obtain the knowledge point label, and annotate the question.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multimodal test question knowledge point annotation method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the multimodal test question knowledge point annotation method according to any one of claims 1 to 6.

Images