Question knowledge point labeling method and system based on multiple modes

Abstract

The invention discloses a multi-mode-based question knowledge point marking method and system, and the method comprises the steps: obtaining original data of the education field, and enabling the original data to be from a physical entity; performing cleaning and enhancement processing on the original data to obtain enhanced training data; based on the enhanced training data, diversified model preferences are established by constructing differentiated data, a collaborative attention mechanism and a gating mechanism are utilized to fuse text and visual features for reasoning, and a multi-modal reasoning result is obtained; and performing integrated prediction according to the multi-modal reasoning result to obtain a question knowledge point label. According to the method, diversified model preferences are established by constructing differentiated data, and interaction of cross-modal information is enhanced by introducing a common attention mechanism and a gating mechanism, so that the model can fuse text and visual features more effectively, and the accuracy and applicability of knowledge point labeling are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of topic knowledge point annotation in the education field, and particularly relates to a multi-modal based topic knowledge point annotation method and system. Background Art

[0002] With the rapid development of information technology, multi-modal technology has been widely applied in various fields. Multi-modal technology refers to obtaining and processing information through multiple senses or multiple data types, and its core lies in fusing data of different modalities to obtain more comprehensive and accurate information. This technology has demonstrated great potential and value in multiple industries such as education, medical care, industry, and entertainment. For example, in the education field, multi-modal technology can combine various forms of teaching resources such as text, images, audio, and video to provide students with a richer and more personalized learning experience. In terms of topic annotation, the application of multi-modal technology is still in the development stage. Currently, most topic annotations mainly rely on a single text modality, that is, relevant knowledge points, problem-solving ideas, and key examination points are annotated through text descriptions.

[0003] However, the existing topic annotation methods mainly rely on a single text modality and have obvious limitations. On the one hand, it is difficult to comprehensively and accurately convey the knowledge and application scenarios involved in the topic only through text descriptions. Especially for topics involving complex structures, graphics, charts, or audio, the lack of intuitive multi-modal presentation may affect understanding and mastery. On the other hand, single-modal annotation is difficult to meet the needs of personalized learning. Different learners may have different abilities to receive and understand information in different modalities, and a single text modality is difficult to adapt to diverse learning styles. In addition, from the perspective of technical logic analysis, the existing technologies have the following main disadvantages: incomplete information representation, single-modal annotation can only capture local features of knowledge points and cannot fuse cross-modal complementary information; semantic fragmentation between modalities, lack of cross-modal alignment and fusion mechanisms, resulting in insufficient semantic consistency in knowledge point annotation; limited generalization and robustness, single-modal annotation models are vulnerable when facing data loss, noise interference, or scene changes, and it is difficult to utilize the advantages of multi-modal information complementarity. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a multi-modal based topic knowledge point annotation method and system to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above object, in the first aspect, the present invention provides a multi-modal based topic knowledge point annotation method, including:

[0006] Obtain the original data in the education field, wherein the original data comes from physical entities;

[0007] Clean and enhance the original data to obtain enhanced training data;

[0008] Based on the enhanced training data, split the training set and validation set with different random seeds to construct differentiated data, and use multiple models to construct differentiated models, thereby establishing diverse model preferences, and use the collaborative attention mechanism and gating mechanism to fuse text and visual features for inference to obtain multimodal inference results;

[0009] According to the multimodal inference results, perform integrated prediction to obtain the topic knowledge point labels.

[0010] Preferably, the steps of obtaining the enhanced training data include:

[0011] Filter the original data based on regular expressions to remove irrelevant noise and obtain the cleaned text;

[0012] Perform data augmentation on the cleaned text, and through operations such as synonym replacement, random deletion or random insertion, increase the diversity of samples and expand the size of the dataset to obtain the enhanced training data.

[0013] Preferably, the steps of obtaining the multimodal inference results include:

[0014] Use the derived model to extract features from the text data to obtain text-based inference logits;

[0015] Use the pre-trained text model and visual model to extract text features and visual features respectively, and fuse them through the collaborative attention mechanism to obtain the fused features;

[0016] Input the fused features into the gating mechanism for adjustment to obtain the gated fused features;

[0017] Perform inference based on the gated fused features to obtain the multimodal inference results.

[0018] Preferably, the steps of obtaining the text-based inference logits include:

[0019] Use the BERT, RoBERTa, and XLNet derived models to extract features from the text data respectively to obtain the text features of different models;

[0020] Input the text features of different models into the linear layer for inference to obtain the text-based inference logits.

[0021] Preferably, the steps of obtaining the fused features include:

[0022] Use the pre-trained RoBERTa model to extract text features;

[0023] Extract visual features using a pre-trained ViT-16 model and perform zero-padding to match the dimensions;

[0024] Project the text features and visual features into the same dimensional space, and calculate the cross-modal interrelationships through a co-attention mechanism to obtain the fused features.

[0025] Preferably, the step of calculating the cross-modal interrelationships through a co-attention mechanism to obtain the fused features includes:

[0026] Use the text features as query vectors, the visual features as key vectors and value vectors, and calculate the cross-modal attention weights through an attention function;

[0027] Weightedly fuse the text features and visual features according to the attention weights to obtain attention-based fused features;

[0028] Adopt skip connections to concatenate the attention-based fused features with the original features to obtain enhanced fused features.

[0029] Preferably, the step of obtaining the gated fused features includes:

[0030] Calculate the contributions of the fused text features and visual features to the gating scores;

[0031] Weightedly fuse the fused text features and visual features according to the gating scores to obtain the gated fused features.

[0032] Preferably, the step of obtaining the topic knowledge point labels includes:

[0033] Input the multi-modal inference results into the Softmax layer to obtain the probability distribution of each knowledge point;

[0034] Select the knowledge point with the highest probability value according to the probability distribution as the final topic knowledge point label.

[0035] Preferably, the method further includes:

[0036] When dealing with missing modal data, introduce a zero-tensor padding strategy to maintain the consistency of the input dimensions.

[0037] In a second aspect, the present invention provides a multi-modal based topic knowledge point annotation system for implementing the method described in the first aspect, including:

[0038] A data acquisition module for acquiring raw data in the education field, wherein the raw data comes from physical entities;

[0039] A data processing module for cleaning and enhancing the raw data to obtain enhanced training data;

[0040] A model inference module, which is used to split the training set and the validation set with different random seeds based on the enhanced training data, construct differential data, use multiple models to construct differential models, thereby establishing diversified model preferences, and use a collaborative attention mechanism and a gating mechanism to fuse text and visual features for inference to obtain multimodal inference results;

[0041] An integrated prediction module, which is used to perform integrated prediction according to the multimodal inference results to obtain topic knowledge point labels.

[0042] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0043] Compared with the prior art, the present invention has the following advantages and technical effects:

[0044] The present invention provides a method for annotating topic knowledge points based on multimodality, including: first, obtaining original data in the education field, where the original data comes from physical entities; second, cleaning and enhancing the original data to obtain enhanced training data; then, based on the enhanced training data, establishing diversified model preferences by constructing differential data, and using a collaborative attention mechanism and a gating mechanism to fuse text and visual features for inference to obtain multimodal inference results; finally, performing integrated prediction according to the multimodal inference results to obtain topic knowledge point labels.

[0045] The present invention combines a model preference-driven strategy, effectively utilizes image features, and improves the accuracy and effect of knowledge point annotation. The present invention can effectively handle graphic and text tasks in complex education scenarios. In addition, by constructing differential data to establish diversified model preferences, introducing a co-attention mechanism and a gating mechanism, enhancing the interaction of cross-modal information, enabling the model to more effectively fuse text and visual features, and improving the accuracy and applicability of knowledge point annotation. Description of the Drawings

[0046] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0047] Figure 1 It is a schematic diagram of the architecture of the method for annotating topic knowledge points based on multimodality according to an embodiment of the present invention;

[0048] Figure 2 It is a schematic diagram of the system according to an embodiment of the present invention;

[0049] Figure 3 Schematic diagram of a computer-readable storage medium according to an embodiment of the present invention. Detailed implementation manners

[0050] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the drawings and in conjunction with the embodiments.

[0051] It should be noted that the steps shown in the flowchart of the drawings may be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than here.

[0052] Embodiment 1

[0053] Different from the input data of the traditional multi-modal model CLIP, the input data of this embodiment is mainly the questions in the education field. The text content is not a simple description of the image, and there is a one-to-many text-image matching situation. This characteristic makes it difficult for the model to directly utilize the image features, greatly increasing the task difficulty. Therefore, compared with the traditional simple text-image matching task, the multi-modal fusion in this embodiment is more challenging, and at the same time, it is more valuable for research and practical significance.

[0054] As Figure 1 shown, this embodiment provides a method for annotating knowledge points of questions based on multi-modal, which includes three parts: Data Augmentation, Preference-Based Inference, and Ensemble Prediction. This embodiment combines the model preference-driven strategy, effectively utilizes the image features, and improves the accuracy and effect of knowledge point annotation.

[0055] Specifically, it includes:

[0056] S1. Obtain the original data in the education field, where the original data comes from physical entities;

[0057] Specifically, for a given multi-modal question , its associated knowledge concept label is , where represents the corresponding text content (such as the question and the answer), represents the image corresponding to this exercise. represents the number of categories of knowledge concepts. The goal of the multi-modal knowledge concept annotation task is to assign a prediction label to .

[0058] S2. Clean and enhance the original data to obtain enhanced training data;

[0059] Further, the steps of obtaining the enhanced training data include:

[0060] S201. Filter the original data based on regular expressions to remove irrelevant noise and obtain the cleaned text;

[0061] S202. Perform data augmentation on the cleaned text. By means of synonym replacement, random deletion, or random insertion operations, increase the diversity of samples and expand the size of the dataset to obtain the enhanced training data.

[0062] Specifically, since the original data may contain irrelevant noise, in this embodiment, noise is filtered out based on regular expressions to obtain the cleaned text. This embodiment performs data augmentation on the input practice text. By selecting one operation from synonym replacement, random deletion, or random insertion, increase the diversity of samples and expand the size of the dataset. To avoid introducing too much noise to hinder semantic understanding, this embodiment augments 10% of the text data and combines the augmented data with the cleaned data to construct an enhanced training set.

[0063] As an additional implementation manner, in this embodiment, when dealing with missing modality data, a zero-tensor filling strategy is introduced to maintain the consistency of the input dimension.

[0064] Specifically, to handle the case of non-multi-modal data, this embodiment introduces a zero-tensor filling strategy when dealing with missing modalities to maintain the consistency of the input dimension. If text is missing, the text input part is input with a zero-tensor of shape [batch_size, 768]; if an image is missing, the image input part is input with a zero-tensor of shape [batch_size, 1536]. This method ensures that the model can still operate normally when facing single-modal data, avoids affecting the overall performance due to missing data modalities, and thus improves the robustness and generalization ability of the model.

[0065] S3. Based on the enhanced training data, split the training set and the validation set with different random seeds, construct differentiated data, and use multiple models to construct differentiated models, thereby establishing diverse model preferences, and use the co-attention mechanism and the gating mechanism to fuse text and visual features for reasoning to obtain multi-modal reasoning results;

[0066] This embodiment believes that due to the deviation between data and the base model, different preference models will vary. To further enhance the diversity and difference between models, thereby improving the effect of ensemble learning or model fusion, this embodiment constructs differentiated preference models in multiple ways. By setting different random seeds to divide the training set and the validation set, data subsets with distribution differences are constructed. At the same time, multiple pre-trained models with different structures (such as BERT, RoBERTa, XLNet) are introduced to construct differentiated preference models from both the data and model levels. This can effectively enhance the diversity between models and provide support for subsequent fusion and optimization.

[0067] Further, the steps of obtaining the multi-modal inference result include:

[0068] S301. Use the pre-trained text model and visual model to extract text features and visual features respectively, and fuse them through the co-attention mechanism to obtain the fused features;

[0069] The steps of obtaining the fused features include:

[0070] (1) Use the pre-trained RoBERTa model to extract text features;

[0071] (2) Use the pre-trained ViT-16 model to extract visual features and perform zero-padding to match the dimensions;

[0072] (3) Project the text features and visual features into the same dimensional space, and calculate the cross-modal mutual relationship through the co-attention mechanism to obtain the fused features.

[0073] The steps of extracting text features include:

[0074] Use derivative models such as BERT, RoBERTa, and XLNet to extract features from the text data respectively to obtain text features of different models;

[0075] Input the text features of different models into a linear layer for inference to obtain text-based inference logits.

[0076] Specifically, for the text, given the input text , this embodiment uses the derivative model to obtain its features, that is, , and then input the feature into a linear layer for inference, that is, . (To avoid ambiguity, this embodiment omits the subscript to simplify the expression.) For example, the feature representation of encoded by the th derivative pre-trained model is ​. Subsequently, this feature is input into the linear layer for inference, obtaining , where .

[0077] Therefore, this embodiment utilizes these derived models to obtain multi-perspective text-based inference logits, which are calculated as follows:

[0078] (1)

[0079] (2)

[0080]

[0081] Where, represents the set of text-based inference logits of the BERT derived model, represents the set of text-based inference logits of the RoBERTa derived model, represents the set of text-based inference logits of the XLNet derived model. To maintain consistency, this embodiment represents these outputs as , where represents the logit of the th derived model, and set .

[0082] Specifically, for the text-image multi-modal data, given the input multi-modal exercise , this embodiment adopts the pre-trained RoBERTa model as the backbone network to extract text features . Similarly, this embodiment uses the pre-trained ViT-16 model to extract visual features and pads them with zeros to represent them as for subsequent residual connections. Subsequently, these two features are fused through the co-attention mechanism, which aims to capture the mutual relationship and complementary information between the features of the two different modalities. By calculating the attention weights of and , this mechanism realizes co-analysis and weighted fusion. Specifically, this embodiment projects the text and visual feature representations into the same dimensional space, which is represented as follows:

[0083]

[0084]

[0085] Where, and are used to obtain the text feature The weight parameter, and is the weight parameter for obtaining visual features .

[0086] In this embodiment, the steps of calculating the cross-modal mutual relationship through the collaborative attention mechanism to obtain the fused features include: First, taking the text features as the query vector, the visual features as the key vector and the value vector, and calculating the cross-modal attention weights through the attention function; Second, performing weighted fusion on the text features and the visual features according to the attention weights to obtain the attention-based fused features; Finally, using skip connection to splice the attention-based fused features with the original features to obtain the enhanced fused features.

[0087] Specifically, this embodiment uses the collaborative attention mechanism to calculate the cross-modal mutual relationship, where is used as the query, is used as the key and the value. Specifically, this embodiment obtains the attention-based image and text features in the following manner :

[0088]

[0089]

[0090] Among them, , represents the dimension of the query vector. Att represents the attention function, and Q, K, V represent the query, key, and value vectors respectively. To improve the reusability of the features, this embodiment further uses skip connection to obtain the enhanced fused features , and its calculation method is as follows: Among them, , [;] represents the feature concatenation operation.

[0091] S302. Input the fused features into the gating mechanism for adjustment to obtain the gated fused features;

[0092] Furthermore, the steps of obtaining the gated fused features include:

[0093] Calculating the contributions of the fused text features and visual features to the gating scores;

[0094] Performing weighted fusion on the fused text features and visual features according to the gating scores to obtain the gated fused features.

[0095] Specifically, and are input into a gating mechanism to analyze and regulate the two input feature representations, and determine and the contribution of each feature dimension in to the gating score, which is calculated as follows:

[0096]

[0097] where , represents the Sigmoid function. and are trainable parameters. This enables selective information filtering and fuses the two features as follows:

[0098]

[0099] where represents the gated fusion feature, represents the Hadamard product between two vectors.

[0100] S304. Perform inference based on the gated fusion feature to obtain a multimodal inference result.

[0101] Specifically, a fully connected layer is finally applied to obtain a multimodal inference logit vector, which is calculated as follows:

[0102]

[0103] where , , . Note that setting can make the result consistent with text-based inference. That is, here and above are combined to obtain the final output logit vector .

[0104] S4. According to the multimodal inference result, perform integrated prediction to obtain the topic knowledge point label.

[0105] Furthermore, the steps to obtain the topic knowledge point label include:

[0106] S401. Input the multimodal inference result into the Softmax layer to obtain the probability distribution of each knowledge point;

[0107] S402. Select the knowledge point with the highest probability value according to the probability distribution as the final question knowledge point label.

[0108] Specifically, given the output logit vector obtained from the multi-perspective preference reasoning stage , in this embodiment, they are input into the Softmax layer as follows:

[0109]

[0110] where represents the corresponding probability.

[0111] Finally, calculate the sum of all to obtain the integrated logit vector , and then take the index of the element with the highest probability value as the predicted category as follows:

[0112]

[0113]

[0114] where , representing the probability distribution of the final c category knowledge points. is the final knowledge point label. For example, is the knowledge point label output by a certain question after passing through this system (such as the properties of a square).

[0115] In summary, in this embodiment, various strategies such as regular expressions, HTML parsing, and string processing are first used to clean the data, and text augmentation techniques are used to expand the data scale. Then, by constructing differential data to establish diverse model preferences, combined with the co-attention mechanism and introducing the gating mechanism, text and visual features are fully integrated for reasoning. The present invention not only significantly improves the effect of knowledge point annotation but also has wide applicability.

[0116] This embodiment is suitable for the graphic-text matching method in complex educational scenarios, performs excellently in the educational scenario dataset, enables it to better handle complex graphic-text relationships, and has good effects and generalization ability.

[0117] Embodiment 2

[0118] As Figure 2 shown, this embodiment also discloses a multi-modal question knowledge point annotation system for implementing the method described in Embodiment 1, including:

[0119] A data acquisition module for acquiring raw data in the field of education, wherein the raw data comes from physical entities;

[0120] A data processing module for cleaning and enhancing the raw data to obtain enhanced training data;

[0121] A model inference module for splitting the training set and the validation set with different random seeds based on the enhanced training data, constructing differentiated data, using multiple models to construct differentiated models, thereby establishing diversified model preferences, and using a collaborative attention mechanism and a gating mechanism to fuse text and visual features for inference to obtain multi-modal inference results;

[0122] An integrated prediction module for performing integrated prediction based on the multi-modal inference results to obtain question knowledge point labels.

[0123] The multi-modal question knowledge point annotation system provided in this embodiment has all the advantages of Embodiment 1.

[0124] Embodiment 3

[0125] As Figure 3 shown, this embodiment also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Embodiment 1 are implemented.

[0126] The above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-modal based method for marking problem knowledge points, characterized in that, It includes the following steps: Obtain the original data in the education field, where the original data comes from physical entities; Clean and enhance the original data to obtain enhanced training data; Based on the enhanced training data, use different random seeds to split the training set and the validation set, construct differential data, and use multiple models to construct differential models, thereby establishing diverse model preferences, and utilize the co-attention mechanism and the gating mechanism to fuse text and visual features for reasoning to obtain multi-modal reasoning results; According to the multi-modal reasoning results, perform integrated prediction to obtain the topic knowledge point labels.

2. The method according to claim 1, wherein: The step of obtaining the enhanced training data includes: Filter the original data based on regular expressions to remove irrelevant noise and obtain the cleaned text; Perform data augmentation on the cleaned text, and through operations such as synonym replacement, random deletion, or random insertion, increase the diversity of samples and expand the size of the dataset to obtain the enhanced training data.

3. The method according to claim 1, wherein: The step of obtaining the multi-modal reasoning results includes: Use pre-trained text models and visual models to extract text features and visual features respectively, and fuse them through the co-attention mechanism to obtain the fused features; Input the fused features into the gating mechanism for adjustment to obtain the gated fused features; Perform reasoning based on the gated fused features to obtain the multi-modal reasoning results.

4. The method according to claim 3, wherein: The step of extracting text features includes: Use BERT, RoBERTa, and XLNet-derived models respectively on the text data for feature extraction to obtain text features of different models; Input the text features of different models into the linear layer for reasoning to obtain the text-based reasoning logits.

5. The method according to claim 3, wherein: The step of obtaining the fused features includes: Use a pre-trained RoBERTa model to extract text features; Use a pre-trained ViT-16 model to extract visual features and perform zero-padding to match the dimensions; Project the text features and visual features into the same dimensional space, and calculate the cross-modal mutual relationship through the co-attention mechanism to obtain the fused features.

6. The method according to claim 5, wherein: The step of calculating the cross-modal mutual relationship through the co-attention mechanism to obtain the fused features includes: Use the text features as the query vector, and the visual features as the key vector and value vector, and calculate the cross-modal attention weights through the attention function; Perform weighted fusion on the text features and visual features according to the attention weights to obtain the attention-based fused features; Adopt skip connection to splice the attention-based fused features with the original features to obtain the enhanced fused features.

7. The method according to claim 3, wherein: The step of obtaining the gated fused features includes: Calculate the contributions of the fused text features and visual features to the gating scores; The fused text features and visual features are weighted and fused according to the gating score to obtain gating fusion features.

8. The method according to claim 1, wherein the steps of obtaining the topic knowledge point labels include: inputting the multi-modal reasoning result into the Softmax layer to obtain the probability distribution of each knowledge point; selecting the knowledge point with the highest probability value according to the probability distribution as the final topic knowledge point label.

9. A multi-modal based question knowledge point annotation system, characterized in that, The apparatus for implementing the method according to any one of claims 1-8 includes: a data acquisition module, configured to acquire original data in the field of education, wherein the original data comes from physical entities; a data processing module, configured to clean and enhance the original data to obtain enhanced training data; a model reasoning module, configured to, based on the enhanced training data, split the training set and the validation set through different random seeds, construct differential data, and use multiple models to construct differential models, thereby establishing diversified model preferences, and performing reasoning by fusing text and visual features using a co-attention mechanism and a gating mechanism to obtain a multi-modal reasoning result; an integrated prediction module, configured to perform integrated prediction according to the multi-modal reasoning result to obtain topic knowledge point labels.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the steps of the method according to any one of claims 1-8 are implemented.

Images