Multi-modal fusion mathematical test question multi-knowledge-point classification method

Abstract

The invention discloses a multi-modal fusion mathematical test question multi-knowledge point classification method, and relates to the field of knowledge point classification, and the method comprises the steps: extracting formula features of mathematical test questions; extracting text features of the mathematical test questions; extracting image features of the mathematical test questions; obtaining image feature representation guided by text features of the mathematical test questions; obtaining text feature representation guided by image features of the mathematical test questions; fusing the image feature representation guided by the text feature of the mathematical test question and the text feature representation guided by the image feature of the mathematical test question based on a multi-modal gating mechanism to obtain a fused feature; filtering the fused features to obtain filtered features; and on the basis of the filtered features, a knowledge point classification result is obtained by adopting KNN classification enhanced by joint contrast learning. According to the method, the accuracy and universality of mathematical test question classification can be effectively improved, and an effective technical solution is provided for tasks such as intelligent test paper composition and personalized test question recommendation in the field of intelligent education.

Description

Technical Field

[0001] The present application relates to the field of knowledge point classification, and in particular to a multi-modal fusion method for classifying multiple knowledge points in mathematics test questions. Background Art

[0002] At present, using artificial intelligence technology to deeply understand diversified online learning resources has become a key research topic in intelligent education. Mathematical test questions, as one of the most basic data units in a large number of learning resources, are a key factor in evaluating the effectiveness of mathematics teaching. Due to the complex semantics and structure of online mathematics test question data, traditional NLP classification methods are no longer applicable.

[0003] Mathematical test questions have unique concepts and logic. They not only contain a large number of professional terms and mathematical symbols in the text mode, but also show intuitive mathematical semantics such as quantity and geometry in the image mode. Both types of information provide important clues for understanding mathematical concepts and knowledge points. Therefore, in the study of multi-knowledge point classification of mathematical test questions, the integration of mathematical text and image information is the key to understanding the conceptuality and logic of mathematical test questions. The logical structure and symbol system in mathematical test questions, as well as their close relationship with knowledge point labels, are not available in other subjects. At present, a large number of studies have ignored the extraction of semantic information of test questions containing mathematical images, which is also crucial for understanding the content of test questions. Therefore, if the classification of mathematical test question data is only regarded as a multi-label text classification task, the classification results still have room for improvement in effectiveness. In addition to the shortcomings of mathematical test questions in semantic extraction, the hierarchy of knowledge points is also a factor that cannot be ignored. Test questions are usually compiled based on the content of textbooks, and the content of textbooks is organized according to levels such as chapters. Therefore, the knowledge points also present a clear hierarchical structure from elementary to advanced. After being refined by level, the number of knowledge points is also becoming more and more abundant. As the level increases, the increase in the number of knowledge points places higher demands on the performance of the classification algorithm and also brings new challenges. Summary of the invention

[0004] The purpose of this application is to provide a multi-modal fusion method for classifying multiple knowledge points in mathematics test questions, effectively integrating local information and contextual logic of mathematics test questions, and performing vector representation for text with formulas, so as to more accurately capture the knowledge points tested in mathematics test questions.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] This application provides a multi-modal fusion mathematics test question multi-knowledge point classification method, including:

[0007] Extract formula features of math test questions;

[0008] Extract text features of math test questions;

[0009] Extract image features of math test questions;

[0010] Obtaining an image feature representation guided by text features of the math test question;

[0011] Obtaining a text feature representation guided by the image features of the math test question;

[0012] Based on a multimodal gating mechanism, the image feature representation guided by the text feature of the math test question and the text feature representation guided by the image feature of the math test question are fused to obtain fused features;

[0013] Filtering the fused features to obtain filtered features;

[0014] Based on the filtered features, a knowledge point classification result is obtained.

[0015] Optionally, the extracting formula features of the math test questions specifically includes the following steps:

[0016] Converting a mathematical formula into a symbol appearance tree and an operator tree; the symbol appearance tree represents the layout structure of the mathematical formula, and the operator tree represents the operation relationship of the mathematical formula;

[0017] Depth-first traversal of the symbol appearance tree and the operator tree to generate a tuple (n1, n2, E, ERP); wherein n1 represents an ancestor node, n2 represents a descendant node, E represents an edge sequence from n1 to n2, and FRP represents a complete path relative to the ancestor node;

[0018] The tuple is input into the trained FastText model to obtain a formula vector.

[0019] Optionally, the extracting text features of mathematics test questions specifically includes the following steps:

[0020] Segment the math test text to obtain a sequence after segmentation;

[0021] Input the segmented sequence into the trained RoBERTa model to obtain the word vector of the math test text;

[0022] Input the formula vector and word vector into the SA-TCNN module to generate a test question text feature map;

[0023] The test question text feature maps are spliced ​​to obtain a test question text feature vector representation.

[0024] Optionally, the test question text feature graphs are concatenated to obtain a test question text feature vector representation using the following formula:

[0025]

[0026] in, Represents the test text feature vectors corresponding to different convolution kernel windows.

[0027] Optionally, the extraction of image features of math test questions specifically adopts the following formula:

[0028]

[0030] Among them, v I represents the image feature vector, W I represents the weight matrix of the linear layer transformation, represents the image vector input, b I Represents the bias vector.

[0031] Optionally, the image feature representation guided by the text features of the math test question is obtained by using the following formula:

[0032]

[0033] Among them, z j represents the output of the jth test sample, represents the weight matrix, Represents the original text features, represents the bias vector, represents vector concatenation, represents the weight matrix, v I represents the image feature vector guided by text features, α j is the attention distribution, representing each image feature v i Corresponding to text features The importance of Indicates z j The weight matrix of represents the bias vector, is the image feature representation guided by text features, α j,i Represents the i-th attention distribution coefficient of the j-th test image.

[0034] Optionally, the method of obtaining the text feature representation guided by the image feature of the math test question specifically adopts the following formula:

[0035]

[0036] Among them, z j ′ represents the output of the jth test sample, represents the weight matrix, represents the bias vector, W T represents the weight matrix, hT represents the test text feature vector, β j represents the attention distribution, represents the weight matrix, represents the bias vector, represents the text feature vector output guided by image features, β j,k represents the kth attention distribution coefficient of the jth test text, Represents the feature vector of the test text after the weight matrix is ​​applied.

[0037] Optionally, the image feature representation guided by the text feature of the math test question and the text feature representation guided by the image feature of the math test question are fused based on a multimodal gating mechanism to obtain fused features, specifically using the following formula:

[0038]

[0039] in, Represents text features and image features, Represents the weight matrix for linear transformation of text features, represents the bias vector, represents the image features after linear transformation, Represents the weight matrix for linear transformation of image features, represents the text feature vector input, represents the bias vector, g j represents the gating signal, σ represents the activation function, W gj represents the weight matrix, represents the text features after linear transformation, m j Represents the fused features.

[0040] Optionally, the fused features are filtered to obtain the filtered features using the following formula:

[0041]

[0042] Among them, s j represents the gating signal, represents the weight matrix, represents the weight matrix, represents the bias vector, r j represents the filtered multimodal features, represents the weight matrix, represents the bias vector, represents the filtered features, Represents the weight matrix of the fused features obtained by linear transformation.

[0043] Optionally, based on the filtered features, the knowledge point classification result is obtained by specifically using the following formula:

[0044]

[0045] in, is the final result of predicting the knowledge point label, λ is the mixed distribution ratio factor of the two prediction label methods, represents the predicted label output by KNN, Represents the predicted label of the fully connected layer output.

[0046] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0047] The present application provides a multi-modal fusion method for classifying multiple knowledge points in mathematics test questions. First, the local information and contextual logic of mathematics test questions are effectively integrated to vectorize the text with formulas, so as to more accurately capture the knowledge points tested by the mathematics test questions; secondly, considering the intrinsic correlation between images, texts and knowledge points in mathematics test questions, the text and image features are comprehensively extracted and integrated, thereby enhancing the multi-modal information processing capability of the model; finally, the similarity between samples and knowledge point labels is used to optimize the multi-label classification performance, and more accurate models and selection strategies are designed in knowledge point classification problems at different levels, which effectively improves the accuracy and universality of mathematics test question classification, thereby providing effective technical solutions for tasks such as intelligent test paper composition and personalized test question recommendation in the field of intelligent education. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 This is a schematic diagram of the overall framework of the MMMQMC model in one embodiment of the present application;

[0050] Figure 2 This is a schematic diagram of a symbol appearance tree in an embodiment of the present application;

[0051] Figure 3 This is a schematic diagram of an operator tree in an embodiment of the present application;

[0052] Figure 4 This is a schematic diagram of the SA-TCNN module structure in one embodiment of the present application;

[0053] Figure 5This is a schematic diagram of the structure of an improved image feature extraction model in an embodiment of the present application;

[0054] Figure 6 This is a schematic diagram of image feature representation guided by text features in one embodiment of the present application;

[0055] Figure 7 A schematic diagram of text feature representation guided by image features in one embodiment of the present application;

[0056] Figure 8 This is a schematic diagram showing the fusion of image and text features in one embodiment of the present application;

[0057] Fig. 9 This is a schematic diagram of the CL-KNN module structure in one embodiment of the present application;

[0058] Fig.10 This is an example of test questions on the test paper composition network in one embodiment of the present application;

[0059] Fig.11 This is an example of local test question source data in one embodiment of the present application;

[0060] Fig.12 This is an example of a test question after cleaning in one embodiment of the present application;

[0061] Fig.13 This is a schematic diagram of the distribution of test questions length in a data set in one embodiment of the present application;

[0062] Fig.14 This is a schematic diagram of the distribution of secondary knowledge points in a data set in an embodiment of the present application;

[0063] Fig.15 This is a schematic diagram of classification results of common primary knowledge points in an embodiment of the present application;

[0064] Fig.16 This is a schematic diagram of the classification results of common secondary knowledge points in one embodiment of the present application. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0066] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0067] In the current research on the classification of knowledge points in math test questions, the application of traditional classification algorithms often fails to fully consider the unique complexity of the mathematical subject. Since math test questions involve many knowledge points, complex structures, and certain logic and reasoning, traditional classification algorithms have certain limitations when dealing with these characteristics. Therefore, these algorithms often do not work well when classifying math test questions, and are unable to accurately identify and classify multiple knowledge points in the test questions. In response to this challenge, combined with the multi-knowledge point and multi-modal characteristics of math test question data, this application proposes the following key issues to be solved:

[0068] (1) In view of the rich subject characteristics contained in the math test text, how to effectively integrate local information and contextual logic to vectorize the text containing mathematical formulas, so as to more accurately capture the knowledge points tested in the math test;

[0069] (2) Considering that mathematical images also carry important mathematical semantics and are intrinsically related to text and knowledge points, how to comprehensively extract and fuse mathematical text and image features to enhance the model’s multimodal information processing capabilities;

[0070] (3) In view of the imbalanced distribution of mathematics test sample sets and the multi-level characteristics of knowledge points, this paper studies how to use the similarities between samples and knowledge point labels to optimize the multi-label classification performance, and designs more accurate models and selection strategies for knowledge point classification problems at different levels.

[0071] Finally, the problem of multi-knowledge point classification of mathematics test questions is abstractly defined as follows:

[0072] The goal is to obtain a mapping from X to Y by training a classification model Θ so that Θ(X)→Y, that is, the model can accurately predict the knowledge points covered by the new test questions. n} represents the input math test question, and x∈{x w ,x e ,x g}, including text description x w 、Mathematical expression x e and the mathematical image x g These types of multimodal information. The specific form of X is a multimodal dataset of length n, where each sample contains a text sequence, a mathematical expression sequence, and image data. And Y = {y1, y2, ..., y m} represents the set of knowledge point labels associated with question X, and each label represents a specific mathematical knowledge point. The specific form of Y is a multi-hot encoded label sequence of length m, where each label corresponds to a knowledge point and takes a value of 0 or 1, indicating whether the knowledge point is related to the question.

[0073] See also Figure 1 , this application proposes a multi-modal math question multi-knowledge point classification model - MMMQMC (Multi-Modal Math Question Multi-Label Classification Model, MMMQMC) model. When extracting the text features of math questions, the structure and semantic information of the formula are captured for the formula part of the question to achieve efficient formula embedding representation. For the pure text part of the question, a text representation method combined with the self-attention mechanism is designed to better capture important contextual information. When extracting the image features of the question, a linear layer is proposed to transform the image features to achieve the dimensional alignment of the image features and the text features. When fusing multimodal features, a multimodal math question data fusion method based on a collaborative attention mechanism is proposed to avoid the problem of loss of correlation information between modalities. When classifying the knowledge points of math questions, a joint contrastive learning-enhanced KNN mechanism (Contrastive Learning-Enhanced KNN, CL-KNN) is proposed. The classification results of the fully connected layer and the sample label results similar to the sample to be predicted obtained by KNN retrieval are weighted and fused as the final predicted multi-label result, thereby improving the performance of the classification model.

[0074] The above model framework consists of three main modules: a test text feature extraction module combined with self-attention mechanism (Self-Attention TextCNN, SA-TCNN), a multimodal test feature fusion module (Cooperative Text-Image Feature Fusion Module, CTIFM) with collaborative attention mechanism, and a KNN classification module enhanced by joint contrastive learning (Contrastive Learning-EnhancedKNN, CL-KNN).

[0075] The specific steps are as follows:

[0076] Step 1: Extract the formula features of math test questions.

[0077] Step 2: Extract text features of math test questions.

[0078] Steps 1 and 2 are the test text feature extraction module SA-TCNN combined with the self-attention mechanism. This module realizes the extraction of math test text features and divides the math test text into two parts: formula and plain text for embedding. First, for the formula part, the Tangent Combined with FastText (TangentCFT) method is used to capture the structure and semantic information of the formula and realize efficient formula embedding representation. The conversion steps of TangentCFT are as follows:

[0079] The mathematical formula is first converted into a symbol layout tree (Symbol LayoutTrees, SLT) and an operator tree (OperatorTrees, OPT). The symbol layout tree is used to represent the layout structure of the mathematical formula, while the operator tree is used to represent the operation relationship in the mathematical formula. 2 +y=5 as an example, the SLT tree of this formula is as follows Figure 2 As shown, the OPT tree is Figure 3 shown.

[0080] In both tree structures, nodes represent various symbols in the formula, such as operators "+", "=", numbers, etc. The representation format of nodes is "Type! Val", where "Type" refers to the type of the node and "Val" is the value of the node. Examples of common node types in the tree structure are shown in Table 1.

[0081] Table 1 Examples of node types in the tree structure

[0082]

[0083] Taking object X as an example, there are 7 types of spatial relationships between nodes in SLT, as shown in Table 2-2, which represent the relationship types and symbols respectively.

[0084] Table 2 Edge relationship types in SLT

[0085]

[0086] In OPT, the edge represents the position of the operator. For those commutative operators (such as the "=" operator), the two sides of the operator are marked as "0" after the order is swapped without affecting the result. Otherwise, the marks on both sides of the operator start from "0" and "1" and increase. Next, the constructed tree structure needs to be traversed to obtain a linear sequence and further parse it into tuples. In this way, the mathematical formula is converted into an embedded representation that the model can handle.

[0087] For example: Formula x 2 The SLT and OPT nodes for +y=5 are shown in Table 3.

[0088] Table 3x 2 +y=5 SLT and OPT nodes

[0089]

[0090] After the depth-first traversal of the tree structure generates a sequence, a tuple of the form (n1, n2, E, FRP) is used to represent the generated tuple of node pairs and their relative positions. Among them, n1 represents the ancestor node, n2 represents the descendant node, E represents the edge sequence from n1 to n2, and FRP represents the complete path relative to the ancestor node, giving the position of n1 on its path starting from the root. Formula x 2 The SLT and OPT traversal tuples for +y=5 are shown in Table 4 below, and FRP is ignored in the tuple sequence.

[0091] Table 4x 2 +y=5Tuple of formula tree traversal

[0092]

[0093] Finally, the elements in each tuple are treated as independent characters, which can be input into the FastText model for training and output a formula vector. The formula vector is used as part of the input of the SA-TCNN module.

[0094] Secondly, for the plain text part, the RoBERTa pre-trained model is used to obtain the word vector of the test text information. The RoBERTa model is an improved version of BERT (A Robustly Optimized BERT, that is, a strongly optimized BERT method). A text representation method combined with a self-attention mechanism is adopted by introducing a self-attention mechanism and a TextCNN structure. Unlike general texts, math test questions often need to consider the logical relationship of the context. Each mathematical concept, condition, and reasoning step in the test question is interrelated, and these relationships are reflected through the semantics of the context. The self-attention mechanism can dynamically assign different attention weights according to the relationship between words, so as to better capture important contextual information. The specific steps are as follows:

[0095] Step 2.1: Load the pre-trained RoBERTa model Method: Use Hugging Face's Transformers library to load a pre-trained RoBERTa model instance.

[0096] Step 2.2: Input the math test text and use the word segmenter to preprocess the text to obtain the result: get the input sequence after word segmentation.

[0097] Step 2.3: Encode the sentence, pass the preprocessed input sequence into the pre-trained RoBERTa model to get the result, and obtain the word vector representation of each token.

[0098] Step 2.4: Extract the hidden layer from the model output to get the word vector representation of the math test text.

[0099] Through the above steps, this application successfully uses the pre-trained language model RoBERTa to represent the plain text part of the math test with word vectors. The word vector is used as another part of the input of the SA-TCNN module.

[0100] The word vector and formula vector are simply concatenated and input into the SA-TCNN module. The structure of the SA-TCNN module is as follows: Figure 4 shown.

[0101] The SA-TCNN model consists of an attention layer, a convolution layer, and a pooling layer. First, the word vector and the formula vector are used as the input of the network and first pass through the first attention layer to give higher weights to the key features of the math test text. Next, convolution kernels of different window sizes are designed to capture math test text features of different granularities. After convolution kernel processing, the generated test text feature map process is shown in the following formula:

[0102]

[0103] C=[C1,C2,...,C k ]

[0104] Among them, l j is the convolution kernel window size, H j Represents the convolution kernel, and C represents the feature map corresponding to all convolution kernels. The maximum pooling operation is applied to the features obtained after convolution to output the important features in the test text, as shown in the following formula:

[0105]

[0106] Finally, the important features of the test text corresponding to different convolution kernel windows are concatenated to obtain the final test text feature vector representation, as shown in the following formula:

[0107]

[0108] Step 3: Extract image features of math test questions.

[0109] Specifically, the deep learning ResNet152 model is used to extract features from math test images, which mainly includes the following steps:

[0110] Step 3.1: Use the pre-trained model library of the deep learning framework PyTorch to load a pre-trained ResNet152 model instance.

[0111] Step 3.2: The math test image is normalized and resized to obtain the result, which is an image tensor suitable for input into the ResNet152 model.

[0112] Step 3.3: Use the ResNet152 model for forward propagation to the result. First, obtain the feature map and use the ResNet152 model for forward propagation, but do not include the final fully connected layer.

[0113] Step 3.4: Apply global average pooling to get a feature vector of length 2048. Use a fully connected layer to reduce the dimension of the feature vector of length 2048 to a feature vector of length 11.

[0114] Step 3.5: Split the feature map into multiple blocks and arrange them sequentially to form a new feature sequence. The output of the last layer Then pass it through a linear layer Convert to vector representation v I , so that the dimensions of image features and text features are consistent. The conversion formula is

[0115] Through the above steps, this application captures the local features of the image and provides a richer feature representation for the classification of math test images. The improved image feature extraction model structure is as follows: Figure 5 shown.

[0116] Step 4: Obtain an image feature representation guided by the text features of the math test question.

[0117] Step 5: Obtain a text feature representation guided by the image features of the math test question.

[0118] In the multi-modal math test multi-knowledge point classification task, the effective fusion of text features and image features is crucial to improving model performance. However, due to the differences in the information expressed by text and images, simply concatenating or weighting the features of these two modalities may cause the model to fail to fully capture the correlation information between the modalities, affecting the performance of the model. To solve this problem, a multimodal test data fusion method based on a collaborative attention mechanism is proposed, called CTIFM (Cooperative Text-Image Feature Fusion Module). The specific implementation of this module is introduced below.

[0119] Obtaining image feature representation guided by test text features

[0120] In order to make the image feature focus more on the part related to the test text, the text-guided attention is used to obtain the image feature representation. The steps are as follows: Figure 6 As shown, the calculation formula for this step is as follows:

[0121]

[0122] Specifically, firstly, the original text features Perform a linear transformation, where is the weight matrix, is the bias vector. The purpose of linear transformation is to map text features to a new space. Then the linear transformation result of text features is combined with image features v I The linear transformation (given by the weight matrix Determine) to concatenate in dimension. Map the concatenated features to the range of [-1,1] and use the softmax function to calculate the attention distribution α j , which represents each image feature v i Corresponding to text features Finally, according to the calculated α j v i Perform weighted summation to obtain image feature representation guided by text features It integrates all image features and emphasizes the parts that are associated with the current text features.

[0123] Obtaining text feature representation guided by test image features

[0124] In order to make the attention of text features focus more on the parts related to the test image, the image-guided attention is used to obtain the text feature representation. The steps are as follows: Figure 7 shown.

[0125] The acquired image feature representation and text feature are used to perform attention calculation and generate the attention distribution. Then, the text features are weighted according to the attention distribution to obtain the test question text feature representation guided by the image feature. The calculation formula for this step is as follows:

[0126]

[0127] Specifically, we first analyze the image features Perform a linear transformation to map it to a new space, where The weight matrix, is the bias vector. Then the linear transformation result of the image feature is combined with the text feature h T The linear transformation result (given by the weight matrix W T Determine) to concatenate in dimension. Map the concatenated features to the range of [-1,1] and use the softmax function to calculate the attention distribution β j , which represents each text feature Corresponding to the image features Finally, according to the calculated β j right Perform weighted summation to obtain text feature representation guided by image features It integrates all text features and emphasizes the parts that are associated with the current image features.

[0128] Step 6: Based on the multimodal gating mechanism, the image feature representation guided by the text features of the math test questions and the text feature representation guided by the image features of the math test questions are fused to obtain fused features.

[0129] Step 7: Filter the fused features to obtain filtered features.

[0130] Through the above-mentioned collaborative attention mechanism, a richer representation of test text and image features can be obtained. In order to further integrate these two features, this application designs a multimodal gating mechanism, such as Figure 8 shown.

[0131] This mechanism can dynamically adjust the weights of text features and image features in the fusion process according to the degree of correlation between them, so that the correlation information between text and image can be captured more flexibly, thereby improving the classification performance. The calculation method of the multimodal gating mechanism is shown in the following formula.

[0132]

[0133] Specifically, firstly, the obtained image features guided by the test text and the text features guided by the test image are linearly transformed to obtain the text features and image features Then apply the weight matrix W gj and the sigmoid activation function will be given by and The concatenated feature vector is output as a gating signal g j , which is used to control the contribution of text features and image features to the final feature representation. Finally, the gating signal is dynamically fused and When j When it is close to 1, the fusion features will mainly contain text features. j When it is close to 0, the fused features will mainly contain image features. Through the multimodal gating mechanism, text and image information can be effectively combined.

[0134] However, since not every math test question has important features related to the text and image, the present invention also introduces a filter gate mechanism to filter the noise in the fusion features of the test question text and image. The introduction of the filter gate helps the model to more accurately identify the key correlation information between the text and the image, and further improves the classification performance. The calculation method of the filter gate mechanism is shown in the following formula.

[0135]

[0136] Specifically, we first analyze the original text features Perform linear transformation, the weight matrix is Then, the multimodal fusion feature m j Perform linear transformation, the weight matrix is Then the linear transformation results of the two features are concatenated in dimension, and the sigmoid function is applied to obtain the gating signal s of the filter gate j , which will determine which features are retained in the final representation. Then, using the weight matrix For m j Re-perform the linear transformation and compare the result with s j Multiply them together to get the filtered multimodal feature r j , this operation is equivalent to retaining only those features that the gated signal considers important. With r j Concatenate and perform linear transformation to obtain the final fusion feature representation It combines text features with filtered multimodal fusion features.

[0137] By proposing the CTIFM module, the correlation between multimodal test data can be better handled. First, the introduction of the attention mechanism enables the model to pay more attention to the parts related to text or images, reducing the interference of noise. Secondly, the multimodal gating mechanism can dynamically adjust the weights according to the degree of correlation between text and image features, allowing the model to fuse the two features more flexibly. Finally, the introduction of the filter gate further reduces the noise in the fused features and improves the robustness of the model.

[0138] Step 8: Based on the filtered features, obtain knowledge point classification results.

[0139] In the task of multi-knowledge point classification of mathematics test questions, the conventional method is to output the knowledge point classification results of the test questions through a fully connected layer. However, the traditional fully connected layer method often fails to fully utilize the label similarity information between samples, resulting in limited classification performance.

[0140] In order to solve this problem, the present application proposes a classification algorithm of a joint contrastive learning-enhanced KNN mechanism (Contrastive Learning-Enhanced KNN, CL-KNN).

[0141] like Fig. 9As shown in the figure, the classification results of the fully connected layer and the sample label results similar to the sample to be predicted obtained by KNN retrieval are weighted and fused as the final result of predicting multiple labels. By designing the objective function of multi-label contrastive learning and the dynamic contrastive learning coefficient, the label similarity information can be better utilized to improve the performance of the classification model.

[0142] In order to assist in the selection of the KNN mechanism, the objective function of multi-label contrastive learning is introduced. This objective function measures the similarity between samples by comparing the label similarity of sample pairs. At the same time, a dynamic contrastive learning coefficient is designed, which is calculated based on the label similarity between samples. The greater the label similarity between samples, the greater the contrastive learning coefficient, thereby increasing the loss value of the sample pair.

[0143] The specific steps are as follows:

[0144] To facilitate understanding of this process, x is used below to represent the feature vector output by the multimodal test feature fusion module. That is, x serves as the input of this module.

[0145] Step 8.1: Build a local storage Datastore for training samples, that is, generate a storage for the mapping relationship between the corresponding fusion features and the corresponding knowledge point classification:

[0146]

[0147] Among them, y i is the corresponding knowledge point corresponding to the fused feature sample j.

[0148] Step 8.2: After the fully connected layer FL, the prediction result is y FL ∈{p|p∈[0,1]} L , using the corresponding knowledge point classification mapping relationship as a query clue, find k adjacent samples in the Datastore:

[0149]

[0150] Step 8.3: According to the k adjacent samples in the above step, give the corresponding KNN prediction label:

[0151] Among them, d refers to the Euclidean distance, τ is the KNN hyperparameter, and α i is the weight of the ith neighbor.

[0152] Step 8.4: The final predicted label for x is expressed as: in is the final result of predicting the knowledge point label, and λ is the mixed distribution ratio factor of the two prediction label methods.

[0153] In the classification algorithm of the present application, the multimodal features of the test questions are first mapped to a low-dimensional space through a fully connected layer, and an intermediate result is output. Then, the KNN mechanism is used to select the training samples that are most similar to the test questions to be classified, and the results of the KNN mechanism are obtained. Finally, the results of the fully connected layer and the KNN mechanism are combined in a weighted fusion manner to obtain the final label prediction result for the new test question sample. The weighted fusion takes into account the importance of the results of the fully connected layer and the KNN mechanism in classification.

[0154] In summary, the beneficial effects of this application are:

[0155] First, the test feature extraction method combined with the self-attention mechanism proposed in this application, on the one hand, the self-attention mechanism can dynamically assign different attention weights according to the relationship between words, so as to better capture important contextual information; on the other hand, it retains the spatial structure information of the image and can better capture the local features in the image.

[0156] Second, the multimodal test feature fusion method proposed in this application can better capture the correlation information between multimodal test data and improve the robustness of the model compared to the method of simply splicing or weighting text and image features. First, the introduction of the attention mechanism enables the model to pay more attention to the parts related to the text or image, reducing the interference of noise. Secondly, the multimodal gating mechanism can dynamically adjust the weights according to the degree of correlation between text and image features, so that the model can more flexibly fuse the two features. Finally, the introduction of the filter gate further reduces the noise in the fused features.

[0157] Third, the KNN classification method enhanced by joint contrastive learning proposed in this application solves the problem of limited classification performance caused by the inability to fully utilize the label similarity information between samples compared with the traditional fully connected layer method.

[0158] Next, in order to verify the effectiveness of the MMMQMC model proposed in this application on a real data set, this application constructs a large-scale mathematics test data set. Through experimental verification, the results of the experimental improvement of the classification of multiple knowledge points in the test questions are deeply analyzed.

[0159] Experimental setup

[0160] The experiment of this application is mainly based on the Pytorch deep learning framework and the pytorch_pretrained_bert library, the processor is Intel(R) Core i5-1035G1 CPU@1.00GHz, and the GPU is RTX2080. In the experimental model setting, for unified processing, the length of the test text is standardized to 180 characters. For texts of insufficient length, a padding strategy is adopted to complete them, otherwise they are truncated. In addition, the test images are uniformly adjusted to a size of 224*224*3, and after processing, each image is converted into a 7x7 feature map. In the design of TextCNN, the window size of the convolution kernel is set to [2,3,4], and the number of convolution kernels of each size is 256. Other important model parameters are shown in Table 5.

[0161] Table 5 Configuration of some model parameters

[0162]

[0163] Experimental data

[0164] There is currently a lack of public Chinese datasets for the research on multi-knowledge point classification of math test questions. To this end, this application built a real dataset using crawler technology. The dataset comes from the online test website and covers high school math test questions including math images and their corresponding knowledge point labels. The construction of this dataset provides a basis for the experiment and ensures the practicality and pertinence of the research work. Fig.10 shown.

[0165] Since the test questions are collected from the Internet, the source data is in Hypertext Markup Language (HTML). After collection, the source data saved in the local database is as follows: Fig.11 As shown, the formula is in Mathematical Markup Language (MathML) and the mathematical images are in the form of https source links.

[0166] Use Python's regular expressions and http request library and other technologies to filter out test questions containing mathematical images, and finally save the images and texts locally separately to form the final data set of all high school mathematics test questions containing mathematical images.

[0167] In the process of building a data set by yourself, after the collection of web test data is completed, data cleaning is required to improve data quality. The cleaning process is divided into several steps: First, remove irrelevant characters in the text, such as spaces, line breaks, etc., while keeping the mathematical formulas in MathML format to ensure the readability and parsability of the formulas. Secondly, for text content other than formulas, the jieba word segmentation tool is used to perform Chinese word segmentation processing to extract valid vocabulary information. Finally, for the mathematical formula part, the TangentCFT parsing method is used to parse the formula content to extract the structured information of the formula. After this series of cleaning steps, pure test text modal data is obtained. In this data, the mathematical formula of each test question is numbered HEL_xxx_WLDOR_x_OL to facilitate identification and reference in subsequent processing. Through such a data cleaning process, the quality of the experimental data is ensured, laying a solid foundation for subsequent model training and performance evaluation. An example of a cleaned math test question is as follows Fig.12 shown.

[0168] The final dataset contains 27,663 math questions, each of which has an average of 69.2 characters, 5.6 math formulas, and one math image attached to each question. Fig.13 shown.

[0169] In addition, we collected the knowledge point labels of all the test questions to form a label dataset. This dataset covers 17 first-level knowledge point labels and 427 second-level knowledge point labels, and the average number of knowledge point labels per test question is 1.93. The distribution of the ID and number of second-level knowledge point labels in the dataset is shown in the figure. Fig.14 shown.

[0170] According to the classification of knowledge points, the distribution of the number of test questions of the 17 first-level knowledge points in the mathematics test data set is shown in Table 6 below.

[0171] Table 6 Distribution of first-level knowledge points

[0172]

[0173] The number of secondary knowledge points is 427. As the level of knowledge points increases, the number of knowledge points also increases, which will lead to a decrease in the number of test questions corresponding to each type of secondary knowledge point. The following shows the distribution of the number of test questions for some secondary knowledge points in the mathematics test data set, as shown in Table 7.

[0174] Table 7 Distribution of some secondary knowledge points

[0175]

[0176] The above collected and cleaned math test questions on the Internet, and performed a simple statistical analysis on the self-built math test question data set. The construction and analysis of this data set provides a data basis for the experiment, which is helpful for the training and evaluation of the model, and further design of the experimental process and analysis of experimental results will be carried out in the future.

[0177] Evaluation indicators

[0178] When evaluating the performance of classification results in multi-label classification tasks, the selection and design of evaluation indicators are crucial. The goal of multi-label classification tasks is to assign matching label subsets based on given samples. It is necessary to select appropriate evaluation indicators to measure the prediction results in order to evaluate the effectiveness of the model.

[0179] Common evaluation indicators for multi-label text classification include: Precision, Recall, F1 value, Hamming Loss, etc.

[0180] In the following formulas, the first formula represents the precision, which measures the proportion of samples predicted by the model to be positive that are actually positive; the second formula represents the recall, which measures the proportion of samples that are actually positive that are correctly predicted by the model to be positive; the third formula is the F1 value, which is the harmonic mean of the precision and recall, and is used to comprehensively evaluate the accuracy and completeness of the model.

[0181]

[0182] The following are the evaluation indicators selected in this application. macro ) is used to measure the prediction accuracy of the model in each category; macro ) evaluates the recall ability of the model in each category. When calculating these indicators, you need to multiply the precision and recall of each category by the proportion of the category in the total samples, and then add the products. In addition, the macro F1 value (F1 macro ) is the harmonic mean of precision and recall, which comprehensively reflects the accuracy and recall of the model; finally, Hamming Loss focuses on the number of labels misclassified by the model. The smaller its value, the better the model effect. These evaluation indicators together provide a comprehensive evaluation of the performance of the model.

[0183]

[0184] Experimental Procedure

[0185] The experimental process of this application mainly includes three stages: data collection, model training and result analysis. The specific steps of each stage are as follows.

[0186] Data collection. Data collection is the basic work of experimental preparation, which involves crawling online test questions, data cleaning, and statistical analysis of data sets.

[0187] Model training. In the model training stage, the fastText pre-trained model is first used to obtain the vector representation of mathematical formulas, and the RoBERTa model is used to extract the word vector of mathematical text information. Then, the cleaned dataset is divided into 80% training set, 10% validation set and 10% test set to ensure the comprehensiveness of model training and the objectivity of evaluation. Based on the proposed MMMQMC model, the model is trained in combination with binary cross entropy loss and multi-label contrastive learning loss. During the training process, the validation set is used for parameter tuning, and the model with the best performance is saved. On the test set, Precision macro 、Recall macro 、F1 macro and HL are used as evaluation indicators to assess the performance of the model.

[0188] Results analysis. The results analysis phase aims to comprehensively evaluate the performance and effectiveness of the model. Through comparative experiments and ablation experiments, the accuracy and rationality of the model in predicting labels of multiple knowledge points in mathematics test questions are analyzed. In addition, the model design and strategy selection at different knowledge point levels are considered to explore the applicability and generalization ability of the model in different scenarios.

[0189] Through the above experimental process, this application aims to comprehensively evaluate the research and implementation of the multi-knowledge point classification method for mathematics test questions based on multimodal fusion. The design and execution of the experimental process follow the scientific and rigorous principles to ensure the reliability of the experimental results and the effectiveness of the research work.

[0190] Experimental results and analysis

[0191] To verify the effectiveness of the proposed MMMQMC model, this application conducted multiple comparative experiments and ablation experiments at different knowledge point levels on a self-built math test data set with multiple knowledge points. The comparative experiment selected the currently popular math test classification algorithm as a benchmark, and compared it with the MMMQMC method at different knowledge point levels to verify its advantages in classification accuracy. The ablation experiment evaluates the contribution of each component to the overall classification performance by gradually removing the multimodal fusion components in the model, thereby verifying the rationality of the model design and the importance of each module.

[0192] Comparative experiment

[0193] In the comparative experiment, first-level knowledge points and second-level knowledge points were selected as labels for multi-classification to verify the effectiveness of the proposed method at different classification granularities. The selected comparative methods include baseline models such as TextCNN and Bi-LSTM, as well as the proposed MMMQMC method. Among them, the first six baseline models only regard math questions as text modalities for feature extraction, while the MMMQMC method adopts a multi-modal fusion strategy. Each experimental method is introduced as follows.

[0194] (1) TextCNN: The text classification model based on convolutional neural network can effectively extract local features of text;

[0195] (2) Bi-LSTM: A text classification model based on a bidirectional long short-term memory network that can fully mine the contextual information of the text;

[0196] (3) CNN-RNN

[57] : A text classification model that combines convolutional neural networks and recurrent neural networks. It can not only extract local features of text, but also capture the sequence information of text.

[0197] (4) Seq2Seq+Att

[58] : A text classification model based on a sequence-to-sequence model and an attention mechanism that can automatically learn text representation and classification.

[0198] (5) SGM

[59] : A sequence generation structure that considers the correlation between labels and automatically obtains the key information of the input text through the attention mechanism;

[0199] (6) RobBERTa+TextCNN: It combines the RoBERTa pre-trained model and the TextCNN text classification model, which can not only take advantage of the pre-trained model but also extract local features of the text;

[0200] (7) MMMQMC: The proposed multi-modal fusion mathematics test multi-knowledge point classification method aims to make full use of multiple modal information such as text and images of mathematics test questions to achieve more accurate classification.

[0201] In the experimental results, the indicator prefix "(-)" means that the smaller the indicator value, the better the model performance; while "(+)" means that the larger the indicator value, the better the model performance. Table 8 shows the comparative experimental results with the first-level knowledge points as labels.

[0202] Table 8 Comparative experimental results under first-level knowledge points

[0203]

[0204] Table 9 presents the comparative experimental results using secondary knowledge points as labels.

[0205] Table 9 Comparative experimental results under secondary knowledge points

[0206]

[0207] Experimental results show that the method proposed in this application outperforms the baseline model regardless of whether the first-level knowledge points or the second-level knowledge points are used as labels.

[0208] Specifically, compared with the TextCNN model, the MMMQMC model proposed in this paper has a higher P macro Improved by 5.88%, R macro Improved by 9.71%, F1 macro Improved by 8.31%; Under the secondary knowledge point P macro Improved by 3.97%, R macro Improved by 3.92%, F1 macro The improvement is 4.36%. These data fully demonstrate that the proposed multimodal fusion model has the advantage of capturing semantic information and has superior performance in the task of multi-knowledge point classification of mathematics test questions.

[0209] When analyzing the experimental results, the present application found that the proposed method showed a significant improvement in the classification performance of the first-level knowledge points. This result reveals that the multimodal fusion method has obvious advantages in dealing with basic and core knowledge points. At the same time, this finding emphasizes the importance of first-level knowledge points in mathematics test questions and provides useful inspiration for further improvement of the model. In addition, the experimental data also shows that the proposed method also surpasses the baseline model in the classification accuracy of second-level knowledge points, indicating that even when faced with more complex and specific knowledge points, the multimodal fusion method still maintains a high level of performance.

[0210] Next, in order to verify the improvement of the multimodal fusion model on the classification performance of knowledge points in mathematics test questions, four common first-level and second-level knowledge points were selected for in-depth analysis. In the experiment, the baseline model SGM was compared with the proposed MMMQMC model. On the coordinate axis, the number of training samples of knowledge point labels shows a decreasing trend. For example, in the first-level knowledge points, the number of test questions in the "plane geometry vectors and their applications" category is relatively large, while the number of test questions in the "probability" category is relatively small. The experimental results show that the MMMQMC model shows a significant improvement in accuracy on knowledge point labels with a small number of samples.

[0211] like Fig.15 As shown in the results of common first-level knowledge point classification, the MMMQMC model is superior to the SGM model in F1 macro In particular, for the relatively rare “probability” knowledge point, the MMMQMC model achieved more F1macro Value improvement.

[0212] Similarly, in the classification results of common secondary knowledge points, such as Fig.16 As shown, the MMMQMC model also shows a higher F1 than the SGM model. macro In addition to the common knowledge point of "Graphs and properties of trigonometric functions", the knowledge point of "Application of derivatives in studying functions" achieved a more significant F1 through the MMMQMC model. macro The value is improved, thereby effectively improving the overall classification performance.

[0213] By comparing the experimental results, we can conclude that it is difficult to support the model to make effective classification decisions by relying solely on single-modal test text information. The introduction of images can supplement the semantic information that is not in the text and improve the classification performance of the model. In addition, the effectiveness of the MMMQMC model in dealing with sample imbalance problems is verified, and the important role of multimodal fusion in improving the accuracy of mathematical test knowledge point classification is emphasized.

[0214] Ablation experiment

[0215] Next, the effectiveness of each component of the proposed model is further verified through ablation experiments, and the contribution of each module in the model to the performance is evaluated. The experiments are still conducted with the first-level and second-level knowledge points as classification labels. The SA-TCNN module, CTIFM module, LAM module and CL-KNN module in the model are removed in turn. Among them, w / o SA-TCNN means the control SA-TCNN layer, w / oCo-att means the control multimodal collaborative attention layer in the CTIFM module; w / o Fus-gate means the control fusion gate mechanism in the CTIFM module; w / o LAM means the control LAM module, w / o KNN means the control KNN auxiliary retrieval in CL-KNN; w / oCL means the control contrast learning mechanism in CL-KNN; w / o CL-KNN means the control complete CL-KNN module.

[0216] Table 10 shows the ablation experiment results under the first-level knowledge points.

[0217] Table 10 Ablation experiment results under first-level knowledge points

[0218]

[0219] First, it is observed that removing the SA-TCNN layer (w / o SA-TCNN) has a greater impact on the classification performance. Compared with the proposed MMMQMC model, the resulting P macro It dropped by 1.98%, R macro F1 fell by 1.97% macroThis result shows that on the first-level knowledge point dataset, the SA-TCNN layer is crucial for extracting local features in text, especially in capturing key concepts and terms in math test questions.

[0220] Second, removing the co-attention layer (w / o Co-att) in CTIFM has relatively little impact on performance, P macro , R macro and F1 macro The decreases in are 1.07%, 1.04%, and 1.56% respectively. This may indicate that on the first-level knowledge point dataset, the correlation between text and label is weak, resulting in limited effect of the collaborative attention layer.

[0221] Further observation shows that the effect of removing the fusion gate mechanism (w / o Fus-gate) in CTIFM is relatively small, with each indicator decreasing by 1.17%, 1.11%, and 1.64% respectively. This may reflect that in the first-level knowledge point dataset, the correlation between text and other modalities is not strong, and the contribution of the multimodal fusion gate is limited.

[0222] However, removing the LAM module (w / o LAM) has a significant impact on the classification performance, P macro , R macro and F1 macro They dropped by 1.95%, 2.14% and 2.70% respectively. This shows that the attention module based on label semantics plays an important role in weighting different input features and understanding the concepts and logical relationships of math questions on the first-level knowledge point dataset.

[0223] Finally, removing the CL-KNN module has little impact on the performance. macro , R macro and F1 macro The decreases of the performance of the model with and without the contrastive learning mechanism (w / o CL) were 0.65%, 0.71% and 1.23% respectively; the decreases of the performance of the model with and without the contrastive learning mechanism (w / o CL) were 0.62%, 0.64% and 1.13% respectively; the decreases of the performance of the model with and without the complete CL-KNN module (w / o CL-KNN) were 0.84%, 0.81% and 1.33% respectively. The results show that the CL-KNN module has little effect on improving the classification performance of the model under the first-level knowledge point.

[0224] In summary, the SA-TCNN layer and LAM module have a significant impact on the classification performance in the first-level knowledge point dataset, while the contribution of other modules to the performance is relatively weak. These findings provide important references for further optimization of the model and adjustment of the experimental design.

[0225] Table 11 shows the ablation experiment results under the second-level knowledge points.

[0226] Table 11 Ablation experiment results under secondary knowledge points

[0227]

[0228] In the case of secondary knowledge points as classification labels, this application continues to conduct an in-depth analysis of the impact of each module to enhance the understanding of the functions of each module under the secondary knowledge point dataset and its contribution to the model performance.

[0229] First, removing the TextCNN layer with self-attention (w / o SA-TCNN) has little impact on the classification performance, P macro , R macro and F1 macro The results show that on the secondary knowledge point dataset, the improved TextCNN layer is not as critical as the basic concepts and definitions in capturing the complex logic in math questions, so its impact on classification performance is weaker than that on the primary knowledge point dataset.

[0230] Secondly, removing the co-attention mechanism (w / o Co-att) and fusion gate mechanism (w / o Fus-gate) in CTIFM has a significant impact on the classification performance. macro , R macro and F1 macro The decreases in the performance of the second-level knowledge points were 0.69%, 0.50%, 0.60% and 0.89%, 0.70%, 0.79% respectively. This shows that on the second-level knowledge point dataset, multimodal collaborative attention and fusion gate mechanisms are essential for integrating feature information from different modalities, which is particularly critical for understanding complex concepts and logical relationships in math test questions. Since second-level knowledge points usually involve the application of multiple concepts and theorems, these modules have a greater impact on classification performance on this dataset.

[0231] Further observation shows that removing the label semantic attention module (w / o LAM) still has a significant impact on the classification performance, P macro , R macro and F1 macro The decreases in the two datasets were 1.40%, 0.90% and 1.15% respectively. This shows that the attention module based on label semantics also plays an important role in the secondary knowledge point dataset, which helps the model better understand the concepts and logical relationships of math questions.

[0232] Finally, the impact of removing the CL-KNN module on the classification performance is more significant on the secondary knowledge point dataset than on the primary knowledge point dataset.macro , R macro and F1 macro The decreases were 0.54%, 0.32% and 0.43% respectively; the decreases after removing the contrastive learning mechanism (w / o CL) were 1.33%, 1.21% and 1.27% respectively; and after removing the complete CL-KNN module (w / o CL-KNN), these indicators decreased by 2.73%, 2.51% and 2.62% respectively. This result shows that under the secondary knowledge point dataset, the KNN retrieval mechanism and contrastive learning play an important role in distinguishing different concepts and logical relationships in mathematics test questions by capturing the differences between positive and negative samples, thus having a greater impact on the classification effect.

[0233] In summary, in the context of the secondary knowledge point dataset, the CTIFM module and the CL-KNN module have a significant impact on the classification performance, while the contributions of other modules to the model performance are relatively limited.

[0234] Through the comprehensive analysis of the ablation experiments with primary and secondary knowledge points as labels, we can observe the correlation between the performance of the model on different knowledge point datasets and each module. On the primary knowledge point dataset, the SA-TCNN module and the LAM module have a significant impact on the classification performance. These two parts are crucial for grasping the basic concepts and logical relationships in math test questions. On the secondary knowledge point dataset, the CTIFM module and the CL-KNN module contribute more to the classification performance. This finding reveals that as the number of knowledge points increases, the complexity of the classification task increases. These two modules effectively distinguish positive and negative samples during the learning process, which plays an important role in deeply understanding the complex concepts and logical relationships in math test questions. The analysis of the mechanism of action of these modules at different knowledge point levels can provide a more effective solution for the multi-knowledge point classification task of math test questions.

[0235] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0236] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0237] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0238] The present application uses specific examples to illustrate the principles and implementation methods of the present application. The above examples are only used to help understand the method and core ideas of the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A multi-modal fusion mathematics test question multi-knowledge point classification method, characterized in that: The multi-modal fusion mathematics test question multi-knowledge point classification method includes: Extract formula features of math test questions; Extract text features of math test questions; Extract image features of math test questions; Obtaining an image feature representation guided by text features of the math test question; Obtaining a text feature representation guided by the image features of the math test question; Based on a multimodal gating mechanism, the image feature representation guided by the text feature of the math test question and the text feature representation guided by the image feature of the math test question are fused to obtain fused features; Filtering the fused features to obtain filtered features; Based on the filtered features, a knowledge point classification result is obtained.

2. The multi-modal fusion mathematics test question multi-knowledge point classification method according to claim 1 is characterized in that: The method of extracting the formula features of the mathematics test questions specifically comprises the following steps: Converting a mathematical formula into a symbol appearance tree and an operator tree; the symbol appearance tree represents the layout structure of the mathematical formula, and the operator tree represents the operation relationship of the mathematical formula; Depth-first traversal of the symbol appearance tree and the operator tree to generate a tuple (n1, n2, E, ERP); wherein n1 represents an ancestor node, n2 represents a descendant node, E represents an edge sequence from n1 to n2, and FRP represents a complete path relative to the ancestor node; The tuple is input into the trained FastText model to obtain a formula vector.

3. The multi-modal fusion mathematics test question multi-knowledge point classification method according to claim 2 is characterized in that: The method of extracting text features of mathematics test questions specifically comprises the following steps: Segment the math test text to obtain a sequence after segmentation; Input the segmented sequence into the trained RoBERTa model to obtain the word vector of the math test text; Input the formula vector and word vector into the SA-TCNN module to generate a test question text feature map; The test question text feature maps are spliced ​​to obtain a test question text feature vector representation.

4. The multi-modal fusion mathematics test question multi-knowledge point classification method according to claim 3 is characterized in that: The test question text feature graphs are spliced ​​to obtain the test question text feature vector representation using the following formula: in, Represents the test text feature vectors corresponding to different convolution kernel windows.

5. The multi-modal fusion mathematics test question multi-knowledge point classification method according to claim 1 is characterized in that: The image features of the math test questions are specifically extracted using the following formula: Among them, v I represents the image feature vector, W I represents the weight matrix of the linear layer transformation, represents the image vector input, b I Represents the bias vector.

6. The multi-modal fusion mathematics test question multi-knowledge point classification method according to claim 5 is characterized in that: The image feature representation guided by the text features of the math test questions is specifically obtained by using the following formula: Among them, z j represents the output of the jth test sample, represents the weight matrix, Represents the original text features, represents the bias vector, represents vector concatenation, represents the weight matrix, v I represents the image feature vector guided by text features, α j is the attention distribution, representing each image feature v i Corresponding to text features The importance of Indicates z j The weight matrix of represents the bias vector, is the image feature representation guided by text features, α j,i Represents the i-th attention distribution coefficient of the j-th test image.

7. The multi-modal fusion mathematics test question multi-knowledge point classification method according to claim 6 is characterized in that: The text feature representation guided by the image feature of the math test question is specifically obtained by using the following formula: Among them, z′ j represents the output of the jth test sample, represents the weight matrix, represents the bias vector, W T represents the weight matrix, h T represents the test text feature vector, β j represents the attention distribution, represents the weight matrix, represents the bias vector, represents the text feature vector output guided by image features, β j,k represents the kth attention distribution coefficient of the jth test text, Represents the feature vector of the test text after the weight matrix is ​​applied.

8. The multi-modal fusion mathematics test question multi-knowledge point classification method according to claim 7 is characterized in that: Based on the multimodal gating mechanism, the image feature representation guided by the text feature of the math test question and the text feature representation guided by the image feature of the math test question are fused to obtain the fused feature, which is specifically formulated as follows: in, Represents text features and image features, Represents the weight matrix for linear transformation of text features, represents the bias vector, represents the image features after linear transformation, Represents the weight matrix for linear transformation of image features, represents the text feature vector input, represents the bias vector, g j represents the gating signal, σ represents the activation function, W gj represents the weight matrix, represents the text features after linear transformation, m j Represents the fused features.

9. The multi-modal fusion mathematics test question multi-knowledge point classification method according to claim 8, characterized in that: The fused features are filtered to obtain the filtered features using the following formula: Among them, s j represents the gating signal, represents the weight matrix, represents the weight matrix, represents the bias vector, r j represents the filtered multimodal features, represents the weight matrix, represents the bias vector, represents the filtered features, Represents the weight matrix of the fused features obtained by linear transformation.

10. The multi-modal fusion mathematics test question multi-knowledge point classification method according to claim 1, characterized in that: Based on the filtered features, the knowledge point classification result is obtained by using the following formula: in, is the final result of predicting the knowledge point label, λ is the mixed distribution ratio factor of the two prediction label methods, represents the predicted label output by KNN, Represents the predicted label of the fully connected layer output.

Images