A multi-modal fusion mathematical test multi-knowledge point classification method

Abstract

This application discloses a multimodal fusion method for classifying multiple knowledge points in mathematical test questions, relating to the field of knowledge point classification. The method includes: extracting formula features from mathematical test questions; extracting text features from mathematical test questions; extracting image features from mathematical test questions; obtaining image feature representations guided by text features of mathematical test questions; obtaining text feature representations guided by image features of mathematical test questions; fusing the image feature representations guided by text features of mathematical test questions and the text feature representations guided by image features of mathematical test questions based on a multimodal gating mechanism to obtain fused features; filtering the fused features to obtain filtered features; and using KNN classification with joint contrastive learning enhancement based on the filtered features to obtain the knowledge point classification result. The above method in this application can effectively improve the accuracy and universality of mathematical test question classification, providing an effective technical solution for tasks such as intelligent test paper generation and personalized test question recommendation in the field of intelligent education.

Description

Technical Field

[0001] This application relates to the field of knowledge point classification, and in particular to a multimodal fusion method for classifying multiple knowledge points in mathematical test questions. Background Technology

[0002] Currently, leveraging artificial intelligence technology to gain a deeper understanding of diverse online learning resources has become a key research topic in intelligent education. Mathematics test questions, as one of the most fundamental data units among a vast amount of learning resources, are a crucial factor in evaluating the effectiveness of mathematics teaching. However, due to the semantic and structural complexity of online mathematics test question data, traditional NLP classification methods are no longer applicable.

[0003] Mathematical test questions possess unique conceptual and logical characteristics. They contain not only a large number of technical terms and mathematical symbols in their textual modality but also intuitive mathematical semantics such as quantity and geometry in their visual modality. Both types of information provide important clues for understanding mathematical concepts and knowledge points. Therefore, in research on the classification of multiple knowledge points in mathematical test questions, the integration of mathematical text and visual information is key to understanding the conceptual and logical nature of mathematical test questions. The logical structure and symbolic system in mathematical test questions, as well as their close association with knowledge point labels, are not found in other disciplines. Currently, many studies neglect the extraction of semantic information from test questions containing mathematical images, yet this visual information is equally crucial for understanding the content of the test questions. Therefore, if the classification of mathematical test question data is only considered as a multi-label text classification task, there is still room for improvement in the effectiveness of the classification results. Besides the shortcomings in semantic extraction from mathematical test questions, the hierarchical nature of knowledge points is also an undeniable factor. Test questions are usually compiled based on textbook content, which is organized according to chapters and other hierarchical levels. Therefore, knowledge points also exhibit a clear hierarchical structure from elementary to advanced levels, and the number of knowledge points increases with each level. As the levels increase and the number of knowledge points grows, higher demands are placed on the performance of classification algorithms, and new challenges are brought about. Summary of the Invention

[0004] The purpose of this application is to provide a multimodal fusion method for classifying multiple knowledge points in mathematical test questions, which effectively integrates local information and contextual logic of mathematical test questions, and performs vector representation of text with formulas, thereby more accurately capturing the knowledge points tested in mathematical test questions.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] This application provides a multimodal fusion method for classifying multiple knowledge points in mathematical test questions, including:

[0007] Extracting the formulaic features from math test questions;

[0008] Extracting textual features from math test questions;

[0009] Extracting image features from math test questions;

[0010] Obtain image feature representations guided by the text features of the mathematical test questions;

[0011] Obtain the text feature representation guided by the image features of the mathematical test question;

[0012] Based on a multimodal gating mechanism, the image feature representation guided by the text features of the math test questions and the text feature representation guided by the image features of the math test questions are fused to obtain the fused features;

[0013] The fused features are filtered to obtain filtered features.

[0014] Based on the filtered features, the knowledge point classification results are obtained.

[0015] Optionally, the extraction of formula features from mathematical test questions specifically includes the following steps:

[0016] The mathematical formula is converted into a symbolic appearance tree and an operator tree; the symbolic appearance tree represents the layout structure of the mathematical formula, and the operator tree represents the operational relationship of the mathematical formula.

[0017] The symbol appearance tree and operator tree are traversed in a depth-first manner to generate a tuple (n1, n2, E, ERP); where n1 represents the ancestor node, n2 represents the descendant node, E represents the edge sequence from n1 to n2, and FRP represents the complete path relative to the ancestor node.

[0018] The tuples are input into the trained FastText model to obtain the formula vector.

[0019] Optionally, the extraction of text features from mathematical test questions specifically includes the following steps:

[0020] The math test text is segmented into words to obtain a segmented sequence.

[0021] The segmented sequence is input into the trained RoBERTa model to obtain word vectors of the math test text;

[0022] The formula vector and word vector are input into the SA-TCNN module to generate a test text feature map;

[0023] The feature maps of the test questions are concatenated to obtain the feature vector representation of the test questions.

[0024] Optionally, the feature maps of the test questions are concatenated to obtain the feature vector representation of the test question text, specifically using the following formula:

[0025]

[0026] in, This represents the feature vector of the test text corresponding to different convolution kernel windows.

[0027] Optionally, the extraction of image features from mathematical test questions specifically employs the following formula:

[0028]

[0029] ∧

[0030] Among them, v I W represents the image feature vector. I V represents the weight matrix of the linear layer transformation. I b represents the image vector input. I This represents the bias vector.

[0031] Optionally, the image feature representation guided by the text features of the mathematical test question specifically adopts the following formula:

[0032]

[0033] Among them, z j This represents the output of the j-th test question sample. Represents the weight matrix. Represents the features of the original text. This represents the bias vector. This represents vector concatenation. Let v represent the weight matrix. I α represents the image feature vector guided by text features. j Let v be the attention distribution, representing each image feature v. i Corresponding to text features The importance of Indicate z j The weight matrix, This represents the bias vector. For image feature representation guided by text features, α j,i Let represent the ith attention distribution coefficient of the j-th question image.

[0034] Optionally, the text feature representation guided by the image features of the mathematical test question is obtained using the following formula:

[0035]

[0036] Among them, z j ' represents the output of the j-th test item sample. Represents the weight matrix. W represents the bias vector.T Let h represent the weight matrix. T β represents the feature vector of the test question text. j Indicates the distribution of attention. Represents the weight matrix. This represents the bias vector. β represents the text feature vector output guided by image features. j,k This represents the k-th attention distribution coefficient of the j-th question text. This represents the feature vector of the test text after the weight matrix has been applied.

[0037] Optionally, based on a multimodal gating mechanism, the image feature representation guided by the text features of the math test question and the text feature representation guided by the image features of the math test question are fused to obtain the fused features, specifically using the following formula:

[0038]

[0039] in, Representing text features and image features, This represents the weight matrix used for the linear transformation of text features. This represents the bias vector. Represents the image features after linear transformation. This represents the weight matrix used for the linear transformation of image features. This represents the text feature vector input. G represents the bias vector. j Let W represent the gating signal, σ represent the activation function, and W represent the activation signal. gj Represents the weight matrix. m represents the text features after linear transformation. j This indicates the characteristics after fusion.

[0040] Optionally, the fused features are filtered to obtain the filtered features using the following formula:

[0041]

[0042] Among them, s j Indicates a gating signal. Represents the weight matrix. Represents the weight matrix. R represents the bias vector. j This represents the filtered multimodal features. Represents the weight matrix. This represents the bias vector. Indicates the filtered features. This represents the weight matrix obtained by linear transformation to fuse the features.

[0043] Optionally, the knowledge point classification result is obtained based on the filtered features using the following formula:

[0044]

[0045] in, This is the final result of predicting knowledge point labels, where λ is the mixed distribution ratio factor of the two label prediction methods. This represents the predicted label output by KNN. This represents the predicted label output by the fully connected layer.

[0046] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0047] This application provides a multimodal fusion method for classifying multiple knowledge points in mathematical test questions. First, it effectively integrates local information and contextual logic of mathematical test questions to generate vector representations for text containing formulas, thereby more accurately capturing the knowledge points tested in the test questions. Second, considering the inherent correlation between images, text, and knowledge points in mathematical test questions, it comprehensively extracts and fuses text and image features, enhancing the model's multimodal information processing capabilities. Finally, it utilizes the similarity between samples and between knowledge point labels to optimize multi-label classification performance, and designs more accurate models and selection strategies for knowledge point classification problems at different levels, effectively improving the accuracy and universality of mathematical test question classification. This provides an effective technical solution for tasks such as intelligent test paper generation and personalized test question recommendation in the field of intelligent education. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the overall framework of the MMMQMC model in one embodiment of this application;

[0050] Figure 2 This is a schematic diagram of the symbol appearance tree in one embodiment of this application;

[0051] Figure 3 This is a schematic diagram of an operator tree in one embodiment of this application;

[0052] Figure 4 This is a schematic diagram of the SA-TCNN module structure in one embodiment of this application;

[0053] Figure 5This is a schematic diagram of the improved image feature extraction model structure in one embodiment of this application;

[0054] Figure 6 This is a schematic diagram of image feature representation guided by text features in one embodiment of this application;

[0055] Figure 7 This is a schematic diagram of image feature-guided text feature representation in one embodiment of this application;

[0056] Figure 8 This is a schematic diagram of image and text feature fusion representation in one embodiment of this application;

[0057] Figure 9 This is a schematic diagram of the CL-KNN module structure in one embodiment of this application;

[0058] Figure 10 This is an example of test questions for a test paper generation network in one embodiment of this application;

[0059] Figure 11 This is an example of local test question source data in one embodiment of this application;

[0060] Figure 12 This is an example of a cleaned test question from one embodiment of this application;

[0061] Figure 13 This is a schematic diagram of the test question length distribution in a dataset according to one embodiment of this application;

[0062] Figure 14 This is a schematic diagram of the distribution of secondary knowledge points in a dataset in one embodiment of this application;

[0063] Figure 15 This is a schematic diagram of the classification results of common primary knowledge points in one embodiment of this application;

[0064] Figure 16 This is a schematic diagram of the classification results of common secondary knowledge points in one embodiment of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] In current research on the classification of knowledge points in mathematics test questions, the application of traditional classification algorithms often fails to fully consider the unique complexity of mathematics. Because mathematics test questions involve numerous knowledge points, complex structures, and possess a certain degree of logic and reasoning, traditional classification algorithms have limitations in handling these characteristics. Therefore, these algorithms often perform poorly in classifying mathematics test questions, failing to accurately identify and classify multiple knowledge points within the questions. To address this challenge, and considering the multi-knowledge-point and multi-modal characteristics of mathematics test question data, this application proposes to solve the following key problems:

[0068] (1) Given the rich subject characteristics contained in the text of mathematical test questions, how to effectively integrate local information and contextual logic to make vector representations of texts with mathematical formulas, so as to more accurately capture the knowledge points tested in mathematical test questions;

[0069] (2) Considering that mathematical images also carry important mathematical semantics and have an inherent connection with text and knowledge points, how to comprehensively extract and integrate mathematical text and image features to enhance the multimodal information processing capability of the model;

[0070] (3) In view of the uneven distribution of mathematical test question set samples and the multi-level characteristics of knowledge points, we study how to use the similarity between samples and between knowledge point labels to optimize the performance of multi-label classification, and design more accurate models and selection strategies in knowledge point classification problems at different levels.

[0071] Finally, the problem of classifying multiple knowledge points in math test questions is abstractly defined as follows:

[0072] The goal is to obtain the mapping from X to Y. This is achieved by training a classification model Θ such that Θ(X)→Y, meaning the model can accurately predict the knowledge points covered in new test questions. Where X = {x1, x2, ..., x...} n} represents the input math problem, where x∈{x w ,x e ,x g}, each including text description x w Mathematical expression x e and mathematical graph x g These are various types of modal information. X is specifically a multimodal dataset of length n, where each sample contains a text sequence, a sequence of mathematical expressions, and image data. Y = {y1, y2, ..., y...} m} represents the set of knowledge point tags associated with question X, where each tag represents a specific mathematical knowledge point. Y is a multi-hot encoded tag sequence of length m, where each tag corresponds to a knowledge point and takes a value of 0 or 1, indicating whether the knowledge point is related to the question.

[0073] See Figure 1 This application proposes a multi-modal math question multi-knowledge point classification model—MMMQMC (Multi-Modal Math Question Multi-Label Classification Model). When extracting text features from math questions, the model captures the structural and semantic information of the formulas, achieving efficient formula embedding representation. For the pure text portion of the questions, a text representation method incorporating a self-attention mechanism is designed to better capture important contextual information. When extracting image features, a linear layer is proposed to transform image features, achieving dimensional alignment between image and text features. When fusing multi-modal features, a multi-modal math question data fusion method based on a collaborative attention mechanism is proposed, avoiding the loss of inter-modal correlation information. When classifying math question knowledge points, a joint contrastive learning-enhanced KNN (CL-KNN) mechanism is proposed. This mechanism weightedly fuses the classification results of the fully connected layer with the label results of similar samples obtained from the KNN retrieval, resulting in the final predicted multi-label result, thus improving the performance of the classification model.

[0074] The above model framework consists of three main modules: a test text feature extraction module combining self-attention mechanism (Self-Attention TextCNN, SA-TCNN), a multimodal test feature fusion module combining collaborative attention mechanism (Cooperative Text-Image Feature FusionModule, CTIFM), and a KNN classification module jointly enhanced by contrastive learning (Contrastive Learning-EnhancedKNN, CL-KNN).

[0075] The specific steps are as follows:

[0076] Step 1: Extract the formula features of the math test questions.

[0077] Step 2: Extract textual features from math test questions.

[0078] Steps 1 and 2 involve the SA-TCNN module, which incorporates a self-attention mechanism for extracting features from mathematical test questions. This module extracts features from the mathematical test questions by dividing the text into two parts: formulas and plain text, which are then embedded. First, for the formula part, the TangentCFT (Tangent Combined with FastText) method is used to capture the structural and semantic information of the formula, achieving efficient formula embedding. The transformation steps of TangentCFT are as follows:

[0079] The mathematical formula is first converted into Symbol Layout Trees (SLT) and Operator Trees (OPT). The Symbol Layout Tree represents the layout structure of the mathematical formula, while the Operator Tree represents the operational relationships within the formula. Taking formula x... 2 Taking +y=5 as an example, the SLT tree of this formula is as follows: Figure 2 As shown, the OPT tree is as follows Figure 3 As shown.

[0080] In both of these tree structures, nodes represent various symbols in the formula, such as operators "+", "=", numbers, etc. The format of a node is "Type!Val", where "Type" refers to the type of the node and "Val" is the value of the node. Examples of common node types in tree structures are shown in Table 1.

[0081] Table 1 Examples of node types in the tree structure

[0082]

[0083] Taking object X as an example, there are 7 types of spatial relationships between nodes in SLT, as shown in Table 2-2, which represent the relationship type and symbol respectively.

[0084] Table 2 shows the edge relation types in SLT.

[0085]

[0086] In OPT, edges represent the positions of operators. For commutative operators (such as the "=" operator), where changing the order does not affect the result, both sides of the operator are marked with "0". Otherwise, the marks on both sides of the operator increment from "0" to "1". Next, the constructed tree structure needs to be traversed to obtain a linear sequence, which is then further parsed into tuples. In this way, mathematical formulas are transformed into embedded representations that the model can process.

[0087] For example: Formula x 2 The SLT and OPT nodes with +y = 5 are shown in Table 3.

[0088] Table 3x 2 SLT and OPT nodes with +y=5

[0089]

[0090] After generating the sequence using a depth-first traversal of the tree structure, tuples of the form (n1, n2, E, FRP) are used to represent the generated tuples of node pairs and their relative positions. Here, n1 represents the ancestor node, n2 represents the descendant node, E represents the edge sequence from n1 to n2, and FRP represents the complete path relative to the ancestor node, giving the position of n1 on its path starting from the root. Formula x 2 The SLT and OPT traversal tuples for +y=5 are shown in Table 4 below, with FRP ignored in the tuple sequence.

[0091] Table 4x 2 +y=5 tuples for formula tree traversal

[0092]

[0093] Ultimately, each element in a tuple is treated as an independent character, allowing it to be fed into the FastText model for training and outputting a formula vector. This formula vector serves as part of the input to the SA-TCNN module.

[0094] Secondly, for the plain text portion, the RoBERTa pre-trained model is used to obtain word vectors for the test question text. The RoBERTa model is an improved version of BERT (A Robustly Optimized BERT). A text representation method combining self-attention and the TextCNN structure is employed. Unlike general text, math test questions often require consideration of the logical relationships within the context. Each mathematical concept, condition, and reasoning step in the question is interconnected, and these relationships are reflected through the semantics of the context. The self-attention mechanism dynamically allocates different attention weights based on the relationships between words, thereby better capturing important contextual information. The specific steps are as follows:

[0095] Step 2.1: Load a pre-trained RoBERTa model. Method: Load a pre-trained RoBERTa model instance using Hugging Face's Transformers library.

[0096] Step 2.2: Input the math test text, and use a word segmenter to preprocess the text to obtain the result: the input sequence after word segmentation.

[0097] Step 2.3: Encode the sentence by feeding the preprocessed input sequence into the pre-trained RoBERTa model to obtain the result and get the word vector representation of each token.

[0098] Step 2.4: Extract the hidden layer from the model output to obtain the word vector representation of the math test text.

[0099] Through the above steps, this application successfully used the pre-trained language model RoBERTa to represent the plain text portion of mathematical test questions using word vectors. These word vectors serve as another input to the SA-TCNN module.

[0100] The word vectors and formula vectors are simply concatenated and input into the SA-TCNN module. The structure of the SA-TCNN module is as follows: Figure 4 As shown.

[0101] The SA-TCNN model consists of attention layers, convolutional layers, and pooling layers. First, word vectors and formula vectors are used as input to the network, passing through the first attention layer to assign higher weights to the key features of the math problem text. Next, convolutional kernels with different window sizes are designed to capture features of the math problem text at different granularities. The generated feature map of the problem text after convolutional kernel processing is shown in the following formula:

[0102]

[0103] C = [C1, C2, ..., C k ]

[0104] Among them, l j H represents the kernel window size. j Let C represent the convolution kernel, and C represent the feature maps corresponding to all convolution kernels. Max pooling is then applied to the features obtained after convolution to output the important features from the test text, as shown in the following formula:

[0105]

[0106] Finally, the important features of the test text corresponding to different convolutional kernel windows are concatenated to obtain the final feature vector representation of the test text, as shown in the following formula:

[0107]

[0108] Step 3: Extract the image features of the math test questions.

[0109] Specifically, the ResNet152 deep learning model is used to extract features from images of math test questions, mainly including the following steps:

[0110] Step 3.1: Load a pre-trained ResNet152 model instance using the pre-trained model library of the deep learning framework PyTorch.

[0111] Step 3.2: Perform image standardization and resizing on the image of the math test questions to obtain the image tensor suitable for input into the ResNet152 model.

[0112] Step 3.3: Use the ResNet152 model for forward propagation to the result. First, obtain the feature map, and then use the ResNet152 model for forward propagation, excluding the final fully connected layer.

[0113] Step 3.4: Apply global average pooling to obtain a feature vector of length 2048. Then, use a fully connected layer to reduce the dimensionality of the feature vector of length 2048 to a feature vector of length 11.

[0114] Step 3.5: Divide the feature map into multiple blocks and arrange them sequentially to form a new feature sequence. The output of the last layer... Then through a linear layer Convert to vector representation v I This ensures that the dimensions of image features and text features are consistent. The conversion formula is:

[0115] Through the above steps, this application captures local features of the image, providing richer feature representations for the classification of mathematical test question images. The improved image feature extraction model structure is as follows: Figure 5 As shown.

[0116] Step 4: Obtain the image feature representation guided by the text features of the math test question.

[0117] Step 5: Obtain the text feature representation guided by the image features of the math test question.

[0118] In multimodal math question classification tasks involving multiple knowledge points, the effective fusion of text and image features is crucial for improving model performance. However, due to the differences in information expressed by text and images, simply concatenating or weighting the features of these two modalities may prevent the model from fully capturing the intermodal relationships, thus affecting its performance. To address this issue, a multimodal question data fusion method based on a collaborative attention mechanism, called CTIFM (Cooperative Text-Image Feature Fusion Module), is proposed. The specific implementation of this module is described below.

[0119] Image feature representation guided by the acquisition of test text features

[0120] To focus more attention on the parts of the image feature representation that are relevant to the test text, text-guided attention is used to obtain image feature representations. The steps are as follows: Figure 6 As shown, the calculation formula for this step is as follows:

[0121]

[0122] Specifically, firstly, the features of the original text... Perform a linear transformation, where, It is a weight matrix. This is the bias vector; the purpose of the linear transformation is to map the text features to a new space. Then, the result of the linear transformation of the text features is compared with the image features v. I Linear transformation (by weight matrix) (Determine) to concatenate the features along the dimension. Map the concatenated features to the range [-1, 1] and use the softmax function to calculate the attention distribution α. j It represents each image feature v i Corresponding to text features The degree of importance. Finally, based on the calculated α j For v i By performing weighted summation, we obtain the image feature representation guided by text features. It integrates all image features, highlighting the parts that are relevant to the current text features.

[0123] Text feature representation guided by the acquisition of test question image features

[0124] To focus more attention on the parts of the text feature representation that are relevant to the test question image, image-guided attention is used to obtain the text feature representation. The steps are as follows: Figure 7 As shown.

[0125] Attention is calculated using the acquired image feature representations and text features to generate an attention distribution. Then, the text features are weighted according to the attention distribution to obtain the image feature-guided text feature representation of the test questions. The calculation method for this step is as follows:

[0126]

[0127] Specifically, firstly, image features Perform a linear transformation to map it to a new space, where is Weight matrix, It is the bias vector. Next, the linear transformation result of the image features is compared with the text features h. T The linear transformation result (from the weight matrix W) T (Determine) to concatenate the features along the dimension. Map the concatenated features to the range [-1, 1] and use the softmax function to calculate the attention distribution β. j It represents each text feature Corresponding to image features The degree of importance. Finally, based on the calculated β j right By performing weighted summation, we obtain the text feature representation guided by image features. It integrates all text features, highlighting the parts that are relevant to the current image features.

[0128] Step 6: Based on the multimodal gating mechanism, fuse the image feature representation guided by the text features of the math test questions and the text feature representation guided by the image features of the math test questions to obtain the fused features.

[0129] Step 7: Filter the fused features to obtain filtered features.

[0130] Through the aforementioned collaborative attention mechanism, richer representations of test text and image features can be obtained. To further integrate these two types of features, this application designs a multimodal gating mechanism, such as... Figure 8 As shown.

[0131] This mechanism dynamically adjusts the weights of text and image features during the fusion process based on the degree of correlation between them. This allows for more flexible capture of the correlation information between text and images, thereby improving classification performance. The calculation method for the multimodal gating mechanism is shown in the following formula.

[0132]

[0133] Specifically, firstly, linear transformations are performed on the obtained image features of the test question text and the text features of the test question image to obtain the text features. and image features Then apply the weight matrix W gj The sigmoid activation function will be determined by and The concatenated feature vector outputs a gated signal g. j This is used to control the contribution of text features and image features to the final feature representation. Finally, the features are dynamically fused based on the gating signal. and When g j When g approaches 1, the fused features will primarily consist of text features; when g... j When the value approaches zero, the fused features will primarily consist of image features. A multimodal gating mechanism can effectively combine textual and image information.

[0134] However, since not all important features of the text and image in a math problem are correlated, this invention also introduces a filtering gate mechanism to filter noise in the fused features of the text and image. The introduction of the filtering gate helps the model more accurately identify the key correlation information between the text and image, further improving classification performance. The calculation method of the filtering gate mechanism is shown in the following formula.

[0135]

[0136] Specifically, first, analyze the features of the original text. Perform a linear transformation, the weight matrix is Next, the multimodal fusion feature m j Perform a linear transformation, the weight matrix is Then, the linear transformation results of the two features are concatenated in the same dimension, and the sigmoid function is applied to obtain the gate signal s of the filter gate. j This will determine which features are retained in the final representation. Then, the weight matrix is ​​used. For m j Perform the linear transformation again, and compare the result with s. j Multiply to obtain the filtered multimodal features r j This operation is equivalent to retaining only the features that the gating signals deem important. Finally, With r j The features are concatenated and then subjected to a linear transformation to obtain the final fused feature representation. It combines textual features with filtered multimodal fusion features.

[0137] By proposing the CTIFM module, we can better handle the correlations between multimodal test data. First, the introduction of the attention mechanism allows the model to focus more on parts related to text or images, reducing noise interference. Second, the multimodal gating mechanism can dynamically adjust weights based on the correlation between text and image features, enabling the model to more flexibly fuse the two types of features. Finally, the introduction of the filtering gate further reduces noise in the fused features, improving the model's robustness.

[0138] Step 8: Based on the filtered features, obtain the knowledge point classification results.

[0139] In multi-knowledge point classification tasks for math test questions, the conventional approach is to use a fully connected layer to output the knowledge point classification results. However, traditional fully connected layer methods often fail to fully utilize the label similarity information between samples, resulting in limited classification performance.

[0140] To address this issue, this application proposes a classification algorithm based on a Joint Contrastive Learning-Enhanced KNN (CL-KNN) mechanism.

[0141] like Figure 9As shown, the classification results of the fully connected layer and the label results of similar samples to the sample to be predicted obtained by KNN are weighted and fused to obtain the final result of multi-label prediction. By designing the objective function and dynamic contrastive learning coefficients for multi-label contrastive learning, label similarity information can be better utilized to improve the performance of the classification model.

[0142] To assist in selecting the KNN mechanism, a multi-label contrastive learning objective function is introduced. This objective function measures the similarity between samples by comparing the label similarity of sample pairs. Simultaneously, a dynamic contrastive learning coefficient is designed, calculated based on the label similarity between samples. The greater the label similarity between samples, the larger the contrastive learning coefficient, thereby increasing the loss value of that sample pair.

[0143] The specific steps are as follows:

[0144] To facilitate understanding of this process, we will use x to represent the feature vector output by the multimodal question feature fusion module. That is, x is used as the input of this module.

[0145] Step 8.1: Construct a local storage (Datastore) for the training samples, which is the storage that generates the mapping relationship between the corresponding fused features and the corresponding knowledge point classifications.

[0146]

[0147] Among them, y i This corresponds to the knowledge point of the fused feature sample j.

[0148] Step 8.2: The prediction result after the fully connected layer FL is y FL ∈{p|p∈[0,1]} L Using the corresponding knowledge point classification mapping relationship as query clues, find k nearest neighbor samples in the Datastore:

[0149]

[0150] Step 8.3: Based on the k nearest neighbors from the above steps, provide the predicted labels for the corresponding KNN:

[0151] Where d refers to the Euclidean distance, τ is the KNN hyperparameter, and α i It is the weight of the i-th neighbor.

[0152] Step 8.4: The final predicted label for x is represented as follows: in λ is the final result of predicting knowledge point labels, and λ is the mixed distribution ratio factor of the two prediction label methods.

[0153] In the classification algorithm of this application, the multimodal features of the test item are first mapped to a low-dimensional space through a fully connected layer, and an intermediate result is output. Then, the KNN mechanism is used to select the training sample most similar to the test item to be classified, and the result of the KNN mechanism is obtained. Finally, the results of the fully connected layer and the KNN mechanism are combined through a weighted fusion to obtain the final label prediction result for the new test item sample. This weighted fusion takes into account the importance of the results of the fully connected layer and the KNN mechanism in the classification.

[0154] In summary, the beneficial effects of this application are as follows:

[0155] First, the test feature extraction method proposed in this application, which combines self-attention mechanism, can dynamically allocate different attention weights according to the relationship between words, thereby better capturing important contextual information. On the other hand, it preserves the spatial structure information of the image and can better capture local features in the image.

[0156] Secondly, the multimodal test item feature fusion method proposed in this application, compared with methods that simply concatenate or weight text and image features, can better capture the correlation information between multimodal test item data, thus improving the robustness of the model. First, the introduction of an attention mechanism allows the model to focus more on parts related to the text or image, reducing noise interference. Second, the multimodal gating mechanism can dynamically adjust the weights according to the degree of correlation between text and image features, enabling the model to fuse the two features more flexibly. Finally, the introduction of a filtering gate further reduces noise in the fused features.

[0157] Third, the KNN classification method with joint contrastive learning enhancement proposed in this application solves the problem of limited classification performance caused by the inability to fully utilize the label similarity information between samples, compared with the traditional fully connected layer method.

[0158] To verify the effectiveness of the proposed MMMQMC model on real-world datasets, this application constructs a large-scale dataset of mathematical test questions. Experimental results are then used to analyze the improved classification of multiple knowledge points in the test questions.

[0159] Experimental setup

[0160] This application's experiments are primarily based on the PyTorch deep learning framework and the pytorch_pretrained_bert library, using an Intel(R) Core i5-1035G1 CPU@1.00GHz and an RTX2080 GPU. In the experimental model settings, for uniform processing, the length of the test text was standardized to 180 characters. For texts shorter than 180 characters, padding was used to complete them; otherwise, they were truncated. Furthermore, the test images were uniformly resized to 224*224*3 pixels, and after processing, each image was converted into a 7x7 feature map. In the TextCNN design, the window size of the convolutional kernels was set to [2,3,4], and the number of convolutional kernels for each size was 256. Other important model parameters are shown in Table 5.

[0161] Table 5 shows the configuration of some model parameters.

[0162]

[0163] Experimental data

[0164] Research on the classification of multiple knowledge points in mathematics test questions currently lacks publicly available Chinese datasets. Therefore, this application constructed a real-world dataset using web scraping technology. This dataset, sourced from an online test paper generation website, covers high school mathematics test questions, including mathematical images, and their corresponding knowledge point labels. The construction of this dataset provides the foundation for the experiment and ensures the practicality and relevance of the research. An example of a mathematics test question before data collection is shown below. Figure 10 As shown.

[0165] Because the test questions were collected from the internet, the source data was in Hypertext Markup Language (HTML). The source data, after collection, was stored in a local database as follows: Figure 11 As shown, the formulas are in Mathematical Markup Language (MathML), and the mathematical images are in the form of HTTPS source links.

[0166] Using Python regular expressions and HTTP request libraries, the test questions containing mathematical images were filtered, and the images and text were saved locally to form the final dataset of all high school math test questions containing mathematical images.

[0167] During the self-built dataset process, after collecting the webpage test question data, data cleaning was required to improve data quality. The cleaning process consisted of several steps: First, irrelevant characters such as spaces and line breaks were removed from the text, while mathematical formulas were preserved in MathML format to ensure readability and parsing. Second, for the text content other than formulas, the jieba word segmentation tool was used for Chinese word segmentation to extract effective vocabulary information. Finally, for the mathematical formulas, the TangentCFT parsing method was used to parse the formula content to extract its structured information. After this series of cleaning steps, clean test question text modal data was obtained. In this data, the mathematical formula for each test question was numbered HEL_xxx_WLDOR_x_OL for easy identification and reference in subsequent processing. This data cleaning process ensured the quality of the experimental data, laying a solid foundation for subsequent model training and performance evaluation. An example of the cleaned mathematical test questions is shown below. Figure 12 As shown.

[0168] The final dataset contained 27,663 math problems, each with an average of 69.2 characters, an average of 5.6 mathematical formulas, and one accompanying mathematical image. The distribution of problem length and number is as follows: Figure 13 As shown.

[0169] In addition, knowledge point tags for all test questions were collected, forming a tag dataset. This dataset covers 17 primary knowledge point tags and 427 secondary knowledge point tags, with an average of 1.93 knowledge point tags per test question. The distribution of secondary knowledge point tag IDs and their quantities in the dataset is shown below. Figure 14 As shown.

[0170] Based on the classification of knowledge points, the distribution of the number of questions for the 17 first-level knowledge points in the mathematics test question dataset is shown in Table 6 below.

[0171] Table 6 Distribution of Level 1 Knowledge Points

[0172]

[0173] There are 427 level-two knowledge points. As the knowledge point level increases, the number of knowledge points also increases, which will lead to a decrease in the sample size of test questions for each level-two knowledge point. The following table shows the distribution of the number of test questions for some level-two knowledge points in the mathematics test question dataset.

[0174] Table 7 shows the distribution of some Level 2 knowledge points.

[0175]

[0176] The above steps involved collecting and cleaning math test questions from the internet, and performing a simple statistical analysis on the self-built math test question dataset. The construction and analysis of this dataset provided the data foundation for the experiment, aiding in model training and evaluation, and facilitating further design of the experimental procedure and analysis of the results.

[0177] Evaluation indicators

[0178] In evaluating the performance of classification results in multi-label classification tasks, the selection and design of evaluation metrics are crucial. The goal of multi-label classification tasks is to assign a matching subset of labels to a given sample, requiring the selection of appropriate evaluation metrics to measure the prediction results and assess the effectiveness of the model.

[0179] Common evaluation metrics for multi-label text classification include: precision, recall, F1 score, and Hamming loss.

[0180] In the following formulas, the first formula represents precision, which measures the proportion of samples that the model predicted as positive but were actually positive; the second formula represents recall, which measures the proportion of samples that were actually positive but were correctly predicted as positive by the model; and the third formula is the F1 score, which is the harmonic mean of precision and recall, used to comprehensively evaluate the accuracy and completeness of the model.

[0181]

[0182] The following are the evaluation metrics selected for this application. Macro precision (Precision) macro Macro recall is used to measure the model's predictive accuracy across various categories; macro Precision and recall are then used to evaluate the model's recall across each class. Calculating these metrics involves multiplying the precision and recall of each class by the proportion of that class in the total sample, and then summing the products. Additionally, the macro F1 score (F1...) is also considered. macro Precision (MR) is the harmonic mean of precision and recall, which comprehensively reflects the model's accuracy and recall capabilities. Finally, Hamming Loss focuses on the number of labels misclassified by the model; a smaller value indicates a better model performance. These evaluation metrics together provide a comprehensive assessment of the model's performance.

[0183]

[0184] Experimental Procedure

[0185] The experimental process of this application mainly includes three stages: data acquisition, model training, and result analysis. The specific steps of each stage are as follows.

[0186] Data collection. Data collection is the foundation of experimental preparation, involving the crawling of online test questions, data cleaning, and statistical analysis of the dataset.

[0187] Model Training. During model training, a fastText pre-trained model is first used to obtain vector representations of mathematical formulas, while a RoBERTa model is used to extract word vectors from the mathematical text. Next, the cleaned dataset is divided into an 80% training set, a 10% validation set, and a 10% test set to ensure comprehensive training and objective evaluation. Based on the proposed MMMQMC model, combined with binary cross-entropy loss and multi-label contrastive learning loss, the model is trained. During training, the validation set is used for parameter tuning, and the best-performing model is saved. On the test set, Precision... macro Recall macro F1 macro HL is used as an evaluation metric to assess the model's performance.

[0188] Results Analysis. The results analysis phase aims to comprehensively evaluate the model's performance and effectiveness. Through comparative and ablation experiments, the accuracy and rationality of the model in predicting multiple knowledge point labels for math exam questions are analyzed. Furthermore, model design and strategy selection at different knowledge point levels are considered to explore the model's applicability and generalization ability in different scenarios.

[0189] Through the above experimental procedure, this application aims to comprehensively evaluate the research and implementation of a multi-modal fusion-based method for classifying multiple knowledge points in mathematical test questions. The design and execution of the experimental procedure followed rigorous scientific principles, ensuring the reliability of the experimental results and the effectiveness of the research.

[0190] Experimental Results and Analysis

[0191] To verify the effectiveness of the proposed MMMQMC model, this application conducted multiple comparative and ablation experiments at different knowledge point levels on a self-built multi-knowledge point dataset of mathematical test questions. The comparative experiments selected currently popular mathematical test question classification algorithms as benchmarks, comparing them with the MMMQMC method at different knowledge point levels to verify its advantage in classification accuracy. The ablation experiments evaluated the contribution of each component to the overall classification performance by progressively removing the multimodal fusion components from the model, thereby verifying the rationality of the model design and the importance of each module.

[0192] Comparative experiment

[0193] In the comparative experiments, first-level and second-level knowledge points were selected as labels for multi-classification to verify the effectiveness of the proposed method at different classification granularities. The selected comparison methods included baseline models such as TextCNN and Bi-LSTM, as well as the proposed MMMQMC method. Among these, the first six baseline models only treated mathematical questions as text modalities for feature extraction, while the MMMQMC method employed a multi-modal fusion strategy. The experimental methods are described below.

[0194] (1) TextCNN: A text classification model using convolutional neural networks, which can effectively extract local features of text;

[0195] (2) Bi-LSTM: A text classification model using a bidirectional long short-term memory network, which can fully exploit the contextual information of the text;

[0196] (3) CNN-RNN

[57] : A text classification model that combines convolutional neural networks and recurrent neural networks, which can extract local features of text and capture the sequence information of text.

[0197] (4) Seq2Seq+Att

[58] : A text classification model based on sequence-to-sequence model and attention mechanism, which can automatically learn the representation and classification of text;

[0198] (5)SGM

[59] : a sequence generation structure that considers the correlation between tags and automatically obtains key information of the input text through an attention mechanism;

[0199] (6) RobBERTa+TextCNN: It combines the RobBERTa pre-trained model and the TextCNN text classification model, which can both take advantage of the pre-trained model and extract local features of the text.

[0200] (7)MMMQMC: The proposed multimodal fusion mathematical test question classification method aims to make full use of the text, image and other modal information of mathematical test questions to achieve more accurate classification.

[0201] In the experimental results, the prefix "(-)" indicates that the smaller the value of the indicator, the better the model performance; while "(+)" indicates that the larger the value of the indicator, the better the model performance. Table 8 shows the comparative experimental results labeled with first-level knowledge points.

[0202] Table 8. Comparative Experiment Results for Level 1 Knowledge Points

[0203]

[0204] Table 9 presents the results of the comparative experiment labeled with secondary knowledge points.

[0205] Table 9. Comparative Experiment Results for Level 2 Knowledge Points

[0206]

[0207] Experimental results show that the proposed method outperforms the baseline model, regardless of whether the labels are first-level or second-level knowledge points.

[0208] Specifically, compared to the TextCNN model, the MMMQMC model proposed in this invention achieves higher performance in P-level knowledge points. macro An increase of 5.88%, R macro An improvement of 9.71%, F1 macro An increase of 8.31%; under the second-level knowledge point, P macro An increase of 3.97%, R macro An improvement of 3.92%, F1 macro The improvement was 4.36%. These data fully demonstrate the advantages of the proposed multimodal fusion model in capturing semantic information and its superior performance in the multi-knowledge point classification task of mathematical test questions.

[0209] In analyzing the experimental results, this application found that the proposed method significantly improves the classification performance of first-level knowledge points. This result reveals the clear advantage of the multimodal fusion method in handling basic and core knowledge points. Simultaneously, this finding emphasizes the importance of first-level knowledge points in mathematical exam questions and provides useful insights for further model improvement. Furthermore, the experimental data also show that the proposed method surpasses the benchmark model in the classification accuracy of second-level knowledge points, indicating that the multimodal fusion method maintains a high performance level even when facing more complex and specific knowledge points.

[0210] Next, to verify the improvement of the multimodal fusion model on the classification performance of mathematical test knowledge points, four common first- and second-level knowledge points were selected for in-depth analysis. In the experiment, the baseline model SGM was compared with the proposed MMMQMC model. On the coordinate axis, the number of training samples for knowledge point labels showed a decreasing trend. For example, among the first-level knowledge points, there were more questions in the category of "plane geometry vectors and their applications," while there were fewer questions in the category of "probability." Experimental results show that the MMMQMC model exhibits a significant accuracy improvement on knowledge point labels with a smaller sample size.

[0211] like Figure 15 As shown, in the classification results of common first-level knowledge points, the MMMQMC model outperforms the SGM model in F1. macro The scores have improved. Especially for the relatively fewer "probability" knowledge points, the MMMQMC model achieved more F1 scores compared to the more common "trigonometric functions" knowledge point.macro Value increased.

[0212] Similarly, in the classification results of common secondary knowledge points, such as Figure 16 As shown, the MMMQMC model also exhibits a higher F1 score than the SGM model. macro Scoring. Beyond the common knowledge point of "Graphs and Properties of Trigonometric Functions," the knowledge point of "Applications of Derivatives in Function Research" achieved a more significant F1 score through the MMMQMC model. macro The value is increased, thereby effectively improving the overall classification performance.

[0213] Analysis of the comparative experimental results leads to the conclusion that relying solely on single-modal test text information is insufficient to support effective classification decisions. Introducing images can supplement semantic information missing from the text, thereby improving the model's classification performance. Furthermore, the effectiveness of the MMMQMC model in handling imbalanced samples is verified, and the crucial role of multimodal fusion in improving the accuracy of knowledge point classification in mathematics test questions is emphasized.

[0214] ablation experiment

[0215] Next, ablation experiments were conducted to verify the effectiveness of each component of the proposed model and to evaluate the contribution of each module to the performance. Experiments were still performed using first- and second-level knowledge points as classification labels. The SA-TCNN, CTIFM, LAM, and CL-KNN modules were removed from the model sequentially. Here, "w / o SA-TCNN" represents the SA-TCNN layer as a reference; "w / o Co-att" represents the multimodal collaborative attention layer in the CTIFM module as a reference; "w / o Fus-gate" represents the fusion gate mechanism in the CTIFM module as a reference; "w / o LAM" represents the LAM module as a reference; "w / o KNN" represents the KNN-assisted retrieval mechanism in the CL-KNN module as a reference; "w / o CL" represents the contrastive learning mechanism in the CL-KNN module as a reference; and "w / o CL-KNN" represents the complete CL-KNN module as a reference.

[0216] Table 10 shows the ablation experiment results under the first-level knowledge points.

[0217] Table 10 Ablation Experiment Results under Level 1 Knowledge Points

[0218]

[0219] First, it was observed that removing the SA-TCNN layer (without SA-TCNN) significantly impacted classification performance. Compared to the proposed MMMQMC model, it resulted in a lower P... macro It decreased by 1.98%, R macro It decreased by 1.97%, F1 macroThe decrease was 2.48%. This result demonstrates that, on a Level 1 knowledge point dataset, the SA-TCNN layer is crucial for extracting local features from text, especially in capturing key concepts and terms in math problems.

[0220] Secondly, removing the co-attention layer (w / o Co-att) in CTIFM has a relatively small impact on performance, P macro R macro and F1 macro The decreases were 1.07%, 1.04%, and 1.56%, respectively. This may indicate that the correlation between text and labels is weak on the first-level knowledge point dataset, resulting in limited effectiveness of the collaborative attention layer.

[0221] Further observation revealed that removing the fusion gate mechanism (without fusion gate) in CTIFM had a relatively small impact, with the decreases in various metrics being 1.17%, 1.11%, and 1.64%, respectively. This may reflect that on the first-level knowledge point dataset, the correlation between text and other modalities is not strong, and the contribution of the multimodal fusion gate is limited.

[0222] However, removing the LAM module (without LAM) had a significant impact on classification performance, P macro R macro and F1 macro The percentages decreased by 1.95%, 2.14%, and 2.70%, respectively. This indicates that the label-based semantic attention module plays a crucial role in weighting different input features and understanding the concepts and logical relationships of mathematical questions on the first-level knowledge point dataset.

[0223] Finally, removing the CL-KNN module has a relatively small impact on performance. Specifically, removing KNN-assisted retrieval (w / oKNN) results in P... macro R macro and F1 macro The decreases were 0.65%, 0.71%, and 1.23%, respectively; the decreases after removing the contrastive learning mechanism (w / o CL) were 0.62%, 0.64%, and 1.13%, respectively; and the decreases after removing the complete CL-KNN module (w / o CL-KNN) were 0.84%, 0.81%, and 1.33%, respectively. These results indicate that the CL-KNN module has a relatively small effect on improving the model's classification performance at the first-level knowledge points.

[0224] In summary, on the Level 1 knowledge point dataset, the SA-TCNN layer and LAM module have a significant impact on classification performance, while other modules contribute relatively little to performance. These findings provide important references for further model optimization and adjustments to experimental design.

[0225] Table 11 shows the ablation experiment results under the secondary knowledge points.

[0226] Table 11 Ablation Experiment Results under Level 2 Knowledge Points

[0227]

[0228] With secondary knowledge points serving as classification labels, this application further analyzes the impact of each module to enhance understanding of the function of each module in the secondary knowledge point dataset and its contribution to model performance.

[0229] First, removing the TextCNN layer that incorporates self-attention (without SA-TCNN) has a relatively small impact on classification performance, P macro R macro and F1 macro The percentages decreased by 0.08%, 0.10%, and 0.09%, respectively. This result indicates that on the second-level knowledge point dataset, the improved TextCNN layer is less critical for capturing complex logic in math problems than for basic concepts and definitions, and therefore its impact on classification performance is weaker compared to the first-level knowledge point dataset.

[0230] Secondly, removing the collaborative attention mechanism (w / o Co-att) and the fusion gate mechanism (w / o Fus-gate) from CTIFM significantly impacted classification performance. macro R macro and F1 macro The decreases were 0.69%, 0.50%, and 0.60%, and 0.89%, 0.70%, and 0.79%, respectively. This indicates that on the secondary knowledge point dataset, multimodal collaborative attention and fusion gate mechanisms are crucial for integrating feature information from different modalities, which is particularly important for understanding complex concepts and logical relationships in mathematical questions. Since secondary knowledge points typically involve the application of multiple concepts and theorems, these modules have a significant impact on classification performance on this dataset.

[0231] Further observation revealed that removing the label semantic attention module (w / o LAM) still had a significant impact on classification performance, P macro R macro and F1 macro The decreases were 1.40%, 0.90%, and 1.15%, respectively. This indicates that the label-based semantic attention module also played an important role on the secondary knowledge point dataset, helping the model to better understand the concepts and logical relationships in math questions.

[0232] Finally, removing the CL-KNN module has a more significant impact on classification performance on the second-level knowledge point dataset compared to the first-level knowledge point dataset. Removing KNN-assisted retrieval (without KNN), Pmacro R macro and F1 macro The decreases were 0.54%, 0.32%, and 0.43%, respectively; the decreases after removing the contrastive learning mechanism (w / o CL) were 1.33%, 1.21%, and 1.27%, respectively; and the decreases after removing the complete CL-KNN module (w / o CL-KNN) were 2.73%, 2.51%, and 2.62%, respectively. This result indicates that, in the secondary knowledge point dataset, the KNN retrieval mechanism and contrastive learning play a crucial role in distinguishing different concepts and logical relationships in mathematical questions by capturing the differences between positive and negative samples, thus having a greater impact on classification performance.

[0233] In summary, in the context of a secondary knowledge point dataset, the CTIFM and CL-KNN modules significantly impacted classification performance, while other modules contributed relatively little to the model performance.

[0234] A comprehensive analysis of ablation experiments using first- and second-level knowledge points as labels reveals the correlation between the model's performance on different knowledge point datasets and the performance of each module. On the first-level knowledge point dataset, the SA-TCNN and LAM modules significantly impact classification performance; these two modules are crucial for grasping the fundamental concepts and logical relationships in mathematical problems. On the second-level knowledge point dataset, the CTIFM and CL-KNN modules contribute significantly to classification performance. This finding reveals that as the number of knowledge points increases and the complexity of the classification task rises, these two modules effectively distinguish between positive and negative samples during the learning process, playing a vital role in deeply understanding the complex concepts and logical relationships in mathematical problems. Analyzing the mechanisms by which these modules function at different knowledge point levels can provide a more effective solution for multi-knowledge point classification tasks in mathematical problems.

[0235] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0236] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0237] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0238] This application uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. In summary, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal fusion method for classifying multiple knowledge points in mathematical test questions, characterized in that, The multimodal fusion method for classifying mathematical test questions based on multiple knowledge points includes: Extracting the formulaic features from math test questions; Extracting textual features from math test questions; Extracting image features from math test questions; Obtain image feature representations guided by the text features of the mathematical test questions; Obtain the text feature representation guided by the image features of the mathematical test question; Based on a multimodal gating mechanism, the image feature representation guided by the text features of the math test questions and the text feature representation guided by the image features of the math test questions are fused to obtain the fused features; The fused features are filtered to obtain filtered features. Based on the filtered features, the knowledge point classification results are obtained; Based on a multimodal gating mechanism, the image feature representation guided by the text features of the math test questions and the text feature representation guided by the image features of the math test questions are fused to obtain the fused features, specifically using the following formula: ； in, Representing text features and image features, This represents the weight matrix used to perform a linear transformation on the text features. This represents the bias vector. Represents the image features after linear transformation. This represents the weight matrix used for the linear transformation of image features. This represents the text feature vector input. This represents the bias vector. Indicates a gating signal. This represents the activation function. Represents the weight matrix. Represents the text features after linear transformation. Indicates the characteristics after fusion; Image feature representation guided by text features; The fused features are filtered to obtain the filtered features using the following formula: ； in, Indicates a gating signal. Represents the weight matrix. Represents the weight matrix. This represents the bias vector. This represents the filtered multimodal features. Represents the weight matrix. This represents the bias vector. Indicates the filtered features. This represents the weight matrix obtained by linear transformation to fuse the features; Represents the features of the original text.

2. The multimodal fusion method for classifying multiple knowledge points in mathematical test questions according to claim 1, characterized in that, The extraction of formula features from mathematical test questions specifically includes the following steps: The mathematical formula is converted into a symbolic appearance tree and an operator tree; the symbolic appearance tree represents the layout structure of the mathematical formula, and the operator tree represents the operational relationship of the mathematical formula. The symbol appearance tree and operator tree are traversed in a depth-first manner to generate a tuple (n1, n2, E, FRP); where n1 represents the ancestor node, n2 represents the descendant node, E represents the edge sequence from n1 to n2, and FRP represents the complete path relative to the ancestor node. The tuples are input into the trained FastText model to obtain the formula vector.

3. The multimodal fusion method for classifying multiple knowledge points in mathematical test questions according to claim 2, characterized in that, The extraction of text features from math test questions specifically includes the following steps: The math test text is segmented into words to obtain a segmented sequence. The segmented sequence is input into the trained RoBERTa model to obtain word vectors of the math test text; The formula vector and word vector are input into the SA-TCNN module to generate a test text feature map; The feature maps of the test questions are concatenated to obtain the feature vector representation of the test questions.

4. The multimodal fusion method for classifying multiple knowledge points in mathematical test questions according to claim 3, characterized in that, The feature maps of the test questions are concatenated to obtain the feature vector representation of the test question text, specifically using the following formula: ； in, This represents the feature vector of the test text corresponding to different convolution kernel windows.

5. The multimodal fusion method for classifying multiple knowledge points in mathematical test questions according to claim 1, characterized in that, The specific formula used to extract the image features of math test questions is as follows: ； in, Represents the image feature vector. The weight matrix represents the linear layer transformation. Represents the image vector input. This represents the bias vector.

6. The multimodal fusion method for classifying multiple knowledge points in mathematical test questions according to claim 5, characterized in that, The image feature representation guided by the text features of the mathematical test question specifically adopts the following formula: ； in, Indicates the first j The output of a sample of test questions, Represents the weight matrix. This represents the bias vector. This represents vector concatenation. Represents the weight matrix. This represents an image feature vector guided by textual features. Let be the attention distribution, representing each image feature. Corresponding to text features The importance of express The weight matrix, This represents the bias vector. Indicates the first j The first question image i Attention distribution coefficients.

7. The multimodal fusion method for classifying multiple knowledge points in mathematical test questions according to claim 6, characterized in that, The text feature representation guided by the image features of the mathematical test question specifically adopts the following formula: ； in, Indicates the first j The output of a sample of test questions, Represents the weight matrix. This represents the bias vector. Represents the weight matrix. This represents the feature vector of the test question text. Indicates the distribution of attention. Represents the weight matrix. This represents the bias vector. This represents the output of text feature vectors guided by image features. Indicates the first j The k-th attention distribution coefficient of a test question text This represents the feature vector of the test text after the weight matrix has been applied.

8. The multimodal fusion method for classifying multiple knowledge points in mathematical test questions according to claim 1, characterized in that, Based on the filtered features, the knowledge point classification result is obtained using the following formula: ； in, It is the final result of predicting knowledge point tags. It is a mixed distribution ratio factor of the two prediction labeling methods. This represents the predicted label output by KNN. This represents the predicted label output by the fully connected layer.

Images