Image-text test question knowledge point labeling method and system based on quaternion hypergraph

Abstract

The invention relates to an image-text test question knowledge point labeling method and system based on a quaternion hypergraph, belongs to the technical field of intelligent education, and aims to solve the problem of low knowledge point labeling accuracy caused by neglect of implicit knowledge features and insufficient multi-stage interaction between modals in the prior art. According to the technical scheme, the method comprises the steps of obtaining explicit and implicit four-dimensional features of two modes of text and vision of an image-text test question; a cross-modal interaction analysis module is constructed, a text hypergraph and a visual hypergraph are constructed based on explicit and implicit feature similarity, and then a unified quaternion hypergraph fusion multi-modal feature is constructed; a modal independence loss function is introduced to keep mutual independence of each modal; proposing a knowledge consistency loss function to optimize cross-modal knowledge consistency; and updating features and optimizing network parameters through a modal complementation module to complete knowledge point labeling. According to the method, high-order association of explicit and implicit features can be effectively mined, and cross-modal semantic alignment precision and knowledge point labeling accuracy are improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent education technology and relates to a method and system for annotating knowledge points in graphic and textual test questions based on quaternion hypergraphs. Background Technology

[0002] In the context of educational informatization, improving education quality and achieving personalized learning for students have become important research directions. Text-and-image test questions, as key learning resources in subject education, contain both question text and accompanying images, conveying knowledge intent from multiple dimensions and helping students gain a deeper understanding of knowledge points. In recent years, the field of artificial intelligence education has seen a surge in intelligent applications based on test question knowledge point annotation, such as question answering, cognitive diagnosis, knowledge tracking, and test paper generation. These applications collect millions of test questions as learning resources to assess students' knowledge mastery and provide personalized learning services. However, the core of realizing these intelligent applications lies in accurately uncovering the hidden knowledge intent within text-and-image test questions and completing precise knowledge point annotation. This makes text-and-image test question knowledge point annotation a key research topic for the efficient organization and management of large-scale question banks.

[0003] With the significant advancements in deep learning technology, research on the annotation of knowledge points in text and image test questions has developed rapidly, and there are currently two main research directions: multimodal feature representation methods and fusion network methods.

[0004] Multimodal feature representation methods aim to extract effective features from the text and images of image-based test questions to support knowledge point annotation. Traditional methods typically utilize deep learning algorithms to learn global textual and visual features. For example, some studies employ bidirectional encoder representation transformation models to extract semantic features from the question text, while others utilize residual networks and visual transformers to extract global visual features from the images. These methods represent each modality as a one-dimensional vector, primarily focusing on global appearance features but neglecting fine-grained representations at the region or word level. To extract more discriminative features, region-based methods combined with attention mechanisms are widely used to highlight key image regions and question words. Some studies have proposed dual attention frameworks that explore intramodal and intermodal interactions through self-attention and directed attention mechanisms, effectively capturing key information in text and images. Similarly, other studies have proposed multi-element hypergraph networks, which filter noise and highlight key information such as keyword positions and sentences by constructing hypergraph structures. However, these methods mainly focus on learning explicit semantic representations of images and sentences, failing to fully explore implicit knowledge features reflecting knowledge intent, such as knowledge keywords in text or knowledge entities in images, thus limiting further improvements in annotation performance.

[0005] Multimodal fusion module design is another key research direction, aiming to effectively learn the interaction between textual and visual modalities. Based on the fusion form, fusion strategies are mainly divided into three types: early fusion, mid-stage fusion, and late-stage fusion. Early fusion methods directly concatenate textual and visual features at the input stage to generate a unified representation, but due to the lack of deep interaction, annotation accuracy is limited. Late-stage fusion methods fuse multimodal features at the end of the model, but similarly fail to fully capture the interaction information between modalities. To overcome these limitations, mid-stage fusion methods achieve dense multimodal interaction in the middle of the neural network, becoming the current mainstream fusion strategy. Research has proposed a visual-language bidirectional encoder representation conversion model that effectively fuses image and textual cues by jointly learning visual and language representations. Similarly, visual-language converter models further simplify computational complexity while maintaining efficient modal interaction. Other research indicates that fusion networks not only need to consider the complementarity of multimodal features but should also retain the independent attributes of each modality to improve annotation performance. However, these models often fail to incorporate the knowledge characteristics of the educational domain into the fusion strategy and lack modeling of implicit knowledge features, resulting in insufficient interpretability and accuracy in knowledge point annotation tasks.

[0006] Current multimodal fusion methods have significant shortcomings in the annotation of knowledge points in text and image test questions: First, existing methods mainly focus on explicit semantic information in text and visual space, such as global semantic features and visual features, while ignoring implicit knowledge information, which is crucial for accurately identifying knowledge intent; second, existing fusion strategies only focus on the complementarity between modalities and fail to fully capture the deep multi-level interactions between explicit and implicit features, thus limiting the improvement of knowledge point annotation performance. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a method and system for labeling knowledge points in text and image test questions based on quaternion hypergraphs, which aims to efficiently and accurately identify knowledge points in text and image test questions, improve the quality of intelligent question bank management and personalized learning services in the education field, and contribute to the development of educational informatization.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A method for annotating knowledge points in test questions based on quaternion hypergraphs includes the following steps:

[0010] S1: Obtain the explicit and implicit feature representations of the text-image test questions, and extract the explicit text features E of the text modality for each test question. t With textual implicit knowledge features K t And the visual dominant feature E of the visual modality v With visual tacit knowledge feature K v ;

[0011] S2: Construct a cross-modal interaction analysis module, build text hypergraphs and visual hypergraphs based on the feature similarity of text modality and visual modality respectively, and build quaternion hypergraphs based on the similarity of explicit and implicit feature spaces. Generate modality-consistent enhanced joint representations through hypergraph neural networks;

[0012] S3: Introduce a modal independence loss function as a local regularization term to maximize the separability of single-modal representations;

[0013] S4: A knowledge consistency loss function is proposed as a local regularization term, and the implicit knowledge features L in the text are measured using Euclidean distance. t With visual tacit knowledge feature K v Correlation, optimize cross-modal knowledge consistency;

[0014] S5: Update the explicit and implicit features of text and visual modalities through the modal complementation module, optimize network parameters by combining modal independence loss function and knowledge consistency loss function, and complete the annotation of knowledge points for text and image test questions.

[0015] Furthermore, in S1, explicit text features E are extracted. t The specific method is as follows: the text of the graphic test questions is represented as a sequence of words. High-level text features are extracted using the pre-trained language model BERT (Bidirectional Encoder Representations from Transformers), and... Label and aggregate text sequence information to form explicit text feature vectors. Where d i Indicates the feature dimension.

[0016] Furthermore, in S1, visual dominant feature E is extracted. v The specific method is as follows: [The original image is then processed / processed]. Reconstructed into a sequence of image patches Where H represents the image height, W represents the image width, C represents the number of channels, P represents the image patch size, and N = HW / P 2 Visual features are extracted using a Transformer encoder, and image patch information is aggregated and labeled with [class] to form a visual explicit feature vector. Where d v Represents the dimension of visual features.

[0017] Furthermore, in S1, implicit knowledge features K are extracted from the text. t The specific method is as follows: construct a knowledge attribute vocabulary K through a knowledge feature mapping network. a ={a1,a2,…,a M}, where M represents the vocabulary size, a heterogeneous graph ρ = (v, ε, θ) is constructed with the question text as the central node, where v represents the set of nodes, ε represents the set of edges, and θ represents the node relationships, and a sliding window C of length p is used. win Scan the context of knowledge attribute words to generate a sequence of neighbor word vectors. Knowledge representation is generated through aggregation operations. Textual implicit knowledge features are generated through latent knowledge space mapping and attention weighting. Where d k This represents the dimension of knowledge characteristics.

[0018] Furthermore, in S1, visual implicit knowledge features K are extracted. v The specific approach is as follows: Knowledge objects in the image are detected using a pre-trained Faster R-CNN (Region-based Convolutional Neural Network), retaining the top q objects with a confidence level higher than a threshold ξ, where ξ is a preset threshold and q is the number of objects; coordinate text is recognized using an OCR (Optical Character Recognition) system, and the text is converted into word vectors using a pre-trained GloVe model; a knowledge hypergraph G is then constructed. k ={V k ,ε k W k} and visual hypergraph G v ={V v ,ε v W v Visual implicit knowledge features are generated by fusing hypergraph convolution and multi-head attention mechanisms.

[0019] Furthermore, the quaternion hypergraph in S2 is constructed as follows: the explicit text features E... t Visual dominant feature E v Textual tacit knowledge features K t and visual tacit knowledge features K v These are respectively taken as the real part and three imaginary parts of the quaternion vector, forming the quaternion vector q = E. t +E v ·i+K t ·j+K v •k; Constructing a text hypergraph G based on the K-nearest neighbor algorithm t ={V t ,ε t W t} and visual hypergraph G v ={V v ,ε v Wv},in Represents the set of text modal vertices. Represent the set of vertices of the visual modality; construct a quaternion hypergraph G by taking the intersection of hyperedges. q ={V q ,ε q W q}, where the correlation matrix H q ∈R n×m satisfy If and only if m represents the number of hyperedges; feature updates are achieved using the quaternion hypergraph convolution operator.

[0020]

[0021] in Represents the quaternion vector of the l-th layer. Represents the quaternion weight matrix. This represents the Hamiltonian product.

[0022] Furthermore, the modal independence loss function L in S3 d The calculation method is as follows:

[0023] explicit features of text and visual dominant features After L2 norm normalization, we get and

[0024] Define a similarity matrix S, where the elements are... Indicates dot product similarity;

[0025] Select the most relevant question set Q using Top-N filtering. h The loss function is expressed as:

[0026]

[0027] Where τ is the temperature hyperparameter and Q is the training set.

[0028] Furthermore, the knowledge consistency loss function L in S4 k The calculation method is as follows:

[0029] Features of implicit knowledge in text K t and visual tacit knowledge features K v After L2 norm normalization, we get and

[0030] Calculate the feature distance using Euclidean distance:

[0031]

[0032] The loss function is expressed as:

[0033]

[0034] Where n is the number of questions.

[0035] Furthermore, the total loss function in S5 is the cross-entropy loss L. c Modal independence loss L d And knowledge consistency loss L k Weighted sum:

[0036] L total =L c +λ1L d +λ2L k

[0037] Where λ1 and λ2 are the loss coefficients, L c The calculation method is as follows:

[0038]

[0039] Where K is the number of knowledge points, and Q is the training set. To predict the label distribution, y i This represents the true label distribution.

[0040] A graphical and textual test question knowledge point annotation system for implementing the method includes:

[0041] The feature extraction module is used to extract explicit text features E using the BERT model. t Visual dominant features E are extracted using a Transformer encoder. v The text implicit knowledge features L are extracted through a knowledge feature mapping network. t Visual implicit knowledge features K are extracted using a knowledge-aware dual hypergraph network. v ;

[0042] The cross-modal interaction analysis module is used to construct quaternion hypergraphs and generate enhanced joint representations through hypergraph neural networks;

[0043] The loss calculation module is used to calculate modal independence loss and knowledge consistency loss;

[0044] The parameter optimization module is used to update network parameters through the backpropagation algorithm and output the knowledge point annotation results.

[0045] The beneficial effects of this invention are as follows:

[0046] (1) This invention breaks through the limitations of traditional multimodal feature representation, and for the first time deconstructs the text and image information in graphic test questions into a dual dimension of explicit semantic features and implicit knowledge features. By constructing a quaternion vector space, explicit text features, explicit visual features, implicit text features, and implicit visual knowledge features are organically integrated to form a unified mathematical representation framework. This structured feature representation method can simultaneously capture surface semantic information and deep knowledge intent, laying a solid foundation for accurate knowledge point recognition.

[0047] (2) The quaternion-based hypergraph modeling method can effectively mine complex high-order similarity relationships between test items. By constructing a unified hypergraph structure through the intersection of textual and visual hypergraphs, this invention achieves deep semantic alignment across modalities. The multi-layer convolutional operations of the hypergraph neural network can capture the non-linear interaction between explicit semantics and implicit knowledge, thereby generating more discriminative enhanced joint representations. This high-order association mining mechanism significantly improves the accuracy and robustness of knowledge point annotation.

[0048] (3) This invention innovatively designs a dual regularization constraint strategy. The modality independence loss function ensures that each modality feature maintains its unique representational characteristics during the fusion process, avoiding performance degradation caused by feature confusion. The knowledge consistency loss function promotes semantic unity of different modalities at the knowledge level through distance measurement of the implicit knowledge space. These two constraint mechanisms work together to maintain the complementary advantages between modalities and strengthen the consistency of knowledge representation.

[0049] (4) Addressing the specific needs of the education field, this invention deeply integrates the characteristics of subject knowledge. By constructing a knowledge feature mapping network and a knowledge perception hypergraph, the system can effectively identify key terms in text and knowledge entities in images, simulating the learner's cognitive process. This deep adaptation to educational scenarios enables this invention to demonstrate better interpretability and practicality in knowledge point annotation tasks, providing reliable technical support for personalized learning services.

[0050] (5) This invention achieves complete process optimization from feature extraction to knowledge point prediction. Through the dynamic feature update mechanism of the modality complementarity module, the system can adaptively adjust the contribution weight of each modality feature. The weighted fusion strategy of the total loss function ensures the stable convergence of the model during training, while taking into account both the performance of the main task and the auxiliary constraint objectives. This end-to-end optimization design makes this invention have better generalization ability and stability in practical applications.

[0051] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0053] Figure 1 This is a schematic diagram of the method of the present invention. Detailed Implementation

[0054] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0055] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0056] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0057] Please see Figure 1This diagram illustrates a method for annotating knowledge points in text-based test questions using quaternions and hypergraphs. This method is primarily used for annotating knowledge points in text-based test questions within educational settings. The key technical problems addressed include insufficient interaction between explicit and implicit features in multimodal text-based test question annotation, feature confusion caused by modal fusion, and inconsistencies in multimodal knowledge representation.

[0058] Step 1: As Figure 1 As shown in the feature extraction module, the purpose of this module is to obtain the explicit and implicit feature representations of the text-image test questions, extracting the explicit and implicit four-dimensional features of each test question in two modalities: text t and visual v, including the explicit text features E of the text modality. t With textual tacit knowledge features L t And the visual dominant feature E of the visual modality v With visual tacit knowledge feature K v First, we introduce the explicit textual features E of the text modality. t Acquisition method: The question text is represented as a sequence of words. This paper employs the pre-trained language model BERT as the text encoder, and extracts high-level text features through a stacked self-attention mechanism and a multilayer perceptron (MLP) module, thereby obtaining the final explicit text feature vector.

[0059] Next, the dominant visual feature E of the visual modality will be introduced. v Acquisition method: Obtain the original image Reconstructed into a series of two-dimensional image blocks As input to the Transformer, image patches are mapped to a fixed-dimensional vector space through a trainable linear projection, generating image patch embeddings, which, along with positional encodings, serve as input to the Transformer. During fine-tuning, a learnable vector is added before the embedded image patch sequence. The tagging process aggregates information from all image patches to form the final visual dominant feature vector.

[0060] Then, we introduce the textual implicit knowledge feature K of the text modality. tAcquisition Method: To acquire implicit textual knowledge features of text modalities, a knowledge feature mapping network was constructed. First, TF-IDF was used to mine subject-specific terms from textbooks / supplementary materials, and TextRank was used to extract scenario keywords from a question bank. The two types of vocabulary were merged and their weights were normalized to form a knowledge attribute vocabulary. Then, a heterogeneous graph was built with the "question text" as the central node and the knowledge attribute words and their context words as related nodes, arranged in word order. Next, a sliding window of length p was used to scan the context of each knowledge word to obtain a sequence of neighboring word vectors. Knowledge word vectors were generated through multi-layer aggregation methods such as summation, averaging, or concatenation, and then concatenated in their original order to form an initial knowledge representation. Then, a "latent knowledge space" was constructed—a scenario association matrix recorded the weights of knowledge words in each knowledge scenario and dynamically adjusted using a decay factor. The knowledge representation was mapped to this space through a learnable projection matrix H, and scenario attention was applied. After focusing on key scenarios, a weighted summation was performed to obtain the final knowledge features. Finally, the semantic vectors extracted by pre-trained Word2Vec and BERT were projected and concatenated with the knowledge features. After weight calculation by a non-linear attention layer, element-wise multiplication was performed to obtain the implicit textual knowledge feature K. t .

[0061] Finally, we introduce the visual implicit knowledge feature E of visual modalities. v Acquisition Method: To acquire visual implicit knowledge features of visual modalities, a knowledge-aware dual hypergraph network was constructed. First, a pre-trained Faster R-CNN was used to detect knowledge objects in the image, retaining the q candidate boxes with the highest confidence and performing mean pooling to obtain object embeddings. Then, OCR was used to recognize coordinates / text in the image, and the text embeddings were obtained through GloVe mapping and mean pooling. The two were concatenated and passed through a fully connected layer to generate the initial knowledge embedding. Subsequently, a knowledge hypergraph was constructed using "typical images" as hyperedges and knowledge entity combinations as connection conditions. Vertex-hyperedge weights were calculated based on entity confidence, and L layers of bidirectional convolution ("vertex→hyperedge→vertex") were used to obtain fine-grained knowledge features. Simultaneously, a CNN was used to extract 2048-dimensional visual embeddings and construct a visual hypergraph, which was then updated through hypergraph convolution to obtain enhanced visual features. Finally, the knowledge features and visual features were concatenated and input into multi-head attention (MHA), and adaptively fused through scaling dot product attention to output visual implicit knowledge features K. v .

[0062] Step Two: As Figure 1 As shown in the cross-modal complementarity module, this module constructs hypergraphs for text and visual modalities respectively to connect similar test items within their respective modalities. Then, based on the similarity between the two in the explicit and implicit feature spaces, a unified quaternion hypergraph is constructed to fuse multimodal features. Finally, a hypergraph neural network is used to capture the higher-order association between explicit semantics and implicit knowledge to generate modally consistent enhanced joint representations, thereby improving the cross-modal semantic alignment accuracy of text and image test items.

[0063] The four-dimensional features of the two modalities are passed through a fully connected layer to map the output features to the h-dimensional dimension, resulting in... The specific details are shown in the formula:

[0064] E t =Linear(E t ;θ et ),E v =Linear(E v ;θ vv ).

[0065] K t =Linear(K) t ;θ kt ),K v =Linear(K) v ;θ kv ).

[0066] Explicit and implicit features are used as the four components of a quaternion vector. t E is considered the real part of the quaternion q. t K t K v The problem's quaternion is represented by the following equation, consisting of three imaginary parts:

[0067] q = E t +E v ·i+K t ·j+K v ·k

[0068] Where i, j, and k are imaginary parts and satisfy the quaternion rule: i 2 =j 2 =k 2 =ijk=-1, the interdependencies and interactions between these components are achieved through Hamiltonian products during training. Natural coding.

[0069] Based on the feature similarity between textual and visual modalities, a text hypergraph G is constructed using the K-nearest neighbor algorithm. t ={V t ,ε t W t} and visual hypergraph G v ={V v ,ε v W v The specific process is as follows:

[0070] For a text modal set of questions, each vertex Initialize as text explicit feature E t and textual implicit knowledge features K tThe joint representation of; For a set of visual modal problems, each vertex Initialize as text explicit feature E t and textual implicit knowledge features K t The joint representation of the text hypergraph. Then construct the hyperedges of the text hypergraph, each hyperedge... by Centered on a point, connect the K adjacent vertices with the most similar features to obtain... The construction of visual hyperedges is consistent with that of text hyperedges, resulting in... Hyperedge weight diagonal matrix W t and W v The result is obtained by normalizing using Euclidean distance, as shown in the equation:

[0071]

[0072] Where dist() represents the Euclidean distance. and They are respectively and The set of K nearest neighbors.

[0073] Based on this, a text-visual hypergraph association matrix is ​​constructed. The association matrix represents the relationship between vertices and hyperedges, as shown in the equation:

[0074]

[0075] If the questions are similar in both textual and visual feature spaces, they belong to the same knowledge point, thus constructing a quaternary hypergraph G. q ={V q ,ε q W q}. Where vertex v q The four-dimensional quaternion vector q = E t +E v ·i+K t ·j+K v ·k; The hyperedge is the intersection of the text and visual hyperedges: ε q =ε t ∩ε v Meanwhile, the correlation matrix H was defined. q ∈R n×m As shown in the formula:

[0076]

[0077] in m is the number of quadruple hyperedges, and n = m (each vertex corresponds to one hyperedge).

[0078] Self-joins are introduced based on the quaternary hypergraph correlation matrix, as shown in the equation:

[0079]

[0080] Where I is the identity matrix. Calculate the vertex degree matrix. and hypermarginality matrix As shown in the formula:

[0081]

[0082] in Let be a diagonal matrix of hyperedge weights, if Then all superedges have equal weight.

[0083] Next, a quaternion hypergraph convolution operator is used to capture higher-order interactions between text and image questions. This is achieved using quaternion vectors. As input to the convolutional layer, it passes through a quaternion filter matrix θ q =θ t +θ v i+θ kv j+θ kt The feature vectors of the image-text test questions are continuously updated in the k-quaternion space. The quaternion hypergraph convolutional layer is defined as follows:

[0084]

[0085] Where σ(·) is a nonlinear activation function, The quaternion vector for the image and text test questions at level l. Let l be the quaternion weight matrix of the l-th layer. This represents the Hamiltonian product, used to calculate two quaternions. and Multiplication, This is the updated quaternion vector at the (l+1)th layer.

[0086] Step 3: As Figure 1 As shown in the modality independence module, the purpose of this module is to maintain the mutual independence of each modality during multimodal fusion by maximizing the separability of the single-modal representation.

[0087] First, L2 norm is used for semantic features. and visual features Normalize, denoted as and Then, the similarity between the two modalities is defined as follows: Where <,> denotes the dot product, thus yielding the similarity matrix S, used to characterize the similarity E between different problems. t and E v The similarity. Then, the semantic relevance score is defined as shown in the equation:

[0088]

[0089] The set of highly relevant questions is selected by TopN and then removed from Q. The final loss function is shown in the following equation:

[0090]

[0091] Where τ is the temperature hyperparameter, which is maximized within the same problem. and By minimizing the feature similarity between different problems after removing highly correlated problems, the modal feature separation degree is maximized.

[0092] Step Four: As Figure 1 As shown in the knowledge consistency module, the purpose of this module is to optimize cross-modal knowledge consistency by measuring the relevance of implicit knowledge features of different modalities through Euclidean distance, and to optimize the fusion of multimodal test features and the rapid convergence of the network by minimizing the distance between text and visual knowledge features.

[0093] First, the L2 norm is used to normalize the textual and visual knowledge features, denoted as . and Then, using the Euclidean distance metric and The distance between them, denoted as , is calculated as follows:

[0094]

[0095] A knowledge consistency loss function was designed based on the relative distance of knowledge feature vectors from different modalities, so that K in the same problem... t and K v They are becoming increasingly similar. The expression for the knowledge consistency loss function is as follows:

[0096]

[0097] Step Four: As Figure 1 As shown in the knowledge point prediction module, the purpose of this module is to update the explicit and implicit features of the text and visual modal through the modality complementarity module, combine independence loss and knowledge consistency loss to jointly optimize the network parameters, further enrich the feature representation of the text and image test questions, and complete the model training through backpropagation to achieve the annotation of knowledge points in the text and image test questions.

[0098] First, the joint feature vector, obtained by concatenating these explicit and implicit features, is input into multiple fully connected (FC) layers to form the image-text test question X. q′ The final representation of it, its functional expression is shown in the following equation:

[0099]

[0100] Where, θq and θ l The parameters for the convolutional layer and the fully connected layer are d, respectively. c This refers to the dimension representing the problem. The final representation. The input is fed into the softmax layer to obtain the confidence score for each knowledge point. The model is trained using the cross-entropy loss function, commonly used in multi-class classification tasks, as defined below:

[0101]

[0102] Where K is the number of knowledge points, and Q is the training set. To predict the label distribution, y i This represents the true label distribution.

[0103] Cross-entropy loss serves as the primary loss function, assisting in the training of the knowledge point annotation task; independence loss and knowledge consistency loss serve as auxiliary losses, further optimizing parameters and accelerating the training process. The total loss is a weighted sum of the three, as shown in the equation below:

[0104] L = L c +λ1L d +λ2L k ,

[0105] λ1+λ2=1

[0106] Among them, L c L d and L k Let λ1 and λ2 represent the cross-entropy loss, independence loss, and knowledge consistency loss, respectively, and λ1 and λ2 be the corresponding loss coefficients. The parameters of the entire network are automatically learned by minimizing the total loss on the training set.

[0107] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it can implement the steps of the method. The storage medium may be, for example, ROM / RAM, magnetic disk, optical disk, etc.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for annotating knowledge points in text and image test questions based on quaternion hypergraphs, characterized in that: Includes the following steps: S1: Obtain the explicit and implicit feature representations of the text-image test questions, and extract the explicit text features E of the text modality for each test question. t With textual implicit knowledge features K t And the visual dominant feature E of the visual modality v With visual tacit knowledge feature K v ; S2: Construct a cross-modal interaction analysis module, build text hypergraphs and visual hypergraphs based on the feature similarity of text modality and visual modality respectively, and build quaternion hypergraphs based on the similarity of explicit and implicit feature spaces. Generate modality-consistent enhanced joint representations through hypergraph neural networks; S3: Introduce a modal independence loss function as a local regularization term to maximize the separability of single-modal representations; S4: A knowledge consistency loss function is proposed as a local regularization term, and the implicit knowledge features K of the text are measured using Euclidean distance. t With visual tacit knowledge feature K v Correlation, optimize cross-modal knowledge consistency; S5: Update the explicit and implicit features of text and visual modalities through the modal complementation module, optimize network parameters by combining modal independence loss function and knowledge consistency loss function, and complete the annotation of knowledge points for text and image test questions.

2. The method for annotating knowledge points in text-based test questions based on quaternion hypergraphs according to claim 1, characterized in that: The explicit text feature E is extracted in S1. t The specific method is as follows: the text of the graphic test questions is represented as a sequence of words. High-level text features are extracted using the pre-trained language model BERT (Bidirectional Encoder Representations from Transformers), and... Label and aggregate text sequence information to form explicit text feature vectors. Where d i Indicates the feature dimension.

3. The method for annotating knowledge points in text-based test questions based on quaternion hypergraphs according to claim 1, characterized in that: Visual dominant feature E is extracted from S1. v The specific method is as follows: [The original image is then processed / processed]. Reconstructed into a sequence of image patches Where H represents the image height, W represents the image width, C represents the number of channels, P represents the image patch size, and N = HW / P 2 Visual features are extracted using a Transformer encoder, and image patch information is aggregated and labeled with [class] to form a visual explicit feature vector. Where d v Represents the dimension of visual features.

4. The method for annotating knowledge points in text-based test questions based on quaternion hypergraphs according to claim 1, characterized in that: The implicit knowledge feature K of the text is extracted in S1. t The specific method is as follows: construct a knowledge attribute vocabulary K through a knowledge feature mapping network. a ={a1,a2,…,a M }, where M represents the vocabulary size, a heterogeneous graph ρ = (v, ε, θ) is constructed with the question text as the central node, where v represents the set of nodes, ε represents the set of edges, and θ represents the node relationships, and a sliding window C of length p is used. win Scan the context of knowledge attribute words to generate a sequence of neighbor word vectors. Knowledge representation is generated through aggregation operations. Textual implicit knowledge features are generated through latent knowledge space mapping and attention weighting. Where d k This represents the dimension of knowledge characteristics.

5. The method for annotating knowledge points in text-based test questions based on quaternion hypergraphs according to claim 1, characterized in that: Visual implicit knowledge feature K is extracted in S1. v The specific approach is as follows: Knowledge objects in the image are detected using a pre-trained Faster R-CNN (Region-based Convolutional Neural Network), retaining the top q objects with a confidence level higher than a threshold ξ, where ξ is a preset threshold and q is the number of objects; coordinate text is recognized using an OCR (Optical Character Recognition) system, and the text is converted into word vectors using a pre-trained GloVe model; a knowledge hypergraph G is then constructed. k ={V k ,ε k W k } and visual hypergraph G v ={V v ,ε v W v Visual implicit knowledge features are generated by fusing hypergraph convolution and multi-head attention mechanisms.

6. The method for annotating knowledge points in text-based test questions based on quaternion hypergraphs according to claim 1, characterized in that: The quaternion hypergraph in S2 is constructed as follows: the explicit text features E are... t Visual dominant feature E v Textual tacit knowledge features K t and visual tacit knowledge features K v These are respectively taken as the real part and three imaginary parts of the quaternion vector, forming the quaternion vector q = E. t +E v ·i+K t ·j+K v •k; Constructing a text hypergraph G based on the K-nearest neighbor algorithm t ={V t ,ε t W t } and visual hypergraph G v ={V v ,ε v W v },in Represents the set of text modal vertices. Represent the set of vertices of the visual modality; construct a quaternion hypergraph G by taking the intersection of hyperedges. q ={V q ,ε q W q }, where the correlation matrix H q ∈R n×m satisfy If and only if m represents the number of hyperedges; feature updates are achieved using the quaternion hypergraph convolution operator. in Represents the quaternion vector of the l-th layer. Represents the quaternion weight matrix. This represents the Hamiltonian product.

7. The method for annotating knowledge points in text-based test questions based on quaternion hypergraphs according to claim 1, characterized in that: The modal independence loss function L in S3 d The calculation method is as follows: explicit features of text and visual dominant features After L2 norm normalization, we get and Define a similarity matrix S, where the elements are... Indicates dot product similarity; Select the most relevant question set Q using Top-N filtering. h The loss function is expressed as: Where τ is the temperature hyperparameter and Q is the training set.

8. The method for annotating knowledge points in text-based test questions based on quaternion hypergraphs according to claim 1, characterized in that: The knowledge consistency loss function L in S4 k The calculation method is as follows: Features of implicit knowledge in text K t and visual tacit knowledge features K v After L2 norm normalization, we get and Calculate the feature distance using Euclidean distance: The loss function is expressed as: Where n is the number of questions.

9. The method for annotating knowledge points in text-based test questions based on quaternion hypergraphs according to claim 1, characterized in that: The total loss function in S5 is the cross-entropy loss L. c Modal independence loss L d And knowledge consistency loss L k Weighted sum: L total =L c +λ1L d +λ2L k Where λ1 and λ2 are the loss coefficients, L c The calculation method is as follows: Where K is the number of knowledge points, and Q is the training set. To predict the label distribution, y i This represents the true label distribution.

10. A graphic and text-based test question knowledge point annotation system for implementing the method of any one of claims 1 to 9, characterized in that: include: The feature extraction module is used to extract explicit text features E using the BERT model. t Visual dominant features E are extracted using a Transformer encoder. v The implicit knowledge feature K of the text is extracted through a knowledge feature mapping network. t Visual implicit knowledge features K are extracted using a knowledge-aware dual hypergraph network. v ; The cross-modal interaction analysis module is used to construct quaternion hypergraphs and generate enhanced joint representations through hypergraph neural networks; The loss calculation module is used to calculate modal independence loss and knowledge consistency loss; The parameter optimization module is used to update network parameters through the backpropagation algorithm and output the knowledge point annotation results.

Images