A multimodal question-answering method and device integrating learner history

Abstract

The present invention provides a multimodal question-answering method and device that integrates a learner's history, relating to the field of multimodal question-answering technology. The method comprises: embedding knowledge visual data through a ResNet model to obtain a knowledge feature representation; embedding question text through a BERT model to obtain a question feature representation; fusing the knowledge feature representation and question feature representation based on time span to obtain a time-fused knowledge feature; fusing history features with time-fused knowledge features to obtain a fused feature representation; calculating the predicted probability of each candidate answer through the question feature representation and the fused feature representation, and selecting the candidate answer with the highest predicted probability as the final answer to the question text. The present invention accurately locates the learner's knowledge shortcomings and interest preferences based on their learning record data, historical performance data, and knowledge visual data, thereby improving the accuracy of question answers.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal question answering, and in particular to a multimodal question answering method and device integrating learner history. Background Art

[0002] With the rapid development of information technology, people's channels for obtaining information are becoming increasingly abundant, and a large amount of multimodal data such as text, images, audio, and video are emerging. How to effectively process and understand these multimodal data has become an important topic in the field of artificial intelligence.

[0003] The development of artificial intelligence (AI) has prompted the pursuit of systems that are closer to human intelligence. Humans can naturally integrate and interact with multiple sensory information when understanding the world. However, traditional AI systems are often limited to processing single-modal information and cannot meet users' needs for querying and understanding multimodal information. In addition, traditional technologies often lack in-depth consideration of individual learners' differences when dealing with learner issues. For example, in the field of education, different students have different learning abilities, knowledge reserves, and interests and hobbies. However, traditional technologies mostly use a universal model and cannot accurately meet the unique needs of each student. In terms of knowledge association, existing technologies have difficulty effectively integrating multimodal information with learners' previous knowledge. Traditional knowledge systems are often in a single textual format. Even if other modal information such as images is included, it is difficult to organically integrate it with learners' historical knowledge. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a multimodal question-answering method and device that integrates learner history.

[0005] The present invention provides a multimodal question-answering method integrating learner history, comprising the steps of:

[0006] S1: Obtain the learner's identity data and question text, and obtain the learner's learning record data, historical performance data, and knowledge visual data through the identity data;

[0007] S2: Preprocessing, feature extraction and feature weighting of learning record data and historical performance data to obtain learners’ historical features;

[0008] S3: Embed the knowledge visual data through the ResNet model to obtain the knowledge feature representation; embed the question text through the BERT model to obtain the question feature representation; fuse the knowledge feature representation and the question feature representation based on the time span to obtain the time-fused knowledge feature;

[0009] S4: Fuse historical features and time-fused knowledge features to obtain a fused feature representation; calculate the predicted probability of each candidate answer through the question feature representation and the fused feature representation, and use the candidate answer with the highest predicted probability as the final answer to the question text.

[0010] Preferably, step S1 is specifically as follows:

[0011] S11: Using a Python crawler to obtain the learner's learning record data and historical performance data from the learning log file of the online learning platform based on the learner's identity data, where the learning record data includes: course type data, teaching resource access data, and course difficulty data;

[0012] S12: Obtain pictures and videos of the corresponding course from the online learning platform as knowledge visual data based on the course type data.

[0013] Preferably, step S2 is specifically as follows:

[0014] S21: Delete incomplete and missing data in the learning record data and historical performance data; standardize the learning record data through one-hot encoding to obtain learning record features; standardize the historical performance data through the Z-score model to obtain historical performance features;

[0015] S22: The learning record features and historical performance features are weightedly represented through the multi-head attention mechanism to obtain the learner's historical features H.

[0016] Preferably, the knowledge feature representation and the problem feature representation are fused based on the time span in step S3 to obtain the time-fused knowledge feature, specifically:

[0017] S31: Through hierarchical recursion, the knowledge feature representation is divided into n-level knowledge feature vectors, and the problem feature representation is divided into n-level problem feature vectors;

[0018] S32: Calculate the similarity between the knowledge feature vectors and the problem feature vectors at each level, use the similarity at each level as a matrix element, and construct a mapping matrix between the problem and the knowledge;

[0019] S33: Through the pyramid attention mechanism and mapping matrix based on time span, the knowledge feature vectors and question feature vectors at each level are aggregated to obtain the time-fused knowledge features.

[0020] Preferred:

[0021] The expression of time fusion knowledge feature G is:

[0022]

[0023] in, is the fused feature vector obtained by fusing the knowledge feature vector and the question feature vector at the first day level, where is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the second weekly level, where It is the fused feature vector obtained by fusing the knowledge feature vector and the question feature vector at the third monthly level.

[0024] Preferred:

[0025] Fusion feature representation The expression is:

[0026]

[0027] in, represents element-wise product, stands for inverse fast Fourier transform, FFT stands for fast Fourier transform, () represents the attention mechanism, Represents the historical feature H applying the Count-Sketch projection function The result after Represents the time fusion knowledge feature G using the Count-Sketch projection function The result after.

[0028] A storage medium stores instructions and data for implementing the multimodal question-answering method integrating learner history.

[0029] A multimodal question-answering device integrating learner history includes: a processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the multimodal question-answering method integrating learner history.

[0030] The present invention has the following beneficial effects:

[0031] This invention accurately identifies learners' knowledge gaps and interests based on their learning records, historical performance data, and knowledge visual data, improving the accuracy of their responses. Regarding knowledge association, this invention deeply integrates multimodal data such as text and images with the learner's history to promote knowledge transfer and application. Regarding assessment, this invention integrates multiple aspects of historical data to objectively assess learning outcomes and provide timely feedback, helping learners adjust their strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0033] Figure 2 This is a structural diagram of the device according to an embodiment of the present invention;

[0034] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0035] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0036] Reference Figure 1 The present invention provides a multimodal question-answering method integrating learner history, comprising the steps of:

[0037] S1: Obtain the learner's identity data and question text, and obtain the learner's learning record data, historical performance data, and knowledge visual data through the identity data;

[0038] As an example:

[0039] Step S1 is specifically as follows:

[0040] S11: Using a Python crawler to obtain the learner's learning record data and historical performance data from the learning log file of the online learning platform based on the learner's identity data, where the learning record data includes: course type data, teaching resource access data, and course difficulty data;

[0041] S12: Obtain pictures and videos of the corresponding course from the online learning platform as knowledge visual data based on the course type data.

[0042] S2: Preprocessing, feature extraction and feature weighting of learning record data and historical performance data to obtain learners’ historical features;

[0043] As an example:

[0044] Step S2 is specifically as follows:

[0045] S21: Delete incomplete and missing data in the learning record data and historical performance data; standardize the learning record data through one-hot encoding to obtain learning record features; standardize the historical performance data through the Z-score model to obtain historical performance features;

[0046] Specifically, one-hot encoding is used for course type attribute data. Assuming that there are five types of courses: Chinese, mathematics, English, physics, and chemistry, for each course, a binary vector of length 5 is created to represent the course type. For example, for Chinese courses, the encoding is [1,0,0,0,0]; for mathematics courses, the encoding is [0,1,0,0,0]; for English courses, the encoding is [0,0,1,0,0]; for physics courses, the encoding is [0,0,0,1,0]; and for chemistry courses, the encoding is [0,0,0,0,1]. The original categorical variables are converted into numerical variables that can be processed by the subsequent model.

[0047] Convert resource access data into sequence data, for example, recording the order and time intervals in which learners access different resources (e.g., videos, documents). For each access record, represent its characteristics (e.g., resource type, access duration, etc.) as a vector and then arrange them in chronological order to form a sequence.

[0048] The difficulty level data of the courses were pre-processed using a data-driven coding method. Based on the course pass rate, courses with a pass rate greater than 80% were coded as "easy", those between 60% and 80% were coded as "relatively easy", those between 40% and 60% were coded as "medium", those between 20% and 40% were coded as "relatively difficult", and those with a pass rate less than 20% were coded as "difficult". The data were then subjected to one-hot encoding, which can better reflect the actual difficulty level of the course.

[0049] The learner's historical scores are pre-processed using Z-score standardization, including:

[0050] Step S211: First, add up all the scores of a course and then divide the sum by the total number of scores to get the average score, i.e. the mean. ;

[0051] Step S212: for each score in the data set, calculate the difference between it and the mean;

[0052] Step S213: Squaring the obtained differences to obtain square differences;

[0053] Step S214: Add the square differences corresponding to all scores and divide by the total number of scores to obtain the variance;

[0054] Step S215: Take the square root of the variance to obtain the standard deviation , the standard deviation reflects the dispersion of scores relative to the mean;

[0055] Step S216: For each score in the data set, use the expression:

[0056]

[0057] in represents the original score, where represents the average score, where represents the standard deviation;

[0058] Step S217: Repeat this step to convert all original scores into standardized scores. After the calculation is completed, all scores are converted into a data distribution with a mean of 0 and a standard deviation of 1.

[0059] S22: The learning record features and historical performance features are weightedly represented through the multi-head attention mechanism to obtain the learner's historical features H.

[0060] Specifically, step S22 is as follows:

[0061] Step S221: Concatenate the preprocessed four types of data vectors by column to form an input matrix , where for each learner’s record, the course type vector is , the teaching resource access vector sequence is , the course difficulty vector is , the historical performance vector is , then input matrix Y, and the calculation process is as follows:

[0062]

[0063] The semicolon here means column-wise concatenation, and Y represents the input matrix;

[0064] Step S222: Input matrix Through three linear transformation matrices respectively 、 、 , get the query vector , key vector Sum vector , the calculation process is as follows:

[0065] , ,

[0066] Step S223: 、 、 Divided into 8 heads, namely , , , the dimension of each head is ,in is the output dimension of the multi-head attention mechanism;

[0067] Step S224: For each header , the formula for calculating the attention score matrix is

[0068]

[0069] in ,in is the output dimension of the multi-head attention mechanism;

[0070] right Perform a softmax operation to obtain the attention weight matrix, which is expressed as follows:

[0071]

[0072] calculate The output of is expressed as follows:

[0073]

[0074] Step S225: Output of each head Putting them together, we get the output of the multi-head attention mechanism, and the calculation process is as follows:

[0075]

[0076] Where H is the learner's historical characteristics, Represents the concatenation function.

[0077] S3: Embed the knowledge visual data through the ResNet model to obtain the knowledge feature representation; embed the question text through the BERT model to obtain the question feature representation; fuse the knowledge feature representation and the question feature representation based on the time span to obtain the time-fused knowledge feature;

[0078] As an example:

[0079] We use a 152-layer ResNet pre-trained on ImageNet data (ResNet is a deep neural network that uses residual blocks to effectively solve the gradient vanishing and gradient degradation problems in deep neural network training, enabling the network to better learn complex features) to extract knowledge feature representations, specifically:

[0080] Preprocess the input knowledge vision to make it meet the input requirements of the model. Let the preprocessing operation function be , then the preprocessed image can be expressed as:

[0081]

[0082] in represents the original knowledge visual data, Indicates that after preprocessing, it can be directly input into Image data of the model;

[0083] The preprocessed image Input into the pre-trained ResNet model, the model will perform a series of convolution, pooling, residual connection and other operations to gradually extract the features of the image. The calculation formula is:

[0084]

[0085] Represents the extracted knowledge visual features, all knowledge features represent .

[0086] As an example:

[0087] To extract as many important semantic features as possible from the text description of the question while reducing the noise caused by colloquialism, we plan to use a fine-tuned BERT model trained on a large-scale educational question-answering corpus (BERT is a pre-trained language model proposed by Google in 2018. Based on the Transformer architecture, it uses a bidirectional training approach. It learns universal language features through two pre-training tasks: a masked language model and next sentence prediction. It can capture long-range dependencies in text and has strong feature extraction capabilities. After fine-tuning, it has achieved good results in various natural language processing tasks). This model will be used as input for subsequent neural network model calculations. The calculation process is as follows:

[0088]

[0089] in Represents the encoding process of the BERT model, A semantic feature representation that represents the learner's question, represents the learner's question, where n represents the length of the question text.

[0090] As an example:

[0091] In step S3, the knowledge feature representation and the problem feature representation are fused based on the time span to obtain the time fused knowledge feature, specifically:

[0092] S31: Through hierarchical recursion, the knowledge feature representation is divided into n-level knowledge feature vectors, and the problem feature representation is divided into n-level problem feature vectors;

[0093] S32: Calculate the similarity between the knowledge feature vectors and the problem feature vectors at each level, use the similarity at each level as a matrix element, and construct a mapping matrix between the problem and the knowledge;

[0094] Specifically, step S32 is as follows:

[0095] Step S321: Considering the hierarchical characteristics of knowledge, if all knowledge is directly considered in the mapping matrix, the dimension of the matrix will be extremely large and too sparse. Based on this, we construct a mapping matrix based on the hierarchy of knowledge, recursively from top to bottom according to the hierarchy of knowledge, and calculate the hierarchical feature representation through hierarchical recursive multiplication. Specifically, for common m-level knowledge , the semantic feature representation of its knowledge is calculated as follows:

[0096]

[0097] Where j represents the level of knowledge, and a value of 1 represents the lowest level of knowledge;

[0098] Step S322: Calculate the similarity between the question feature vector and the knowledge feature vector using the Manhattan distance method. and The Manhattan distance between them is: , where the smaller the Manhattan distance, the higher the similarity, and the similarity value range is between 0 and 1. When the two vectors are exactly the same, the Manhattan distance is 0 and the similarity is 1. When the distance between the two vectors is farther, the similarity is closer to 0;

[0099] Step S323: Convert the Manhattan distance to similarity. The calculation formula is:

[0100]

[0101] in Represents two vectors and The Manhattan distance between The Manhattan distance calculated by this formula is Convert to problem feature vector and knowledge feature vector similarity between

[0102] Step S324: Based on the similarity calculation results, a mapping matrix between the question and the knowledge is constructed, and the similarity values ​​are used as elements in the mapping matrix. The higher the similarity, the stronger the correlation between the question and the corresponding knowledge.

[0103] Step S325: Perform statistical analysis on all similarity values, observe their distribution, draw a histogram of the similarity values, check the concentration area and dispersion of the similarity values, set the threshold to 0.6, and assign a higher value to the corresponding position in the mapping matrix when the similarity is greater than the threshold, otherwise assign a lower value;

[0104] Step S326: The mapping matrix is ​​represented based on the problem characteristics obtained above. And all knowledge feature representations It is constructed and dynamically maintained based on the learner's interaction. The maintained correlation is intended to define a scoring function based on translation distance:

[0105]

[0106] in Indicates the characteristics of the problem, represents all knowledge features, where express and The L2 norm of the difference;

[0107] The problem-knowledge mapping matrix can help construct the knowledge hierarchy in the time-span-based pyramid attention mechanism. By analyzing factors such as the degree of relevance between knowledge and problems in the matrix and the time it takes to learn the knowledge, knowledge can be divided into different time-span levels. This also provides a screening range for subsequent time-span-based knowledge feature aggregation.

[0108] S33: Through the pyramid attention mechanism and mapping matrix based on time span, the knowledge feature vectors and question feature vectors at each level are aggregated to obtain the time-fused knowledge features.

[0109] Specifically, step S33 is as follows:

[0110] Step S331: Considering that the memory curve actually affects learning efficiency, and that content that was learned more recently is far less helpful for current learning than content that was learned more recently, a pyramid attention mechanism based on time span is proposed for knowledge feature aggregation.

[0111] Step S332: Divide the knowledge into three levels according to the time span: day, week, and month. is the feature vector of the knowledge unit set learned in the last three days, is the feature vector of the knowledge unit set learned in the last 7 days (excluding the last 3 days), The feature vector of the knowledge unit set learned in the last 30 days (excluding the last 7 days);

[0112] Step S333: At the daily level (layer 1), calculate the attention weight and convergence features for the set of knowledge units learned in the last three days. They are query matrix, key matrix and value matrix respectively;

[0113] , ,

[0114] in is a matrix composed of the eigenvectors of the knowledge unit set learned in the last three days, where , , is a learnable weight matrix;

[0115] Calculate the attention score, the calculation formula is:

[0116]

[0117] in is the dimension of the key vector, where represents the query matrix;

[0118] According to its attention score and value matrix, the aggregated features at the day level are obtained :

[0119]

[0120] Similarly, at the weekly level (layer 2), for the knowledge units learned in the last 7 days (excluding the last 3 days), calculate the attention weights and convergence features; let They are query matrix, key matrix and value matrix respectively;

[0121] , ,

[0122] in is a matrix composed of the eigenvectors of the knowledge unit set learned in the last 7 days (excluding the last 3 days), where , , is a learnable weight matrix;

[0123] Calculate the attention score, the calculation formula is:

[0124]

[0125] in is the dimension of the key vector, where represents the query matrix;

[0126] According to its attention score and value matrix, the aggregated features of the weekly level are obtained :

[0127]

[0128] Similarly, at the monthly level (layer 3), for the knowledge units learned in the last 30 days (excluding the last 7 days), the attention weights and convergence features are calculated; let They are query matrix, key matrix and value matrix respectively;

[0129] , ,

[0130] in is a matrix composed of the eigenvectors of the knowledge unit set learned in the last 30 days (excluding the last 7 days), where , , is a learnable weight matrix;

[0131] Calculate the attention score, the calculation formula is:

[0132]

[0133] in is the dimension of the key vector, where represents the query matrix;

[0134] According to its attention score and value matrix, the aggregated features at the day level are obtained :

[0135]

[0136] Step S334: constructing a C-tree and fusion of features;

[0137] A C-tree is formed according to the hierarchy. Here we assume that we start from the bottom layer (day level) and perform feature fusion upwards;

[0138] Assume that the output of the day level is the leaf node of the C-tree. We combine the features of these nodes according to the structure of the C-tree. In the first layer of fusion, C adjacent day-level feature vectors are combined together. Assume From the day-level features The C parts obtained by division ( The length of C is divisible by C), the feature vector obtained by the first fusion :

[0139]

[0140] in represents the splicing function;

[0141] For the weekly level, its features are also divided and fused according to the C-tree structure. From the day-level features The C parts obtained by division ( The length of the second fusion is divisible by C); the feature vector obtained by the second fusion :

[0142]

[0143] in represents the splicing function;

[0144] Finally, for the monthly level, let From the day-level features The C parts obtained by division ( The length of is divisible by C), the feature vector obtained by the third fusion :

[0145]

[0146] in represents the splicing function;

[0147] Step S335: finally combining the features fused at different levels to obtain the final time-fused knowledge feature G;

[0148] The expression of time fusion knowledge feature G is:

[0149]

[0150] in, is the fused feature vector obtained by fusing the knowledge feature vector and the question feature vector at the first day level, where is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the second weekly level, where It is the fused feature vector obtained by fusing the knowledge feature vector and the question feature vector at the third monthly level.

[0151] S4: Fuse historical features and time-fused knowledge features to obtain a fused feature representation; calculate the predicted probability of each candidate answer through the question feature representation and the fused feature representation, and use the candidate answer with the highest predicted probability as the final answer to the question text.

[0152] Specifically, step S4 is as follows:

[0153] Step S41: Fusing historical features and time-fused knowledge features to obtain fused feature representation;

[0154] Furthermore, the step S41 is specifically as follows:

[0155] Step S411: From the learner's historical feature representation and temporal fusion knowledge feature representation ;in , is the quantity dimension, , is the characteristic dimension;

[0156] Step S412: Apply the Count-Sketch projection function to the learner's historical features and time-integrated knowledge features get and ;

[0157] Step S413: According to the convolution theorem, the Count-Sketch of the outer product of two vectors can be expressed as the convolution of two Count-Sketch, which is calculated in the frequency domain. and Perform fast Fourier transform (FFT) to obtain and ;

[0158] Step S414: Calculate their element-by-element product in the frequency domain to obtain a fused feature representation;

[0159] Fusion feature representation The expression is:

[0160]

[0161] in, represents element-wise product, stands for inverse fast Fourier transform, FFT stands for fast Fourier transform, () represents the attention mechanism, Represents the historical feature H applying the Count-Sketch projection function The result after Represents the time fusion knowledge feature G using the Count-Sketch projection function The result after.

[0162] Step S52: The fused features , the feature representation of problem w , candidate answers Feature representation Input a nonlinear layer and then pass it through the softmax function to get the predicted probability , that is, the candidate answer As the probability of the correct answer to the question w, the candidate answer with the highest probability is output as the final answer, where the expression is:

[0163]

[0164] represents the u-th candidate answer, Yes The feature representation of is the number of candidate answers, and F represents a feed-forward neural network layer used to predict the final answer.

[0165] See Figure 2 , Figure 2 4 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, wherein the hardware device specifically includes: a multimodal question-answering device 401 integrating learner history, a processor 402 and a storage medium 403.

[0166] A multimodal question-answering device 401 integrating learner history: The multimodal question-answering device 401 integrating learner history implements the multimodal question-answering method integrating learner history.

[0167] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the multimodal question-answering method integrating learner history.

[0168] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the multimodal question-answering method integrating learner history.

[0169] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0170] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal question answering method integrating learner history, characterized by: Including steps: S1: Obtain the learner's identity data and question text, and obtain the learner's learning record data, historical performance data, and knowledge visual data through the identity data; S2: Preprocessing, feature extraction and feature weighting of learning record data and historical performance data to obtain learners’ historical features; S3: Embed the knowledge visual data through the ResNet model to obtain the knowledge feature representation; embed the question text through the BERT model to obtain the question feature representation; The knowledge feature representation and the problem feature representation are fused based on the time span to obtain the time-fused knowledge feature; S4: Fuse historical features and time fusion knowledge features to obtain fusion feature representation; The question feature representation and fusion feature representation are used to calculate the predicted probability of each candidate answer, and the candidate answer with the highest predicted probability is used as the final answer to the question text; The learning record data includes: course type data, teaching resource access data, and course difficulty data; according to the course type data, pictures and videos of the corresponding courses are obtained from the online learning platform as knowledge visual data; In step S3, the knowledge feature representation and the problem feature representation are fused based on the time span to obtain the time fused knowledge feature, specifically: S31: Through hierarchical recursion, the knowledge feature representation is divided into n-level knowledge feature vectors, and the problem feature representation is divided into n-level problem feature vectors; S32: Calculate the similarity between the knowledge feature vectors and the problem feature vectors at each level, use the similarity at each level as a matrix element, and construct a mapping matrix between the problem and the knowledge; S33: Through the pyramid attention mechanism and mapping matrix based on time span, the knowledge feature vectors and question feature vectors at each level are aggregated to obtain the time-fused knowledge features.

2. The multimodal question-answering method integrating learner history according to claim 1, characterized in that: Step S1 is specifically as follows: According to the learner's identity data, the learner's learning record data and historical performance data are obtained from the learning log file of the online learning platform using Python crawler.

3. The multimodal question-answering method integrating learner history according to claim 1, characterized in that: Step S2 is specifically as follows: S21: Delete incomplete and missing data in the learning record data and historical performance data; standardize the learning record data through one-hot encoding to obtain learning record features; standardize the historical performance data through the Z-score model to obtain historical performance features; S22: The learning record features and historical performance features are weightedly represented through the multi-head attention mechanism to obtain the learner's historical features H.

4. The multimodal question-answering method integrating learner history according to claim 1, characterized in that: The expression of time fusion knowledge feature G is: in, is the fused feature vector obtained by fusing the knowledge feature vector and the question feature vector at the first day level, where is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the second weekly level, where It is the fused feature vector obtained by fusing the knowledge feature vector and the question feature vector at the third monthly level.

5. The multimodal question-answering method integrating learner history according to claim 1, characterized in that: Fusion feature representation The expression is: in, represents element-wise product, stands for inverse fast Fourier transform, FFT stands for fast Fourier transform, () represents the attention mechanism, Represents the historical feature H applying the Count-Sketch projection function The result after Represents the time fusion knowledge feature G using the Count-Sketch projection function The result after.

6. A storage medium, characterized in that: The storage medium stores instructions and data for implementing the multimodal question-answering method integrating learner history as described in any one of claims 1 to 5.

7. A multimodal question-answering device that integrates learner history, characterized by: include: A processor and a storage medium; the processor loads and executes instructions and data in the storage medium to implement the multimodal question-answering method integrating learner history as described in any one of claims 1 to 5.

Images