Multi-modal question and answer method and device fusing history of learner

Abstract

The invention provides a multi-modal question and answer method and device fusing the history of a learner, and relates to the technical field of multi-modal question and answer, and the method comprises the steps: carrying out the embedding operation of knowledge visual data through a ResNet model, and obtaining knowledge feature representation; performing embedding operation on the question text through a BERT model to obtain question feature representation; performing time span-based fusion on the knowledge feature representation and the problem feature representation to obtain time fusion knowledge features; the historical features and the time fusion knowledge features are fused, and fusion feature representation is obtained; and calculating through the question feature representation and the fusion feature representation to obtain the prediction probability of each candidate answer, and taking the candidate answer with the highest prediction probability as the final answer of the question text. According to the learning record data, the historical score data and the knowledge visual data of the learner, the knowledge weakness and interest preference of the learner are accurately positioned, and the question answering accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal question answering, and particularly to a multimodal question answering method and device that integrates the history of learners. Background Art

[0002] With the rapid development of information technology, the channels for people to obtain information are becoming increasingly rich, and a large amount of multimodal data such as text, images, audio, and video has emerged. How to effectively process and understand this multimodal data has become an important topic in the field of artificial intelligence.

[0003] The development of artificial intelligence has prompted people to pursue systems that are closer to human intelligence. When humans perceive the world, they can naturally integrate various sensory information and interact. However, traditional artificial intelligence systems are often limited to the processing of single-modal information and cannot meet users' needs for querying and understanding multimodal information. In addition, when dealing with learners' questions, traditional technologies often lack in-depth consideration of learners' individual differences. For example, in the field of education, different students have different learning abilities, knowledge reserves, and interests and hobbies. However, most traditional technologies adopt a general model and cannot accurately meet the unique needs of each student. In terms of knowledge association, existing technologies are difficult to effectively integrate multimodal information with learners' past knowledge. Traditional knowledge systems are often in a single text form, and even if there are other modal information such as images, it is difficult to organically combine them with learners' historical knowledge. Summary of the Invention

[0004] To solve the above problems, the present invention provides a multimodal question answering method and device that integrates the history of learners.

[0005] The present invention provides a multimodal question answering method that integrates the history of learners, including the steps of: S1: Obtain the identity data and question text of the learner, and obtain the learning record data, historical performance data, and knowledge visual data of the learner through the identity data; S2: Preprocess, extract features, and weight features from the learning record data and historical performance data to obtain the historical features of the learner; S3: Perform an embedding operation on the knowledge visual data through a ResNet model to obtain a knowledge feature representation; perform an embedding operation on the question text through a BERT model to obtain a question feature representation; fuse the knowledge feature representation and the question feature representation based on the time span to obtain a time-fused knowledge feature; S4: Fuse the historical features and the time-fused knowledge features to obtain a fused feature representation; calculate the prediction probabilities of each candidate answer through the question feature representation and the fused feature representation, and use the candidate answer with the highest prediction probability as the final answer to the question text.

[0006] Preferably, step S1 is specifically as follows: S11: From the learning log file of the online learning platform according to the identity data of the learner, use Python crawler to obtain the learning record data and historical score data of the learner, where the learning record data includes: course type data, teaching resource access data, and course difficulty data; S12: According to the course type data, obtain the corresponding course pictures and videos from the online learning platform as knowledge visual data.

[0007] Preferably, step S2 is specifically as follows: S21: Delete the incomplete data and missing data in the learning record data and historical score data; standardize the learning record data through one-hot encoding to obtain learning record features; standardize the historical score data through the Z-score model to obtain historical score features; S22: Use the multi-head attention mechanism to perform weighted representation on the learning record features and historical score features to obtain the historical features H of the learner.

[0008] Preferably, in step S3, the fusion of the knowledge feature representation and the problem feature representation based on the time span to obtain the time-fused knowledge feature is specifically as follows: S31: Through hierarchical recursion, divide the knowledge feature representation into knowledge feature vectors at n levels, and divide the problem feature representation into problem feature vectors at n levels; S32: Calculate the similarity between the knowledge feature vectors and problem feature vectors at each level, and use the similarities at each level as matrix elements to construct a mapping matrix of problems and knowledge; S33: Through the pyramid attention mechanism based on the time span and the mapping matrix, perform feature aggregation on the knowledge feature vectors and problem feature vectors at each level to obtain the time-fused knowledge feature.

[0009] Preferably: The expression of the time-fused knowledge feature G is:

[0010] Where, is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the first daily level, where is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the second weekly level, where is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the third monthly level.

[0011] Preferably: The expression of the fused feature representation is:

[0012] Among them, represents element-wise multiplication, represents the inverse fast Fourier transform, and FFT represents the fast Fourier transform. () represents the attention mechanism, represents the result after applying the Count-Sketch projection function to the historical feature H, and represents the result after applying the Count-Sketch projection function to the temporal fusion knowledge feature G. and

[0013] A storage medium stores instructions and data for implementing the multimodal question-answering method that fuses the learner's history.

[0014] A multimodal question-answering device that fuses the learner's history includes: a processor and a storage medium; the processor loads and executes the instructions and data in the storage medium for implementing the multimodal question-answering method that fuses the learner's history.

[0015] The present invention has the following beneficial effects: Based on the learner's learning record data, historical performance data, and knowledge visual data, the present invention accurately locates the learner's knowledge shortfalls and interest preferences, improving the accuracy of question answering. In terms of knowledge association, the present invention deeply fuses multimodal data such as text and images with the learner's history, promoting knowledge transfer and application. In terms of evaluation, the present invention comprehensively considers various historical data, objectively evaluates the learning effect, and provides timely feedback to assist the learner in adjusting strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flowchart of the method according to an embodiment of the present invention; Figure 2 is a structural diagram of the device according to an embodiment of the present invention; The implementation, functional features, and advantages of the object of the present invention will be further described in conjunction with the embodiments with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] Referring to Figure 1 , the present invention provides a multimodal question-answering method that fuses the learner's history, including the steps of: S1: Obtain the identity data and question text of the learner, and obtain the learner's learning record data, historical performance data, and knowledge visual data through the identity data; As an embodiment: Step S1 is specifically as follows: S11: From the learning log file of the online learning platform according to the identity data of the learner, use Python crawler to obtain the learning record data and historical score data of the learner. The learning record data includes: course type data, teaching resource access data, and course difficulty data; S12: According to the course type data, obtain the corresponding course pictures and videos from the online learning platform as knowledge visual data.

[0019] S2: Preprocess, extract features, and weight features for the learning record data and historical score data to obtain the historical features of the learner; As an embodiment: Step S2 is specifically as follows: S21: Delete the incomplete data and missing data in the learning record data and historical score data; standardize the learning record data through one-hot encoding to obtain learning record features; standardize the historical score data through the Z-score model to obtain historical score features; Specifically, for the course type attribute data, use one-hot encoding (One - Hot Encoding). Assume that there are 5 course types: Chinese, Mathematics, English, Physics, and Chemistry. For each course, create a binary vector of length 5 to represent the course type. For example, for the Chinese course, the encoding is [1,0,0,0,0]; for the Mathematics course, the encoding is [0,1,0,0,0]; for the English course, the encoding is [0,0,1,0,0]; for the Physics course, the encoding is [0,0,0,1,0]; for the Chemistry course, the encoding is [0,0,0,0,1], converting the original categorical variable into a numerical variable that can be processed by subsequent models; For the teaching resource access data, convert it into sequence data. For example, record the access order and time interval of the learner to different teaching resources (such as videos, documents). For each access record, represent its features (such as resource type, access duration, etc.) as a vector, and then arrange them in chronological order to form a sequence; For the course difficulty data, use a data-driven encoding method for preprocessing. Based on the course pass rate, encode the courses with a pass rate greater than 80% as "easy", those between 60% - 80% as "relatively easy", those between 40% - 60% as "medium", those between 20% - 40% as "relatively difficult", and those less than 20% as "difficult", and then perform one-hot encoding. This encoding can better reflect the actual difficulty of the course; For the historical scores of the learner, use Z-score standardization for preprocessing operations, including: Step S211: First, sum up all the scores of a certain course, and then divide the sum by the total number of scores to obtain the average score, that is, the mean value. ; Step S212: For each score in the dataset, calculate the difference between it and the mean value. Step S213: Square the obtained differences respectively to get the squared differences. Step S214: Add up the squared differences corresponding to all scores, and then divide by the total number of scores to obtain the variance. Step S215: Take the square root of the variance to obtain the standard deviation. , and the standard deviation reflects the degree of dispersion of the scores relative to the mean value. Step S216: For each score in the dataset, use the expression:

[0020] where represents the original score, where represents the average score, where represents the standard deviation. Step S217: Repeat this step to convert all the original scores into standardized scores. After the calculation, all the scores are converted into a data distribution with a mean value of 0 and a standard deviation of 1. S22: Use the multi-head attention mechanism to perform weighted representation on the learning record features and historical score features to obtain the historical features H of the learner.

[0021] Specifically, step S22 is as follows: Step S221: Concatenate the four types of preprocessed data vectors by columns to form an input matrix , where for each learner's record, the course type vector is , the teaching resource access vector sequence is , the course difficulty vector is , and the historical score vector is , then the input matrix Y, and its calculation process is as follows:

[0022] Here, the semicolon indicates concatenation by columns, and Y represents the input matrix; Step S222: Pass the input matrix through three linear transformation matrices , , respectively to obtain the query vector , the key vector and the value vector , and its calculation process is as follows: , ,

[0023] Step S223: Divide , , into 8 heads respectively, namely , , , and the dimension of each head is , where is the output dimension of the multi-head attention mechanism; Step S224: For each head , the formula for calculating the attention score matrix is

[0024] where , where is the output dimension of the multi-head attention mechanism; Perform a softmax operation on to obtain the attention weight matrix, and its expression is as follows:

[0025] Calculate the output of , and its expression is as follows:

[0026] Step S225: Concatenate the outputs of each head to obtain the output of the multi-head attention mechanism, and its calculation process is as follows:

[0027] where H is the historical feature of the learner, and where represents the concatenation function.

[0028] S3: Perform an embedding operation on the knowledge visual data through the ResNet model to obtain a knowledge feature representation; perform an embedding operation on the question text through the BERT model to obtain a question feature representation; fuse the knowledge feature representation and the question feature representation based on the time span to obtain a time-fused knowledge feature; As an example: Use a 152-layer ResNet pre-trained on ImageNet data (ResNet is a deep neural network that effectively solves the problems of gradient disappearance and gradient degradation in the training of deep neural networks, enabling the network to better learn complex features) to extract the knowledge feature representation, specifically: Preprocess the input knowledge vision to meet the input requirements of the model. Let the preprocessing operation function be , then the preprocessed image can be expressed as:

[0029] where represents the original knowledge vision data, represents the image data that can be directly input into the model after preprocessing; Input the preprocessed image into the pre-trained ResNet model. The model will perform a series of operations such as convolution, pooling, and residual connection to gradually extract the features of the image. The calculation formula is:

[0030] represents the extracted knowledge vision features, and all knowledge feature representations .

[0031] As an example: To extract as many important semantic features as possible from the text description of the problem and reduce the noise caused by colloquialism, a fine-tuned BERT model trained on a large-scale educational Q&A corpus using the BERT model (BERT is a pre-trained language model proposed by Google in 2018, based on the Transformer architecture, using a bidirectional training method, learning general language features through two pre-training tasks: masked language model and next sentence prediction, capable of capturing long-range dependencies in text and having powerful feature extraction capabilities, and can achieve good results after fine-tuning in a variety of natural language processing tasks) will be used and used as the input for the subsequent neural network model calculation. The calculation process is as follows:

[0032] where represents the encoding process of the BERT model, represents the semantic feature representation of the learner's question, represents the learner's question, where n represents the length of the question text.

[0033] As an example: In step S3, the knowledge feature representation and the question feature representation are fused based on the time span to obtain the time-fused knowledge features. Specifically: S31: Through hierarchical recursion, the knowledge feature representation is divided into knowledge feature vectors at n levels, and the question feature representation is divided into question feature vectors at n levels; S32: Calculate the similarity between the knowledge feature vectors and the problem feature vectors at each level, and use the similarities at each level as matrix elements to construct a mapping matrix of problems and knowledge; Specifically, step S32 is as follows: Step S321: Considering the hierarchical characteristics of knowledge, if all knowledge is directly considered in the mapping matrix, the dimension of the matrix will be extremely large and too sparse. Based on this, we construct the mapping matrix based on the hierarchy of knowledge, recursively from top to bottom according to the hierarchy of knowledge, and calculate the hierarchical feature representation by multiplying recursively through levels. Specifically, for common m-level knowledge , the calculation method of its semantic feature representation is as follows:

[0034] where j represents the level of knowledge, and the value of 1 represents the knowledge at the lowest level; Step S322: Use the Manhattan distance method to calculate the similarity between the problem feature vector and the knowledge feature vector. For two vectors and , the Manhattan distance between them is: , where the smaller the Manhattan distance, the higher the similarity, and the value range of the similarity is between 0 and 1. When the two vectors are exactly the same, the Manhattan distance is 0 and the similarity is 1. When the two vectors are farther apart, the similarity approaches 0; Step S323: Convert the Manhattan distance to similarity, and the calculation formula is:

[0035] where represents the Manhattan distance between two vectors and , where refers to converting the calculated Manhattan distance to the similarity between the problem feature vector and the knowledge feature vector ; Step S324: According to the similarity calculation result, construct a mapping matrix of problems and knowledge, and use the similarity value as an element in the mapping matrix. The higher the similarity, the stronger the association degree between the problem and the corresponding knowledge; Step S325: Conduct statistical analysis on all similarity values, observe their distribution, draw a histogram of similarity values, view the concentrated area and dispersion degree of similarity values, set the threshold to 0.6, and assign a higher value to the corresponding position in the mapping matrix when the similarity is greater than the threshold, otherwise assign a lower value; Step S326: The mapping matrix is based on the problem feature representation obtained above and all knowledge feature representations are constructed and dynamically maintained according to the interactions of learners. The maintained relevance is defined by a scoring function based on the translational distance:

[0036] where represents the problem feature, represents all knowledge features, where represents and is the L2 norm of the difference between; The problem-knowledge mapping matrix can assist in constructing the knowledge hierarchy in the pyramid attention mechanism based on the time span. By analyzing factors such as the association degree between knowledge and problems in the matrix and the learning time of knowledge, the knowledge can be divided into different time span levels, and it also provides a screening range for subsequent knowledge feature aggregation based on the time span; S33: Feature aggregation is performed on the knowledge feature vectors and problem feature vectors of each level through the pyramid attention mechanism based on the time span and the mapping matrix to obtain the time-fused knowledge features.

[0037] Specifically, step S33 is as follows: Step S331: Considering that the impact of the memory curve on learning efficiency actually exists, the content with a longer learning time distance is much less helpful for the current learning than the content with a shorter learning time distance, and it is proposed to use the pyramid attention mechanism based on the time span for knowledge feature aggregation; Step S332: Divide the knowledge into three levels according to the time span: daily, weekly, and monthly. Let be the feature vector of the set of knowledge units learned within the last 3 days, be the feature vector of the set of knowledge units learned within the last 7 days (excluding the last 3 days), be the feature vector of the set of knowledge units learned within the last 30 days (excluding the last 7 days); Step S333: For the daily level (the first level), which is the set of knowledge units learned within the last 3 days, calculate the attention weights and the aggregated features. Let be the query matrix, key matrix, and value matrix respectively; , ,

[0038] where is the matrix composed of the feature vectors of the set of knowledge units learned within the last 3 days, where , , is a learnable weight matrix; Calculate the attention score, and the calculation formula is:

[0039] where is the dimension of the key vector, where represents the query matrix; Obtain the aggregated features at the daily level according to its attention score and the value matrix :

[0040] Similarly for the weekly level (layer 2), for the knowledge units learned within the last 7 days (excluding the last 3 days), calculate the attention weights and the aggregated features; let be the query matrix, key matrix, and value matrix respectively; , ,

[0041] where is the matrix composed of the feature vectors of the set of knowledge units learned within the last 7 days (excluding the last 3 days), where , , is the learnable weight matrix; Calculate the attention score, and the calculation formula is:

[0042] where is the dimension of the key vector, where represents the query matrix; Obtain the aggregated features at the weekly level according to its attention score and the value matrix :

[0043] Similarly for the monthly level (layer 3), for the knowledge units learned within the last 30 days (excluding the last 7 days), calculate the attention weights and the aggregated features; let be the query matrix, key matrix, and value matrix respectively; , ,

[0044] where is the matrix composed of the feature vectors of the set of knowledge units learned within the last 30 days (excluding the last 7 days), where , , is the learnable weight matrix; Calculate the attention score, and the calculation formula is:

[0045] Among them is the dimension of the key vector, where represents the query matrix; According to its attention scores and value matrix, the aggregated features at the daily level are obtained :

[0046] Step S334: Construct a C-ary tree and perform feature fusion; A C-ary tree is formed according to the levels. Here, it is assumed that we start from the bottom level (daily level) and perform feature fusion upward; Let the output of the daily level be the leaf nodes of the C-ary tree. We combine the features of these nodes according to the structure of the C-ary tree. In the first-level fusion, C adjacent daily-level feature vectors are combined together. Let be the C parts obtained by dividing from the daily-level feature (the length of is divisible by C), and the feature vector obtained by the first fusion is:

[0047] Among them represents the concatenation function; For the weekly level, its features are also divided and fused according to the C-ary tree structure. Let be the C parts obtained by dividing from the daily-level feature (the length of is divisible by C); the feature vector obtained by the second fusion is:

[0048] Among them represents the concatenation function; Finally, for the monthly level, let be the C parts obtained by dividing from the daily-level feature (the length of is divisible by C), and the feature vector obtained by the third fusion is:

[0049] Among them represents the concatenation function; Step S335: Combine the features after fusion at different levels to obtain the final temporal fusion knowledge feature G; The expression of the temporal fusion knowledge feature G is:

[0050] Among them, is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the first daily level, where is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the second weekly level, where is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the third monthly level.

[0051] S4: Fuse the historical feature and the time-fused knowledge feature to obtain a fused feature representation; calculate the prediction probability of each candidate answer through the problem feature representation and the fused feature representation, and take the candidate answer with the highest prediction probability as the final answer to the problem text.

[0052] Specifically, step S4 is as follows: Step S41: Fuse the historical feature and the time-fused knowledge feature to obtain a fused feature representation; Furthermore, step S41 is specifically as follows: Step S411: The historical feature representation from the learner and the time-fused knowledge feature representation ; where , is the quantity dimension, , is the feature dimension; Step S412: Apply the Count-Sketch projection function to the learner's historical feature and the time-fused knowledge feature to obtain and ; Step S413: According to the convolution theorem, the Count-Sketch of the outer product of two vectors can be expressed as the convolution of two Count-Sketches and calculated in the frequency domain, that is, perform a fast Fourier transform (FFT) on and to obtain and ; Step S414: Calculate their element-wise product in the frequency domain to obtain a fused feature representation; The fused feature representation has the following expression:

[0053] Among them, represents the element-wise product, represents the inverse fast Fourier transform, FFT represents the fast Fourier transform, () represents the attention mechanism, represents the result after applying the Count-Sketch projection function to the historical feature H, represents the result after applying the Count-Sketch projection function to the temporal fusion knowledge feature G.

[0054] Step S52: Input the fused feature , the feature representation of the question w , candidate answers feature representation into a non-linear layer, and then obtain the prediction probability through the softmax function , that is, the probability that the candidate answer is the correct answer to the question w. The candidate answer with the highest probability is output as the final answer. The expression:

[0055] represents the u-th candidate answer, yes feature representation, is the number of candidate answers, and F represents a forward neural network layer for predicting the final answer.

[0056] Please refer to Figure 2 , Figure 2 which is the schematic diagram of the hardware device working in the embodiment of the present invention. The hardware device specifically includes: a multi-modal question answering device 401 that fuses the learner's history, a processor 402, and a storage medium 403.

[0057] A multi-modal question answering device 401 that fuses the learner's history: The multi-modal question answering device 401 that fuses the learner's history implements the multi-modal question answering method that fuses the learner's history.

[0058] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the multi-modal question answering method that fuses the learner's history.

[0059] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the multi-modal question answering method that fuses the learner's history.

[0060] ​​It should be noted that in this text, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or system including that element.

[0061] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A multimodal question answering method that integrates the learner's history, characterized in that, Including the steps: S1: Obtain the identity data and question text of the learner, and obtain the learning record data, historical score data, and knowledge visual data of the learner through the identity data; S2: Preprocess, extract features, and weight features from the learning record data and historical score data to obtain the historical features of the learner; S3: Perform an embedding operation on the knowledge visual data through the ResNet model to obtain a knowledge feature representation; perform an embedding operation on the question text through the BERT model to obtain a question feature representation; Fuse the knowledge feature representation and the question feature representation based on the time span to obtain a time-fused knowledge feature; S4: Fuse the historical features and the time-fused knowledge features to obtain a fused feature representation; through Calculate the prediction probabilities of each candidate answer based on the question feature representation and the fused feature representation, and take the candidate answer with the highest prediction probability as the final answer to the question text.

2. The multimodal question answering method for integrating learners' history according to claim 1, wherein The specific steps of S1 are as follows: S11: According to the identity data of the learner, use Python crawlers to obtain the learning record data and historical score data of the learner from the learning log file of the online learning platform, where the learning record data includes: course type data, teaching resource access data, and course difficulty data; S12: Obtain the pictures and videos of the corresponding courses from the online learning platform according to the course type data as the knowledge visual data.

3. The multimodal question answering method for integrating the learner's history according to claim 1, wherein The specific steps of S2 are as follows: S21: Delete the incomplete data and missing data in the learning record data and historical score data; standardize the learning record data through one-hot encoding to obtain learning record features; standardize the historical score data through the Z-score model to obtain historical score features; S22: Perform weighted representation on the learning record features and historical score features through the multi-head attention mechanism to obtain the historical feature H of the learner.

4. The multimodal question-answering method for integrating learner history according to claim 1, wherein, In step S3, the fusion of the knowledge feature representation and the question feature representation based on the time span to obtain a time-fused knowledge feature is specifically as follows: S31: Through hierarchical recursion, divide the knowledge feature representation into knowledge feature vectors at n levels, and divide the question feature representation into question feature vectors at n levels; S32: Calculate the similarity between the knowledge feature vectors and question feature vectors at each level, and use the similarities at each level as matrix elements to construct a mapping matrix of questions and knowledge; S33: Perform feature aggregation on the knowledge feature vectors and question feature vectors at each level through the pyramid attention mechanism based on the time span and the mapping matrix to obtain a time-fused knowledge feature.

5. The multi-modal question-answering method for fusing the history of the learner according to claim 4, characterized in that: The expression of the time-fused knowledge feature G is: Among them, is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the first daily level, where is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the second weekly level, where is the fused feature vector obtained by fusing the knowledge feature vector and the problem feature vector at the third monthly level.

6. The multi-modal question-answering method for fusing the history of the learner according to claim 1, characterized in that: Fused feature representation The expression is as follows: Among them, represents element-wise multiplication, represents the inverse fast Fourier transform, and FFT represents the fast Fourier transform, () represents the attention mechanism, represents the result after applying the Count-Sketch projection function to the historical feature H . represents the result after applying the Count-Sketch projection function to the time-fused knowledge feature G .

7. A storage medium, characterized in that: The storage medium stores instructions and data for implementing the multi-modal question-answering method for fusing the history of the learner according to any one of claims 1 to 6.

8. A multimodal question-answering device that integrates the learner's history, characterized in that: Including: A processor and a storage medium; the processor loads and executes the instructions and data in the storage medium for implementing the multi-modal question-answering method for fusing the history of the learner according to any one of claims 1 to 6.

Images