Metacognitive level prediction method based on sounding thinking driven retrieval enhancement

Abstract

The invention discloses a meta-cognition level prediction method based on voiced thinking driven retrieval enhancement, which is characterized in that an information extraction method in a deep learning technology is adopted to extract answer features, a voiced thinking response analysis library is constructed, a correlation mode between various answer contents and corresponding meta-cognition scores is established, and a prediction result is obtained. The method comprises the following steps: establishing a mapping model, bridging the internal relation between the answer content and the meta-cognitive level by using the established mapping model, matching the sound thinking answer with the existing answer sample in a sound thinking response analysis library to realize the prediction of the meta-cognitive level, and specifically comprising the steps of data collection, data enhancement, big five personality correction, mapping model establishment, matching, retrieval and the like. Compared with the prior art, the method has the advantages that a more scientific and targeted problem design scheme can be provided on the basis of real-time evaluation of the meta-cognition level of the student when the type of problems proposed to the student is determined, so that the method has remarkable advantages in the aspects of education evaluation and personalized learning support.

Description

Technical Field

[0001] The present invention relates to the field of metacognition technology, in particular to a metacognition level prediction method based on think-aloud driven retrieval enhancement. Background Art

[0002] Metacognition is a general term for an individual's knowledge, experience and monitoring mechanism of his or her own cognitive process, results and related activities. It covers two dimensions: metacognitive knowledge and metacognitive monitoring. Specifically, metacognitive knowledge involves an individual's cognition of his or her own or others' cognitive activities, processes, results and related information, while metacognitive monitoring refers to the dynamic process of an individual planning, monitoring and regulating his or her own cognitive activities. Metacognition plays a core role in an individual's learning process, work task execution and problem solving, and it helps individuals to efficiently manage cognitive resources, thereby improving the efficiency and quality of cognitive activities. With the in-depth development of cognitive science and educational psychology, the core position of metacognition in individual cognitive activities has become increasingly prominent. Therefore, the effective evaluation of metacognition has become an important issue in research and practice.

[0003] At present, there are a variety of methods and technical systems for metacognition assessment. Among them, the self-report method is widely used because of its simplicity. This method collects individuals' cognitive and experiential information about their own cognitive processes, results and related activities through questionnaire filling or self-statement. However, this method is susceptible to the subjective bias and memory bias of the subjects, and cannot achieve real-time metacognitive prediction. Especially in the scenario of large educational models, it is difficult to instantly evaluate the user's metacognitive level to generate customized content. Another commonly used method is the thinking aloud method, which requires the subjects to express all their thinking activities in words when performing tasks to evaluate their metacognitive level. Although this method can intuitively show the subject's thinking trajectory, it is easily restricted by language expression ability and task difficulty, and professionals are required to analyze the answer content to evaluate the metacognitive level. Summary of the invention

[0004] The purpose of the present invention is to provide a metacognitive level prediction method based on think-aloud driven retrieval enhancement for the deficiencies of the prior art. The method of constructing a think-aloud response analysis library (RAG library) and a deep mapping model is adopted to establish the correlation pattern between multiple answer contents and corresponding metacognitive scores, and the Big Five personality factors are introduced to eliminate the influence of language expression ability differences. The deep mapping model between metacognitive level and answer content is constructed by deep feature extraction technology, and the user's think-aloud answer to a specific question is efficiently matched with the existing answer samples in the RAG library, and the most likely metacognitive score is directly output. This process does not need to rely on traditional questionnaire measurement methods, and does not require professional personnel to analyze one by one, and can achieve accurate prediction of the user's metacognitive level, thereby improving the evaluation efficiency while ensuring the accuracy and practicality of the evaluation. In the application scenario of the educational big data model, the present invention provides a solid theoretical basis for model design, especially when deciding what type of questions to ask students, it can provide a more scientific and targeted problem design scheme based on the real-time evaluation of the student's metacognitive level, thereby showing significant advantages in educational evaluation and personalized learning support, and has a good application prospect.

[0005] The specific technical scheme for achieving the purpose of the present invention is: a metacognitive level prediction method based on think-aloud driven retrieval enhancement, which is characterized by using a constructed think-aloud response analysis library (RAG library), associating answer content with metacognitive scores, introducing the Big Five personality theory to correct personalized language deviations, constructing a mapping model to connect answers with metacognitive levels, and using deep learning technology to efficiently match think-aloud answers with existing answer samples in the think-aloud response analysis library to achieve prediction of metacognitive levels.

[0006] The construction of the think-aloud response analysis library includes answer samples at different metacognitive levels, and the specific steps include:

[0007] Step 1: Data collection and metacognitive level measurement

[0008] Collect a certain number of users' answers to selected questions, ask them to elaborate on their thinking process, and after completing all the questions and answers, accurately obtain the user's metacognitive level through questionnaire measurement. In addition, collect their basic information, such as gender, education, major, etc., so as to conduct more targeted searches later.

[0009] Step 2: Multi-agent Data Augmentation

[0010] Based on the actual data obtained, possible answers covering different metacognitive levels are simulated and generated, and agent technology is used to enrich and optimize the data set. Multi-agent data enhancement specifically includes:

[0011] (I) Construction of a multi-agent system, which includes the following components:

[0012] a) Agents: Each agent represents a user with a specific metacognitive level. The number of agents should be large enough to cover different levels of metacognitive levels from low to high.

[0013] b) Interactive environment: Design a simulation environment that can receive input from the agent (such as question answers) and return corresponding feedback. This environment is built based on the actual data collected in step 1 to ensure that the generated answers are consistent with the actual situation.

[0014] c) Data generation mechanism: During the interaction between the agent and the environment, a mechanism is designed to record and generate data. This data should include the agent’s state (such as metacognitive level), actions (such as answered questions), and feedback from the environment (such as correct answers or evaluations of questions).

[0015] (II) Data Enhancement

[0016] a) Agent state representation

[0017] Assuming metacognitive level s imeta It is an important feature in the state vector. The state vector s of the agent i It is expressed by the following formula (1):

[0018] s i =[s i1 ,s i2 ,…,s in ] (1).

[0019] Among them, s ij Represents the j-th characteristic of agent i (such as metacognitive level, knowledge mastery, learning style, etc.).

[0020] b) Data generating function

[0021] Define the data generating function g(s i ,q), which accepts the agent's state vector s i and question q as input, and returns the agent's answer to the question a i , which is expressed by the following formula (2):

[0022] a i =g(s i ,q) (2).

[0023] Data generating function g(s i ,q) is a deep learning model that generates corresponding answers based on the input features and questions. i .

[0024] c) Data Augmentation

[0025] For the state vector s of the agent i Fine-tune and apply the data generating function g(s i ,q) generates a new answer. The process can be expressed by the following formula (3):

[0026] {(s1,q,a1),(s2,q,a2),…,(s m ,q,a m )} (3).

[0027] Among them, s1, s2, s m Represents agents in different states, q is the question, a1, a2, a m is the corresponding answer.

[0028] Step 3: Big Five Personality Correction

[0029] Accurately establish the mapping relationship between the user's answer content and metacognitive level, and introduce the Big Five personality theory for correction to eliminate the influence of personality traits on language expression. The specific steps are as follows:

[0030] (I) Apply the Big Five personality theory and design a questionnaire or scale to evaluate the user's personality traits. The Big Five personality includes five dimensions: Neuroticism, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness. Determine the user's personality trait score through statistical analysis methods (such as factor analysis, regression analysis, etc.).

[0031] (ii) Construct a correction model that takes personality trait scores as input and certain features of the answer content (such as vocabulary usage, sentence structure, etc.) as output to estimate the impact of personality traits on language expression. Use a machine learning algorithm such as linear regression to train the correction model so that it can accurately predict the impact of personality traits on language expression. The linear regression model is expressed by the following formula (4):

[0032] A corrected =AW·P (4).

[0033] Among them, A is the feature vector of the answer content, P is the personality trait score, and A corrected is the corrected answer content feature, and W is the weight matrix.

[0034] The weight matrix W is solved by the least square method shown in the following formula (5), so that the corrected answer content feature A corrected Minimize the correlation with the personality trait score P:

[0035] W = (P′P) -1 P′A (5).

[0036] Among them, A is the corrected answer content feature A corrected , P is the personality trait score matrix, and P′ is the transpose of the personality trait score matrix P.

[0037] Step 4: Mapping model construction

[0038] After the correction, the answer content features A corrected After that, a mapping model is constructed to predict the user's metacognitive level, using a neural network as the implementation of the nonlinear function f. It is expressed by the following formula (6):

[0039]

[0040] Among them, σ is the activation function, W hidden and W outPut is the weight matrix of the neural network, b hidden and b output are the bias terms respectively.

[0041] The thinking-aloud answer is matched with the existing answer samples in the thinking-aloud response analysis library. The thinking-aloud response analysis library is matched and searched according to the actual answer of the user, and the matching degree between the matching result and the answer corresponding to each metacognitive score is calculated, and the matching result with the highest score is selected as the output of the metacognitive level prediction. The specific steps include:

[0042] Step 5: Matching and Retrieval

[0043] A think-aloud response analysis library (RAG library) containing answer samples at different metacognitive levels is constructed. According to the user's actual answer, a matching search is performed in the RAG library, and the matching degree between the retrieved matching results and the answer corresponding to each metacognitive score is calculated. The matching result with the highest score is selected as the output. The specific steps are as follows:

[0044] 1. Feature extraction

[0045] After correcting the Big Five personality traits of the user's answer, the word embedding model BERT is used to convert the text into a vector representation and extract additional features such as text length and keyword frequency, which helps improve the accuracy of retrieval. At the same time, the user's basic information (such as gender, education, major, etc.) is converted into a quantifiable feature vector for comprehensive consideration in the matching process.

[0046] (II) Building an index

[0047] For each answer in the training set, its vector representation and additional features are calculated, and these answers are stored together with the corresponding metacognitive level in an index structure such as an inverted index, hash table, or vector database.

[0048] (III) Similarity calculation

[0049] When a new corrected user answer is received, its vector representation and additional features are calculated, and the answer most similar to this answer is searched in the index. In some cases, different features may contribute differently to the similarity calculation. Therefore, a weight can be assigned to each feature, and the weighted cosine similarity can be used to calculate the similarity. The specific calculation is shown in the following formula (7):

[0050]

[0051] Among them, w i is the weight of the i-th feature, q i and d i are the i-th components of the query vector and document vector respectively.

[0052] 4. Prediction of metacognitive level

[0053] For the most similar answer (or answer set) retrieved, take its corresponding metacognitive level as the prediction result. If multiple similar answers are retrieved, we can calculate the average or weighted average of the metacognitive levels of these answers as the final prediction. For the most similar answer (or answer set) retrieved, not only its corresponding metacognitive level should be considered, but also the similarity between the user's basic information and the background information of the answer sample should be considered. This is achieved by calculating the similarity between the user's basic information features and the answer sample's basic information features. These similarities are weighted and summed with the similarity of the metacognitive level to obtain the final matching score.

[0054] The similarity Sim_weighted(q,d) between the basic information features of the user and the basic information features of the answer sample is calculated by the following formula (8):

[0055] Sim_weighted(q,d)=(∑(w i *q i *d i )) / (√(∑(w i *q i ^2))*√(∑(w i *d i ^2)))(8).

[0056] Among them, w_i weight parameter indicates the importance of each feature; q i represents the i-th eigenvalue in the user's basic information eigenvector; di is the i-th eigenvalue in the basic information eigenvector of the answer sample.

[0057] The similarity between the user's basic information features and the metacognitive level Final_Score(q,d) is calculated by the following formula (9):

[0058] Final_Score(q,d)=w_text×Sim_text(q,d)+w_extra×Sim_extra(q,d)+w_info×Sim_info(q_info,d_info)+w_meta×Sim_meta_adjusted(q_meta,d_meta)(9).

[0059] Among them, w_text is the weight of text content similarity; Sim text(q,d) is the text content similarity between query q and document d; w_extra is the weight of the additional feature similarity; Sim extra(q,d) is the additional feature similarity between query q and document d; w_info is the weight of the similarity of user basic information;

[0060] is the similarity between the user basic information of query q and the user basic information of document d; w_meta is the weight of the metacognitive level similarity; Sim_meta_adjusted is the similarity between the metacognitive levels of query q.

[0061] A SVM model is trained to distinguish answers at different metacognitive levels, and the model is used to calculate the similarity or distance between the query answer and the training answer. In SVM, a kernel function is defined as shown in the following formula (10) to calculate the similarity between two vectors:

[0062]

[0063] Among them, σ is a free parameter, which determines the smoothness of the kernel function, ||xx′|| 2 Represents the square of the Euclidean distance between two vectors x and x′.

[0064] Compared with the existing technology, the present invention has stronger interpretability and real-time performance. In particular, when deciding what type of questions to ask students, it can provide more scientific and targeted question design solutions based on real-time assessment of students' metacognitive levels, thereby showing significant advantages in educational assessment and personalized learning support. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Diagram of the model constructed for the present invention. DETAILED DESCRIPTION

[0066] The present invention is further described in detail with reference to the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0067] Example 1

[0068] See also Figure 1 , the real-time evaluation and prediction based on students’ metacognitive level is carried out in the following steps:

[0069] Step 1: Collect 100 users' think-aloud responses to the selected question. For example, this question could be: "When solving complex math problems (such as proving a math theorem or solving a word problem), how do you plan your problem-solving steps, monitor your problem-solving process, and adjust your strategy as needed?" Ask users to elaborate on their thinking process and strategy adjustments. Subsequently, accurately obtain the user's metacognitive level through questionnaire measurement. In addition, their basic information, such as gender, education, major, etc., should be collected to facilitate more targeted searches later.

[0070] Step 2: Use multi-agent technology to simulate and generate possible answers covering different metacognitive levels to enrich and optimize the data set. The constructed multi-agent system includes agents representing different metacognitive levels, which interact in a simulated environment and generate answers based on their states (such as metacognitive levels).

[0071] Step 3: Big Five Personality Correction

[0072] In order to eliminate the influence of personality traits on language expression, the Big Five Personality Theory is introduced for correction. The user's personality traits are assessed through a questionnaire, and a correction model is constructed to estimate the influence of personality traits on language expression.

[0073] For example, a Big Five personality questionnaire was administered to 100 students, and a correction model was constructed based on the assessment results. It was found that a student with a high extraversion score might be more inclined to use positive and vivid words, while a student with a high neuroticism score might be more inclined to use negative and anxious words. Through the correction model, these word uses can be adjusted to more accurately reflect the student's metacognitive level.

[0074] Step 4: Mapping model construction

[0075] After obtaining the corrected answer content features, a mapping model is constructed to predict the user's metacognitive level, using a neural network as the implementation of a nonlinear function. A mapping model is constructed using a neural network, taking the corrected answer content features as input and the metacognitive level as output. By training the model, the mapping relationship between the answer content and the metacognitive level can be learned. For example, when the model receives an answer that explains the problem-solving steps and strategies in detail, it may predict that the student has a high metacognitive level.

[0076] Step 5: Matching and Retrieval

[0077] The user's actual answer is matched in the RAG library, and the user's basic information and metacognitive level are comprehensively considered to calculate the matching degree. A RAG library containing answer samples of different metacognitive levels is constructed, and on this basis, the user's basic information, such as gender, education, major, etc., is further integrated as additional screening and matching basis.

[0078] When a new user's answer is received, it is first converted into a vector representation using the BERT model for efficient matching and retrieval in the RAG library. At the same time, based on the user's basic information, the answer samples in the RAG library are preliminarily screened, and samples that are similar or relevant to the user's background are selected for more detailed matching. In the matching process, not only the similarity of the answer content is focused on, but also a comprehensive evaluation is conducted in combination with the user's metacognitive level. For example, if a new user's answer is very similar in content to an answer sample with a high metacognitive level in the RAG library, and the user's background information such as education and major is consistent with the background information of the sample, then the model may give the user a higher matching score and predict that he or she also has a high metacognitive level.

[0079] Finally, after comprehensively considering multiple factors such as answer content similarity, metacognitive level, and user basic information, the result with the highest matching score is selected as the output to provide users with more accurate and personalized feedback.

Claims

1. A method for predicting metacognitive level based on think-aloud driven retrieval enhancement, characterized in that: By extracting answer features and applying the information extraction method in deep learning technology, a think-aloud response analysis library is constructed to establish the association pattern between multiple answer contents and corresponding metacognitive scores, and the constructed mapping model bridges the intrinsic connection between the answer content and the metacognitive level, and the think-aloud answer is matched with the existing answer samples in the think-aloud response analysis library to achieve the prediction of the metacognitive level. The construction of the think-aloud response analysis library includes answer samples at different metacognitive levels. The specific steps include: Step 1: Data collection and metacognitive level measurement Collect no less than 100 users' think-aloud answers to the selected questions and the users' basic information. After completing all the questions and answers, obtain the users' metacognitive level S through questionnaire measurement. The users' basic information includes: gender, education background and major. Step 2: Multi-agent Data Augmentation 2-1: Multi-agent system construction The multi-agent system includes three components: multiple agents, an interactive environment, and a data generation mechanism. The multiple agents cover different levels of metacognition from low to high, where each agent represents a user with a specific metacognition level; the interactive environment is a simulation environment built based on collected data, which will receive the input of the agent, i.e., answer questions, and return corresponding feedback; the data generation mechanism records the process of interaction between the agent and the simulation environment, and generates data including: the state of the agent, action, and environmental feedback; 2-2: Data Enhancement 2-2-1: State vector of the agent Assuming that the metacognitive level S is the feature s in the state vector s, and its value domain is set according to the user's metacognitive level S, the state vector s of the agent is expressed by the following formula (1): s=[s,s,…,s] (1); Among them, s is the j-th feature of agent i; 2-2-2: Data generation function Define the data generating function g(s,q) represented by the following formula (2): a=g(s,q) (2); Among them, s is the state vector; q is the input question; a is the agent's answer to question q; 2-2-3: Data Enhancement The data enhancement is performed by fine-tuning the state vector s of the agent, and a data generation function g(s,q) is applied to generate a new answer. The data enhancement process is represented by the following (3): {(s1,q,a1),(s2,q,a2),…,(s m ,q,a m )} (3); Among them, s1, s2, s m are agents in different states; q is the question; a1, a2, a m For the corresponding answer; Step 3: Big Five Personality Correction 3-1: Use a questionnaire or scale designed based on the Big Five personality theory to assess the user's personality traits, and use a statistical method of factor analysis or regression analysis to determine the user's personality trait score P. The Big Five personality includes five dimensions: neuroticism, extraversion, openness, agreeableness, and conscientiousness; 3-2: Factor analysis is used to construct a correction model for estimating the impact of personality traits on language expression. The model takes personality trait scores as input and the characteristics of answer vocabulary usage and sentence structure as output. The correction model is trained using a machine learning algorithm of a linear regression model to obtain the corrected answer content feature A corrected The machine learning algorithm of the linear regression model is expressed by the following formula (4): A corrected =A-W·P (4); Among them, A is the feature vector of the answer content; P is the personality trait score; W is the weight matrix; The weight matrix W is solved by the least square method shown in the following formula (5), so that the corrected answer content feature A corrected Minimize the correlation with the personality trait score P: W=(P′P) -1 P′A (5); Among them, A is the corrected answer content feature A corrected ; P is the personality trait score matrix; P′ is the transpose of the personality trait score matrix P; Step 4: Mapping model construction According to the corrected answer content feature A corrected , using the following (6) neural network as the nonlinear function f, construct a prediction of the user's metacognitive level The mapping model: Among them, σ is the activation function; W hidden and W output is the weight matrix of the neural network; b hidden and b output is the bias term; is the nonlinear function f; g is the data generating function; The thinking-aloud answer is matched with the existing answer samples in the thinking-aloud response analysis library. The thinking-aloud response analysis library is matched and searched according to the actual answer of the user, and the matching degree between the matching result and the answer corresponding to each metacognitive score is calculated, and the matching result with the highest score is selected as the output of the metacognitive level prediction. The specific steps include: 5-1: Feature extraction The user's answer is corrected for the Big Five personality traits, and the word embedding model BERT is used to convert the text into a vector representation, extract additional features including text length and keyword frequency, and convert the user's basic information such as gender, education level, and major into a quantifiable feature vector; 5-2: Building an index For each answer in the training set, calculate its vector representation and additional features, and store these answers together with the corresponding metacognitive level in an index structure of an inverted index, hash table or vector database; 5-3: Similarity calculation When a corrected new user answer is received, its vector representation is calculated, and additional features are extracted. A weight is assigned to each additional feature, and similarity is calculated using weighted cosine similarity. The most similar answer or answer set is searched in the index based on the similarity. The similarity is calculated using the weighted cosine similarity shown in the following formula (7): Among them, w i is the weight of the i-th feature; q i ,d i are the i-th components of the query vector and document vector respectively; 5-4: Metacognitive level prediction 5-4-1: Take the most similar answer or answer set retrieved, and take its corresponding metacognitive level as the prediction result. If multiple similar answers are retrieved, take the average or weighted average of the metacognitive levels of these answers as the prediction result; if the most similar answer or answer set is retrieved, take the weighted sum of the similarity between the basic information features of the user and the basic information features of the answer sample, and the similarity between the basic information features of the user and the metacognitive level, and get the final matching score; The similarity Sim_weighted(q,d) between the basic information features of the user and the basic information features of the answer sample is calculated by the following formula (8): Among them, w i is the weight parameter, indicating the importance of each feature; q i is the i-th eigenvalue in the user's basic information eigenvector; d i To answer the i-th eigenvalue in the basic information eigenvector of the sample; The similarity between the user's basic information features and the metacognitive level Final_Score(q,d) is calculated by the following formula (9): Final_Score(q,d)=w_text×Sim_text(q,d)+w_extra×Sim_extra(q,d)+w_info×Sim_info(q_info,d_info)+w_meta×Sim_meta_adjusted(q_meta,d_meta)(9); Among them, w_text is the weight of text content similarity; Sim_text(q,d) is the text content similarity between query q and document d; w_extra is the weight of additional feature similarity; Sim_extra(q,d) is the additional feature similarity between query q and document d; w_info is the weight of user basic information similarity; Sim_info(q_info,d_info) is the similarity between the user basic information of query q and the user basic information of document d; w_meta is the weight of the similarity of metacognitive level; Sim_meta_adjusted is the similarity between the metacognitive levels of query q; 5-4-2: Use a trained SVM model to distinguish answers at different metacognitive levels, and use the model to calculate the similarity or distance between the query answer and the training answer. In SVM, a kernel function is defined as shown in the following formula (10) to calculate the similarity between two vectors: Among them, σ is a free parameter, which determines the smoothness of the kernel function, ||xx′|| 2 Represents the square of the Euclidean distance between two vectors x and x′.

Images