An Adaptive Learning Intervention Method Based on Multimodal Analysis

Abstract

This invention discloses an adaptive learning intervention method based on multimodal analysis, belonging to the fields of smart education and artificial intelligence technology. It solves the problem of insufficient accuracy in existing learning intervention methods, including: Step S1: Constructing a student state vector; Step S2: Creating a time series of the student state vector and identifying teaching events based on the time series; Step S3: Customizing a dedicated model; when a teaching event is triggered, extracting the corresponding video stream data and transmitting it to the dedicated model to generate a structured cognitive representation; Step S4: Parsing the structured cognitive representation to generate prompt text and displaying it to the student; Step S5: Recording the video stream data and structured cognitive representation to generate a student's problem file and providing learning suggestions; Step S6: Synchronously transmitting the student's problem file and learning suggestions. This invention achieves ultra-high precision personalized tutoring by training a dedicated model.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence and educational science, and in particular to a method and system for providing highly personalized and adaptive tutoring to learners by utilizing real-time multimodal perception, advanced AI model analysis, and closed-loop feedback mechanisms. Background Technology

[0002] Existing technologies or products related to student learning tutoring have the following shortcomings:

[0003] Passive interaction: The vast majority of systems rely on learners’ active input, such as submitting answers, clicking buttons, or asking questions; the system lacks the ability to actively perceive and cannot intervene when learners are most in need of help during their “confusion moments”.

[0004] Superficial data analysis: The system's analysis is mostly based on the final result (such as whether the answer is right or wrong) or simple behavioral data (such as page dwell time), which cannot delve into the learner's cognitive process to understand "why they made mistakes" and "where they got stuck".

[0005] Static personalization: Many so-called "personalizations" are based on pre-set user profiles or simple rule engines, which are difficult to dynamically adjust strategies according to the learner's real-time status changes during a learning process, and lack true adaptability.

[0006] Therefore, there is an urgent need in this field for a new type of intelligent tutoring technology that can proactively perceive, deeply understand, and dynamically adapt to the learner's real-time state; to this end, we propose an adaptive learning intervention method based on multimodal analysis. Summary of the Invention

[0007] In view of the shortcomings of existing technologies, the core objective of this invention is to provide an AI tutoring method that can simulate the ability of excellent human teachers to "read between the lines" and "teach according to aptitude," thereby overcoming the limitations of existing technologies.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: an adaptive learning intervention method based on multimodal analysis, the specific working process of each step of which is as follows:

[0009] Step S1: Monitor students' learning status in real time to obtain video stream data, process the video stream data, and construct student state vectors;

[0010] Step S2: Update the student state vector to obtain the time series of the student state vector. Extract feature sequence segments through self-labeling. Set feature thresholds based on the feature sequence segments. Analyze the time series based on the feature thresholds to identify teaching events.

[0011] Step S3: Obtain the open-source multimodal base model, fine-tune the open-source multimodal base model to customize the exclusive model; when the teaching event is triggered, extract the corresponding video stream data, transmit the video stream data to the exclusive model, and generate structured cognitive representations;

[0012] Step S4: Analyze the structured cognitive representation, use the analysis results as the question input, repeatedly call the dedicated model to generate prompt text, and display the prompt text to the students;

[0013] Step S5: Record the video stream data and structured cognitive representations corresponding to the teaching events to obtain a record file. Persist in storing the record file, summarize the data comprehensively based on the stored record file, generate the student's problem file, and provide learning suggestions to the student.

[0014] Step S6: Through access control, synchronize and transmit the student's problem file and learning suggestions to other authorized clients.

[0015] Furthermore, the specific steps of step S1 are as follows:

[0016] Step S11: Decompose the video stream data, extract the image frames from the video stream data, count the number of image frames, and denote the number of image frames as ts. Count the number of pixels in each image frame and denote the number of pixels as is×js. Based on the number of pixels, obtain the image pixels for each pixel and denote the image pixels as txs. t (i, j), txs t (i, j) represents the image pixel at the i-th row and j-th column of the t-th image frame;

[0017] Image pixels txs from different image frames t (i, j), calculate the pixel changes of the image frame to obtain the student's motion activity degree yhd(t);

[0018] ;

[0019] Among them: txs t＋1 (i, j) represents the image pixel at the i-th row and j-th column of the (t+1)-th image frame;

[0020] Step S12: Perform pose estimation on the image frame and identify the student's body nodes; record the position of the body nodes in the image frame, establish a base point based on the position of the body nodes in the image frame, and calculate the student's pose quantization value through the positional relationship between the base point and other body nodes to obtain the pose features.

[0021] Step S13: Statistically analyze the degree of motion and posture characteristics, construct a two-dimensional student state vector, and denote the student state vector as V.

[0022] Furthermore, the specific steps of step S12 are as follows:

[0023] Step S121: Obtain the number of student body nodes, denoted as ds; collect the position of each body node to obtain the node position jsw. t (d); jsw t (d) = (i) t (d), j t (d));

[0024] Based on the node positions in different image frames, the numerical changes in node positions are calculated to obtain the change in node positions bhl(d).

[0025] ;

[0026] Where: i t (d), j t (d) represents the parameter indicating the node position in frame t;

[0027] The changes at different node positions are statistically analyzed to obtain changes from bhl(1) to bhl(ds); the smallest change among the changes from bhl(1) to bhl(ds) is extracted, and the node corresponding to it is taken as the base point. The position of the base point is denoted as (jdi). t jdj t );

[0028] Step S122: Calculate the distance between the base point and the body node on each video frame, integrate all distances to obtain the student's pose quantization value, and record the student's pose quantization value as the pose feature ztz(t).

[0029] Furthermore, the specific steps of step S2 are as follows:

[0030] Step S21: Update and statistically analyze the student state vector in real time according to the time series to obtain the time series of the student state vector. Label the time series to obtain the feature sequence segment. Extract the student state vector from the feature sequence segment, analyze the value of the student state vector, and set the feature threshold.

[0031] Step S22: Obtain the segment length of the feature sequence fragment, set the trigger length according to the segment length of the feature sequence fragment, identify the student status according to the feature threshold, and identify the teaching event based on the trigger length and the student status.

[0032] Furthermore, the specific steps of step S21 also include:

[0033] Step S211: Obtain the student state vectors under different video frames according to the temporal sequence, and denote them as V(t), where V(t) represents the student state vector under the t-th video frame; count the student state vectors to form a time series of student state vectors, and label the time series to obtain static sequence segments and non-static sequence segments.

[0034] Step S212: Count the number of video frames in the still sequence segment, denoted as jzs. Obtain the motion activity degree of the student state vector under each video frame in the still sequence segment, denoted as jyhd(jz). Calculate the mean of the motion activity degree jyhd(jz) based on the number of video frames jzs in the still sequence segment to obtain the mean static activity value jjz.

[0035] The static fluctuation value jbd is calculated by combining the mean static activity with the degree of motion activity of the student state vector in the static sequence segment.

[0036] The number of video frames in the non-static sequence segment is counted and denoted as fjs. The motion activity degree of the student state vector under each video frame in the non-static sequence segment is obtained and denoted as fyhd(fj). Based on the number of video frames fjs in the non-static sequence segment, the mean of motion activity degree fyhd(fj) is calculated to obtain the mean of non-static activity fjz.

[0037] The non-stationary fluctuation value fbd is calculated by combining the mean of non-stationary activity with the degree of motion activity of the student state vector in the non-stationary sequence segment.

[0038] Step S213: Calculate the characteristic threshold tyz based on the static activity mean jjz, static fluctuation value jbd, non-static activity mean fjz, and non-static fluctuation value fbd;

[0039] .

[0040] Furthermore, the specific steps of step S22 are as follows:

[0041] Step S221: Obtain the number of segments of the still sequence fragment, denoted as ps, count the segment length of each still sequence fragment, denoted as still segment length jpc(1) to jpc(ps), extract the minimum value among the still segment lengths jpc(1) to jpc(ps) to obtain the trigger length;

[0042] Step S222: Obtain the time series of the student's state vector, use the feature threshold as a judgment condition to traverse and judge the time series of the student's state vector, extract the student's motion activity degree in the time series, if the motion activity degree is less than the feature threshold, judge that the student is in a static state, extract the posture features in the static state, verify the static state based on the posture features, count the continuous time series of the student in the static state, compare the continuous time series with the trigger length, if the continuous time series is greater than the trigger length, it indicates that the student needs teaching intervention, trigger the teaching event.

[0043] Furthermore, the specific steps of step S3 are as follows:

[0044] Step S31: Fine-tune the open-source multimodal base model. Through supervised fine-tuning, interactive control is performed on the base model. Fixed interaction steps are set. Through preference fine-tuning, the output results of the model are standardized. The fine-tuned model is saved to obtain the custom model.

[0045] Step S32: When a teaching event is triggered, the dedicated model is invoked, and the video stream data at the time of the teaching event is input into the dedicated model for problem identification. The identified problems are generated using a preset pattern to obtain a structured cognitive representation.

[0046] Furthermore, the specific steps of step S31 are as follows:

[0047] Step S311: High-resolution images of student assignments, draft paper, and textbook exercises are acquired to obtain image information. The image information is then transmitted to the open-source multimodal basic model for image recognition, and the recognition results are output. The recognition results are compared with the image information through manual verification, and the incorrectly recognized parts are extracted and manually labeled. Data is entered into the open-source multimodal basic model, and the open-source multimodal basic model is trained through multiple recognitions to enable it to accurately recognize images.

[0048] Step S312: Based on the image recognition results, repeatedly call the open-source multimodal basic model, set prompt words, analyze the image recognition results, store the analysis results in JSON format, and use the stored JSON format as the student's structured cognitive representation.

[0049] Step S313: Based on the trained model, multi-faceted intervention feedback is generated for the problem input. A preference dataset is generated based on the intervention feedback. The direct preference optimization algorithm is used, and the model is further optimized in combination with the preference dataset to obtain a custom model.

[0050] Furthermore, the specific steps of step S5 are as follows:

[0051] Step S51: After the teaching event is completed, the video stream data and structured cognitive representation corresponding to the teaching event are statistically analyzed to obtain a record file; the record file is persistently stored, an analysis trigger threshold is set for the stored record file, and the record file is analyzed to obtain the student's learning summary;

[0052] Step S52: Based on the students' learning summaries, extract the problems that exist in the students, generate the students' problem files, perform frequency statistics on the learning summaries based on the students' problem files, analyze the impact weight of different problems on students, and provide learning suggestions to students based on the impact weight of the problems.

[0053] Furthermore, the specific steps of step S52 are as follows:

[0054] Step S521: Perform text analysis on the students' learning summaries, decompose the students' learning summaries to obtain multiple keywords, automatically filter the keywords through a large model, extract keywords related to the learning content, count the number of keywords to obtain the number of keywords gs; record the keywords as gjc(1) to gjc(gs) based on the number of keywords, count the keywords, and generate the students' question files;

[0055] Step S522: Extract each keyword from the student's question file, use the keywords as search conditions to search the student's learning summary, record the number of search results, obtain the keyword frequency, and denote the keyword frequency as gpc(g); calculate the weight based on the keyword frequency to obtain the keyword influence weight gqz(g).

[0056] The influence weights gqz(g) of the keywords are sorted in descending order to obtain a weight ranking list. Learning suggestions are given to students based on the weight ranking list, and students study the learning content associated with the keywords corresponding to the weight ranking list.

[0057] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0058] Turning passive into proactive: By continuously sensing the student's status, the system can proactively identify opportunities for intervention, achieving "unobtrusive" protection and "timely" assistance.

[0059] From appearance to cognition: By leveraging the capabilities of multimodal large models, the system can "understand" students' learning content and processes, enabling in-depth analysis of students' cognitive levels.

[0060] From static to dynamic: student profiles are continuously evolving, and the system's intervention strategies are dynamically generated based on real-time analysis, truly achieving a high degree of adaptability that is tailored to each individual and each situation. Attached Figure Description

[0061] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0062] Figure 1 This is a schematic diagram of the method of the present invention;

[0063] Figure 2 This is a schematic diagram of the model recognition training of the present invention;

[0064] Figure 3 This is a schematic diagram illustrating the preference optimization of the present invention; Detailed Implementation

[0065] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0066] Example 1

[0067] Please see Figure 1 This invention provides an adaptive learning intervention method based on multimodal analysis, belonging to the fields of smart education and artificial intelligence technology. The method includes:

[0068] Step S1: Monitor students' learning status in real time to obtain video stream data, process the video stream data, and construct student state vectors;

[0069] Step S11: Decompose the video stream data, extract the image frames from the video stream data, count the number of image frames, and denote the number of image frames as ts. Count the number of pixels in each image frame and denote the number of pixels as is×js. Based on the number of pixels, obtain the image pixels for each pixel and denote the image pixels as txs. t (i, j), txs t (i, j) represents the image pixel at the i-th row and j-th column of the t-th image frame;

[0070] Image pixels txs from different image frames t (i, j), calculate the pixel changes of the image frame to obtain the student's motion activity degree yhd(t);

[0071] ;

[0072] Among them: txs t＋1 (i, j) represents the image pixel at the i-th row and j-th column of the (t+1)-th image frame;

[0073] It should be noted that by calculating the absolute value of the pixel difference between adjacent image frames, the calculation error caused by the mutual cancellation of pixel changes is prevented, thus improving the calculation accuracy. At the same time, by traversing and summing is×js pixels, the comprehensiveness of the motion activity calculation is ensured.

[0074] It should be noted that when students exercise, the image pixels change synchronously. Based on the changes in the image pixel values, the student's exercise status can be effectively fed back. At the same time, the image pixels can intuitively display minute numerical changes, accurately recording the student's exercise activities.

[0075] Step S12: Perform pose estimation on the image frame using a lightweight pose estimation algorithm (such as MediaPipe Pose) to identify the student's body nodes; record the position of the body nodes in the image frame, establish a base point based on the position of the body nodes in the image frame, and calculate the student's pose quantization value through the positional relationship between the base point and other body nodes to obtain the pose features.

[0076] It should be noted that pose estimation algorithms are an important branch of computer vision, which aim to analyze the spatial pose (such as joint angles, limb positions, etc.) of human bodies, animals, or objects through image or video data.

[0077] Step S121: Obtain the number of student body nodes, denoted as ds; collect the position of each body node to obtain the node position jsw. t (d); jsw t (d) = (i) t (d), j t (d));

[0078] It should be noted that the node determination is based on the analysis of image frames. The node's position corresponds to the position of a pixel in the image. The node's position is numerically extracted using is×js pixels. t (d), j t (d) can take values ​​within the range of is×js;

[0079] Based on the node positions in different image frames, the numerical changes in node positions are calculated to obtain the change in node positions bhl(d).

[0080] ;

[0081] Where: i t (d), j t (d) represents the parameter indicating the node position in frame t;

[0082] It should be noted that the calculation is performed by using the mean and the node positions of different frames to improve the stability of the calculation. At the same time, the changes in node positions are decomposed into numerical changes in different directions, which facilitates intuitive judgment of the changes. Combining the two ensures the comprehensiveness and accuracy of the calculation results.

[0083] The changes at different node positions are statistically analyzed to obtain changes from bhl(1) to bhl(ds); the smallest change among the changes from bhl(1) to bhl(ds) is extracted, and the node corresponding to it is taken as the base point. The position of the base point is denoted as (jdi). t jdj t );

[0084] Step S122: Calculate the distance between the base point and the body node on each video frame, integrate all distances to obtain the student's pose quantization value, and record the student's pose quantization value as the pose feature ztz(t).

[0085] ;

[0086] It should be noted that by analyzing and statistically analyzing the distance relationships between body nodes, the student's posture characteristics can be quantified, which facilitates the judgment of posture characteristics.

[0087] Step S13: Statistically analyze the degree of motion and posture characteristics, construct a two-dimensional student state vector, and denote the student state vector as V, V = [yhd, ztz].

[0088] Step S2: Update the student state vector to obtain the time series of the student state vector. Extract feature sequence segments through self-labeling. Set feature thresholds based on the feature sequence segments. Analyze the time series based on the feature thresholds to identify teaching events.

[0089] Step S21: Update and statistically analyze the student state vector in real time according to the time series to obtain the time series of the student state vector. Label the time series to obtain the feature sequence segment. Extract the student state vector from the feature sequence segment, analyze the value of the student state vector, and set the feature threshold.

[0090] It should be noted that: labeling time series refers to judging and recording students' action states (such as writing, stillness) through analysis by students themselves and educational experts.

[0091] Step S211: Obtain the student state vectors under different video frames according to the temporal sequence, and denote them as V(t), where V(t) represents the student state vector under the t-th video frame; count the student state vectors to form a time series of student state vectors, and label the time series to obtain static sequence segments and non-static sequence segments.

[0092] Step S212: Count the number of video frames in the still sequence segment, denoted as jzs. Obtain the motion activity degree of the student state vector under each video frame in the still sequence segment, denoted as jyhd(jz). Calculate the mean of the motion activity degree jyhd(jz) based on the number of video frames jzs in the still sequence segment to obtain the mean static activity value jjz.

[0093] ;

[0094] The static fluctuation value jbd is calculated by combining the mean static activity with the degree of motion activity of the student state vector in the static sequence segment.

[0095] ;

[0096] The number of video frames in the non-static sequence segment is counted and denoted as fjs. The motion activity degree of the student state vector under each video frame in the non-static sequence segment is obtained and denoted as fyhd(fj). Based on the number of video frames fjs in the non-static sequence segment, the mean of motion activity degree fyhd(fj) is calculated to obtain the mean of non-static activity fjz.

[0097] ;

[0098] The non-stationary fluctuation value fbd is calculated by combining the mean of non-stationary activity with the degree of motion activity of the student state vector in the non-stationary sequence segment.

[0099] ;

[0100] Step S213: Calculate the characteristic threshold tyz based on the static activity mean jjz, static fluctuation value jbd, non-static activity mean fjz, and non-static fluctuation value fbd;

[0101] ;

[0102] It should be noted that the range of values ​​between the mean of static activity and the mean of non-static activity is allocated by the fluctuation value ratio. Based on the allocation result, a single value is extracted to judge the student's degree of physical activity, and the judgment result is accurate.

[0103] Step S22: Obtain the segment length of the feature sequence segment, set the trigger length according to the segment length of the feature sequence segment, identify the student status according to the feature threshold, and identify the teaching event based on the trigger length and the student status.

[0104] Step S221: Obtain the number of segments of the still sequence fragment, denoted as ps, count the segment length of each still sequence fragment, denoted as still segment length jpc(1) to jpc(ps), extract the minimum value among the still segment lengths jpc(1) to jpc(ps) to obtain the trigger length;

[0105] Step S222: Obtain the time series of student state vectors, use the feature threshold as a judgment condition to traverse and judge the time series of student state vectors, extract the student's motion activity degree in the time series, if the motion activity degree is less than the feature threshold, judge that the student is in a static state, extract the posture features in the static state, verify the static state based on the posture features, count the continuous time series of students in a static state, compare the continuous time series with the trigger length, if the continuous time series is greater than the trigger length, it indicates that the student needs teaching intervention, trigger the teaching event;

[0106] It should be noted that verifying the static state based on posture characteristics ensures that students are in a learning state, accurately identifies the triggering causes of teaching events, and guarantees the accuracy of teaching event triggering.

[0107] Please see Figure 2 Step S3: Obtain the open-source multimodal base model, fine-tune the open-source multimodal base model to customize the exclusive model; when the teaching event is triggered, extract the corresponding video stream data, transmit the video stream data to the exclusive model, and generate a structured cognitive representation.

[0108] Step S31: Fine-tune the open-source multimodal base model. Through supervised fine-tuning, interactive control is performed on the base model. Fixed interaction steps are set. Through preference fine-tuning, the output results of the model are standardized. The fine-tuned model is saved to obtain the custom model.

[0109] It should be noted that Supervised Fine-Tuning (SFT) is a commonly used technique in machine learning, especially on the basis of pre-trained models (such as BERT, GPT, ResNet, etc.), to further optimize the model performance with a small amount of labeled data, so that it can be better adapted to specific tasks.

[0110] Preference Fine-Tuning (PFT) is an optimization technique for Large Language Models (LLMs). Its core goal is to enable the model to learn and distinguish "better" responses by comparing different responses output by the model, thereby making its performance on subjective tasks such as tone, style, and creativity more in line with human expectations.

[0111] Step S311: Collect high-definition images of student assignments, draft paper, and textbook exercises in real learning scenarios to obtain image information. Transmit the image information to the open-source multimodal basic model for image recognition and output the recognition results. Through manual verification, compare the recognition results with the image information, extract the incorrectly recognized parts, manually label the incorrectly recognized parts, input data into the open-source multimodal basic model, and train the open-source multimodal basic model through multiple recognitions to enable it to accurately recognize images.

[0112] Step S312: Based on the image recognition results, repeatedly call the open-source multimodal basic model, set prompt words, analyze the image recognition results, store the analysis results in JSON format, and use the stored JSON format as the student's structured cognitive representation.

[0113] It should be noted that the basic model undergoes full or efficient parameter fine-tuning. Through learning and training, the model learns to recognize questions, understand handwriting, judge right and wrong, and analyze error types, and stably generates JSON output that conforms to preset patterns, ensuring the system's subsequent automated processing.

[0114] Please see Figure 3 Step S313: Based on the trained model, multi-faceted intervention feedback is generated for the problem input. A preference dataset is constructed based on the intervention feedback. The Direct Preference Optimization (DPO) algorithm is used to further optimize the model in combination with the preference dataset to obtain a custom model.

[0115] Step S32: When a teaching event is triggered, the dedicated model is invoked, and the video stream data at the time of the teaching event is input into the dedicated model for problem identification. The identified problems are generated using a preset pattern to obtain a structured cognitive representation.

[0116] Step S4: Analyze the structured cognitive representation, use the analysis results as the question input, repeatedly call the dedicated model to generate prompt text, and display the prompt text to the students;

[0117] The system analyzes structured cognitive representations, generates prompts based on the analysis results, transforms the analysis results into question inputs using the prompts, and then uses a dedicated model to answer the questions. The system also analyzes the answer results, generates prompt text, and uses the prompt text to interact with students and guide their learning direction.

[0118] Step S5: Record the video stream data and structured cognitive representations corresponding to the teaching events to obtain a record file. Persist in storing the record file, summarize the data comprehensively based on the stored record file, generate the student's problem file, and provide learning suggestions to the student.

[0119] Step S51: After the teaching event is completed, the video stream data and structured cognitive representation corresponding to the teaching event are statistically analyzed to obtain a record file; the record file is persistently stored in a PostgreSQL database, an analysis trigger threshold is set for the stored record file, the record file is analyzed, and the student's learning summary is obtained.

[0120] Step S52: Based on the students' learning summaries, extract the problems that exist in the students, generate the students' problem files, perform frequency statistics on the learning summaries based on the students' problem files, analyze the impact weight of different problems on students, and provide learning suggestions to students based on the impact weight of the problems.

[0121] Step S521: Perform text analysis on the students' learning summary, decompose the students' learning summary to obtain multiple keywords, automatically filter the keywords through a large model, extract keywords related to the learning content, count the number of keywords to obtain the number of keywords gs; record the keywords as gjc(1) to gjc(gs) based on the number of keywords, count the keywords to generate the students' question files;

[0122] Step S522: Extract each keyword from the student's question file, use the keywords as search conditions to search the student's learning summary, record the number of search results, obtain the keyword frequency, and denote the keyword frequency as gpc(g); calculate the weight based on the keyword frequency to obtain the keyword influence weight gqz(g).

[0123] ;

[0124] The influence weights gqz(g) of the keywords are sorted in descending order to obtain a weight ranking list. Learning suggestions are given to students based on the weight ranking list, and students study the learning content associated with the keywords corresponding to the weight ranking list.

[0125] Step S6: Through access control, synchronize and transmit the student's problem file and learning suggestions to other authorized clients;

[0126] Students' problem files and learning suggestions are stored in the cloud, and access to the cloud is controlled by permissions. Data is transmitted to other clients through authorization. Other clients can assist students' learning based on the students' problem files and learning suggestions.

[0127] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of adaptive learning intervention based on multi-modal analysis, characterized in that, The application comprises the following steps: Step S1: Real-time monitoring of the learning status of students to obtain video stream data, processing the video stream data, and constructing a student state vector; Step S2: Updating the student state vector to obtain a time series of the student state vector, extracting a feature sequence segment through autonomous labeling, setting a feature threshold based on the feature sequence segment, analyzing the time series based on the feature threshold, and identifying a teaching event; Step S3: Obtaining an open-source multi-modal base model, customizing a dedicated model by fine-tuning the open-source multi-modal base model; when the teaching event is triggered, extracting the corresponding video stream data, transmitting the video stream data to the dedicated model, and generating a structured cognitive representation; Step S4: Analyzing the structured cognitive representation, inputting the analysis result as a question, repeatedly calling the dedicated model to generate a prompt text, and displaying the prompt text to the student; Step S5: Recording the video stream data corresponding to the teaching event and the structured cognitive representation to obtain a record file, persistently storing the record file, summarizing based on the stored record file, generating a student problem file, and providing learning suggestions to the student; Step S6: Synchronizing the student problem file and learning suggestions to other authorized clients through permission control; The specific steps of step S1 are as follows: Step S11: decompose the video stream data, extract the image frames of the video stream data, count the number of image frames, record the number of image frames as ts, count the number of pixel points of each image frame, record the number of pixel points as is×js, according to the number of pixel points, acquire the image pixel of each pixel point, record the image pixel as txs t (i,j), txs t (i,j) represents the image pixel of the pixel point at the i-th row and the j-th column of the t-th image frame; By image pixels txs t (i,j) of different image frames, the pixel change of the image frames is calculated to obtain the motion activity degree yhd(t) of the student; ； wherein: txs t＋1 (i,j) represents the image pixel of the pixel point at the i-th row and j-th column of the t+1-th image frame. It should be noted that when the student is moving, the image pixel changes synchronously, and the change value of the image pixel can effectively reflect the student's movement status. At the same time, the image pixel can intuitively display the small numerical change, and accurately record the student's movement activity; Step S12: Pose estimation of the image frame to identify the body nodes of the student; recording the position of the body nodes in the image frame, setting a base point based on the position of the body nodes in the image frame, calculating the pose quantitative value of the student based on the positional relationship between the base point and other body nodes, and obtaining the pose feature; Step S13: Statistically constructing a two-dimensional student state vector, denoted as V.

2. The self-adaptive learning intervention method based on multi-modal analysis according to claim 1, characterized in that, The specific steps of step S12 are as follows: Step S121: Obtain the number of body nodes of the student, denoted as ds; collect the position of each body node to obtain node position jsw t (d); jsw t (d) = (i t (d), j t (d)); It should be noted that the judgment of the node is based on the analysis of the image frame, so the position of the node is consistent with the position of the pixel point of the image, and the position of the node is extracted by is×js pixel points; t (d), j t The value of (d) is in the range of is×js. According to the node position under different image frames, the numerical change of the node position is calculated to obtain the change amount bhl(d) of the node position; ； wherein: i t (d), j t (d) denotes parameters of the node position of the t-th frame; The variation amounts of different node positions are counted to obtain variation amounts bhl(1) to bhl(ds); the minimum variation amount among the variation amounts bhl(1) to bhl(ds) is extracted, and the node corresponding to the minimum variation amount is taken as a base point; and the position of the base point is recorded as (jdi t , jdj t ). Step S122: Calculating the distance between the base point and the body nodes on each video frame, integrating all the distances, and obtaining the pose quantitative value of the student, denoted as the pose feature ztz(t).

3. The self-adaptive learning intervention method based on multi-modal analysis according to claim 1, characterized in that, The specific steps of step S2 are as follows: Step S21: Real-time updating and statistical analysis of the student state vector based on the time sequence to obtain a time series of the student state vector, labeling the time series to obtain a feature sequence segment, extracting the student state vector in the feature sequence segment, analyzing the numerical value of the student state vector, and setting a feature threshold; Step S22: Obtaining the segment length of the feature sequence segment, setting a trigger length based on the segment length of the feature sequence segment, identifying the student state based on the feature threshold, and identifying a teaching event based on the trigger length and the student state.

4. The self-adaptive learning intervention method based on multi-modal analysis according to claim 3, characterized in that, The specific steps of step S21 further comprise: Step S211: Obtain the student state vector under different video frames from the time sequence, denoted as V(t), V(t) represents the student state vector under the tth video frame; perform statistics on the student state vector to form a time sequence of the student state vector, label the time sequence to obtain a static sequence segment and a non-static sequence segment; Step S212: Count the number of video frames in the static sequence segment, denoted as jzs, obtain the motion activity degree of the student state vector under each video frame in the static sequence segment, denoted as jyhd(jz), and calculate the mean value of the motion activity degree jyhd(jz) according to the number of video frames jzs in the static sequence segment to obtain a static activity mean value jjz; According to the static activity mean value and the motion activity degree of the student state vector in the static sequence segment, a static fluctuation value jbd is obtained; Count the number of video frames in the non-static sequence segment, denoted as fjs, obtain the motion activity degree of the student state vector under each video frame in the non-static sequence segment, denoted as fyhd(fj), and calculate the mean value of the motion activity degree fyhd(fj) according to the number of video frames fjs in the non-static sequence segment to obtain a non-static activity mean value fjz; According to the non-static activity mean value and the motion activity degree of the student state vector in the non-static sequence segment, a non-static fluctuation value fbd is obtained; Step S213: Calculate the feature threshold tyz according to the static activity mean value jjz, the static fluctuation value jbd, the non-static activity mean value fjz, and the non-static fluctuation value fbd; 。 5. The self-adaptive learning intervention method based on multi-modal analysis according to claim 3, characterized in that, The specific steps of the step S22 are as follows: Step S221: Obtain the number of segments of the static sequence segment, denoted as ps, count the segment length of each static sequence segment, denoted as static segment length jpc(1) to jpc(ps), and extract the minimum value in the static segment length jpc(1) to jpc(ps) to obtain a trigger length; Step S222: Obtain the time sequence of the student state vector, traverse the time sequence of the student state vector with the feature threshold as a judgment condition, extract the motion activity degree of the student in the time sequence, if the motion activity degree is less than the feature threshold, judge that the student is in a static state, extract the posture feature in the static state, verify the static state according to the posture feature, count the continuous time sequence in which the student is in the static state, and compare the continuous time sequence with the trigger length, if the continuous time sequence is greater than the trigger length, it indicates that the student needs teaching intervention, and a teaching event is triggered.

6. The self-adaptive learning intervention method based on multi-modal analysis according to claim 1, characterized in that, The specific steps of the step S3 are as follows: Step S31: Fine-tune based on an open-source multi-modal base model, perform interactive control on the base model through supervised fine-tuning, set a fixed interactive step, standardize the output result of the model through preference fine-tuning, save the fine-tuned model to obtain a dedicated model; Step S32: When the teaching event triggers, the exclusive model is called, the video stream data at the time of the teaching event trigger is input into the exclusive model for problem identification, and the identified problem is generated in a preset mode to obtain a structured cognitive representation.

7. The self-adaptive learning intervention method based on multi-modal analysis according to claim 6, characterized in that, The specific steps of the step S31 are as follows: Step S311: High-definition images of student homework, scratch paper, and textbook exercises are collected to obtain image information, the image information is transmitted to an open-source multi-modal basic model for image recognition, the recognition result is output, the recognition result is compared with the image information through manual inspection, the error recognition part is extracted, the error recognition part is manually labeled, data entry is performed on the open-source multi-modal basic model, the open-source multi-modal basic model is trained through multiple recognitions, and the open-source multi-modal basic model can accurately recognize images; Step S312: According to the image recognition result, the open-source multi-modal basic model is repeatedly called, prompt words are set, the image recognition result is analyzed, the analysis result is stored in a json format, and the stored json format is used as the structured cognitive representation of the student; Step S313: Based on the trained model, the problem input is subjected to multi-aspect intervention feedback generation, a preference data set is constructed according to the intervention feedback generation, a direct preference optimization algorithm is adopted, the model is optimized again in combination with the preference data set, and the exclusive model is obtained.

8. The self-adaptive learning intervention method based on multi-modal analysis according to claim 1, characterized in that, The specific steps of the step S5 are as follows: Step S51: After the teaching event is completed, the video stream data corresponding to the teaching event and the structured cognitive representation are counted to obtain a record file; the record file is persistently stored, the analysis trigger threshold is set for the stored record file, the record file is analyzed, and the learning summary of the student is obtained; Step S52: According to the learning summary of the student, the problems existing in the student are extracted, the problem file of the student is generated, the frequency of the learning summary is counted according to the problem file of the student, the influence weight of different problems on the student is analyzed, and learning suggestions are made to the student according to the influence weight of the problem.

9. The self-adaptive learning intervention method based on multi-modal analysis according to claim 8, characterized in that, The specific steps of the step S52 are as follows: Step S521: The learning summary of the student is analyzed, the learning summary of the student is decomposed to obtain a plurality of keywords, the keywords are automatically screened, the keywords associated with the learning content are extracted, the number of keywords is counted to obtain a keyword number gs; the keywords are recorded as gjc(1) to gjc(gs) according to the keyword number, the keywords are counted to generate the problem file of the student; Step S522: According to the problem file of the student, each keyword is extracted, the keyword is used as a retrieval condition to retrieve the learning summary of the student, the number of retrieval results is recorded, the keyword frequency is obtained, and the keyword frequency is recorded as gpc(g); the keyword frequency is used for weight calculation to obtain the influence weight gqz(g) of the keyword; The influence weight gqz(g) of the keyword is sorted in descending order to obtain a weight sorting list, learning suggestions are made to the student according to the weight sorting list, and the student learns the learning content associated with the keyword according to the keyword corresponding to the weight sorting list.

Images