Adaptive learning intervention method based on multi-modal analysis

Abstract

The invention discloses an adaptive learning intervention method based on multi-modal analysis, belongs to the technical field of wisdom education and artificial intelligence, solves the problem that an existing learning intervention method is insufficient in accuracy, and comprises the following steps: S1, constructing a student state vector; s2, creating a time sequence of the student state vector, and identifying a teaching event according to the time sequence; s3, customizing the exclusive model; when a teaching event is triggered, extracting corresponding video stream data, transmitting the video stream data to the exclusive model, and generating a structured cognitive representation; s4, analyzing the structured cognitive representation, generating a prompt text, and displaying the prompt text to students; s5, recording the video stream data and the structured cognitive representation, generating a question file of the student, and giving a learning suggestion to the student; s6, synchronously transmitting the question files and the learning suggestions of the students; according to the invention, by training a proprietary model, ultrahigh-precision personalized tutoring is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the cross field of artificial intelligence and education science, and in particular to a method and system for highly personalized and adaptive tutoring of learners using real-time multi-modal perception, advanced AI model analysis and closed-loop feedback mechanism. BACKGROUND

[0002] The existing technologies or products related to student learning tutoring have the following shortcomings: Passive interaction: Most systems rely on the active input of learners, such as submitting answers, clicking buttons or initiating questions; the system lacks active perception ability and cannot intervene at the "moment of confusion" when the learner needs help most.

[0003] Shallow data analysis: The analysis of the system is mostly based on the final result (such as correct or incorrect answers) or simple behavior data (such as page dwell time), which cannot go deep into the cognitive process of the learner, understand why the learner makes mistakes and where the learner gets stuck.

[0004] Static personalization: Many so-called "personalization" is based on pre-set user profiles or simple rule engines, which is difficult to dynamically adjust strategies according to the real-time state changes of the learner in a learning process, and lacks true adaptability.

[0005] Therefore, there is an urgent need in the art for a new intelligent tutoring technology that can actively perceive, deeply understand and dynamically adapt to the real-time state of the learner; for this purpose, we propose an adaptive learning intervention method based on multi-modal analysis. SUMMARY

[0006] In view of the shortcomings of the prior art, the core purpose of the present application is to provide an AI tutoring method that can simulate the "reading between the lines" and "teaching according to the aptitude" ability of excellent human teachers, to overcome the limitations of the prior art.

[0007] In order to achieve the above purpose, the present application adopts the following technical solution: an adaptive learning intervention method based on multi-modal analysis, the specific working process of each step is as follows: Step S1: Real-time monitoring of the learning status of the student, obtaining video stream data, processing the video stream data, and constructing a student state vector; Step S2: Updating the student state vector, obtaining the time series of the student state vector, extracting feature sequence segments through autonomous labeling, setting feature threshold values according to the feature sequence segments, analyzing the time series according to the feature threshold values, and identifying teaching events; Step S3: Obtain the open-source multimodal base model, fine-tune the open-source multimodal base model to customize the exclusive model; when the teaching event is triggered, extract the corresponding video stream data, transmit the video stream data to the exclusive model, and generate structured cognitive representations; Step S4: Analyze the structured cognitive representation, use the analysis results as the question input, repeatedly call the dedicated model to generate prompt text, and display the prompt text to the students; Step S5: Record the video stream data and structured cognitive representations corresponding to the teaching events to obtain a record file. Persist in storing the record file, summarize the data comprehensively based on the stored record file, generate the student's problem file, and provide learning suggestions to the student. Step S6: Through access control, synchronize and transmit the student's problem file and learning suggestions to other authorized clients.

[0008] Furthermore, the specific steps of step S1 are as follows: Step S11: Decompose the video stream data, extract the image frames from the video stream data, count the number of image frames, and denote the number of image frames as ts. Count the number of pixels in each image frame and denote the number of pixels as is×js. Based on the number of pixels, obtain the image pixels for each pixel and denote the image pixels as txs. t (i, j), txs t (i, j) represents the image pixel at the i-th row and j-th column of the t-th image frame; Image pixels txs from different image frames t (i, j), calculate the pixel changes of the image frame to obtain the student's motion activity degree yhd(t); ; Among them: txs t＋1 (i, j) represents the image pixel at the i-th row and j-th column of the (t+1)-th image frame; Step S12: Perform pose estimation on the image frame and identify the student's body nodes; record the position of the body nodes in the image frame, establish a base point based on the position of the body nodes in the image frame, and calculate the student's pose quantization value through the positional relationship between the base point and other body nodes to obtain the pose features. Step S13: Statistically analyze the degree of motion and posture characteristics, construct a two-dimensional student state vector, and denote the student state vector as V.

[0009] Furthermore, the specific steps of step S12 are as follows: Step S121: Obtain the number of student body nodes, denoted as ds; collect the position of each body node to obtain the node position jsw.t (d); jsw t (d) = (i) t (d), j t (d)); Based on the node positions in different image frames, the numerical changes in node positions are calculated to obtain the change in node positions bhl(d). ; Where: i t (d), j t (d) represents the parameter indicating the node position in frame t; The changes at different node positions are statistically analyzed to obtain changes from bhl(1) to bhl(ds); the smallest change among the changes from bhl(1) to bhl(ds) is extracted, and the node corresponding to it is taken as the base point. The position of the base point is denoted as (jdi). t jdj t ); Step S122: Calculate the distance between the base point and the body node on each video frame, integrate all distances to obtain the student's pose quantization value, and record the student's pose quantization value as the pose feature ztz(t).

[0010] Furthermore, the specific steps of step S2 are as follows: Step S21: Update and statistically analyze the student state vector in real time according to the time series to obtain the time series of the student state vector. Label the time series to obtain the feature sequence segment. Extract the student state vector from the feature sequence segment, analyze the value of the student state vector, and set the feature threshold. Step S22: Obtain the segment length of the feature sequence fragment, set the trigger length according to the segment length of the feature sequence fragment, identify the student status according to the feature threshold, and identify the teaching event based on the trigger length and the student status.

[0011] Furthermore, the specific steps of step S21 also include: Step S211: Obtain the student state vectors under different video frames according to the temporal sequence, and denote them as V(t), where V(t) represents the student state vector under the t-th video frame; count the student state vectors to form a time series of student state vectors, and label the time series to obtain static sequence segments and non-static sequence segments. Step S212: Count the number of video frames in the still sequence segment, denoted as jzs. Obtain the motion activity degree of the student state vector under each video frame in the still sequence segment, denoted as jyhd(jz). Calculate the mean of the motion activity degree jyhd(jz) based on the number of video frames jzs in the still sequence segment to obtain the mean static activity value jjz. The static fluctuation value jbd is calculated by combining the mean static activity with the degree of motion activity of the student state vector in the static sequence segment. The number of video frames in the non-static sequence segment is counted and denoted as fjs. The motion activity degree of the student state vector under each video frame in the non-static sequence segment is obtained and denoted as fyhd(fj). Based on the number of video frames fjs in the non-static sequence segment, the mean of motion activity degree fyhd(fj) is calculated to obtain the mean of non-static activity fjz. The non-stationary fluctuation value fbd is calculated by combining the mean of non-stationary activity with the degree of motion activity of the student state vector in the non-stationary sequence segment. Step S213: Calculate the characteristic threshold tyz based on the static activity mean jjz, static fluctuation value jbd, non-static activity mean fjz, and non-static fluctuation value fbd; .

[0012] Furthermore, the specific steps of step S22 are as follows: Step S221: Obtain the number of segments of the still sequence fragment, denoted as ps, count the segment length of each still sequence fragment, denoted as still segment length jpc(1) to jpc(ps), extract the minimum value among the still segment lengths jpc(1) to jpc(ps) to obtain the trigger length; Step S222: Obtain the time series of the student's state vector, use the feature threshold as a judgment condition to traverse and judge the time series of the student's state vector, extract the student's motion activity degree in the time series, if the motion activity degree is less than the feature threshold, judge that the student is in a static state, extract the posture features in the static state, verify the static state based on the posture features, count the continuous time series of the student in the static state, compare the continuous time series with the trigger length, if the continuous time series is greater than the trigger length, it indicates that the student needs teaching intervention, trigger the teaching event.

[0013] Furthermore, the specific steps of step S3 are as follows: Step S31: Fine-tune the open-source multimodal base model. Through supervised fine-tuning, interactive control is performed on the base model. Fixed interaction steps are set. Through preference fine-tuning, the output results of the model are standardized. The fine-tuned model is saved to obtain the custom model. Step S32: When a teaching event is triggered, the dedicated model is invoked, and the video stream data at the time of the teaching event is input into the dedicated model for problem identification. The identified problems are generated using a preset pattern to obtain a structured cognitive representation.

[0014] Furthermore, the specific steps of step S31 are as follows: Step S311: High-resolution images of student assignments, draft paper, and textbook exercises are acquired to obtain image information. The image information is then transmitted to the open-source multimodal basic model for image recognition, and the recognition results are output. The recognition results are compared with the image information through manual verification, and the incorrectly recognized parts are extracted and manually labeled. Data is entered into the open-source multimodal basic model, and the open-source multimodal basic model is trained through multiple recognitions to enable it to accurately recognize images. Step S312: Based on the image recognition results, repeatedly call the open-source multimodal basic model, set prompt words, analyze the image recognition results, store the analysis results in JSON format, and use the stored JSON format as the student's structured cognitive representation. Step S313: Based on the trained model, multi-faceted intervention feedback is generated for the problem input. A preference dataset is generated based on the intervention feedback. The direct preference optimization algorithm is used, and the model is further optimized in combination with the preference dataset to obtain a custom model.

[0015] Furthermore, the specific steps of step S5 are as follows: Step S51: After the teaching event is completed, the video stream data and structured cognitive representation corresponding to the teaching event are statistically analyzed to obtain a record file; the record file is persistently stored, an analysis trigger threshold is set for the stored record file, and the record file is analyzed to obtain the student's learning summary; Step S52: Based on the students' learning summaries, extract the problems that exist in the students, generate the students' problem files, perform frequency statistics on the learning summaries based on the students' problem files, analyze the impact weight of different problems on students, and provide learning suggestions to students based on the impact weight of the problems.

[0016] Furthermore, the specific steps of step S52 are as follows: Step S521: Perform text analysis on the students' learning summaries, decompose the students' learning summaries to obtain multiple keywords, automatically filter the keywords through a large model, extract keywords related to the learning content, count the number of keywords to obtain the number of keywords gs; record the keywords as gjc(1) to gjc(gs) based on the number of keywords, count the keywords, and generate the students' question files; Step S522: Extract each keyword from the student's question file, use the keywords as search conditions to search the student's learning summary, record the number of search results, obtain the keyword frequency, and denote the keyword frequency as gpc(g); calculate the weight based on the keyword frequency to obtain the keyword influence weight gqz(g). The influence weights gqz(g) of the keywords are sorted in descending order to obtain a weight ranking list. Learning suggestions are given to students based on the weight ranking list, and students study the learning content associated with the keywords corresponding to the weight ranking list.

[0017] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: Turning passive into proactive: By continuously sensing the student's status, the system can proactively identify opportunities for intervention, achieving "unobtrusive" protection and "timely" assistance.

[0018] From appearance to cognition: By leveraging the capabilities of multimodal large models, the system can "understand" students' learning content and processes, enabling in-depth analysis of students' cognitive levels.

[0019] From static to dynamic: student profiles are continuously evolving, and the system's intervention strategies are dynamically generated based on real-time analysis, truly achieving a high degree of adaptability that is tailored to each individual and each situation. Attached Figure Description

[0020] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0021] Figure 1 This is a schematic diagram of the method of the present invention; Figure 2 This is a schematic diagram of the model recognition training of the present invention; Figure 3 This is a schematic diagram illustrating the preference optimization of the present invention; Detailed Implementation The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1 Please see Figure 1 This invention provides an adaptive learning intervention method based on multimodal analysis, belonging to the fields of smart education and artificial intelligence technology. The method includes: Step S1: Monitor students' learning status in real time to obtain video stream data, process the video stream data, and construct student state vectors; Step S11: Decompose the video stream data, extract the image frames from the video stream data, count the number of image frames, and denote the number of image frames as ts. Count the number of pixels in each image frame and denote the number of pixels as is×js. Based on the number of pixels, obtain the image pixels for each pixel and denote the image pixels as txs. t (i, j), txs t (i, j) represents the image pixel at the i-th row and j-th column of the t-th image frame; Image pixels txs from different image frames t (i, j), calculate the pixel changes of the image frame to obtain the student's motion activity degree yhd(t); ; Among them: txs t＋1 (i, j) represents the image pixel at the i-th row and j-th column of the (t+1)-th image frame; It should be noted that by calculating the absolute value of the pixel difference between adjacent image frames, the calculation error caused by the mutual cancellation of pixel changes is prevented, thus improving the calculation accuracy. At the same time, by traversing and summing is×js pixels, the comprehensiveness of the motion activity calculation is ensured. It should be noted that when students exercise, the image pixels change synchronously. Based on the changes in the image pixel values, the student's exercise status can be effectively fed back. At the same time, the image pixels can intuitively display minute numerical changes, accurately recording the student's exercise activities. Step S12: Perform pose estimation on the image frame using a lightweight pose estimation algorithm (such as MediaPipe Pose) to identify the student's body nodes; record the position of the body nodes in the image frame, establish a base point based on the position of the body nodes in the image frame, and calculate the student's pose quantization value through the positional relationship between the base point and other body nodes to obtain the pose features. It should be noted that pose estimation algorithms are an important branch of computer vision, which aim to analyze the spatial pose (such as joint angles, limb positions, etc.) of human bodies, animals, or objects through image or video data.

[0023] Step S121: Obtain the number of student body nodes, denoted as ds; collect the position of each body node to obtain the node position jsw. t (d); jsw t (d) = (i) t (d), j t (d)); It should be noted that the node determination is based on the analysis of image frames. The node's position corresponds to the position of a pixel in the image. The node's position is numerically extracted using is×js pixels. t (d), j t (d) can take values ​​within the range of is×js; Based on the node positions in different image frames, the numerical changes in node positions are calculated to obtain the change in node positions bhl(d). ; Where: i t (d), j t (d) represents the parameter indicating the node position in frame t; It should be noted that the calculation is performed by using the mean and the node positions of different frames to improve the stability of the calculation. At the same time, the changes in node positions are decomposed into numerical changes in different directions, which facilitates intuitive judgment of the changes. Combining the two ensures the comprehensiveness and accuracy of the calculation results.

[0024] The changes at different node positions are statistically analyzed to obtain changes from bhl(1) to bhl(ds); the smallest change among the changes from bhl(1) to bhl(ds) is extracted, and the node corresponding to it is taken as the base point. The position of the base point is denoted as (jdi). t jdj t ); Step S122: Calculate the distance between the base point and the body node on each video frame, integrate all distances to obtain the student's pose quantization value, and record the student's pose quantization value as the pose feature ztz(t). ; It should be noted that by analyzing and statistically analyzing the distance relationships between body nodes, the student's posture characteristics can be quantified, which facilitates the judgment of posture characteristics. Step S13: Statistically analyze the degree of motion and posture characteristics, construct a two-dimensional student state vector, and denote the student state vector as V, V = [yhd, ztz].

[0025] Step S2: Update the student state vector to obtain the time series of the student state vector. Extract feature sequence segments through self-labeling. Set feature thresholds based on the feature sequence segments. Analyze the time series based on the feature thresholds to identify teaching events. Step S21: Update and statistically analyze the student state vector in real time according to the time series to obtain the time series of the student state vector. Label the time series to obtain the feature sequence segment. Extract the student state vector from the feature sequence segment, analyze the value of the student state vector, and set the feature threshold. It should be noted that: labeling time series refers to judging and recording students' action states (such as writing, stillness) through analysis by students themselves and educational experts.

[0026] Step S211: Obtain the student state vectors under different video frames according to the temporal sequence, and denote them as V(t), where V(t) represents the student state vector under the t-th video frame; count the student state vectors to form a time series of student state vectors, and label the time series to obtain static sequence segments and non-static sequence segments. Step S212: Count the number of video frames in the still sequence segment, denoted as jzs. Obtain the motion activity degree of the student state vector under each video frame in the still sequence segment, denoted as jyhd(jz). Calculate the mean of the motion activity degree jyhd(jz) based on the number of video frames jzs in the still sequence segment to obtain the mean static activity value jjz. ; The static fluctuation value jbd is calculated by combining the mean static activity with the degree of motion activity of the student state vector in the static sequence segment. ; The number of video frames in the non-static sequence segment is counted and denoted as fjs. The motion activity degree of the student state vector under each video frame in the non-static sequence segment is obtained and denoted as fyhd(fj). Based on the number of video frames fjs in the non-static sequence segment, the mean of motion activity degree fyhd(fj) is calculated to obtain the mean of non-static activity fjz. ; The non-stationary fluctuation value fbd is calculated by combining the mean of non-stationary activity with the degree of motion activity of the student state vector in the non-stationary sequence segment. ; Step S213: Calculate the characteristic threshold tyz based on the static activity mean jjz, static fluctuation value jbd, non-static activity mean fjz, and non-static fluctuation value fbd; ; It should be noted that the range of values ​​between the mean of static activity and the mean of non-static activity is allocated by the fluctuation value ratio. Based on the allocation result, a single value is extracted to judge the student's degree of physical activity, and the judgment result is accurate.

[0027] Step S22: Obtain the segment length of the feature sequence segment, set the trigger length according to the segment length of the feature sequence segment, identify the student status according to the feature threshold, and identify the teaching event based on the trigger length and the student status. Step S221: Obtain the number of segments of the still sequence fragment, denoted as ps, count the segment length of each still sequence fragment, denoted as still segment length jpc(1) to jpc(ps), extract the minimum value among the still segment lengths jpc(1) to jpc(ps) to obtain the trigger length; Step S222: Obtain the time series of student state vectors, use the feature threshold as a judgment condition to traverse and judge the time series of student state vectors, extract the student's motion activity degree in the time series, if the motion activity degree is less than the feature threshold, judge that the student is in a static state, extract the posture features in the static state, verify the static state based on the posture features, count the continuous time series of students in a static state, compare the continuous time series with the trigger length, if the continuous time series is greater than the trigger length, it indicates that the student needs teaching intervention, trigger the teaching event; It should be noted that verifying the static state based on posture characteristics ensures that students are in a learning state, accurately identifies the triggering causes of teaching events, and guarantees the accuracy of teaching event triggering.

[0028] Please see Figure 2 Step S3: Obtain the open-source multimodal base model, fine-tune the open-source multimodal base model to customize the exclusive model; when the teaching event is triggered, extract the corresponding video stream data, transmit the video stream data to the exclusive model, and generate a structured cognitive representation. Step S31: Fine-tune the open-source multimodal base model. Through supervised fine-tuning, interactive control is performed on the base model. Fixed interaction steps are set. Through preference fine-tuning, the output results of the model are standardized. The fine-tuned model is saved to obtain the custom model. It should be noted that Supervised Fine-Tuning (SFT) is a commonly used technique in machine learning, especially on the basis of pre-trained models (such as BERT, GPT, ResNet, etc.), to further optimize the model performance with a small amount of labeled data, so that it can be better adapted to specific tasks. Preference Fine-Tuning (PFT) is an optimization technique for Large Language Models (LLMs). Its core goal is to enable the model to learn and distinguish "better" responses by comparing different responses output by the model, thereby making its performance on subjective tasks such as tone, style, and creativity more in line with human expectations. Step S311: Collect high-definition images of student assignments, draft paper, and textbook exercises in real learning scenarios to obtain image information. Transmit the image information to the open-source multimodal basic model for image recognition and output the recognition results. Through manual verification, compare the recognition results with the image information, extract the incorrectly recognized parts, manually label the incorrectly recognized parts, input data into the open-source multimodal basic model, and train the open-source multimodal basic model through multiple recognitions to enable it to accurately recognize images. Step S312: Based on the image recognition results, repeatedly call the open-source multimodal basic model, set prompt words, analyze the image recognition results, store the analysis results in JSON format, and use the stored JSON format as the student's structured cognitive representation. It should be noted that the basic model undergoes full or efficient parameter fine-tuning. Through learning and training, the model learns to recognize questions, understand handwriting, judge right and wrong, and analyze error types, and stably generates JSON output that conforms to preset patterns, ensuring the system's subsequent automated processing.

[0029] Please see Figure 3 Step S313: Based on the trained model, multi-faceted intervention feedback is generated for the problem input. A preference dataset is constructed based on the intervention feedback. The Direct Preference Optimization (DPO) algorithm is used to further optimize the model in combination with the preference dataset to obtain a custom model.

[0030] Step S32: When a teaching event is triggered, the dedicated model is invoked, and the video stream data at the time of the teaching event is input into the dedicated model for problem identification. The identified problems are generated using a preset pattern to obtain a structured cognitive representation.

[0031] Step S4: Analyze the structured cognitive representation, use the analysis results as the question input, repeatedly call the dedicated model to generate prompt text, and display the prompt text to the students; The system analyzes structured cognitive representations, generates prompts based on the analysis results, transforms the analysis results into question inputs using the prompts, and then uses a dedicated model to answer the questions. The system also analyzes the answer results, generates prompt text, and uses the prompt text to interact with students and guide their learning direction.

[0032] Step S5: Record the video stream data and structured cognitive representations corresponding to the teaching events to obtain a record file. Persist in storing the record file, summarize the data comprehensively based on the stored record file, generate the student's problem file, and provide learning suggestions to the student. Step S51: After the teaching event is completed, the video stream data and structured cognitive representation corresponding to the teaching event are statistically analyzed to obtain a record file; the record file is persistently stored in a PostgreSQL database, an analysis trigger threshold is set for the stored record file, the record file is analyzed, and the student's learning summary is obtained. Step S52: Based on the students' learning summaries, extract the problems that exist in the students, generate the students' problem files, perform frequency statistics on the learning summaries based on the students' problem files, analyze the impact weight of different problems on students, and provide learning suggestions to students based on the impact weight of the problems. Step S521: Perform text analysis on the students' learning summaries, decompose the students' learning summaries to obtain multiple keywords, automatically filter the keywords through a large model, extract keywords related to the learning content, count the number of keywords to obtain the number of keywords gs; record the keywords as gjc(1) to gjc(gs) based on the number of keywords, count the keywords, and generate the students' question files; Step S522: Extract each keyword from the student's question file, use the keywords as search conditions to search the student's learning summary, record the number of search results, obtain the keyword frequency, and denote the keyword frequency as gpc(g); calculate the weight based on the keyword frequency to obtain the keyword influence weight gqz(g). ; The influence weights gqz(g) of the keywords are sorted in descending order to obtain a weight ranking list. Learning suggestions are given to students based on the weight ranking list, and students study the learning content associated with the keywords corresponding to the weight ranking list.

[0033] Step S6: Through access control, synchronize and transmit the student's problem file and learning suggestions to other authorized clients; Students' problem files and learning suggestions are stored in the cloud, and access to the cloud is controlled by permissions. Data is transmitted to other clients through authorization. Other clients can assist students' learning based on the students' problem files and learning suggestions.

[0034] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive learning intervention method based on multimodal analysis, characterized in that, include: Step S1: Monitor students' learning status in real time to obtain video stream data, process the video stream data, and construct student state vectors; Step S2: Update the student state vector to obtain the time series of the student state vector. Extract feature sequence segments through self-labeling. Set feature thresholds based on the feature sequence segments. Analyze the time series based on the feature thresholds to identify teaching events. Step S3: Obtain the open-source multimodal base model, fine-tune the open-source multimodal base model to customize the exclusive model; when the teaching event is triggered, extract the corresponding video stream data, transmit the video stream data to the exclusive model, and generate structured cognitive representations; Step S4: Analyze the structured cognitive representation, use the analysis results as the question input, repeatedly call the dedicated model to generate prompt text, and display the prompt text to the students; Step S5: Record the video stream data and structured cognitive representations corresponding to the teaching events to obtain a recording file. Persist in storing the recording file, summarize the data based on the stored recording file, generate student problem files, and provide learning suggestions to students. Step S6: Through access control, synchronize and transmit the student's problem file and learning suggestions to other authorized clients.

2. The adaptive learning intervention method based on multimodal analysis according to claim 1, characterized in that, The specific steps of step S1 are as follows: Step S11: Decompose the video stream data, extract the image frames from the video stream data, count the number of image frames, and denote the number of image frames as ts. Count the number of pixels in each image frame and denote the number of pixels as is×js. Based on the number of pixels, obtain the image pixels for each pixel and denote the image pixels as txs. t (i, j), txs t (i, j) represents the image pixel at the i-th row and j-th column of the t-th image frame; Image pixels txs from different image frames t (i, j), calculate the pixel changes of the image frame to obtain the student's motion activity degree yhd(t); ； Among them: txs t＋1 (i, j) represents the image pixel at the i-th row and j-th column of the (t+1)-th image frame; Step S12: Perform pose estimation on the image frame and identify the student's body nodes; record the position of the body nodes in the image frame, establish a base point based on the position of the body nodes in the image frame, and calculate the student's pose quantization value through the positional relationship between the base point and other body nodes to obtain the pose features. Step S13: Statistically analyze the degree of motion and posture characteristics, construct a two-dimensional student state vector, and denote the student state vector as V.

3. The adaptive learning intervention method based on multimodal analysis according to claim 2, characterized in that, The specific steps of step S12 are as follows: Step S121: Obtain the number of student body nodes, denoted as ds; collect the position of each body node to obtain the node position jsw. t (d); jsw t (d) = (i) t (d), j t (d)); Based on the node positions in different image frames, the numerical changes in node positions are calculated to obtain the change in node positions bhl(d). ； Where: i t (d), j t (d) represents the parameter indicating the node position in frame t; The changes at different node positions are statistically analyzed to obtain changes from bhl(1) to bhl(ds); the smallest change among the changes from bhl(1) to bhl(ds) is extracted, and the node corresponding to it is taken as the base point. The position of the base point is denoted as (jdi). t jdj t ); Step S122: Calculate the distance between the base point and the body node on each video frame, integrate all distances to obtain the student's pose quantization value, and record the student's pose quantization value as the pose feature ztz(t).

4. The adaptive learning intervention method based on multimodal analysis according to claim 1, characterized in that, The specific steps of step S2 are as follows: Step S21: Update and statistically analyze the student state vector in real time according to the time series to obtain the time series of the student state vector. Label the time series to obtain the feature sequence segment. Extract the student state vector from the feature sequence segment, analyze the value of the student state vector, and set the feature threshold. Step S22: Obtain the segment length of the feature sequence fragment, set the trigger length according to the segment length of the feature sequence fragment, identify the student status according to the feature threshold, and identify the teaching event based on the trigger length and the student status.

5. The adaptive learning intervention method based on multimodal analysis according to claim 4, characterized in that, The specific steps of step S21 also include: Step S211: Obtain the student state vectors under different video frames according to the temporal sequence, and denote them as V(t), where V(t) represents the student state vector under the t-th video frame; count the student state vectors to form a time series of student state vectors, and label the time series to obtain static sequence segments and non-static sequence segments. Step S212: Count the number of video frames in the still sequence segment, denoted as jzs. Obtain the motion activity degree of the student state vector under each video frame in the still sequence segment, denoted as jyhd(jz). Calculate the mean of the motion activity degree jyhd(jz) based on the number of video frames jzs in the still sequence segment to obtain the mean static activity value jjz. The static fluctuation value jbd is calculated by combining the mean static activity with the degree of motion activity of the student state vector in the static sequence segment. The number of video frames in the non-static sequence segment is counted and denoted as fjs. The motion activity degree of the student state vector under each video frame in the non-static sequence segment is obtained and denoted as fyhd(fj). Based on the number of video frames fjs in the non-static sequence segment, the mean of motion activity degree fyhd(fj) is calculated to obtain the mean of non-static activity fjz. The non-stationary fluctuation value fbd is calculated by combining the mean of non-stationary activity with the degree of motion activity of the student state vector in the non-stationary sequence segment. Step S213: Calculate the characteristic threshold tyz based on the static activity mean jjz, static fluctuation value jbd, non-static activity mean fjz, and non-static fluctuation value fbd; 。 6. The adaptive learning intervention method based on multimodal analysis according to claim 4, characterized in that, The specific steps of step S22 are as follows: Step S221: Obtain the number of segments of the still sequence fragment, denoted as ps, count the segment length of each still sequence fragment, denoted as still segment length jpc(1) to jpc(ps), extract the minimum value among the still segment lengths jpc(1) to jpc(ps) to obtain the trigger length; Step S222: Obtain the time series of the student's state vector, use the feature threshold as a judgment condition to traverse and judge the time series of the student's state vector, extract the student's motion activity degree in the time series, if the motion activity degree is less than the feature threshold, judge that the student is in a static state, extract the posture features in the static state, verify the static state based on the posture features, count the continuous time series of the student in the static state, compare the continuous time series with the trigger length, if the continuous time series is greater than the trigger length, it indicates that the student needs teaching intervention, trigger the teaching event.

7. The adaptive learning intervention method based on multimodal analysis according to claim 1, characterized in that, The specific steps of step S3 are as follows: Step S31: Fine-tune the open-source multimodal base model. Through supervised fine-tuning, interactive control is performed on the base model. Fixed interaction steps are set. Through preference fine-tuning, the output results of the model are standardized. The fine-tuned model is saved to obtain the custom model. Step S32: When a teaching event is triggered, the dedicated model is invoked, and the video stream data at the time of the teaching event is input into the dedicated model for problem identification. The identified problems are generated using a preset pattern to obtain a structured cognitive representation.

8. The adaptive learning intervention method based on multimodal analysis according to claim 7, characterized in that, The specific steps of step S31 are as follows: Step S311: High-resolution images of student assignments, draft paper, and textbook exercises are acquired to obtain image information. The image information is then transmitted to the open-source multimodal basic model for image recognition, and the recognition results are output. The recognition results are compared with the image information through manual verification, and the incorrectly recognized parts are extracted and manually labeled. Data is entered into the open-source multimodal basic model, and the open-source multimodal basic model is trained through multiple recognitions to enable it to accurately recognize images. Step S312: Based on the image recognition results, repeatedly call the open-source multimodal basic model, set prompt words, analyze the image recognition results, store the analysis results in JSON format, and use the stored JSON format as the student's structured cognitive representation. Step S313: Based on the trained model, multi-faceted intervention feedback is generated for the problem input. A preference dataset is generated based on the intervention feedback. The direct preference optimization algorithm is used, and the model is further optimized in combination with the preference dataset to obtain a custom model.

9. The adaptive learning intervention method based on multimodal analysis according to claim 1, characterized in that, The specific steps of step S5 are as follows: Step S51: After the teaching event is completed, the video stream data and structured cognitive representation corresponding to the teaching event are statistically analyzed to obtain a record file; the record file is persistently stored, an analysis trigger threshold is set for the stored record file, and the record file is analyzed to obtain the student's learning summary; Step S52: Based on the students' learning summaries, extract the problems that exist in the students, generate the students' problem files, perform frequency statistics on the learning summaries based on the students' problem files, analyze the impact weight of different problems on students, and provide learning suggestions to students based on the impact weight of the problems.

10. The adaptive learning intervention method based on multimodal analysis according to claim 9, characterized in that, The specific steps of step S52 are as follows: Step S521: Perform text analysis on the students' learning summary, decompose the students' learning summary to obtain multiple keywords, automatically filter the keywords, extract keywords related to the learning content, count the number of keywords to obtain the number of keywords gs; record the keywords as gjc(1) to gjc(gs) based on the number of keywords, count the keywords to generate the students' question files; Step S522: Extract each keyword from the student's question file, use the keywords as search conditions to search the student's learning summary, record the number of search results, obtain the keyword frequency, and denote the keyword frequency as gpc(g); calculate the weight based on the keyword frequency to obtain the keyword influence weight gqz(g). The influence weights gqz(g) of the keywords are sorted in descending order to obtain a weight ranking list. Learning suggestions are given to students based on the weight ranking list, and students study the learning content associated with the keywords corresponding to the weight ranking list.

Images