Student learning preference prediction method, device and equipment based on multi-modal large model

Abstract

The invention relates to the technical field of learning accompanying, solves the problem that the learning hobbies of students cannot be effectively predicted in real time in the prior art, and provides a multi-modal large model-based student learning hobbies prediction method, device and equipment. The method comprises the following steps: acquiring real-time video data in a learning accompanying scene of a student; according to the real-time video data, in combination with a preset time interval, extracting a key frame image; performing feature extraction on the key frame image by using a multi-modal large model, and outputting first feature information and second feature information related to student learning preference prediction; constructing a student learning preference prediction model according to the first feature information and the second feature information; and processing new video data through the student learning preference prediction model, and outputting a student learning preference prediction result. According to the invention, the learning hobbies of the students can be effectively predicted in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of learning accompaniment, and in particular to a method, device and equipment for predicting student learning preferences based on a multimodal large model. Background Art

[0002] Nowadays, understanding students' learning preferences is crucial to education. Understanding students' learning preferences can help educators better design courses, teaching methods and resources to meet students' needs and improve their learning outcomes. Since each student has his or her own unique learning preferences and styles, understanding students' learning preferences can help educators better personalize teaching, because different students may have very different ways of accepting and liking learning content. Personalized learning can better stimulate students' learning interest and motivation and improve their learning outcomes. At the same time, understanding students' learning preferences can help educators design more attractive and interactive teaching activities. When students participate in learning content related to their interests, they are more likely to actively participate and are more likely to stay focused and engaged. Secondly, when students feel that their learning preferences are respected and valued, they are more likely to actively participate in learning and are more likely to show their ability to learn independently during the learning process. Understanding students' learning preferences can stimulate their autonomy and initiative in learning, thereby cultivating their ability to learn independently. Finally, by understanding students' learning preferences, educators can understand each student more deeply and establish a closer connection with them. This relationship is based on mutual respect and understanding, which helps to establish a positive learning atmosphere and promote communication and cooperation between educators and students. In summary, understanding students' learning preferences is crucial for personalized teaching, improving learning outcomes, promoting autonomous learning, and establishing good teacher-student relationships. Existing technologies rely on students' online behavior data or traditional questionnaires to obtain learning preference information. However, this method is limited by students' voluntary participation and honest answers, and cannot fully capture students' true preferences.

[0003] The existing Chinese patent CN116304973A discloses a classroom teaching emotion recognition method and system based on multimodal fusion, the method comprising: collecting text, voice and video data, and performing feature extraction preprocessing on the text, voice and video data respectively; establishing a multimodal emotion recognition model, using the preprocessed text, voice and video data to train the multimodal emotion recognition model to obtain a trained multimodal emotion recognition model; collecting classroom students' text information, voice signals and video signals and performing feature extraction preprocessing respectively; inputting the preprocessed text information, voice signals and video signals into the trained multimodal emotion recognition model for emotion analysis, and obtaining the emotion state classification result; The students' classroom emotional state is calculated based on the emotional state classification results. The collection of text, voice and video data is to construct a training set for subsequent model training. CH-SIMS is used to collect the training set. 60 original videos are collected. These videos come from different movies, TV series and variety shows that have reference value for character emotion recognition. These videos are cropped to obtain video clips. These video clips need to include the character's spontaneous expressions, head postures, different degrees of occlusion, different lighting effects, different voice tones and different lines. Each video clip is manually marked, and the corresponding emotional state classification results of the video clip are marked as -1 (negative), 0 (neutral) and 1 (positive). After the marking is completed, the text, voice and video data in the corresponding video clip are collected. However, if the above scheme is applied to predicting students' learning preferences, since the videos used for emotion recognition model training data are from movies, TV series and variety shows, the contexts, expression styles and plot promotions of these videos are often very different from the actual classroom environment, and are not enough to train a model that can accurately understand students' emotional states in a natural classroom environment; at the same time, the recognition of emotional states is not directly equivalent to the determination of learning preferences. A student's emotional expression in class (such as happiness, sadness, neutrality) may be affected by many factors, such as personal life events, interactions with classmates, etc., which are not necessarily directly related to the preferences for learning content. Therefore, judging students' preferences for specific subjects or course content only by emotional states may not be accurate enough.

[0004] Therefore, how to predict students' learning preferences in real time and effectively is an urgent problem to be solved. Summary of the invention

[0005] In view of this, the present invention provides a method, device and equipment for predicting students' learning preferences based on a multimodal large model, so as to solve the problem in the prior art that students' learning preferences cannot be predicted in real time and effectively.

[0006] The technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a method for predicting student learning preferences based on a multimodal large model, the method comprising:

[0008] Obtain real-time video data in the student learning and accompanying scenario;

[0009] Extracting key frame images according to the real-time video data and in combination with a preset time interval;

[0010] Using a multimodal large model, extracting features from the key frame image, and outputting first feature information and second feature information related to the prediction of the student's learning preference;

[0011] Constructing a student learning preference prediction model based on the first feature information and the second feature information;

[0012] The new video data is processed by the student learning preference prediction model to output the student learning preference prediction result.

[0013] Preferably, the use of a multimodal large model to extract features from the key frame image and output first feature information and second feature information related to the prediction of the student's learning preference includes:

[0014] Inputting the key frame image into a pre-trained multimodal large model, and outputting first feature information related to the teaching material;

[0015] The key frame image is input into a pre-trained multimodal large model, and second feature information related to the student's reaction is output.

[0016] Preferably, the step of inputting the key frame image into a pre-trained multimodal large model and outputting first feature information related to the teaching material comprises:

[0017] Inputting the key frame image into a pre-trained multimodal large model, and outputting a first feature vector and first text information related to the teaching material;

[0018] The first feature vector is annotated using the first text information, and the annotated first feature vector is output as the first feature information.

[0019] Preferably, the step of inputting the key frame image into a pre-trained multimodal large model and outputting second feature information related to the student's reaction comprises:

[0020] Inputting the key frame image into a pre-trained multimodal large model, and outputting a second feature vector and second text information related to the student's response;

[0021] The second feature vector is labeled using the second text information, and the labeled second feature vector is output as the second feature information.

[0022] Preferably, the multimodal large model at least includes: a BL IP model, a CLIP model and a Vi LBERT model.

[0023] Preferably, constructing a student learning preference prediction model based on the first feature information and the second feature information includes:

[0024] Determine comprehensive feature information based on the first feature information and the second feature information;

[0025] Obtaining predefined learning preference feature template information;

[0026] The preset neural network model is trained using the comprehensive feature information and the learning preference feature template information, and the trained model is used as the student learning preference prediction model.

[0027] Preferably, the obtaining of predefined learning preference feature template information includes:

[0028] Obtaining first label information corresponding to a preset student learning material category and second label information corresponding to a preset student response category;

[0029] Determining learning preference label information according to the first label information and the second label information;

[0030] The learning preference label information is assigned to corresponding preset feature information to determine the learning preference feature template information.

[0031] In a second aspect, the present invention provides a device for predicting student learning preferences based on a multimodal large model, the device comprising:

[0032] Video data acquisition module, used to obtain real-time video data in the student learning and accompanying scenario;

[0033] A key frame image extraction module, used to extract key frame images according to the real-time video data in combination with a preset time interval;

[0034] A feature extraction module, used to extract features from the key frame image using a multimodal large model, and output first feature information and second feature information related to the prediction of the student's learning preference;

[0035] A model building module, used for building a student learning preference prediction model based on the first feature information and the second feature information;

[0036] The learning preference prediction module is used to process new video data through the student learning preference prediction model and output the student learning preference prediction result.

[0037] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method of the first aspect in the above-mentioned embodiment.

[0038] In a fourth aspect, an embodiment of the present invention further provides a storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method of the first aspect in the above-mentioned embodiment is implemented.

[0039] In summary, the beneficial effects of the present invention are as follows:

[0040] The present invention provides a method, device and equipment for predicting student learning preferences based on a multimodal large model. The method includes: acquiring real-time video data in a student learning accompaniment scenario; extracting key frame images based on the real-time video data and a preset time interval; using the multimodal large model to perform feature extraction on the key frame images, and output first feature information and second feature information related to the prediction of student learning preferences; constructing a student learning preference prediction model based on the first feature information and the second feature information; processing new video data through the student learning preference prediction model, and outputting a student learning preference prediction result. The present invention uses real-time video data to capture the latest changes in students' learning behaviors, and can reflect students' behavior patterns and emotional states in real time. This real-time nature enhances the timeliness and relevance of learning preference predictions, making educational interventions more timely and personalized. In view of the fact that learning is a large inertial behavior, key frames are extracted from real-time video data at preset time intervals. The extraction of key frames can reduce the complexity of data processing while helping to focus on those moments that show students' reactions, thereby better understanding students' interest in and reactions to specific teaching content. Through a multimodal large model, feature extraction is performed on key frame images. The large model can extract rich and in-depth features from each independent key frame image, including the visual content of the image and the associated text description, allowing for in-depth analysis of the details of each learning moment, capturing instantaneous performance or subtle changes that are ignored in the overall sequence analysis. The use of single-frame feature vectors provides processing The flexibility of the data. In some cases, you can choose to focus on analyzing specific frames or moments based on your needs, rather than being forced to process the entire video sequence. This can effectively save computing resources and focus on key moments. At the same time, processing single-frame data allows the large model to give output faster, because each frame can be processed independently without waiting for the entire sequence to be processed. This is very useful for application scenarios that require fast feedback (such as real-time teaching interaction and instant adjustment of teaching strategies). In a real-time learning environment, the ability to instantly analyze each key frame image and provide feedback helps teachers or teaching systems adjust teaching content or methods based on students' immediate reactions. Through the first and second feature information extracted by the large model, a student learning preference prediction model is constructed, which can effectively predict students' learning preferences in real time. Accurate learning preference prediction can help educators better understand students' learning motivations and obstacles, thereby designing more effective teaching interventions. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings required for use in the embodiment of the present invention will be briefly introduced below. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work, and these are all within the protection scope of the present invention.

[0042] Figure 1 It is a schematic diagram of the overall working process of the method for predicting student learning preferences based on a multimodal large model in Example 1 of the present invention;

[0043] Figure 2 This is a schematic diagram of a process for extracting features from a key frame image in Embodiment 1 of the present invention;

[0044] Figure 3 This is a schematic diagram of a process for determining first characteristic information related to teaching materials in Embodiment 1 of the present invention;

[0045] Figure 4 This is a schematic diagram of a process for determining second characteristic information related to a student's reaction in Embodiment 1 of the present invention;

[0046] Figure 5 This is a schematic diagram of the process of constructing a student learning preference prediction model in Example 1 of the present invention;

[0047] Figure 6 This is a schematic diagram of the process of obtaining predefined learning preference feature template information in Example 1 of the present invention;

[0048] Figure 7 It is a structural block diagram of a device for predicting student learning preferences based on a multimodal large model in Example 2 of the present invention;

[0049] Figure 8 It is a schematic diagram of the structure of the electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the orientation or position relationship indicated by the terms "center", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the phrase "comprising..." do not exclude the existence of other identical elements in the process, method, article or device comprising the elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments can be combined with each other, all within the protection scope of the present invention.

[0051] Example 1

[0052] See also Figure 1 Embodiment 1 of the present invention discloses a method for predicting student learning preferences based on a multimodal large model, the method comprising:

[0053] Obtain real-time video data in the student learning and accompanying scenario;

[0054] Specifically, real-time video data of students learning captured by a camera placed flat on the table in a learning accompaniment scenario is obtained. The camera placed flat on the table is mainly used to capture students' facial expressions, gestures and other behavioral patterns that affect the learning experience during the learning process. In this way, students' learning behaviors and emotional reactions are recorded in real time, so as to analyze students' participation, concentration and learning barriers. The real-time video data provides valuable perspectives and basis for subsequent educational interventions, helping educators better understand and support the learning needs of each student.

[0055] Extracting key frame images according to the real-time video data and in combination with a preset time interval;

[0056] Specifically, key frame images in the video are extracted for analysis at preset time intervals. For example, a key frame image is extracted every 6 seconds. This is based on the fact that learning behavior has a large inertia, that is, the changes in students' behavior and emotional state will not fluctuate drastically in a very short period of time. This method can reduce the burden of data processing while retaining sufficient information to observe and analyze students' key behaviors and expression changes in the learning process, thereby effectively evaluating and supporting their learning status.

[0057] Using a multimodal large model, extracting features from the key frame image, and outputting first feature information and second feature information related to the prediction of the student's learning preference;

[0058] Specifically, a multimodal large model is used to process the video key frame images in the learning accompaniment scenario, and the first feature information and the second feature information related to the prediction of students' learning preferences are extracted therefrom, wherein the multimodal large model includes at least a BLIP model, a CLIP model and a Vi LBERT model, and the above models can process image and text data, making them particularly suitable for understanding and analyzing the relationship between image content and text descriptions (such as text labels or explanatory annotations in images). Two important feature information are extracted from the video key frames through the analysis of the large model: the first feature information includes information related to the teaching content, such as teaching materials of different subjects or themes, such as mathematics, Chinese, and English; the second feature information includes students' different reactions to these teaching materials, such as interest, concentration, and participation. Extracting features from key frames through a large model helps to accurately predict students' learning preferences, such as which teaching content or methods can arouse students' interest and positive response. Such information is very valuable for customizing personalized educational experience and improving teaching effectiveness.

[0059] In one embodiment, see Figure 2 , the method of using a multimodal large model to extract features from key frame images and outputting first feature information and second feature information related to the prediction of students' learning preferences includes:

[0060] Inputting the key frame image into a pre-trained multimodal large model, and outputting first feature information related to the teaching material;

[0061] Specifically, during the learning process, the first feature information related to the teaching material is extracted by inputting the selected key frame image into the pre-trained multimodal large model. The pre-trained multimodal large model, such as BLIP, CLIP or ViLBERT, can understand and analyze the association between images and texts, so it can effectively identify and parse the visual and contextual information related to the teaching content from the image. The extracted first feature information includes at least the visual representation of the teaching material, such as the page image of the textbook.

[0062] In one embodiment, see Figure 3 , the step of inputting the key frame image into a pre-trained multimodal large model and outputting first feature information related to the teaching material comprises:

[0063] Inputting the key frame image into a pre-trained multimodal large model, and outputting a first feature vector and first text information related to the teaching material;

[0064] Specifically, a pre-trained multimodal large model, such as the BLIP model, is used to extract visual and text features closely related to the teaching materials from the video key frame images. First, each key frame image I frame is input into the BLIP model, which is specially designed to process and understand images and their associated text information, so that it can extract deep content and semantic information from a single image. In this case, the model extracts features from the key frame image, and the extraction formula is: matreial =Bl ip(I frame ), the first eigenvector F of the output materialRich in details about teaching materials, such as text visible in the image and the visual style of teaching tools. The BLIP large model is able to extract rich and in-depth features from each independent keyframe image, including the visual content of the image and the associated text description, allowing in-depth analysis of the details of each learning moment and capturing instantaneous performance or subtle changes that are ignored in the overall sequence analysis. The use of single-frame feature vectors provides flexibility in processing data. In some cases, you can choose to focus on analyzing specific frames or moments as needed, rather than being forced to process the entire video sequence, which can effectively save computing resources and focus on key moments; at the same time, processing single-frame data allows the large model to give output faster, because each frame can be processed independently without waiting for the entire sequence to be processed, which is very useful for application scenarios that require fast feedback (such as real-time teaching interaction and instant adjustment of teaching strategies); in a real-time learning environment, the ability to instantly analyze each keyframe image and provide feedback helps teachers or teaching systems adjust teaching content or methods based on students' immediate reactions; on the other hand, the BLIP large model has The ability to process multiple data types, including images, text, voice, etc., enables it to understand and analyze the complex information involved in the learning process from different perspectives. These models can process data from different sources simultaneously and combine them to provide more comprehensive and in-depth learning references and understanding. In the scenario of learning preference prediction, students' learning behaviors and preferences are often affected by multiple factors, such as subject content, teaching methods, student personality, etc. Through the BLIP large model, data can be obtained from multiple perspectives such as students' facial expressions, language descriptions, and learning environment, so as to more accurately understand students' learning status and preferences. For example, the model can analyze students' facial expression changes, text feedback, and voice intonation during the learning process to infer their preferences and understanding of specific teaching content. This comprehensive data analysis capability makes the multimodal large model highly applicable and accurate in predicting learning preferences, providing more effective decision-making support and personalized teaching suggestions for educational practice.

[0065] The first feature vector is annotated using the first text information, and the annotated first feature vector is output as the first feature information.

[0066] Specifically, the first feature vector is annotated with text. This step is based on the text output provided by the BLIP model at the same time. For example, if the key frame image shows the equipment and process of a chemical experiment, the model will output the first text information related to it, such as "test tubes and chemical reactions in a chemical laboratory". The first text information is used to annotate the corresponding first feature vector to ensure that each feature vector has a clear and accurate semantic label, which is helpful for further analysis and application. Finally, the annotated first feature vector is output as the first feature information. The output first feature information is not only rich and detailed, but also has undergone precise semantic enhancement, so that subsequent educational applications, such as personalized teaching content recommendation, teaching effect evaluation, or the establishment of a teaching material database, can be more accurately optimized and adjusted based on these data.

[0067] The key frame image is input into a pre-trained multimodal large model, and second feature information related to the student's reaction is output.

[0068] Specifically, the key frame images in the student learning scene are input into the pre-trained multimodal large model to obtain the second feature information related to the student's reaction. The multimodal large model is used to identify the student's behavior and emotional response from the key frame image, and convert it into an understanding and analysis of the student's reaction. The second feature information includes at least feature vectors corresponding to the student's facial expression, body language, posture, etc. and text descriptions, such as the student's feedback, questions or comments on the teaching content. By fully understanding the students' emotional state, understanding and participation in the learning process, educators can be provided with deeper insights and personalized teaching support.

[0069] In one embodiment, see Figure 4 , the step of inputting the key frame image into a pre-trained multimodal large model and outputting second feature information related to the student's reaction comprises:

[0070] Inputting the key frame image into a pre-trained multimodal large model, and outputting a second feature vector and second text information related to the student's response;

[0071] Specifically, first, the image of the student area in the video frame is input into a pre-trained multimodal large model, such as the BLIP model, to identify and classify the student's behavioral response. The BLIP model extracts rich semantic information from the key frame image. The extraction formula is: reaction =BLIP(I student ), where I student An image representing the student area in a video frame, F reaction is the second eigenvector of student reactions; the second eigenvector describes the student’s behavioral and emotional reactions at a specific moment, such as concentration, excitement, and confusion.

[0072] The second feature vector is labeled using the second text information, and the labeled second feature vector is output as the second feature information.

[0073] Specifically, the second feature vector is annotated based on the second text information provided by the BLIP model. For example, if the student's response is identified as focused and positive, the BLIP model will output the relevant second text information, such as "students participate in teaching activities attentively and show a positive learning attitude". The second text information will be used to annotate the corresponding second feature vector to ensure that each feature vector has a clear and accurate semantic description. Finally, the annotated second feature vector is output as the second feature information. The second feature information contains the emotions and reactions of students during the learning process. After in-depth semantic analysis and annotation, this data is easier to understand and use. This detailed and meticulous student response information helps educators better understand students' learning status and needs, thereby providing more accurate and personalized teaching support.

[0074] Constructing a student learning preference prediction model based on the first feature information and the second feature information;

[0075] Specifically, a student learning preference prediction model is constructed based on the extracted first feature information and the second feature information, and the students' preference for different learning content or teaching methods is predicted by comprehensively considering their behavioral responses and emotional states. By combining the first feature information and the second feature information, the student learning preference prediction model can establish a comprehensive understanding of the students' learning preferences. The method for constructing a student learning preference prediction model includes using a machine learning algorithm or a deep learning method, using the extracted first feature information and the second feature information as input to train the model. The trained student learning preference prediction model will be able to accurately predict the students' preference for different teaching content or teaching methods, providing important guidance for educators, helping them to better design personalized teaching plans, and improving students' learning effects and satisfaction.

[0076] In one embodiment, see Figure 5 The step of constructing a student learning preference prediction model based on the first feature information and the second feature information includes:

[0077] Determine comprehensive feature information based on the first feature information and the second feature information;

[0078] Specifically, the two annotated feature vectors are spliced ​​using the extracted first feature information and the second feature information, so as to combine the teaching content and the student response in the student learning process to obtain a more comprehensive and integrated feature representation of the student's learning preferences, and obtain a annotated comprehensive feature vector, namely:

[0079] Feombined=Concat(Fmaterial,Freaction)

[0080] Among them, Fcombine d Represents a comprehensive feature vector, and outputs the labeled comprehensive feature vector as comprehensive feature information, which can more comprehensively reflect the students' learning status and preference tendencies, and provide richer and more accurate input for the training of the student learning preference prediction model, thereby helping the student learning preference prediction model to more accurately predict the students' preference for different teaching contents or teaching methods, and provide more effective guidance and support for educators.

[0081] Obtaining predefined learning preference feature template information;

[0082] Specifically, obtaining predefined learning preference feature template information means first determining and collecting student learning preference feature template information before building a student learning preference prediction model. The learning preference feature template information is determined through previous research or practical experience, and can also be customized according to specific learning scenarios and teaching objectives. These feature templates can cover various factors that may affect students' learning preferences, such as subject preferences, teaching method preferences, learning environment preferences, etc.

[0083] In one embodiment, see Figure 6 , the obtaining of predefined learning preference feature template information includes:

[0084] Obtaining first label information corresponding to a preset student learning material category and second label information corresponding to a preset student response category;

[0085] Specifically, the first label information corresponding to the preset student learning material category and the second label information corresponding to the student response category are obtained. In this step, the student learning material category and the student response category need to be predefined, and the first label information and the second label information are assigned to each category. When defining the classification criteria for learning material types and student behaviors, the diversity of teaching materials and the diversity of student behaviors should be considered first. For the classification of learning materials, they should be classified according to subjects, themes, difficulty, etc., such as mathematics, Chinese, English, etc. At the same time, the form of teaching resources can also be considered, such as textbooks, exercise books, teaching videos, etc.; and for the classification of student behaviors, they can be classified according to students' responses to teaching materials, such as interest level, concentration, participation, etc. The design of these classification criteria should be clear and operational to facilitate subsequent data collection and analysis.

[0086] Determining learning preference label information according to the first label information and the second label information;

[0087] Specifically, based on the first label information and the second label information, the learning preference label information is determined, and the learning preference label information can comprehensively reflect the student's preference for different learning materials and the corresponding student response category. For example, if a student shows a high interest and concentration in mathematics courses, then the corresponding learning preference label information is "highly interested in mathematics and highly concentrated".

[0088] The learning preference label information is assigned to corresponding preset feature information to determine the learning preference feature template information.

[0089] Specifically, the determined learning preference label information is assigned to the corresponding preset feature information to determine the learning preference feature template information, which should include the learning preference label information and the corresponding learning material feature vector, student response feature vector, etc. By associating the learning preference label information with the sample comprehensive feature vector composed of the learning material feature vector and the student response feature vector in the sample video data, a mapping relationship between the student learning preference and the specific learning material and the student response is established, providing an important data basis for the subsequent construction of the learning preference prediction model.

[0090] The preset neural network model is trained using the comprehensive feature information and the learning preference feature template information, and the trained model is used as the student learning preference prediction model.

[0091] Specifically, the preset neural network model is trained using comprehensive feature information and learning preference feature template information, and the trained model is used as a student learning preference prediction model. In this process, the LSTM (Long Short-Term Memory) model is selected as the neural network model to learn the inherent laws of students' learning habits. First, the comprehensive feature information is used, and the comprehensive feature information includes the first feature vector corresponding to the labeled teaching materials and the second feature vector corresponding to the labeled student response. The comprehensive feature information is used as the training input of the LSTM model. Then, during the training process, the model parameters are optimized by minimizing the prediction error between the comprehensive feature information and the learning preference feature template information. The LSTM model will gradually learn and understand the patterns and laws in the student learning process, thereby improving the ability to predict students' learning preferences. Through repeated iterative training, the LSTM model will continuously adjust parameters to improve its accuracy and generalization ability in predicting students' learning preferences. The training process is as follows:

[0092] M trained =Train(F combined , Y habit )

[0093] Among them, Train represents the training function, Yhabit Represents the learning preference feature template information, M trained Represents a student learning preference prediction model; ultimately, the trained LSTM model will be used as a student learning preference prediction model to predict students' preference for different learning content or teaching methods. The student learning preference prediction model can automatically identify students' learning habits and preferences based on their learning behaviors and reactions, providing important guidance and support for personalized teaching.

[0094] The new video data is processed by the student learning preference prediction model to output the student learning preference prediction result.

[0095] Specifically, the newly collected student learning video data is processed with the same video content recognition and student response recognition to obtain new comprehensive feature information, that is, a new labeled comprehensive feature vector, and then the trained model M is used to identify the new comprehensive feature vector. trained Predict the learning preference for the new comprehensive feature information, namely:

[0096] H predict =Pred ict(M trained , F new )

[0097] Among them, Pred ict represents the prediction function, F new represents the new comprehensive feature information, H predict is the predicted learning preference.

[0098] Example 2

[0099] See also Figure 7 Embodiment 2 of the present invention further provides a device for predicting student learning preferences based on a multimodal large model, the device comprising:

[0100] Video data acquisition module, used to obtain real-time video data in the student learning and accompanying scenario;

[0101] A key frame image extraction module, used to extract key frame images according to the real-time video data in combination with a preset time interval;

[0102] A feature extraction module, used to extract features from the key frame image using a multimodal large model, and output first feature information and second feature information related to the prediction of the student's learning preference;

[0103] A model building module, used for building a student learning preference prediction model based on the first feature information and the second feature information;

[0104] The learning preference prediction module is used to process new video data through the student learning preference prediction model and output the student learning preference prediction result.

[0105] Specifically, a device for predicting student learning preferences based on a multimodal large model provided in an embodiment of the present invention is used, and the device includes: a video data acquisition module, used to acquire real-time video data in a student learning accompaniment scenario; a key frame image extraction module, used to extract key frame images according to the real-time video data in combination with a preset time interval; a feature extraction module, used to use the multimodal large model to perform feature extraction on the key frame images, and output first feature information and second feature information related to the prediction of student learning preferences; a model construction module, used to construct a student learning preference prediction model according to the first feature information and the second feature information; a learning preference prediction module, used to process new video data through the student learning preference prediction model, and output a student learning preference prediction result. This device uses real-time video data to capture the latest changes in students' learning behaviors, and can reflect students' behavioral patterns and emotional states in real time. This real-time nature enhances the timeliness and relevance of learning preference predictions, making educational interventions more timely and personalized. Taking into account the fact that learning is a large inertial behavior, key frames are extracted from real-time video data at preset time intervals. The extraction of key frames can reduce the complexity of data processing while helping to focus on those moments that show students' reactions, thereby better understanding students' interest in and reactions to specific teaching content. Through a multimodal large model, feature extraction is performed on key frame images. The large model can extract rich and in-depth features from each independent key frame image, including the visual content of the image and the associated text description, allowing for in-depth analysis of the details of each learning moment, capturing instantaneous performance or subtle changes that are ignored in the overall sequence analysis. The use of single-frame feature vectors provides processing The flexibility of the data. In some cases, you can choose to focus on analyzing specific frames or moments based on your needs, rather than being forced to process the entire video sequence. This can effectively save computing resources and focus on key moments. At the same time, processing single-frame data allows the large model to give output faster, because each frame can be processed independently without waiting for the entire sequence to be processed. This is very useful for application scenarios that require fast feedback (such as real-time teaching interaction and instant adjustment of teaching strategies). In a real-time learning environment, the ability to instantly analyze each key frame image and provide feedback helps teachers or teaching systems adjust teaching content or methods based on students' immediate reactions. Through the first and second feature information extracted by the large model, a student learning preference prediction model is constructed, which can effectively predict students' learning preferences in real time. Accurate learning preference prediction can help educators better understand students' learning motivations and obstacles, thereby designing more effective teaching interventions.

[0106] Example 3

[0107] In addition, combined Figure 1 The method for predicting students' learning preferences based on a multimodal large model of the embodiment 1 of the present invention described above can be implemented by an electronic device. Figure 8 A schematic diagram of the hardware structure of an electronic device provided in Embodiment 3 of the present invention is shown.

[0108] The electronic device may include a processor and a memory storing computer program instructions.

[0109] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.

[0110] The memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory may be inside or outside a data processing device. In a specific embodiment, the memory is a non-volatile solid-state memory. In a specific embodiment, the memory includes a read-only memory (ROM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0111] The processor implements any one of the student learning preference prediction methods based on a multimodal large model in the above-mentioned embodiments by reading and executing computer program instructions stored in the memory.

[0112] In one example, the electronic device may further include a communication interface and a bus. Figure 8 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.

[0113] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.

[0114] The bus includes hardware, software or both, and the parts of the device are coupled to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnection, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnection, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. In suitable cases, the bus may include one or more buses. Although the embodiments of the present invention describe and illustrate specific buses, the present invention considers any suitable bus or interconnection.

[0115] Example 4

[0116] In addition, in combination with the method for predicting student learning preferences based on a multimodal macro model in the above embodiment 1, embodiment 4 of the present invention can also provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the methods for predicting student learning preferences based on a multimodal macro model in the above embodiments is implemented.

[0117] In summary, the embodiments of the present invention provide a method, device and apparatus for predicting students' learning preferences based on a multimodal large model.

[0118] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0119] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0120] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0121] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A method for predicting students' learning preferences based on a multimodal large model, characterized in that: The method comprises: Obtain real-time video data in the student learning and accompanying scenario; Extracting key frame images according to the real-time video data and in combination with a preset time interval; Using a multimodal large model, extracting features from the key frame image, and outputting first feature information and second feature information related to the prediction of the student's learning preference; Constructing a student learning preference prediction model based on the first feature information and the second feature information; The new video data is processed by the student learning preference prediction model to output the student learning preference prediction result.

2. The method for predicting student learning preferences based on a multimodal large model according to claim 1 is characterized in that: The method of using the multimodal large model to extract features from the key frame image and outputting first feature information and second feature information related to the prediction of the student's learning preference includes: Inputting the key frame image into a pre-trained multimodal large model, and outputting first feature information related to the teaching material; The key frame image is input into a pre-trained multimodal large model, and second feature information related to the student's reaction is output.

3. The method for predicting student learning preferences based on a multimodal large model according to claim 2 is characterized in that: The step of inputting the key frame image into a pre-trained multi-modal large model and outputting first feature information related to the teaching material comprises: Inputting the key frame image into a pre-trained multimodal large model, and outputting a first feature vector and first text information related to the teaching material; The first feature vector is annotated using the first text information, and the annotated first feature vector is output as the first feature information.

4. The method for predicting student learning preferences based on a multimodal large model according to claim 2 is characterized in that: The step of inputting the key frame image into a pre-trained multi-modal large model and outputting second feature information related to the student's reaction comprises: Inputting the key frame image into a pre-trained multimodal large model, and outputting a second feature vector and second text information related to the student's response; The second feature vector is labeled using the second text information, and the labeled second feature vector is output as the second feature information.

5. The method for predicting student learning preferences based on a multimodal large model according to claim 2 is characterized in that: The multimodal large model at least includes: a BLIP model, a CLIP model and a Vi LBERT model.

6. The method for predicting student learning preferences based on a multimodal large model according to any one of claims 1 to 5, characterized in that: The constructing of the student learning preference prediction model according to the first feature information and the second feature information includes: Determine comprehensive feature information based on the first feature information and the second feature information; Obtaining predefined learning preference feature template information; The preset neural network model is trained using the comprehensive feature information and the learning preference feature template information, and the trained model is used as the student learning preference prediction model.

7. The method for predicting student learning preferences based on a multimodal large model according to claim 6 is characterized in that: The obtaining of predefined learning preference feature template information includes: Obtaining first label information corresponding to a preset student learning material category and second label information corresponding to a preset student response category; Determining learning preference label information according to the first label information and the second label information; The learning preference label information is assigned to corresponding preset feature information to determine the learning preference feature template information.

8. A device for predicting student learning preferences based on a multimodal large model, characterized in that: The device comprises: Video data acquisition module, used to obtain real-time video data in the student learning and accompanying scenario; A key frame image extraction module, used to extract key frame images according to the real-time video data in combination with a preset time interval; A feature extraction module, used to extract features from the key frame image using a multimodal large model, and output first feature information and second feature information related to the prediction of the student's learning preference; A model building module, used for building a student learning preference prediction model based on the first feature information and the second feature information; The learning preference prediction module is used to process new video data through the student learning preference prediction model and output the student learning preference prediction result.

9. An electronic device, characterized in that: include: At least one processor, at least one memory and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, implement the method according to any one of claims 1 to 7.

10. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images