Dynamically corrected classroom student state multi-mode identification method and system

Abstract

The invention belongs to a dynamically corrected classroom student state multi-mode identification method and system, and the method comprises the steps: obtaining a video image in a classroom, carrying out the character detection and interception of the video image, and obtaining an intercepted image carrying student information in a video; classifying student states based on the intercepted image to obtain a classification result, and identifying student identities based on the intercepted image to obtain student identity information; dynamically correcting the classification result by using a multi-modal large language model to obtain a student state time axis; and performing student state identification based on the student identity information and the student state time axis to obtain a state identification result and a behavior evaluation result of the students in the classroom. According to the invention, the identification accuracy and the identification efficiency can be improved.

Description

Technical Field

[0001] The present application belongs to the field of multimodal recognition, and more specifically, to a dynamically corrected multimodal recognition method and system for classroom student status. Background Art

[0002] With the continuous deepening of educational informatization, real-time monitoring and specific evaluation and analysis of students' status in the classroom have become the key to evaluating students' status in class and improving teaching quality. Existing technologies still have many limitations in identifying students' status. Currently, the commonly used method is to obtain classroom video images for image analysis and image recognition. However, due to the complexity of multimodal fusion, the difficulty of sample acquisition, and the consumption of system computing resources in the current image processing process, there are problems such as inaccurate recognition results and poor recognition efficiency in the process of classroom student status recognition.

[0003] Therefore, how to improve the accuracy and efficiency of student status recognition is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of this application is to provide a dynamically corrected multimodal recognition method and system for classroom student status, aiming to solve the problems of poor accuracy and efficiency in classroom student status recognition.

[0005] In a first aspect, the present application provides a dynamically modified multimodal recognition method for classroom student status, comprising:

[0006] The present application provides a dynamically modified multimodal recognition method for classroom student status, comprising:

[0007] Acquire video images in the classroom, perform person detection and interception on the video images, and obtain intercepted images carrying student information in the video;

[0008] Classifying the student status based on the intercepted image to obtain a classification result, and identifying the student identity based on the intercepted image to obtain the student identity information;

[0009] Dynamically modify the classification results using a multimodal large language model to obtain a student status timeline;

[0010] The student status is identified based on the student identity information and the student status timeline to obtain status identification results and behavior evaluation results of the students in the classroom.

[0011] The present application implements efficient person detection, precise state classification, accurate identity recognition and intelligent dynamic correction step by step through multiple steps including person detection and interception, identity recognition, rough state classification, multimodal large language model correction and state evaluation. It processes from multiple different angles, effectively improving the recognition accuracy and efficiency of student states. It first performs preliminary detection through the model and then corrects it through the multimodal large language model, which reduces the demand for computing resources and thus improves the recognition efficiency.

[0012] Optionally, before dynamically correcting the classification result using the multimodal large language model, the method further includes:

[0013] According to the time information of each frame of the captured image, determine the student's initial state time axis;

[0014] The multimodal large language model is used to dynamically modify the classification result to obtain the student status timeline, including:

[0015] Determining whether the confidence level of the classification result is lower than a preset threshold;

[0016] When the confidence level is lower than the preset threshold, the state time axis is marked, and the captured image corresponding to the mark is input into the multimodal large language model to obtain detailed state information of the student;

[0017] The detailed status information is combined with the initial status timeline before correction to obtain a corrected student status timeline.

[0018] This application uses a multimodal large language model to dynamically correct the classification results, and can further optimize the accuracy of the state timeline when the judgment confidence is lower than the preset threshold. By inputting uncertain classification results into the model, the system can obtain more contextual information from multimodal data, thereby more accurately identifying the student's state.

[0019] Optionally, the student status identification is performed based on the student identity information and the student status timeline to obtain status identification results and behavior evaluation results of students in the classroom, including:

[0020] The student identity information and the student status timeline are input into the multimodal large language model for identification and evaluation, and the status identification result and behavior evaluation result of the student are obtained.

[0021] Optionally, the acquiring of a video image in the classroom, performing person detection and interception on the video image, and obtaining an intercepted image carrying student information in the video includes:

[0022] Acquire video images in the classroom from a camera;

[0023] Inputting the video image into a person detection model to obtain a detection result of the video image;

[0024] Determine that the detection result is a person image, intercept the person image, and obtain a plurality of small images carrying student information as the intercepted images.

[0025] Optionally, classifying the student status based on the captured image to obtain a classification result includes:

[0026] Inputting the captured image into a classification model, classifying the student status into eight categories, and obtaining the classification result;

[0027] The classification model is established based on the target detection technology YOLO and is trained based on image samples and classification labels corresponding to the image samples.

[0028] Optionally, the identifying the student identity based on the captured image to obtain the student identity information includes:

[0029] Inputting the captured image into a recognition model, recognizing the code corresponding to the captured image, and obtaining the student identity information;

[0030] The recognition model is established based on the face recognition algorithm arcface, collects facial information of all students as identity information samples, and is trained based on the identity information samples and the recognition labels corresponding to the identity information samples.

[0031] Optionally, the classification results of the eight categories of student status include: concentration state, reading state, writing state, sleeping state, looking around state, hand-raising state, standing state and leisure state.

[0032] In a second aspect, the present application also provides a dynamically modified multimodal recognition system for classroom student status, comprising:

[0033] A detection and capture unit, used to obtain video images in the classroom, perform person detection and capture on the video images, and obtain a captured image carrying student information in the video;

[0034] A classification and identification unit, used to classify the student status based on the intercepted image to obtain a classification result, and to identify the student identity based on the intercepted image to obtain the student identity information;

[0035] A correction unit, used for dynamically correcting the classification result by using a multimodal large language model to obtain a student status timeline;

[0036] The status evaluation unit is used to identify the student status based on the student identity information and the student status timeline, and obtain the status identification results and behavior evaluation results of the students in the classroom.

[0037] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0039] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0040] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0041] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0042] (1) This application implements efficient person detection, precise state classification, accurate identity recognition and intelligent dynamic correction step by step through multiple steps including person detection and interception, identity recognition, state rough classification, multimodal large language model correction and state evaluation. It processes from multiple different angles, effectively improving the recognition accuracy and efficiency of student states. It first performs preliminary detection through the model and then corrects it through the multimodal large language model, which reduces the demand for computing resources and thus improves the recognition efficiency.

[0043] (2) This application uses character detection technology to accurately identify students in classroom videos and ensure the relevance of captured images to students, thereby improving the efficiency and accuracy of subsequent processing. By using captured student images to classify student states and combining face recognition technology, the student's identity information can be accurately identified. The multimodal large language model can be used to combine the student's state classification results and timeline information to dynamically correct the student's state recognition results. Dynamic correction can make state recognition more precise and detailed, further improving the accuracy of student state recognition.

[0044] (3) This application evaluates the students' entire class status by accessing a multimodal large language model, and can provide a class status report for each student, which is helpful for teaching management and evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is one of the flow charts of the dynamically modified multimodal identification method of classroom student status provided in the embodiment of the present application;

[0046] Figure 2 This is the second flow chart of the dynamically modified multimodal identification method of classroom student status provided in the embodiment of the present application;

[0047] Figure 3 This is the third flow chart of the dynamically modified multimodal identification method of classroom student status provided in the embodiment of the present application;

[0048] Figure 4 It is a structural diagram of a dynamically modified classroom student status multimodal recognition system provided in an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0052] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0053] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0054] Reference Figure 1 The present application provides a dynamically modified multimodal recognition method for classroom student status, comprising:

[0055] S101. Obtaining a video image in the classroom, performing character detection and capture on the video image, and obtaining a captured image carrying student information in the video;

[0056] S102. Classifying the student status based on the captured image to obtain a classification result, and identifying the student identity based on the captured image to obtain the student identity information;

[0057] S103. Dynamically modify the classification result using a multimodal large language model to obtain a student status timeline;

[0058] S104. Perform student status recognition based on the student identity information and the student status timeline to obtain status recognition results and behavior evaluation results of students in the classroom.

[0059] Specifically, the detailed description of each step in the embodiments of the present application is as follows:

[0060] S101 video image acquisition and person detection.

[0061] Specifically including: obtaining video images in the classroom from the camera;

[0062] Inputting the video image into a person detection model to obtain a detection result of the video image;

[0063] Determine that the detection result is a person image, intercept the person image, and obtain a plurality of small images carrying student information as the intercepted images.

[0064] First, video images are acquired from the classroom, usually through a camera or other video acquisition device. Then, person detection is performed on the acquired video images, such as through a target detection algorithm, so as to accurately identify the students in the classroom. The detected students can be cut out as small images to generate separate images.

[0065] S102 Student status classification and identification.

[0066] In this step, image processing and machine learning techniques are used to classify the status of each captured student image. The status classification in this step is a rough classification, which can be achieved through the trained classification model to output the eight classification results of the student status.

[0067] The eight-category classification results of student status include: concentration state, reading state, writing state, sleeping state, looking around state, hand-raising state, standing state and leisure state (i.e. playing with mobile phones or other electronic devices).

[0068] The captured student images are identified through face recognition or other identity verification technologies. This is done by comparing the student's facial features with the information in the known database to ensure that each student's status timeline can be correctly associated with the corresponding identity.

[0069] S103 dynamically corrects the student status timeline.

[0070] The student status classification results are input into the multimodal large language model, which can integrate multiple information such as images and possible voice and text to gain a deeper understanding of the student's actual status. The model can dynamically adjust and correct the initial classification results to improve the accuracy and refinement of the student's status.

[0071] S104 Student status identification and behavior evaluation.

[0072] Combined with the corrected student status timeline and student identity information, the system can accurately identify the current status of each student in the classroom. Based on the identified student status, the system can generate behavioral evaluation results.

[0073] The embodiment of the present application implements efficient person detection, precise state classification, accurate identity recognition and intelligent dynamic correction step by step through multiple steps of person detection and interception, identity recognition, coarse state classification, multimodal large language model correction and state evaluation, effectively improving the recognition accuracy and efficiency of student states, and first performs preliminary detection through the model, and then corrects it through the multimodal large language model, reducing the demand for computing resources and thereby improving recognition efficiency.

[0074] Optionally, before dynamically correcting the classification result using the multimodal large language model, the method further includes:

[0075] According to the time information of each frame of the captured image, determine the student's initial state time axis;

[0076] The multimodal large language model is used to dynamically modify the classification result to obtain the student status timeline, including:

[0077] Determining whether the confidence level of the classification result is lower than a preset threshold;

[0078] When the confidence level is lower than the preset threshold, the state time axis is marked, and the captured image corresponding to the mark is input into the multimodal large language model to obtain detailed state information of the student;

[0079] The detailed status information is combined with the initial status timeline before correction to obtain a corrected student status timeline.

[0080] In this embodiment, images are captured frame by frame from the classroom video, and the time information of each frame is extracted to ensure that the student status can be accurately matched to the time point. The status of each captured student image is classified.

[0081] Determine whether the confidence of each classification result is lower than the preset threshold. If the confidence of the classification result of a frame is lower than the threshold, it may mean that the model is not confident enough about the state classification of the frame. If the confidence is lower than the threshold, mark the time point of the frame as a point that requires more detailed analysis. The corresponding captured image will be input into the multimodal large language model.

[0082] The multimodal large language model can comprehensively consider the visual information of the captured image and possible voice and text information to gain a deeper understanding of the student's status. The model outputs more detailed and accurate student status information. The detailed status information output by the multimodal large language model is combined with the original status timeline to complete the dynamic correction of the status timeline. The corrected timeline includes not only the basic status classification, but also a more detailed description of the student status provided by the multimodal information.

[0083] The embodiment of the present application realizes obtaining the student status timeline from the video image, and realizes dynamic correction and refinement through a multimodal large language model, thereby further improving the accuracy of student status recognition.

[0084] Optionally, the student status identification is performed based on the student identity information and the student status timeline to obtain status identification results and behavior evaluation results of students in the classroom, including:

[0085] The student identity information and the student status timeline are input into the multimodal large language model for identification and evaluation, and the status identification result and behavior evaluation result of the student are obtained.

[0086] Optionally, classifying the student status based on the captured image to obtain a classification result includes:

[0087] Inputting the captured image into a classification model, classifying the student status into eight categories, and obtaining the classification result;

[0088] The classification model is established based on the target detection technology YOLO and is trained based on image samples and classification labels corresponding to the image samples.

[0089] In this embodiment, YOLO (You Only Look Once) target detection technology is used to process each captured video image frame. YOLO can quickly and accurately detect targets in the image and give their positions and bounding boxes. Based on YOLO, its ability is expanded to detect student status.

[0090] The YOLO model training process is as follows: Based on the image samples and the classification labels corresponding to these image samples, prepare a dataset for training the YOLO model. The dataset should cover scenes of various typical student states to ensure that the model performs well in various environments and conditions. Using the prepared dataset, train the YOLO object detection model so that it can accurately detect the state of the student in the image. During the training process, the model will learn how to distinguish different states and generate corresponding bounding boxes and state labels.

[0091] Optionally, the identifying the student identity based on the captured image to obtain the student identity information includes:

[0092] Inputting the captured image into a recognition model, recognizing the code corresponding to the captured image, and obtaining the student identity information;

[0093] The recognition model is established based on the face recognition algorithm arcface, collects facial information of all students as identity information samples, and is trained based on the identity sample information and the recognition labels corresponding to the identity sample information.

[0094] In this embodiment, arcface is a deep learning-based face recognition algorithm used to verify and identify the identity of students.

[0095] The model built based on the face recognition algorithm is used for face recognition. The training process of the model is as follows: collect the facial information of all students as identity information samples, and associate each sample with a unique identity label. Use the collected facial information samples and corresponding identity labels to train the ArcFace model. During the training process, the model learns how to extract recognizable features from facial images to distinguish the identities of different students.

[0096] After obtaining the trained model, the facial region of each student detected by YOLO is extracted and fed into the trained ArcFace model as input. The ArcFace model outputs the identity label corresponding to each facial region, that is, the identified student identity information.

[0097] Reference Figure 2 The complete process of the dynamically modified multimodal identification method for classroom student status provided by the embodiment of the present application includes:

[0098] Camera input photos;

[0099] People detection part and interception;

[0100] Rough classification of status;

[0101] Identity recognition obtains student identity information;

[0102] The multimodal large language model is modified to obtain the state timeline;

[0103] Input into the large language model to obtain the overall classroom behavior evaluation of students.

[0104] Reference Figure 3 The correction process in the dynamically corrected multimodal identification method for classroom student status provided in the embodiment of the present application includes the following steps:

[0105] The interception results output by the person detection and interception part are roughly classified into states;

[0106] Determine whether the confidence of the classification result is lower than the threshold;

[0107] If yes, input the multimodal large language model for correction, otherwise combine them to get the state timeline.

[0108] Reference Figure 4 The present application also provides a dynamically modified multimodal recognition system for classroom student status, including:

[0109] A detection and interception unit 410 is used to obtain a video image in the classroom, perform person detection and interception 420 on the video image, and obtain an intercepted image carrying student information in the video;

[0110] The classification and identification unit 430 is used to classify the student status based on the intercepted image to obtain a classification result, and to identify the student identity based on the intercepted image to obtain the student identity information;

[0111] A correction unit 440 is used to dynamically correct the classification result using a multimodal large language model to obtain a student status timeline;

[0112] The status evaluation unit 450 is used to identify the student status based on the student identity information and the student status timeline, and obtain the status identification results and behavior evaluation results of the students in the classroom.

[0113] Optionally, a time axis determination unit is also included:

[0114] According to the time information of each frame of the captured image, determine the student's initial state time axis;

[0115] The correction unit is specifically used for:

[0116] Determining whether the confidence level of the classification result is lower than a preset threshold;

[0117] When the confidence level is lower than the preset threshold, the state time axis is marked, and the captured image corresponding to the mark is input into the multimodal large language model to obtain detailed state information of the student;

[0118] The detailed status information is combined with the initial status timeline before correction to obtain a corrected student status timeline.

[0119] Optionally, the state evaluation unit is specifically used to:

[0120] The student identity information and the student status timeline are input into the multimodal large language model for identification and evaluation, and the status identification result and behavior evaluation result of the student are obtained.

[0121] Optionally, the detection and interception unit is specifically used to:

[0122] Acquire video images in the classroom from a camera;

[0123] Inputting the video image into a person detection model to obtain a detection result of the video image;

[0124] Determine that the detection result is a person image, intercept the person image, and obtain a plurality of small images carrying student information as the intercepted images.

[0125] Optionally, the classification and identification unit is specifically used to:

[0126] Inputting the captured image into a classification model, classifying the student status into eight categories, and obtaining the classification result;

[0127] The classification model is established based on the target detection technology YOLO and is trained based on image samples and classification labels corresponding to the image samples.

[0128] Optionally, the classification and identification unit is specifically used to:

[0129] Inputting the captured image into a recognition model, recognizing the code corresponding to the captured image, and obtaining the student identity information;

[0130] The recognition model is established based on the face recognition algorithm arcface, collects facial information of all students as identity information samples, and is trained based on the identity sample information and the recognition labels corresponding to the identity sample information.

[0131] Optionally, the classification results of the eight categories of student status include: concentration state, reading state, writing state, sleeping state, looking around state, hand-raising state, standing state and leisure state.

[0132] Reference Figure 5Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, which may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the method in the above embodiment.

[0133] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0134] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0135] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0136] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0137] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0138] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0139] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0140] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A dynamically modified multimodal recognition method for classroom student status, characterized in that: include: Acquire video images in the classroom, perform person detection and interception on the video images, and obtain intercepted images carrying student information in the video; Classifying the student status based on the intercepted image to obtain a classification result, and identifying the student identity based on the intercepted image to obtain the student identity information; Dynamically modify the classification results using a multimodal large language model to obtain a student status timeline; The student status is identified based on the student identity information and the student status timeline to obtain status identification results and behavior evaluation results of the students in the classroom.

2. The method for dynamically correcting multimodal identification of classroom student status according to claim 1 is characterized in that: Before dynamically correcting the classification result using the multimodal large language model, the method further includes: According to the time information of each frame of the captured image, determine the student's initial state time axis; The multimodal large language model is used to dynamically modify the classification result to obtain the student status timeline, including: Determining whether the confidence level of the classification result is lower than a preset threshold; When the confidence level is lower than the preset threshold, the state time axis is marked, and the captured image corresponding to the mark is input into the multimodal large language model to obtain detailed state information of the student; The detailed status information is combined with the initial status timeline before correction to obtain a corrected student status timeline.

3. The method for dynamically correcting multimodal identification of classroom student status according to claim 1 is characterized in that: The student status recognition is performed based on the student identity information and the student status timeline to obtain the status recognition results and behavior evaluation results of the students in the classroom, including: The student identity information and the student status timeline are input into the multimodal large language model for identification and evaluation, and the status identification result and behavior evaluation result of the student are obtained.

4. The method for dynamically correcting multimodal identification of classroom student status according to claim 1 is characterized in that: The method of acquiring a video image in the classroom, performing person detection and interception on the video image, and obtaining an intercepted image carrying student information in the video includes: Acquire video images in the classroom from a camera; Inputting the video image into a person detection model to obtain a detection result of the video image; Determine that the detection result is a person image, intercept the person image, and obtain a plurality of small images carrying student information as the intercepted images.

5. The method for dynamically correcting multimodal identification of classroom student status according to claim 1 is characterized in that: The classifying the student status based on the intercepted image to obtain the classification result includes: Inputting the captured image into a classification model, classifying the student status into eight categories, and obtaining the classification result; The classification model is established based on the target detection technology YOLO and is trained based on image samples and classification labels corresponding to the image samples.

6. The method for dynamically correcting multimodal identification of classroom student status according to claim 1 is characterized in that: The identifying of the student based on the captured image to obtain the student identity information includes: Inputting the captured image into a recognition model, recognizing the code corresponding to the captured image, and obtaining the student identity information; The recognition model is established based on the face recognition algorithm arcface, collects facial information of all students as identity information samples, and is trained based on the identity information samples and the recognition labels corresponding to the identity information samples.

7. The method for dynamically correcting multimodal identification of classroom student status according to claim 5 is characterized in that: The classification results of the eight categories of student status include: concentration state, reading state, writing state, sleeping state, looking around state, hand-raising state, standing state and leisure state.

8. A dynamically modified multimodal recognition system for classroom student status, characterized in that: include: A detection and capture unit, used to obtain video images in the classroom, perform person detection and capture on the video images, and obtain a captured image carrying student information in the video; A classification and identification unit, used to classify the student status based on the intercepted image to obtain a classification result, and to identify the student identity based on the intercepted image to obtain the student identity information; A correction unit, used for dynamically correcting the classification result by using a multimodal large language model to obtain a student status timeline; The status evaluation unit is used to identify the student status based on the student identity information and the student status timeline, and obtain the status identification results and behavior evaluation results of the students in the classroom.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images