Student classroom behavior analysis method and device based on large model, and medium

Abstract

The invention provides a student classroom behavior analysis method based on a large model, and the method comprises the steps: obtaining video data and audio data through a video collection device and an audio collection device which are disposed in a classroom, inputting the video data and the audio data into a trained student classroom behavior model, processing a YOLOv5 model of a picture sample data set, and carrying out the processing of the YOLOv5 model. The Whisper model is used for processing the audio sample data set; based on the YOLOv5 model, identifying student behaviors in the classroom real-time monitoring image, and comparing student behavior characteristics with predetermined standard behavior characteristics to obtain a first behavior characteristic value; furthermore, on the basis of the Whisper model, identifying a text of the audio sample data set, and comparing the text with a predetermined hot word to obtain a second behavior feature value; and integrating the first behavior characteristic value and the second behavior characteristic value to obtain a total score of the classroom behaviors of the students and reporting the total score. The invention further discloses a large model-based student classroom behavior analysis device and a storage medium.

Description

Technical Field

[0001] The present invention relates to the technical field of smart education and intelligent monitoring, and in particular to a method and device for analyzing student classroom behavior based on a large model, and a computer-readable storage medium. Background Art

[0002] With the rapid development of technologies such as the Internet, big data, and artificial intelligence, educational informatization has become an important means to promote educational modernization. Smart classroom behavior analysis technology came into being in this context, using advanced information technology to provide more accurate and efficient support for education.

[0003] Classroom behavior analysis aims to study the internal mechanism of teachers' teaching activities and students' academic development in the classroom, help teachers and students reflect on their own classroom performance, and thus promote the improvement of classroom teaching quality and effectiveness. Traditional classroom teaching behavior analysis mostly collects and analyzes data through self-evaluation, manual supervision, classroom inspections and other methods. It has the disadvantages of strong subjectivity of personal evaluation, small sample size, time-consuming and labor-intensive, resulting in low interpretability and scalability. The popularization of artificial intelligence technology has brought opportunities to improve these shortcomings. Smart classroom behavior analysis technology is to collect classroom behavior data of teachers and students in the context of educational informatization, integrate big data, artificial intelligence and other advanced technologies, and analyze them, which can help teachers better understand students' learning situation, adjust teaching strategies in time, and improve teaching effectiveness.

[0004] Classroom behavior analysis uses computer vision, video subtitle extraction, and large model technology to monitor the behavior of teachers and students in real time and capture the content of teachers' lectures and student interactions. It combines the above information to analyze the summary of teachers' lectures, sort out classroom keywords, analyze teachers' emotions during lectures, teacher-student interactions, classroom activity, statistics on teachers' and students' lateness, early departure, and absences, and capture students' behaviors (such as lying on the table, playing with mobile phones, raising hands, fighting, writing, etc.). Combining the above analysis results, a classroom quality report is obtained to help teachers and students understand the classroom situation and provide more accurate and efficient support for education. Summary of the invention

[0005] The present invention provides a method, device and computer-readable storage medium for analyzing student classroom behavior based on a large model. The large model is used to comprehensively evaluate student classroom behavior through multimodal data such as sound and behavior, so as to accurately evaluate student behavior and teaching effectiveness.

[0006] In a first aspect, a method for analyzing student classroom behavior based on a large model is provided, comprising: acquiring video data and audio data respectively through a video acquisition device and an audio acquisition device arranged in a classroom, and performing preprocessing operations on the video data and the audio data to generate a picture sample data set and an audio sample data set of student classroom behavior; inputting the picture sample data set and the audio sample data set into a trained student classroom behavior model, wherein the student classroom behavior model comprises at least two of the following: a YOLOv5 model for processing the picture sample data set, and a Whisper model for processing the audio sample data set; based on the YOLOv5 model, identifying student behavior in a real-time monitoring image of the classroom, comparing student behavior characteristics with predetermined standard behavior characteristics, and obtaining a first behavior feature value; and, based on the Whisper model, identifying text in the audio sample data set, comparing the text with predetermined hot words, and obtaining a second behavior feature value; integrating the first behavior feature value and the second behavior feature value to obtain a total score of the student classroom behavior and report it.

[0007] In some embodiments, the image sample data set includes: images of student behavior states, marked as sleeping, looking up at the class, looking down at reading and writing, standing or discussing; the audio sample data set includes: the number of times, duration and frequency of speaking.

[0008] In some embodiments, identifying student behavior in a real-time classroom monitoring image includes: grayscale processing the image sample data set to convert it into a grayscale image; matching the grayscale image with a predetermined student behavior state, and calculating the matching degree based on the following calculation formula:

[0009]

[0010] Wherein, L represents the matching degree, W and H represent the width and height of the grayscale image respectively, x and y represent the horizontal coordinate and vertical coordinate of the pixel point respectively, f(x, y) represents the pixel value of the grayscale image, and γ' represents the Euler formula.

[0011] In some embodiments, comparing the student behavior characteristics with the predetermined standard behavior characteristics includes: assigning points to the image sample data set and the predetermined standard behavior characteristics to obtain the first behavior characteristic value.

[0012] In some embodiments, identifying the text of the audio sample data set includes: transcribing the audio sample data set based on the Whisper model; and aligning the timestamps of the text generated by the transcription.

[0013] In some embodiments, the parameters of the Whisper model are optimized using the audio sample data set, including: using an encoder to perform automatic speech recognition on the input audio sample data set to obtain a recognition result; and adjusting the parameters of the Whisper model based on the recognition result.

[0014] In some embodiments, the image sample data set is divided into a training set, a validation set, and a test set, and the divided samples are annotated and data labeled; data enhancement processing is performed on the images in the training set, and brightness, contrast, and saturation are randomly adjusted on the images in the training set.

[0015] In a second aspect, a device for analyzing student classroom behavior based on a large model is provided, characterized in that it includes: a data acquisition module, which is used to obtain video data and audio data respectively through a video acquisition device and an audio acquisition device set in the classroom, and pre-process the video data and the audio data to generate a picture sample data set and an audio sample data set of the student classroom behavior; a model processing module, which is used to input the picture sample data set and the audio sample data set into a trained student classroom behavior model, and the student classroom behavior model includes at least two of the following: a YOLOv5 model for processing the picture sample data set, and a Whisper model for processing the audio sample data set; a processing module, which is used to identify the student behavior in the real-time monitoring image of the classroom based on the YOLOv5 model, compare the student behavior characteristics with the predetermined standard behavior characteristics, and obtain a first behavior feature value; and, based on the Whisper model, identify the text of the audio sample data set, compare the text with a predetermined hot word, and obtain a second behavior feature value; a result generation module, which is used to integrate the first behavior feature value and the second behavior feature value to obtain a total score of the student classroom behavior and report it.

[0016] In a third aspect, the present invention provides an electronic device, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned large-model-based student classroom behavior analysis method is implemented.

[0017] In a fourth aspect, the present invention further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores program code, and the program code can be called by a processor to execute the above-mentioned large-model-based student classroom behavior analysis method.

[0018] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0019] First, the student classroom behavior analysis method can evaluate students based on their learning behavior and performance, helping them better master knowledge and skills.

[0020] Second, by analyzing students' facial expressions, voice intonation, etc., we can identify students' emotional states, help teachers understand students' psychological needs in a timely manner, and provide emotional support.

[0021] Third, it can help teachers promptly identify problematic behaviors in the classroom, such as distraction and daydreaming, and take appropriate intervention measures to maintain classroom order.

[0022] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0024] Figure 1 A flow chart of a method for analyzing student classroom behavior based on a large model provided in an embodiment of the present application is shown;

[0025] Figure 2 is a schematic block diagram of a device for analyzing student classroom behavior based on a large model provided in an embodiment of the present application;

[0026] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of an application scenario of an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0029] The term "including" and its variations used in this document mean open inclusion, that is, "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0030] This application provides a method for analyzing student classroom behavior based on a large model. Figure 1 , which is a schematic diagram of the first embodiment of the present application. Figure 1 The first embodiment of the present application provides a method for analyzing student classroom behavior based on a large model and is described in detail.

[0031] The present application provides a method 100 for analyzing student classroom behavior based on a large model. The processing flow of the method 100 may include the following steps:

[0032] Step S102: Acquire video data and audio data, that is, respectively acquire video data and audio data through a video acquisition device and an audio acquisition device set up in the classroom, and perform preprocessing operations on the video data and the audio data to generate a picture sample data set and an audio sample data set of students' classroom behavior.

[0033] In some embodiments, the image sample data set includes: images of student behavior states, marked as sleeping, looking up at the class, looking down at reading and writing, standing or discussing; the audio sample data set includes: the number of times, duration and frequency of speaking.

[0034] In some embodiments, the audio data is preprocessed to convert it into a format suitable for model processing, such as a uniform sampling rate and audio encoding format. This step ensures the stability and consistency of the audio data in the subsequent speech recognition process.

[0035] In some embodiments, the RTSP real-time stream can be accessed to obtain video frames, and the trained YOLOv5 model can be used to perform real-time behavior analysis on the frames. The number of frames to be detected per second can be configured according to server resources, and the analysis results (such as behavior category, confidence, etc.) can be transmitted to the front end in real time for annotation. Real-time feedback: Through real-time behavior analysis, teachers can instantly understand students' attention and participation, so as to adjust teaching strategies in time. This instant feedback mechanism helps to improve the effectiveness of classroom teaching and ensure that students can better understand and master knowledge.

[0036] Step S104: inputting the trained student classroom behavior model, that is, inputting the image sample data set and the audio sample data set into the trained student classroom behavior model, wherein the student classroom behavior model includes at least two of the following: a YOLOv5 model for processing the image sample data set, and a Whisper model for processing the audio sample data set.

[0037] Step S106: Identification and comparison, that is, based on the YOLOv5 model, identify the student behavior in the real-time classroom monitoring image, compare the student behavior characteristics with the predetermined standard behavior characteristics, and obtain a first behavior feature value; and, based on the Whisper model, identify the text of the audio sample data set, compare the text with predetermined hot words, and obtain a second behavior feature value.

[0038] In some embodiments, identifying student behavior in a real-time classroom monitoring image includes: grayscale processing the image sample data set to convert it into a grayscale image; matching the grayscale image with a predetermined student behavior state, and calculating the matching degree based on the following calculation formula:

[0039]

[0040] Wherein, L represents the matching degree, W and H represent the width and height of the grayscale image respectively, x and y represent the horizontal coordinate and vertical coordinate of the pixel point respectively, f(x, y) represents the pixel value of the grayscale image, and γ' represents the Euler formula.

[0041] In some embodiments, comparing the student behavior characteristics with the predetermined standard behavior characteristics includes: assigning points to the image sample data set and the predetermined standard behavior characteristics to obtain the first behavior characteristic value.

[0042] In some embodiments, identifying the text of the audio sample data set includes: transcribing the audio sample data set based on the Whisper model; and aligning the timestamps of the text generated by the transcription.

[0043] In some embodiments, relevant hot words can be collected manually or automatically. For example, for math teaching videos, key terms in math formulas, names of mathematicians, etc. are collected; for language teaching videos, new popular words, grammatical terms, etc. are collected. These hot words are added to the input of the model in a specific format (such as the form of <code start>hotwords<code end> ​​parameters) so that they can be given priority by the model during the transcription process.

[0044] Step S108: integration and reporting, that is, integrating the first behavior feature value and the second behavior feature value to obtain the total score of the student's classroom behavior and reporting it. In this way, personalized teaching can be achieved: the system can provide personalized learning suggestions based on the student's behavior data to help teachers provide differentiated teaching according to the needs of different students. This personalized teaching method can improve students' interest and enthusiasm in learning and promote their all-round development.

[0045] In some embodiments, the method for analyzing student classroom behavior based on a large model may also include: optimizing the parameters of the Whisper model using the audio sample data set, including: using an encoder to perform automatic speech recognition on the input audio sample data set to obtain a recognition result; and adjusting the parameters of the Whisper model based on the recognition result.

[0046] In some embodiments, the method for analyzing student classroom behavior based on a large model may also include: dividing the image sample data set into a training set, a validation set, and a test set, and annotating and data labeling the divided samples; performing data enhancement processing on the images in the training set, and randomly adjusting the brightness, contrast, and saturation of the images in the training set.

[0047] In some embodiments, the total score of the student's classroom behavior can be reported by class, course, or teacher, and a report can be generated and sent to the front end, which receives the analysis results and annotates and displays them on the user interface. The report can also be exported according to needs; teachers can view the behavior status of students in real time and make corresponding teaching adjustments.

[0048] Figure 2 2 is a schematic block diagram of a device 200 for analyzing student classroom behavior based on a large model according to an exemplary embodiment. The device 200 for analyzing student classroom behavior based on a large model includes: a data acquisition module 202, a large model module 204, a processing module 206 and a result generation module 208, wherein:

[0049] The data acquisition module 202 is used to obtain video data and audio data respectively through the video acquisition device and audio acquisition device set in the classroom, and perform preprocessing operations on the video data and the audio data to generate a picture sample data set and an audio sample data set of students' classroom behavior.

[0050] The large model module 204 is used to input the image sample data set and the audio sample data set into a trained student classroom behavior model, wherein the student classroom behavior model includes at least two of the following: a YOLOv5 model for processing the image sample data set, and a Whisper model for processing the audio sample data set;

[0051] The processing module 206 is used to identify the student behavior in the real-time classroom monitoring image based on the YOLOv5 model, compare the student behavior characteristics with the predetermined standard behavior characteristics, and obtain a first behavior feature value; and identify the text of the audio sample data set based on the Whisper model, compare the text with the predetermined hot words, and obtain a second behavior feature value;

[0052] The result generation module 208 is used to integrate the first behavior feature value and the second behavior feature value to obtain a total score of the student's classroom behavior and report it.

[0053] In some embodiments, the processing module 206 identifies the student behavior in the real-time classroom monitoring image, which may include: grayscale processing the image sample data set to convert it into a grayscale image; matching the grayscale image with a predetermined student behavior state, and calculating the matching degree based on the following calculation formula:

[0054]

[0055] Wherein, L represents the matching degree, W and H represent the width and height of the grayscale image respectively, x and y represent the horizontal coordinate and vertical coordinate of the pixel point respectively, f(x, y) represents the pixel value of the grayscale image, and γ' represents the Euler formula.

[0056] In some embodiments, the analysis device 200 for student classroom behavior based on a large model may also include: a model optimization module. For example: the model optimization module may use the audio sample data set to optimize the parameters of the Whisper model, including: using a decoder to perform automatic speech translation on the input audio sample data set to obtain a first recognition result; using an encoder to perform automatic speech recognition on the input audio sample data set to obtain a second recognition result; combining the first recognition result and the second recognition result to adjust the parameters of the Whisper model. For another example: the model optimization module may also divide the image sample data set into a training set, a validation set, and a test set, and annotate and data label the divided samples; perform data enhancement processing on the images in the training set, and randomly adjust the brightness, contrast, and saturation of the images in the training set.

[0057] Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3As shown, the electronic device 300 may include a processor 3001 and a memory 3002. Optionally, the electronic device 300 may further include a transceiver 3003. The processor 3001, the memory 3002 and the transceiver 3003 may be connected, for example, via a communication bus. The memory 3002 stores computer-readable instructions, and when the computer-readable instructions are executed by the processor 3001, the steps of the above-mentioned method for analyzing student classroom behavior based on a large model are implemented.

[0058] In a specific implementation, as an embodiment, the processor 3001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0059] In a specific implementation, as an embodiment, the electronic device 300 may also include multiple processors, such as Figure 3 3001 and processor 3004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0060] The memory 3002 is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor 3001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0061] The transceiver 3003 is used to communicate with a network device or a terminal device.

[0062] Optionally, the transceiver 3003 may include a receiver and a transmitter, wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0063] Optionally, the transceiver 3003 may be integrated with the processor 3001 , or may exist independently and be coupled to the processor 3001 via an interface circuit of the electronic device 300 , which is not specifically limited in the embodiment of the present invention.

[0064] It should be noted that Figure 3 The structure of the electronic device 300 shown in the figure does not constitute a limitation on the electronic device. The actual electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects of the electronic device 300 can refer to the technical effects of the above-mentioned method embodiment, which will not be repeated here.

[0065] like Figure 4The application scenario diagram of the illustrated embodiment is a schematic diagram of an application scenario of the analysis method of student classroom behavior based on a large model provided in the embodiment of the present application. In the application scenario diagram, a terminal device 401 and a server 402 are included. The terminal device 401 and the server 402 communicate with each other through a communication network.

[0066] The terminal device 401 is an electronic device used by the target object, which may be a personal computer, a mobile phone, a tablet computer, a notebook, an e-book reader, a vehicle-mounted terminal, etc. In addition, a client related to the analysis method of student classroom behavior based on a large model may be installed on the terminal device 401, and the client may be software (for example, an APP, a browser, etc.), or a web page, a small program, etc. By implementing the above-mentioned analysis method of student classroom behavior based on a large model on the terminal device 401, relevant operations of data processing of student classroom behavior based on a large model are performed.

[0067] Server 402 can be an independent physical server or an edge device 402 in the field of cloud computing. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, cloud functions, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (English name: Content Delivery Network, abbreviated as CDN), as well as big data and artificial intelligence platforms.

[0068] There is no restriction on the number of the terminal devices 401 and / or servers 402 .

[0069] It should be noted that the analysis method of student classroom behavior based on a large model in the embodiment of the present application can be executed by the terminal device 401 or the server 402 alone, or can be executed by the terminal device 401 and the server 402 together. For example, when the terminal device 401 and the server 402 are executed together, the server 402 can provide the terminal device 401 with a video acquisition device and an audio acquisition device in the classroom to obtain video data and audio data respectively. For example, the image sample data set may include: a student behavior state picture, and annotated as sleeping, raising the head to listen to the class, lowering the head to read and write, standing or discussing; the audio sample data set may include: the number of times, duration and frequency of speaking. The terminal device 401 can provide the server 402 with the YOLOv5 model to identify the student behavior in the real-time monitoring image of the classroom, compare the student behavior characteristics with the predetermined standard behavior characteristics, and obtain the first behavior feature value; and, based on the Whisper model, identify the text of the audio sample data set, compare the text with the predetermined hot words, and obtain the second behavior feature value; integrate the first behavior feature value and the second behavior feature value to obtain the total score of the student's classroom behavior and report it.

[0070] In an exemplary embodiment, the present invention further provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the steps of the above-mentioned method for analyzing student classroom behavior based on a large model. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0071] An embodiment of the present invention also provides an electronic device, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned large-model-based student classroom behavior analysis method is implemented.

[0072] An embodiment of the present invention provides a computer-readable storage medium, characterized in that program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the above-mentioned large-model-based student classroom behavior analysis method.

[0073] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0074] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0075] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0076] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0077] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0079] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0080] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0083] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0084] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for analyzing student classroom behavior based on a large model, characterized in that: include: Acquire video data and audio data respectively through a video acquisition device and an audio acquisition device set in the classroom, and perform preprocessing operations on the video data and the audio data to generate a picture sample data set and an audio sample data set of the student's classroom behavior; Inputting the image sample dataset and the audio sample dataset into a trained student classroom behavior model, wherein the student classroom behavior model includes at least two of the following: a YOLOv5 model for processing the image sample dataset, and a Whisper model for processing the audio sample dataset; Based on the YOLOv5 model, identify the student behavior in the real-time monitoring image of the classroom, compare the student behavior characteristics with the predetermined standard behavior characteristics, and obtain a first behavior characteristic value; Furthermore, based on the Whisper model, identifying the text of the audio sample data set, comparing the text with a predetermined hot word, and obtaining a second behavior feature value; The first behavior characteristic value and the second behavior characteristic value are integrated to obtain a total score of the student's classroom behavior and report it.

2. The method for analyzing student classroom behavior based on a large model according to claim 1 is characterized in that: The image sample dataset includes: images of students’ behavior states, annotated as sleeping, looking up at the class, looking down at the class, standing or discussing; The audio sample data set includes: the number of utterances, duration and frequency.

3. The method for analyzing student classroom behavior based on a large model according to claim 1 is characterized in that: Identify student behaviors in real-time classroom surveillance images, including: Converting the image sample data set into a grayscale image by grayscale processing; The grayscale image is matched with a predetermined student behavior state, and the matching degree is calculated based on the following calculation formula: Wherein, L represents the matching degree, W and H represent the width and height of the grayscale image respectively, x and y represent the horizontal coordinate and vertical coordinate of the pixel point respectively, f(x, y) represents the pixel value of the grayscale image, and γ' represents the Euler formula.

4. The method for analyzing student classroom behavior based on a large model according to claim 1 is characterized in that: Compare student behavior characteristics to predetermined standard behavior characteristics, including: The picture sample data set and the predetermined standard behavior feature are scored to obtain the first behavior feature value.

5. The method for analyzing student classroom behavior based on a large model according to claim 1 is characterized in that: Recognizing the text of the audio sample dataset, comprising: Based on the Whisper model, transcribing the audio sample dataset; Align the timestamps of the transcribed text.

6. The method for analyzing student classroom behavior based on a large model according to claim 1 is characterized in that: Also includes: Optimizing the parameters of the Whisper model using the audio sample data set includes: Using an encoder to perform automatic speech recognition on the input audio sample data set to obtain a recognition result; The Whisper model is parameterized based on the recognition result.

7. The method for analyzing student classroom behavior based on a large model according to claim 6 is characterized in that: Also includes: Dividing the image sample data set into a training set, a validation set, and a test set, and annotating and data labeling the divided samples; Data enhancement is performed on the images in the training set, and brightness, contrast and saturation are randomly adjusted on the images in the training set.

8. A device for analyzing student classroom behavior based on a large model, characterized in that: include: A data acquisition module, used to acquire video data and audio data respectively through a video acquisition device and an audio acquisition device set in the classroom, and perform preprocessing operations on the video data and the audio data to generate a picture sample data set and an audio sample data set of students' classroom behavior; A large model module, used to input the image sample data set and the audio sample data set into a trained student classroom behavior model, wherein the student classroom behavior model includes at least two of the following: a YOLOv5 model for processing the image sample data set, and a Whisper model for processing the audio sample data set; A processing module, used for identifying student behaviors in the real-time classroom monitoring image based on the YOLOv5 model, comparing student behavior characteristics with predetermined standard behavior characteristics, and obtaining a first behavior characteristic value; Furthermore, based on the Whisper model, identifying the text of the audio sample data set, comparing the text with a predetermined hot word, and obtaining a second behavior feature value; The result generation module is used to integrate the first behavior feature value and the second behavior feature value to obtain the total score of the student's classroom behavior and report it.

9. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method for analyzing student classroom behavior based on a large model as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the large-model-based student classroom behavior analysis method as described in any one of claims 1 to 7.

Images