Real-time course patrol method and device based on video processing and medium

Abstract

The invention provides a real-time class patrol method based on video processing, and the method comprises the steps: collecting the real-time video data of the behaviors of students in a classroom, extracting the individual images of the students in the classroom according to frames, and carrying out the behavior state recognition; detecting and tracking the individual image through a DeepSort algorithm according to a behavior state recognition result, and comparing detection frame boundaries to obtain recognition results, namely listening to class by raising the head, reading and writing by lowering the head and standing; extracting an image of each student according to frames from a classroom video under a time dimension in which individual behaviors of each student need to be continuously monitored and managed in real time, and inputting the image into the student behavior recognition model for behavior recognition; and counting various behavior durations of each student in the classroom video, displaying the distribution of each behavior duration of each student every day, performing visual analysis on each behavior duration, and uploading an analysis result to a server for teaching reference. The invention further discloses a real-time course patrol device based on video processing and a storage medium.

Description

Technical Field

[0001] The present invention relates to the technical field of video intelligent analysis, and in particular to a real-time class inspection method and device based on video processing, and a computer-readable storage medium. Background Art

[0002] With the popularity of online education, educators hope to understand students' learning status and participation in real time. Traditional evaluation methods are often lagging and cannot reflect students' learning effects in a timely manner. Therefore, real-time behavior analysis technology has emerged to provide instant feedback and help teachers adjust their teaching strategies.

[0003] Real-time behavioral analysis refers to the real-time monitoring and analysis of user behavior and interaction through computer vision and machine learning technology. This technology can be used to identify user emotions, attention, engagement, etc., thereby providing feedback to educators and helping to optimize teaching content and methods.

[0004] Real-time behavioral analysis often relies on computer vision techniques, especially deep learning models such as convolutional neural networks (CNNs), which can extract features from video streams and identify and classify different behavioral patterns.

[0005] At present, deep learning dominates the field of real-time behavior analysis, and many open source and commercial technologies have emerged. The most similar to the present invention is Yolov5, which is open sourced by the Ultralytics team. It is based on the convolutional neural network (CNN) architecture and adopts the design concept of the improved YOLO (You Only Look Once) series of models. YOLOv5 achieves efficient real-time performance in target detection tasks and combines a variety of deep learning technologies, such as feature pyramid networks (FPN) and adaptive anchor boxes (Adaptive Anchor Boxes) to improve detection accuracy and speed.

[0006] However, the existing technology still has shortcomings in teaching real-time behavior analysis. When faced with complex behavior patterns, YOLOv5 may make misjudgments or miss judgments, especially when the behaviors change rapidly or overlap. This limits its application effect in certain high-dynamic scenarios, leaving room for innovation and breakthroughs for the present invention. Summary of the invention

[0007] The present invention provides a real-time classroom inspection method, device and computer-readable storage medium based on video processing, which utilizes a model to monitor students' behaviors (such as attention and participation) in real time through video, and provides timely feedback to teachers and students for adjustments, so as to better serve educational and teaching activities.

[0008] In a first aspect, a real-time class inspection method based on video processing is provided, including: collecting real-time video data of students' classroom behaviors, extracting individual images of students in the classroom frame by frame for behavioral state recognition; according to the results of behavioral state recognition, detecting and tracking the individual images through the DeepSort algorithm, and then comparing the boundaries of the detection frame to obtain recognition results of looking up to listen to the class, looking down to read and write, and standing respectively; extracting each student image frame by frame from the classroom video under the time dimension of the individual behavior of each student that needs to be monitored and managed in real time, and inputting the image into the student behavior recognition model for behavior recognition; counting the duration of various behaviors of each student in the classroom video, and displaying the distribution of the duration of each student's various behaviors every day, visually analyzing the duration of each behavior, and uploading the analysis results to the server for teaching reference.

[0009] In some embodiments, real-time video data of students' behavior in class is collected, and individual images of students in the class are extracted frame by frame for behavior status recognition, including: using a part of the individual images as a training set and another part as a test set, inputting the training set into a YOLOv5 network model to train an individual behavior recognition model, and verifying it with the test set; annotating the body status of the students in the frame images in the individual images, and converting them into txt format files required by YOLOv5, and adding the txt format files generated by each annotation to the training set; inputting the txt format files into the YOLOv5 network model, and using the test set for verification to optimize parameters.

[0010] In some embodiments, based on the result of behavioral state recognition, the individual image is detected and tracked by the DeepSort algorithm, and the recognition results obtained by comparing the detection frame boundaries are respectively looking up to listen to the class, looking down to read and write, and standing, including: dividing the individual image into at least two areas; based on any two consecutive frames of individual images, using the DeepSort tracking algorithm, matching the student behavioral states in the at least two areas, and obtaining the predicted state of the successfully matched student behavioral states in the at least two areas in the next frame.

[0011] In some embodiments, the YOLOv5 network model includes: an input end, a Backbone network, a Neck network, and a Head output end.

[0012] In some embodiments, it is necessary to continuously monitor and manage the individual behavior of each student in real time in the classroom video in the time dimension to extract each student image frame by frame, including: continuously monitoring and managing the image of each student in real time, and accumulating the number of frames of each detected behavior in chronological order to determine.

[0013] In some embodiments, a visual analysis is performed on the duration of each behavior, and the analysis results are uploaded to a server for teaching reference, including: uploading the analysis results to a server, and using a ReactJS framework to display the analysis results on a software display page.

[0014] In some embodiments, the real-time video data is stored in a video acquisition device with uninterrupted power supply and storage function.

[0015] In a second aspect, a real-time class inspection device based on video processing is provided, including: a video acquisition module, used to collect real-time video data of students' behavior in the classroom, extract individual images of students in the classroom frame by frame for behavior state recognition; wherein, a part of the individual images is used as a training set, and the other part is used as a test set, the training set is input into a YOLOv5 network model to train an individual behavior recognition model, and is verified by the test set; the body state of the students in the frame picture in the individual image is annotated, and converted into a txt format file required by YOLOv5, and the txt format file generated by each annotation is added to the training set; the txt format file is input into the YOLOv5 network model, and the training set is input into the YOLOv5 network model, and the training set is input into the YOLOv5 network model, and the training set is input into the YOLOv5 network model, and the training set is input into the YOLOv5 network model, and the training set is input into the YOLOv5 network model, and the training set is input into the YOLOv5 network model, and the training set is verified by the test set; ... In the network model, the test set is used for verification and optimization of parameters; the detection module is used to detect and track the individual image through the DeepSort algorithm according to the result of behavioral state recognition, and then compare the boundaries of the detection box to obtain the recognition results of looking up to listen to the class, looking down to read and write, and standing; the recognition module is used to extract each student image frame by frame from the classroom video under the time dimension of the time dimension where real-time continuous monitoring and management of each student's individual behavior is required, and input it into the student behavior recognition model for behavior recognition; the display module is used to count the duration of various behaviors of each student in the classroom video, and display the distribution of the duration of each student's various behaviors every day, perform visual analysis on the duration of each behavior, and upload the analysis results to the server for teaching reference.

[0016] In a third aspect, the present invention provides an electronic device, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned real-time class inspection method based on video processing is implemented.

[0017] In a fourth aspect, the present invention further provides a computer-readable storage medium, characterized in that program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the above-mentioned real-time class inspection method based on video processing.

[0018] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: First, through real-time behavior analysis, teachers can instantly understand students' attention and participation, so as to adjust teaching strategies in time. This instant feedback mechanism helps to improve the effectiveness of classroom teaching and ensure that students can better understand and master knowledge.

[0019] Second, by analyzing student behavior, teachers can identify which students are active in class and which students may need more attention. This attention can motivate students to participate more actively in class discussions and activities, and enhance their learning motivation.

[0020] Third, the system can generate detailed behavior analysis reports to help teachers and schools evaluate teaching effectiveness and provide data-based feedback. This evaluation mechanism can promote the professional development of teachers and improve their teaching ability.

[0021] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0023] Figure 1 A flowchart of real-time class inspection based on video processing provided by an embodiment of the present application is shown; Figure 2 is a schematic block diagram of a real-time class inspection device based on video processing provided in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application; Figure 4 It is a schematic diagram of an application scenario of an embodiment of the present application. DETAILED DESCRIPTION

[0024] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0026] This application provides a real-time classroom inspection method based on video processing. Figure 1 , which is a schematic diagram of the first embodiment of the present application. Figure 1 A first embodiment of the present application provides a real-time class inspection method based on video processing and is described in detail.

[0027] The present application provides a real-time class inspection method 100 based on video processing. The processing flow of the method 100 may include the following steps: Step S102: Acquire real-time video data of students' behaviors in the classroom, that is, collect real-time video data of students' behaviors in the classroom, extract individual images of students in the classroom frame by frame to identify the behavior status.

[0028] In some embodiments, real-time video data of students' behavior in class is collected, and individual images of students in the class are extracted frame by frame for behavior status recognition, including: using a part of the individual images as a training set and another part as a test set, inputting the training set into a YOLOv5 network model to train an individual behavior recognition model, and verifying it with the test set; annotating the body status of the students in the frame images in the individual images, and converting them into txt format files required by YOLOv5, and adding the txt format files generated by each annotation to the training set; inputting the txt format files into the YOLOv5 network model, and using the test set for verification to optimize parameters.

[0029] Step S104: Establish a student behavior recognition model, that is, based on the results of behavior state recognition, detect and track the individual image through the DeepSort algorithm, and then compare the boundaries of the detection frame to obtain the recognition results of looking up to listen to the class, looking down to read and write, and standing. In this way, by analyzing student behavior, teachers can identify which students are active in class and which students may need more attention. This kind of attention can motivate students to participate more actively in classroom discussions and activities and enhance their learning motivation.

[0030] In some embodiments, based on the result of behavioral state recognition, the individual image is detected and tracked by the DeepSort algorithm, and the recognition results obtained by comparing the detection frame boundaries are respectively looking up to listen to the class, looking down to read and write, and standing. This may include: dividing the individual image into at least two areas; according to any two consecutive frames of individual images, using the DeepSort tracking algorithm, matching the student behavioral states in the at least two areas, and obtaining the predicted state of the successfully matched student behavioral states in the at least two areas in the next frame. Through behavioral state recognition and real-time behavioral analysis, teachers can instantly understand students' attention and participation, so as to adjust teaching strategies in time. This instant feedback mechanism helps to improve the effectiveness of classroom teaching and ensure that students can better understand and master knowledge.

[0031] In some embodiments, the YOLOv5 network model includes: an input end, a Backbone network, a Neck network, and a Head output end.

[0032] Step S106: extracting images for behavior recognition, that is, extracting each student image frame by frame from the classroom video in the time dimension that requires real-time continuous monitoring and management of each student's individual behavior, and inputting the image into the student behavior recognition model for behavior recognition.

[0033] In some embodiments, the classroom video in the time dimension that requires real-time and continuous monitoring and management of each student's individual behavior is extracted frame by frame to obtain each student's image, which can include: real-time and continuous monitoring and management of each student's image, and the number of frames of each detected behavior must be accumulated in chronological order to determine.

[0034] Step S108: Visualize the analysis and upload the results, that is, count the duration of each student's various behaviors in the classroom video, and display the distribution of each student's daily behavior duration, perform a visual analysis of each behavior duration, and upload the analysis results to the server for teaching reference. In this way, the system can generate a detailed behavior analysis report to help teachers and schools evaluate teaching effectiveness and provide data-based feedback. This evaluation mechanism can promote the professional development of teachers and improve their teaching ability.

[0035] In some embodiments, a visual analysis is performed on the duration of each behavior, and the analysis results are uploaded to a server for teaching reference, which may include: uploading the analysis results to a server, and displaying the analysis results on a software display page using a ReactJS framework. In this way, by analyzing classroom behavior data, school administrators and teachers can identify problems and improvement points in teaching, thereby formulating more effective teaching plans and strategies. This data-driven decision support can improve the quality of education and promote teaching reform. The system can generate detailed behavior analysis reports to help teachers and schools evaluate teaching effectiveness and provide data-based feedback. This evaluation mechanism can promote the professional development of teachers and improve their teaching abilities.

[0036] In some embodiments, the real-time video data is stored in a video acquisition device with uninterrupted power supply and storage function.

[0037] Figure 2 2 is a schematic block diagram of a real-time class inspection device 200 based on video processing according to an exemplary embodiment. The real-time class inspection device 200 based on video processing includes: a video acquisition module 202, a detection module 204, a recognition module 206 and a display module 208, wherein: The video acquisition module 202 is used to collect real-time video data of students' behavior in the classroom, extract individual images of students in the classroom frame by frame for behavior status recognition; wherein, a part of the individual images is used as a training set and the other part is used as a test set, the training set is input into the YOLOv5 network model to train the individual behavior recognition model, and is verified by the test set; the body state of the students in the frame picture in the individual image is annotated and converted into a txt format file required by YOLOv5, and the txt format file generated by each annotation is added to the training set; the txt format file is input into the YOLOv5 network model, and the test set is used for verification to optimize the parameters.

[0038] The detection module 204 is used to detect and track the individual image through the DeepSort algorithm according to the result of the behavior state recognition, and then compare the boundaries of the detection frame to obtain the recognition results of raising the head to listen to the class, lowering the head to read and write, and standing.

[0039] The recognition module 206 is used to extract each student image frame by frame from the classroom video in the time dimension where real-time continuous monitoring and management of each student's individual behavior is required, and input the image into the student behavior recognition model for behavior recognition.

[0040] The display module 208 is used to count the duration of each student's various behaviors in the classroom video, and to display the distribution of the duration of each student's various behaviors every day, to perform a visual analysis of the duration of each behavior, and to upload the analysis results to the server for teaching reference.

[0041] Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device 300 may include a processor 3001 and a memory 3002. Optionally, the electronic device 300 may further include a transceiver 3003. The processor 3001, the memory 3002 and the transceiver 3003 may be connected, for example, via a communication bus. The memory 3002 stores computer-readable instructions, and when the computer-readable instructions are executed by the processor 3001, the steps of the real-time class inspection method based on video processing are implemented.

[0042] In a specific implementation, as an embodiment, the processor 3001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0043] In a specific implementation, as an embodiment, the electronic device 300 may also include multiple processors, such as Figure 3 3001 and processor 3004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0044] The memory 3002 is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor 3001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0045] The transceiver 3003 is used to communicate with a network device or a terminal device.

[0046] Optionally, the transceiver 3003 may include a receiver and a transmitter, wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0047] Optionally, the transceiver 3003 may be integrated with the processor 3001 , or may exist independently and be coupled to the processor 3001 via an interface circuit of the electronic device 300 , which is not specifically limited in the embodiment of the present invention.

[0048] It should be noted that Figure 3 The structure of the electronic device 300 shown in the figure does not constitute a limitation on the electronic device. The actual electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects of the electronic device 300 can refer to the technical effects of the above-mentioned method embodiment, which will not be repeated here.

[0049] like Figure 4 The application scenario diagram of the illustrated embodiment is a schematic diagram of an application scenario of the real-time class inspection method based on video processing provided by the embodiment of the present application. In the application scenario diagram, a terminal device 401 and a server 402 are included. Among them, the terminal device 401 and the server 402 communicate with each other through a communication network.

[0050] The terminal device 401 is an electronic device used by the target object, and the electronic device may be a personal computer, a mobile phone, a tablet computer, a notebook, an e-book reader, a vehicle-mounted terminal, etc. In addition, a client related to the real-time class inspection method based on video processing may be installed on the terminal device 401, and the client may be software (for example, an APP, a browser, etc.), or a web page, a small program, etc. By implementing the real-time class inspection method based on video processing on the terminal device 401, the related operations of the real-time class inspection method based on video processing are performed.

[0051] Server 402 may be an independent physical server or an edge device in the field of cloud computing. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, cloud functions, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (English name: Content Delivery Network, abbreviated as CDN), as well as big data and artificial intelligence platforms.

[0052] There is no restriction on the number of the terminal devices 401 and / or servers 402 .

[0053] It should be noted that the real-time class inspection method based on video processing in the embodiment of the present application can be executed by the terminal device 401 or the server 402 alone, or it can be executed by the terminal device 401 and the server 402 together. For example, when the terminal device 401 and the server 402 are executed together, the server 402 can provide the terminal device 401 with real-time video data of students' behavior in the classroom, and extract individual images of students in the classroom frame by frame for behavior state recognition. The terminal device 401 can detect and track the individual images according to the result of behavior state recognition to the server 402 through the DeepSort algorithm, and then compare the detection frame boundary to obtain the recognition results of head-up listening, head-down reading and writing, and standing respectively; the classroom video under the time dimension of the individual behavior of each student that needs to be monitored and managed in real time and continuously is extracted frame by frame, and the student behavior recognition model is input for behavior recognition; the various behavior durations of each student in the classroom video are counted, and the distribution of each behavior duration of each student every day is displayed, and each behavior duration is visually analyzed, and the analysis results are uploaded to the server for teaching reference.

[0054] In an exemplary embodiment, the present invention further provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the steps of the real-time class inspection method based on video processing as described above. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0055] An embodiment of the present invention further provides an electronic device, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the above-mentioned real-time class inspection method based on video processing is implemented.

[0056] An embodiment of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores program code, and the program code can be called by a processor to execute the above-mentioned real-time class inspection method based on video processing.

[0057] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0058] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0059] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0060] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0061] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0062] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0063] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0064] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0065] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0066] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0067] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0068] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A real-time class inspection method based on video processing, characterized in that: include: Collect real-time video data of students' behavior in class, extract individual images of students in class frame by frame to identify their behavior status; According to the results of behavioral state recognition, the individual images are detected and tracked by DeepSort algorithm, and then the recognition results are compared with the boundaries of the detection frame, which are head-up listening, head-down reading and writing, and standing. Extract each student image frame by frame from the classroom video in the time dimension where each student's individual behavior needs to be monitored and managed in real time and continuously, and input the image into the student behavior recognition model for behavior recognition; Count the duration of each student's various behaviors in the classroom video, and display the distribution of each student's various behavior durations every day. Perform a visual analysis of each behavior duration, and upload the analysis results to the server for teaching reference.

2. The real-time class inspection method based on video processing according to claim 1 is characterized in that: Collect real-time video data of students' behavior in class, extract individual images of students in the class frame by frame for behavior status recognition, including: Part of the individual images is used as a training set, and the other part is used as a test set. The training set is input into a YOLOv5 network model to train an individual behavior recognition model, and the training set is verified by the test set. Annotate the student's physical state in the frame image in the individual image and convert it into a txt format file required by YOLOv5, and add the txt format file generated by each annotation to the training set; Input the txt format file into the YOLOv5 network model, and use the test set to verify and optimize the parameters.

3. The real-time class inspection method based on video processing according to claim 2 is characterized in that: According to the results of behavioral state recognition, the individual image is detected and tracked by DeepSort algorithm, and the recognition results obtained by comparing the detection frame boundaries are head-up listening, head-down reading and writing, and standing, including: dividing the individual image into at least two regions; According to any two consecutive frames of individual images, the DeepSort tracking algorithm is used to match the student behavior states in the at least two regions to obtain the predicted state of the successfully matched student behavior states in the at least two regions in the next frame.

4. The real-time class inspection method based on video processing according to claim 3 is characterized in that: The YOLOv5 network model includes: an input end, a Backbone network, a Neck network and a Head output end.

5. The real-time class inspection method based on video processing according to claim 1 is characterized in that: Extract each student's image frame by frame from the classroom video in the time dimension where real-time continuous monitoring and management of each student's individual behavior is required, including: The real-time continuous monitoring manages the images of each pupil and the number of frames for each detected behavior shall be accumulated in time sequence.

6. The real-time class inspection method based on video processing according to claim 1 is characterized in that: Visually analyze the duration of each behavior, and upload the analysis results to the server for teaching reference, including: The analysis results are uploaded to the server and displayed on the software display page using the ReactJS framework.

7. The real-time class inspection method based on video processing according to any one of claims 1 to 6, characterized in that: The real-time video data is stored in a video acquisition device with uninterrupted power supply and storage function.

8. A real-time class inspection device based on video processing, characterized in that: include: The video acquisition module is used to collect real-time video data of students' behavior in the classroom, extract individual images of students in the classroom frame by frame for behavior state recognition; wherein, a part of the individual images is used as a training set, and the other part is used as a test set, the training set is input into the YOLOv5 network model to train the individual behavior recognition model, and is verified by the test set; the body state of the students in the frame pictures in the individual images is annotated, and converted into the txt format file required by YOLOv5, and the txt format file generated by each annotation is added to the training set; the txt format file is input into the YOLOv5 network model, and the test set is used for verification and parameter optimization; The detection module is used to detect and track the individual image through the DeepSort algorithm according to the result of the behavior state recognition, and then compare the boundaries of the detection frame to obtain the recognition results of looking up to listen to the class, looking down to read and write, and standing; A recognition module is used to extract each student image frame by frame from the classroom video in the time dimension where each student's individual behavior needs to be monitored and managed in real time and continuously, and input the image into the student behavior recognition model for behavior recognition; The display module is used to count the duration of each student's various behaviors in the classroom video, and to display the distribution of each student's various behavior durations every day, to conduct a visual analysis of each behavior duration, and to upload the analysis results to the server for teaching reference.

9. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the real-time class inspection method based on video processing as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the real-time class inspection method based on video processing as described in any one of claims 1 to 7.

Images