Classroom behavior analysis method, system, computer and storage medium

By combining teacher voice and student image data, using OpenPose algorithm and knowledge distillation technology to perform student behavior analysis, the problem of poor analysis results in the existing technology is solved, the accuracy of classroom behavior analysis and the quality of concentration evaluation are improved, and the Real-time teaching effect.

CN118823834BActive Publication Date: 2024-11-29ZIGUANG MODU EDUCATION TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411295919.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-18
Publication Date
2024-11-29
Estimated Expiration
2044-09-18

AI Technical Summary

Technical Problem

In the prior art, it is difficult to accurately judge the student's concentration through student posture analysis, it is impossible to distinguish between reading and daze in the book, and when looking up, it is impossible to distinguish between looking at the blackboard and daze in the blackboard, resulting in a low accuracy of concentration evaluation.

Method used

Combining teacher voice data and student image data, OpenPose algorithm is used for node detection, and teacher models and student models are built on cloud and edge devices through knowledge distillation algorithm to match behavioral features and remind them to determine whether students' behavior is abnormal.

Benefits of technology

It improves the accuracy of classroom behavior analysis and the quality of concentration evaluation, realizes real-time analysis, and enhances teaching effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118823834B_ABST
    Figure CN118823834B_ABST
Patent Text Reader

Abstract

The present invention provides a classroom behavior analysis method, system, computer and storage medium, the method comprising the following steps: extracting the teacher's voice data in real time based on a microphone to obtain the teacher's instructions; acquiring the student's image data based on a monitoring device, performing joint point detection on the image data, and obtaining the student's behavior characteristics; performing consistency matching based on the teacher's instructions and the behavior characteristics to determine whether the behavior characteristics are abnormal behavior characteristics; if the behavior characteristics are abnormal behavior characteristics, reminding the student based on the number. The teacher's voice data is extracted in real time based on the teacher's microphone to analyze and obtain the teacher's instructions, and then the student's behavior is analyzed according to the monitoring image, and by performing consistency matching, it is determined whether the student's behavior is abnormal, and the student's behavior is evaluated in combination with the teacher's instructions, thereby improving the analysis effect and the evaluation quality of the student's concentration.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of classroom analysis, and in particular to a classroom behavior analysis method, system, computer and storage medium. Background Art

[0002] By performing posture recognition on students' images in surveillance videos, students' behaviors can be analyzed to evaluate their concentration in class based on their behaviors.

[0003] In the prior art, since the information source of behavioral analysis only relies on the students' posture, when a student lowers his head, it is impossible to judge whether the student is in a daze or reading; when a student raises his head, it is impossible to judge whether the student is in a daze or looking at the blackboard, resulting in low accuracy of concentration evaluation and poor analysis effect. Summary of the invention

[0004] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a classroom behavior analysis method, system, computer and storage medium, aiming to solve the technical problem of poor analysis effect in the prior art.

[0005] In order to achieve the above object, in a first aspect, the present invention provides a classroom behavior analysis method, comprising the following steps:

[0006] Extracting the teacher's voice data in real time based on the microphone, and extracting text features from the voice data to obtain the teacher's instructions;

[0007] Acquire image data of students based on monitoring equipment, and number the students based on coordinate positions in the image data;

[0008] Performing joint point detection on the image data based on the OpenPose algorithm to obtain a posture skeleton model corresponding to the joint points, and analyzing and matching the posture skeleton model to obtain the behavioral characteristics of the student;

[0009] Based on the consistency matching between the teacher instruction and the behavior feature, determining whether the behavior feature is an abnormal behavior feature;

[0010] If the behavior characteristic is an abnormal behavior characteristic, the student is reminded based on the number.

[0011] According to one aspect of the above technical solution, the step of obtaining the behavioral characteristics of the student specifically includes:

[0012] A teacher model and a student model are constructed in the cloud and the edge device respectively by using a knowledge distillation algorithm, and joint point detection is performed on the image data based on the OpenPose algorithm by using the teacher model and the student model;

[0013] The joint point confidence prediction information obtained according to the teacher model is transmitted to the student model in the form of soft labels for guidance, so as to obtain the student's behavioral characteristics based on the student model.

[0014] According to one aspect of the above technical solution, the step of performing joint point detection on the image data based on the OpenPose algorithm through the teacher model and the student model specifically includes:

[0015] Performing real-time detection on the feature similarity of the teacher model and the student model to obtain the matching degree between the teacher model and the student model;

[0016] According to the difference between the soft label and the joint point confidence map of the student model, a loss function is constructed to balance the teacher model and the student model based on the loss function.

[0017] According to one aspect of the above technical solution, the step of obtaining the matching degree between the teacher model and the student model specifically includes:

[0018] The matching degree between the teacher model and the student model is calculated based on the following formula:

[0019] ;

[0020] ;

[0021] In the formula, is the matching degree, represents the similarity between the i-th pixel and the j-th pixel, and Respectively represent the similarity between the i-th pixel and the j-th pixel of the teacher model and the student model, represents the feature vector of the i-th pixel, represents the feature vector of the j-th pixel, W and H represent the width and height of the feature map respectively, and R is a real number.

[0022] According to one aspect of the above technical solution, the loss function The expression is as follows:

[0023] ;

[0024] ;

[0025] In the formula, is the difference, and They represent the confidence graphs of the kth joint point predicted by the teacher model and the student model, respectively, where K is the number of joint points, is the balance coefficient between the difference degree and the matching degree.

[0026] According to one aspect of the above technical solution, the step of performing joint point detection on the image data based on the OpenPose algorithm to obtain a posture skeleton model corresponding to the joint points specifically includes:

[0027] The image data is input into a human body posture estimation framework model, and the key point positions of the detection target are predicted through a feedforward network to obtain a two-dimensional confidence map;

[0028] The affinity of each joint point is calculated, and the confidence map and the affinity are analyzed, and each joint point is matched and connected to obtain a posture skeleton model corresponding to the joint point.

[0029] In a second aspect, the present application also provides a classroom behavior analysis system, comprising:

[0030] An instruction module is used to extract the teacher's voice data in real time based on the sound receiving microphone, and perform text feature extraction on the voice data to obtain the teacher's instruction;

[0031] A monitoring module, used for acquiring image data of students based on a monitoring device, and numbering the students based on coordinate positions in the image data;

[0032] A behavior module is used to detect joint points of the image data based on the OpenPose algorithm to obtain a posture skeleton model corresponding to the joint points, analyze and match the posture skeleton model, and obtain the behavior characteristics of the student;

[0033] A matching module, used for matching the teacher's instruction with the behavior feature to determine whether the behavior feature is an abnormal behavior feature;

[0034] A reminder module is used to remind the student based on the number if the behavior feature is an abnormal behavior feature.

[0035] According to one aspect of the above technical solution, the behavior module is specifically used for:

[0036] A teacher model and a student model are constructed in the cloud and the edge device respectively by using a knowledge distillation algorithm, and joint point detection is performed on the image data based on the OpenPose algorithm by using the teacher model and the student model;

[0037] The joint point confidence prediction information obtained according to the teacher model is transmitted to the student model in the form of soft labels for guidance, so as to obtain the student's behavioral characteristics based on the student model.

[0038] According to one aspect of the above technical solution, the behavior module is also used for:

[0039] Performing real-time detection on the feature similarity of the teacher model and the student model to obtain the matching degree between the teacher model and the student model;

[0040] According to the difference between the soft label and the joint point confidence map of the student model, a loss function is constructed to balance the teacher model and the student model based on the loss function.

[0041] According to one aspect of the above technical solution, the behavior module is also used for:

[0042] The matching degree between the teacher model and the student model is calculated based on the following formula:

[0043] ;

[0044] ;

[0045] In the formula, is the matching degree, represents the similarity between the i-th pixel and the j-th pixel, and Respectively represent the similarity between the i-th pixel and the j-th pixel of the teacher model and the student model, represents the feature vector of the i-th pixel, represents the feature vector of the j-th pixel, W and H represent the width and height of the feature map respectively, and R is a real number.

[0046] In a third aspect, an embodiment of the present application provides a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a classroom behavior analysis method as described in the first aspect is implemented.

[0047] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements a classroom behavior analysis method as described in the first aspect.

[0048] Compared with the prior art, the beneficial effects of the present invention are: the teacher's voice data is extracted in real time based on the teacher's receiving microphone to analyze and obtain the teacher's instructions, and then the student's behavior is analyzed according to the monitoring image, and it is determined whether the student's behavior is abnormal by matching it for consistency, and the student's behavior is evaluated in combination with the teacher's instructions, thereby improving the analysis effect and the evaluation quality of the student's concentration. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of the classroom behavior analysis method in the first embodiment of the present invention;

[0050] Figure 2 It is a structural block diagram of a classroom behavior analysis system in a second embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of the hardware structure of a computer in the third embodiment of the present application;

[0052] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0053] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0054] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0056] Embodiment 1

[0057] See also Figure 1 , shown is a classroom behavior analysis method in a first embodiment of the present invention, comprising the following steps:

[0058] Step S100, extracting the teacher's voice data in real time based on the receiving microphone, and performing text feature extraction on the voice data to obtain the teacher's instructions.

[0059] Specifically, in this embodiment, the above-mentioned sound pickup microphone can be a microphone that the teacher wears at any time, or is fixed on the podium and is used for picking up and amplifying sound.

[0060] Step S200, obtaining the image data of the students based on the monitoring device, and numbering the students based on the coordinate positions in the image data. In some application scenarios of this embodiment, the seat of each student is fixed, that is, the number corresponding to the position of each seat in the image corresponds to the student's student number, so as to facilitate the analysis and statistics of data in the smart classroom.

[0061] Step S300, performing joint point detection on the image data based on the OpenPose algorithm to obtain a posture skeleton model corresponding to the joint points, analyzing and matching the posture skeleton model to obtain the behavioral characteristics of the student.

[0062] For easy understanding, the OpenPose algorithm is an open source real-time multi-person pose estimation library developed by Carnegie Mellon University. It estimates the human body's pose by analyzing the key points of the human body in an image or video, identifying the various parts of the body, and inferring the human body's pose information.

[0063] Specifically, in this embodiment, the step of performing joint point detection on the image data based on the OpenPose algorithm to obtain a posture skeleton model corresponding to the joint points specifically includes:

[0064] The image data is input into a human body posture estimation framework model, and the key point positions of the detection target are predicted through a feedforward network to obtain a two-dimensional confidence map;

[0065] The affinity of each joint point is calculated, and the confidence map and the affinity are analyzed, and each joint point is matched and connected to obtain a posture skeleton model corresponding to the joint point.

[0066] It should be noted that the model trained by the OpenPose algorithm performs well in the recognition accuracy of human joints. However, the disadvantage is that in order to achieve good accuracy, the internal parameters of the OpenPose method are too complex and redundant, sacrificing high computational costs and large memory. In order to pursue excellent performance, the model can only be deployed on high-performance servers.

[0067] In some application scenarios of this embodiment, in order to reduce costs and consider the lightweight of the model, the step of obtaining the behavioral characteristics of the student specifically includes:

[0068] A teacher model and a student model are constructed in the cloud and the edge device respectively by using a knowledge distillation algorithm, and joint point detection is performed on the image data based on the OpenPose algorithm by using the teacher model and the student model;

[0069] The joint point confidence prediction information obtained according to the teacher model is transmitted to the student model in the form of soft labels for guidance, so as to obtain the student's behavioral characteristics based on the student model.

[0070] The implementation idea of ​​knowledge distillation is to train a larger teacher model based on the source data, and then convolute the output feature map to provide a knowledge source for a smaller student model. The student model is supervised and trained by a higher-precision teacher model to make the recognition accuracy of the two similar. In some application scenarios of this embodiment, due to the large scale of the teacher model and the relatively complex calculation amount, the teacher model is deployed on a server with a high-performance GPU for iterative training to obtain better human posture estimation accuracy; and the student model, due to its lightweight advantage, is deployed in a lower-performance edge device as the final training result model. It can be understood that the above-mentioned edge device, that is, the processor, is also used to receive and recognize teacher instructions to facilitate later behavior recognition.

[0071] In this embodiment, the teacher model transmits the joint point confidence prediction information corresponding to the soft label to the student model, so that the student model can obtain the semantic information required for high precision through a small amount of convolution.

[0072] Specifically, in this embodiment, the step of detecting joint points of the image data based on the OpenPose algorithm by using the teacher model and the student model specifically includes:

[0073] Performing real-time detection on the feature similarity of the teacher model and the student model to obtain the matching degree between the teacher model and the student model;

[0074] According to the difference between the soft label and the joint point confidence map of the student model, a loss function is constructed to balance the teacher model and the student model based on the loss function.

[0075] Furthermore, in this embodiment, the step of obtaining the matching degree between the teacher model and the student model specifically includes:

[0076] The matching degree between the teacher model and the student model is calculated based on the following formula:

[0077] ;

[0078] ;

[0079] In the formula, is the matching degree, represents the similarity between the i-th pixel and the j-th pixel, and Respectively represent the similarity between the i-th pixel and the j-th pixel of the teacher model and the student model, represents the feature vector of the i-th pixel, represents the feature vector of the j-th pixel, W and H represent the width and height of the feature map respectively, and R is a real number.

[0080] Furthermore, the above loss function The expression is as follows:

[0081] ;

[0082] ;

[0083] In the formula, is the difference, and They represent the confidence graphs of the kth joint point predicted by the teacher model and the student model, respectively, where K is the number of joint points, is the balance coefficient between the difference degree and the matching degree.

[0084] Step S400, performing consistency matching between the teacher instruction and the behavior feature to determine whether the behavior feature is an abnormal behavior feature.

[0085] It can be understood that when the teacher's instruction is identified as "look at the blackboard" through analysis of text features, and the above-mentioned behavioral feature analysis result is lowering the head, it can be determined that the student corresponding to the target has an inconsistent instruction state and is marked as abnormal behavior; when the teacher's instruction is identified as "read the textbook", and the above-mentioned behavioral feature analysis result is raising the head, this behavior is also marked as abnormal behavior.

[0086] Step S500, if the behavior feature is an abnormal behavior feature, remind the student based on the number. In this embodiment, the amount of data processing and analysis can be reduced through knowledge distillation, thereby achieving real-time analysis of the classroom. The reminder method can be to remind the teacher to remind the students to pay attention to the lecture, or to remind through the desktop device configured on each seat.

[0087] In summary, the classroom behavior analysis method in the above embodiment of the present invention extracts the teacher's voice data in real time based on the teacher's microphone to analyze and obtain the teacher's instructions, and then analyzes the student's behavior based on the monitoring image, and determines whether the student's behavior is abnormal by matching it for consistency. It combines the teacher's instructions to evaluate the student's behavior, improves the analysis effect, and the evaluation quality of the student's concentration, and can reduce the amount of data processing and analysis through knowledge distillation, thereby realizing real-time analysis of the classroom and improving the teaching quality.

[0088] Embodiment 2

[0089] The second embodiment of the present application also provides a kind of classroom behavior analysis system, and this system is used to realize described embodiment and preferred implementation mode, and has been described no more.As used below, the term "module", "unit", "subunit" etc. can realize the combination of software and / or hardware of predetermined function.Although the system described in the following embodiment is preferably realized with software, the realization of hardware, or the combination of software and hardware is also possible and conceived.

[0090] like Figure 2 As shown, the system includes: an instruction module 100, a monitoring module 200, a behavior module 300, a matching module 400 and a reminder module 500;

[0091] The instruction module 100 is used to extract the teacher's voice data in real time based on the sound receiving microphone, and perform text feature extraction on the voice data to obtain the teacher's instruction;

[0092] The monitoring module 200 is used to obtain image data of students based on monitoring equipment, and number the students based on the coordinate positions in the image data;

[0093] The behavior module 300 is used to detect joint points of the image data based on the OpenPose algorithm to obtain a posture skeleton model corresponding to the joint points, analyze and match the posture skeleton model, and obtain the behavior characteristics of the student;

[0094] The matching module 400 is used to perform consistency matching based on the teacher instruction and the behavior feature to determine whether the behavior feature is an abnormal behavior feature;

[0095] The reminder module 500 is used to remind the student based on the number if the behavior feature is an abnormal behavior feature.

[0096] Specifically, in this embodiment, the behavior module 300 is specifically used for:

[0097] A teacher model and a student model are constructed in the cloud and the edge device respectively by using a knowledge distillation algorithm, and joint point detection is performed on the image data based on the OpenPose algorithm by using the teacher model and the student model;

[0098] The joint point confidence prediction information obtained according to the teacher model is transmitted to the student model in the form of soft labels for guidance, so as to obtain the student's behavioral characteristics based on the student model.

[0099] Furthermore, in this embodiment, the behavior module 300 is also used for:

[0100] Performing real-time detection on the feature similarity of the teacher model and the student model to obtain the matching degree between the teacher model and the student model;

[0101] According to the difference between the soft label and the joint point confidence map of the student model, a loss function is constructed to balance the teacher model and the student model based on the loss function.

[0102] Preferably, in this embodiment, the behavior module 300 is further used for:

[0103] The matching degree between the teacher model and the student model is calculated based on the following formula:

[0104] ;

[0105] ;

[0106] In the formula, is the matching degree, represents the similarity between the i-th pixel and the j-th pixel, and Respectively represent the similarity between the i-th pixel and the j-th pixel of the teacher model and the student model, represents the feature vector of the i-th pixel, represents the feature vector of the j-th pixel, W and H represent the width and height of the feature map respectively, and R is a real number.

[0107] In summary, the classroom behavior analysis method in the above-mentioned embodiment of the present invention extracts the teacher's voice data in real time based on the teacher's microphone through the instruction module 100 to analyze and obtain the teacher's instructions, and then analyzes the student's behavior according to the monitoring image of the monitoring module 200 and the behavior module 300, and performs consistency matching through the matching module 400 to determine whether the student's behavior is abnormal. The student's behavior is evaluated in combination with the teacher's instructions, which improves the analysis effect and the evaluation quality of the student's concentration. The amount of data processing and analysis can be reduced through knowledge distillation, thereby realizing real-time analysis of the classroom and improving the teaching quality.

[0108] It should be noted that each module can be a functional module or a program module, and can be implemented by software or hardware. For modules implemented by hardware, each module can be located in the same processor; or each module can be located in different processors in any combination.

[0109] Embodiment 3

[0110] The third embodiment of the present application provides a computer. It can be understood that the principles mentioned in the classroom behavior analysis system in this embodiment correspond to the classroom behavior analysis method in the first embodiment of the present application. For details of the relevant principles not described, please refer to the first embodiment and will not be elaborated here.

[0111] The computer may include a processor 81 and a memory 82 storing computer program commands.

[0112] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0113] Among them, the memory 82 may include a large-capacity memory for data or commands. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0114] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program commands executed by the processor 81 .

[0115] The processor 81 implements any one of the classroom behavior analysis methods in the above embodiments by reading and executing computer program commands stored in the memory 82.

[0116] In some embodiments, the computer may further include a communication interface 83 and a bus 80. Figure 3 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0117] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0118] The bus 80 includes hardware, software or both, and couples the components of the computer to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0119] Embodiment 4

[0120] In combination with the classroom behavior analysis method in the above embodiments, the fourth embodiment of the present application provides a readable storage medium. The readable storage medium stores computer program commands; when the computer program commands are executed by a processor, any one of the classroom behavior analysis methods in the above embodiments is implemented.

[0121] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method of classroom behavior analysis, It is characterized in that The following steps are involved: Extracting the teacher's voice data in real time based on the microphone, and extracting text features from the voice data to obtain the teacher's instructions; Acquire image data of students based on monitoring equipment, and number the students based on coordinate positions in the image data; Performing joint point detection on the image data based on the OpenPose algorithm to obtain a posture skeleton model corresponding to the joint points, and analyzing and matching the posture skeleton model to obtain the behavioral characteristics of the student; Based on the consistency matching between the teacher instruction and the behavior feature, determining whether the behavior feature is an abnormal behavior feature; If the behavior characteristic is an abnormal behavior characteristic, reminding the student based on the number; The step of obtaining the student's behavioral characteristics specifically includes: A teacher model and a student model are constructed in the cloud and the edge device respectively by using a knowledge distillation algorithm, and joint point detection is performed on the image data based on the OpenPose algorithm by using the teacher model and the student model; The prediction information of the joint point confidence obtained by the teacher model is transmitted to the student model in the form of soft labels for guidance, so as to obtain the behavior characteristics of the student based on the student model; Among them, the step of performing joint point detection on the image data based on the OpenPose algorithm through the teacher model and the student model specifically includes: Performing real-time detection on the feature similarity of the teacher model and the student model to obtain the matching degree between the teacher model and the student model; Constructing a loss function according to the difference between the soft label and the joint point confidence map of the student model, so as to balance the teacher model and the student model based on the loss function; The step of obtaining the matching degree between the teacher model and the student model specifically includes: The matching degree between the teacher model and the student model is calculated based on the following formula: ; ; In the formula, is the matching degree, represents the similarity between the i-th pixel and the j-th pixel, and Respectively represent the similarity between the i-th pixel and the j-th pixel of the teacher model and the student model, represents the feature vector of the i-th pixel, represents the feature vector of the j-th pixel, W and H represent the width and height of the feature map respectively, and R is a real number.

2. The classroom behavior analysis method according to claim 1, It is characterized in that The loss function The expression is as follows: ; ; In the formula, is the difference, and They represent the confidence graphs of the kth joint point predicted by the teacher model and the student model, respectively, where K is the number of joint points, is the balance coefficient between the difference degree and the matching degree.

3. The classroom behavior analysis method according to any one of claims 1-2, It is characterized in that The steps of performing joint point detection on the image data based on the OpenPose algorithm to obtain a posture skeleton model corresponding to the joint points specifically include: The image data is input into a human body posture estimation framework model, and the key point positions of the detection target are predicted through a feedforward network to obtain a two-dimensional confidence map; The affinity of each joint point is calculated, and the confidence map and the affinity are analyzed, and each joint point is matched and connected to obtain a posture skeleton model corresponding to the joint point.

4. A classroom behavior analysis system, It is characterized in that include: An instruction module is used to extract the teacher's voice data in real time based on the sound receiving microphone, and perform text feature extraction on the voice data to obtain the teacher's instruction; A monitoring module, used for acquiring image data of students based on a monitoring device, and numbering the students based on coordinate positions in the image data; A behavior module is used to detect joint points of the image data based on the OpenPose algorithm to obtain a posture skeleton model corresponding to the joint points, analyze and match the posture skeleton model, and obtain the behavior characteristics of the student; A matching module, used for matching the teacher's instruction with the behavior feature to determine whether the behavior feature is an abnormal behavior feature; A reminder module, configured to remind the student based on the number if the behavior feature is an abnormal behavior feature; Wherein, the behavior module is specifically used for: A teacher model and a student model are constructed in the cloud and the edge device respectively by using a knowledge distillation algorithm, and joint point detection is performed on the image data based on the OpenPose algorithm by using the teacher model and the student model; The prediction information of the joint point confidence obtained by the teacher model is transmitted to the student model in the form of soft labels for guidance, so as to obtain the behavior characteristics of the student based on the student model; Wherein, the behavior module is also used for: Performing real-time detection on the feature similarity of the teacher model and the student model to obtain the matching degree between the teacher model and the student model; Constructing a loss function according to the difference between the soft label and the joint point confidence map of the student model, so as to balance the teacher model and the student model based on the loss function; The behavior module is also used to: The matching degree between the teacher model and the student model is calculated based on the following formula: ; ; In the formula, is the matching degree, represents the similarity between the i-th pixel and the j-th pixel, and Respectively represent the similarity between the i-th pixel and the j-th pixel of the teacher model and the student model, represents the feature vector of the i-th pixel, represents the feature vector of the jth pixel, W and H represent the width and height of the feature map respectively, and R is a real number.

5. A computer comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the classroom behavior analysis method according to any one of claims 1 to 3 is implemented.

6. A storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the classroom behavior analysis method described in any one of claims 1 to 3 is implemented.

Citation Information

Patent Citations

  • Real-time monitoring method, device and equipment for student discipline violation based on visual perception

    CN113298005A