Classroom scene-based student behavior identification method and identification device

Abstract

The embodiment of the invention relates to the technical field of communication, and discloses a student behavior recognition method and recognition device based on a classroom scene, and the method comprises the steps: analyzing a real-time classroom video stream, obtaining a target image of a target person having a preset classroom behavior, and if the face features analyzed from the target image meet a preset condition, determining that the target person has the preset classroom behavior; taking the analyzed face feature as a target face feature, and determining a character role of the target character according to the target face feature and a teacher information base; if the face feature analyzed from the target image does not meet the preset condition, analyzing human body information of the target image, obtaining a target human body feature of the target person, and determining a character role of the target person according to the target human body feature; and outputting the target image of which the character role is the target character of the student and the corresponding classroom behavior. According to the invention, role identification can be carried out according to the human body features, so that the classroom behaviors in the classroom can be effectively counted.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of target recognition, and in particular to a method and device for identifying student behavior based on a classroom scenario. Background Art

[0002] In the classroom observation project, visual algorithms are used to identify the behavior data of people in the picture to count the classroom behavior. During this process, the teacher will also walk around the classroom, which may be misjudged by the visual algorithm as students standing up or interacting on the stage, so it is necessary to distinguish the identity attributes of students and teachers.

[0003] The traditional solution is to collect the facial information of all teachers before the visual algorithm is deployed and run, analyze the behavioral data through the visual algorithm, obtain the facial information of the person, and compare the acquired facial information with the face library. If the similarity between the facial information and a certain face in the face library is greater than the threshold, it is judged as a teacher, otherwise it is a student.

[0004] The above method requires the collection of teachers' facial information in advance, which not only poses a risk of user privacy leakage, but also requires the collection of all teachers' facial information in advance, which is difficult to implement in practice and raises the threshold for product use. In addition, for people with their backs to the collection device, detailed facial information cannot be collected, and thus accurate identification cannot be performed through the collected facial information; for people whose faces are blocked, detailed facial information cannot be collected either. In these cases, the identity attributes of the person to whom the behavior data belongs cannot be accurately determined, and thus classroom behavior in the classroom cannot be effectively counted. Summary of the invention

[0005] The embodiment of the present invention provides a student behavior recognition method and recognition device based on a classroom scenario to solve the problem of user privacy leakage and the inability to perform role recognition when detailed facial information cannot be collected.

[0006] The purpose of the embodiment of the present invention is achieved through the following technical solutions:

[0007] In order to solve the above technical problems, in a first aspect, an embodiment of the present invention provides a method for identifying student behaviors based on a classroom scenario, comprising:

[0008] Analyze the real-time classroom video stream to obtain the target image of the target person who exhibits the preset classroom behavior;

[0009] Analyze the facial information of the target image, and if the facial features analyzed from the target image meet the preset conditions, use the analyzed facial features as the target facial features, and determine the character of the target person according to the target facial features and the teacher information database;

[0010] If the facial features analyzed from the target image do not meet the preset conditions, the human body information of the target image is analyzed to obtain the target human body features of the target person, and the character role of the target person is determined according to the target human body features;

[0011] Output the target image of the target person whose character is the student and the corresponding classroom behavior.

[0012] In one embodiment, if the facial features analyzed from the target image do not meet the preset conditions, analyzing the human body information of the target image to obtain the target human body features of the target person, and determining the character role of the target person according to the target human body features includes:

[0013] If the facial features analyzed from the target image do not meet the preset conditions, obtaining the target body features of the target person;

[0014] Obtaining a human body identifier corresponding to the target human body feature from a temporary classroom library, determining a face identifier corresponding to the human body identifier according to a mapping relationship, and obtaining the target face feature according to the face identifier;

[0015] Determine whether there is a teacher's facial feature matching the target facial feature in the teacher information database;

[0016] If it exists, the target person is a teacher;

[0017] If not present, the target person is a student.

[0018] In one embodiment, the process of constructing the mapping relationship includes:

[0019] Acquire a real-time classroom video stream, track and identify the classroom video stream, assign a tracking identifier to the tracked person, and acquire facial features and body features of the tracked person;

[0020] Assigning a face identifier to the face feature and assigning a body identifier to the body feature;

[0021] Associating the face identifier and the body identifier belonging to the same tracking identifier to obtain a mapping relationship;

[0022] The facial features carrying the facial identifier, the body features carrying the body identifier, and the mapping relationship are added to the temporary classroom library.

[0023] In one embodiment, after determining whether there is a teacher's facial feature matching the target facial feature in the teacher information database, the method further includes:

[0024] If the target person is a teacher, in the teacher information database, the time tag of the matched teacher's facial feature is updated to the current timestamp, wherein the time tag is used to represent the most recent time when the teacher's facial feature is identified as a teacher.

[0025] In one embodiment, the method for identifying student behavior based on classroom scenarios further includes:

[0026] Acquire real-time classroom video stream, track and identify the classroom video stream, obtain the facial features of the teacher in this class, and update the teacher information database according to the teacher facial features.

[0027] In one embodiment, updating the teacher information database according to the teacher's facial features includes:

[0028] Determine whether the teacher's facial features exist in the teacher information database;

[0029] If it does not exist in the teacher information database, then determine whether the teacher's facial features exist in the temporary teacher database;

[0030] If it does not exist in the temporary teacher database, the teacher's facial features are added to the temporary teacher database, and the number of records of the teacher's facial features is initialized to 1;

[0031] If it exists in the temporary teacher database, the number of records of the teacher's facial features will be increased by 1, and it will be determined whether the number of records is greater than the set number threshold. If it is greater than the set number threshold, the teacher's facial features will be added to the teacher information database.

[0032] In one embodiment, the real-time classroom video stream is obtained, and the classroom video stream is tracked and identified to obtain the facial features of the teacher in the class, including:

[0033] Acquire a real-time classroom video stream, track the characters appearing in the classroom video stream, and acquire facial features of the tracked characters;

[0034] Obtain the facial features that appear in the podium area for the longest time in the first N minutes of the classroom video stream and whose face quality score exceeds a set threshold, and determine whether to use the facial features as teacher facial features in combination with the human features corresponding to the facial features;

[0035] and / or,

[0036] Obtaining the facial features of the longest standing time in the classroom video stream and whose facial quality score exceeds a set threshold, and determining whether to use the facial features as teacher facial features in combination with the human features corresponding to the facial features;

[0037] and / or,

[0038] Acquire the facial features of the human body that moves most frequently in the classroom video stream and whose face quality scores exceed a set threshold, and determine whether to use the facial features as teacher facial features in combination with the human body features corresponding to the facial features.

[0039] In one embodiment, the determining whether to use the facial feature as the teacher's facial feature in combination with the human body feature corresponding to the facial feature includes:

[0040] Acquire the human body features corresponding to the facial features;

[0041] Determine whether the physical characteristics meet the teacher's height requirements and / or dress requirements;

[0042] If the height requirement and / or dress requirement of a teacher is not met, the facial features are not the facial features of a teacher;

[0043] If the height requirement and / or dress requirement for teachers are met, the facial features are used as teacher facial features.

[0044] In one embodiment, the teacher information database stores teacher facial features and time tags corresponding to the teacher facial features;

[0045] The updating process of the teacher information database also includes:

[0046] For the teacher facial features existing in the teacher information database, determine whether the time difference between the current time and the time tag of the teacher facial features is greater than a set time threshold;

[0047] If it is greater than the set time threshold, the corresponding teacher facial features will be deleted from the teacher information database.

[0048] In one embodiment, the teacher information database also stores the algorithm version number corresponding to the teacher's facial features;

[0049] The updating process of the teacher information database also includes:

[0050] Obtaining the current algorithm version number for performing facial feature recognition on the classroom video stream;

[0051] For the teacher facial features existing in the teacher information database, determining whether the current algorithm version number is compatible with the algorithm version number of the teacher facial features;

[0052] If they are incompatible, the corresponding teacher facial features will be deleted from the teacher information database.

[0053] In one embodiment, the preset classroom behaviors include: one or more of: raising hands, standing up, interacting on the podium, and lying on the table.

[0054] In order to solve the above technical problems, in a second aspect, an embodiment of the present invention provides a student behavior recognition device based on a classroom scene, comprising:

[0055] The acquisition module is used to obtain real-time classroom video stream;

[0056] The analysis module is used to analyze the real-time classroom video stream to obtain a target image of a target person who has a preset classroom behavior; and analyze the facial information of the target image. If the facial features analyzed from the target image meet the preset conditions, the analyzed facial features are used as the target facial features. If the facial features analyzed from the target image do not meet the preset conditions, the human body information of the target image is analyzed to obtain the target human body features of the target person;

[0057] The determination module is used to determine the character role of the target person according to the target person's facial features and the teacher information database; or, to determine the character role of the target person according to the target person's body features;

[0058] The output module is used to output the target image of the target person whose character is a student and the corresponding classroom behavior.

[0059] In order to solve the above technical problems, in a third aspect, an embodiment of the present invention provides a student behavior recognition device, comprising:

[0060] at least one processor; and,

[0061] a memory communicatively connected to the at least one processor; wherein,

[0062] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the student behavior identification method based on classroom scenarios described in the first aspect.

[0063] To solve the above technical problems, in a fourth aspect, an embodiment of the present invention provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the student behavior recognition method based on classroom scenarios described in the first aspect.

[0064] Compared with the prior art, the beneficial effect of the present invention is that in the present invention, the teacher information database does not directly store teacher face pictures, but stores teacher face features, thereby avoiding the risk of information leakage. When the face features that meet the preset conditions cannot be obtained, the role can be identified based on the body features, and the situation where the person is facing away from the collection device or the person's face is blocked can be dealt with. The role of the person to whom the classroom behavior belongs can be effectively identified, and the classroom behavior in the classroom can be effectively counted. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise specified, the figures in the drawings do not constitute proportional limitations.

[0066] Figure 1 is a scene schematic diagram of a method for identifying student behavior based on a classroom scene provided by an embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of another method for identifying student behavior based on a classroom scenario provided by an embodiment of the present invention;

[0068] Figure 3 It is a flow chart of a method for identifying student behavior based on classroom scenarios provided by an embodiment of the present invention;

[0069] Figure 4 is a flow chart of another method for identifying student behavior based on classroom scenarios provided by an embodiment of the present invention;

[0070] Figure 5 The embodiment of the present invention provides Figure 3 Specific flow diagram of step 40;

[0071] Figure 6 is a schematic diagram of a human body detection frame provided by an embodiment of the present invention;

[0072] Figure 7 The embodiment of the present invention provides Figure 4 A first specific flow chart of step 10;

[0073] Figure 8 The embodiment of the present invention provides Figure 4 A second specific flow chart of step 10;

[0074] Fig. 9 The embodiment of the present invention provides Figure 4 A third specific flow diagram of step 10 in FIG.

[0075] Fig.10 The embodiment of the present invention provides Figure 4 A fourth specific flow chart of step 10 in FIG.

[0076] Fig.11 The embodiment of the present invention provides Figure 4 A fifth specific flow chart of step 10;

[0077] Fig.12 The embodiment of the present invention provides Figure 4 A sixth specific flow chart of step 10;

[0078] Fig.13 It is a structural schematic diagram of a student behavior recognition device based on a classroom scene provided by an embodiment of the present invention;

[0079] Fig.14 It is a structural schematic diagram of another student behavior recognition device based on classroom scenarios provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0081] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0082] It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, and all are within the protection scope of the present invention. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flow chart.

[0083] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0084] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0085] The traditional solution is to collect all teachers’ facial information before the visual algorithm is deployed and run, analyze the behavioral data through the visual algorithm, obtain the person’s facial information, and compare the acquired facial information with the facial database. If the facial information and a certain face in the facial database have a similarity greater than a threshold, it is determined to be a teacher, otherwise it is a student. The aforementioned method requires collecting teachers’ facial information in advance, which not only poses a risk of user privacy leakage, but also requires collecting all teachers’ facial information in advance. The actual implementation is difficult and the threshold for product use is raised. In addition, for people with their backs to the collection device, the corresponding facial information cannot be collected; for people with their faces blocked, detailed facial information cannot be collected. In these cases, the identity attributes of the person to whom the behavioral data belongs cannot be determined, and thus the classroom behavior in the classroom cannot be effectively counted.

[0086] In order to facilitate understanding of this solution, the scenario to which the embodiment of the present invention is applicable is first described. In a classroom scenario, such as Figure 1 A first acquisition device is deployed at the front of the classroom to obtain the classroom video stream. In this case, only one acquisition device needs to be deployed in the classroom, which is low in cost. However, the teacher's facial features and body features can only be collected when the teacher walks down from the podium and faces the first acquisition device.

[0087] In another classroom scenario, Figure 2 , a first acquisition device is deployed at the front of the classroom. The first acquisition device shoots from the podium to the back of the classroom, mainly capturing student behavior; a second acquisition device is deployed at the back of the classroom, shooting from the back of the classroom to the podium, mainly capturing teacher behavior. The first acquisition device mainly captures classroom video streams containing students, and the second acquisition device mainly captures classroom video streams containing teachers. In this case, two acquisition devices need to be deployed in the classroom, which is costly. When collecting teacher information, there is no need to impose too many restrictions on the teacher's location. If the teacher faces the second acquisition device, the teacher's facial features and body features can be collected through the second acquisition device; if the teacher faces the first acquisition device, the teacher's facial features and body features can also be collected through the first acquisition device.

[0088] In the embodiment of the present invention, for Figure 1In the scenario, the first acquisition device is used to collect the classroom video stream for classroom analysis, and it is also necessary to obtain the teacher's facial features and body features from the classroom video stream; Figure 2 In the scenario, the classroom video stream used for classroom analysis comes from the first acquisition device, and the video stream acquired by the second acquisition device is mainly used to acquire the facial features and body features of the teacher.

[0089] In actual application scenarios, whether Figure 1 The scene shown is still Figure 2 The scenario shown is applicable to the student behavior recognition method based on classroom scenario of the embodiment of the present invention. The difference is that in each class, the judgment rule for determining which person is the teacher is changed, which will be specifically explained in the following embodiments.

[0090] Embodiment 1:

[0091] To solve the above problems, see Figure 3 This embodiment provides a method for identifying student behavior based on a classroom scenario, including:

[0092] Step 20: Analyze the real-time classroom video stream to obtain the target image of the target person who exhibits the preset classroom behavior.

[0093] The preset classroom behaviors include: one or more of: raising hands, standing up, interacting on the podium, and lying on the table.

[0094] Assuming that the teacher walks down from the podium and stands in the student area of ​​the classroom, it will also be judged that the preset classroom behavior has occurred; or, if the teacher walks back and forth in the classroom, it is also easy to be judged that the preset classroom behavior has occurred. It is necessary to judge the role of the target person who exhibits the preset classroom behavior, and only filter out students with the preset classroom behavior, so as to count the classroom teaching situation.

[0095] In actual application scenarios, if the facial features of the target person collected meet the preset conditions, the role of the target person is judged based on the facial features; if the facial features of the target person collected do not meet the preset conditions, for example, the face is blocked, or the person is facing away from the collection device, then the facial features collected cannot meet the preset conditions. In this case, the role of the target person is judged based on the body features. The preset conditions are set according to actual conditions and are not specifically limited here. For example, the preset conditions can be a preset face quality score. If the facial features of the target person collected are greater than the preset face quality score, it means that the facial features of the target person collected meet the preset conditions, otherwise, they do not meet the preset conditions.

[0096] In this embodiment, the face of the target person is analyzed. If the facial features of the target person analyzed meet the preset conditions, step 30 is executed; if the facial features analyzed from the target image do not meet the preset conditions, step 40 is executed.

[0097] Step 30: Analyze the facial information of the target image. If the facial features analyzed from the target image meet the preset conditions, use the analyzed facial features as target facial features, and determine the character role of the target person based on the target facial features and the teacher information database.

[0098] In this embodiment, if the target facial features that meet the conditions can be obtained, it is determined whether there are teacher facial features that match the target facial features in the teacher information database; if so, the target person is a teacher, and in the teacher information database, the time tag of the matched teacher facial features is updated to the current timestamp, wherein the time tag is used to represent the most recent time when the teacher facial features are identified as a teacher. In actual use, the teacher information database can be updated according to the time tag. If not, the target person is a student.

[0099] Step 40: If the facial features analyzed from the target image do not meet the preset conditions, the human body information of the target image is analyzed to obtain the target human body features of the target person, and the character role of the target person is determined according to the target human body features.

[0100] In this embodiment, if the facial features analyzed from the target image do not meet the preset conditions, the target human features of the target person are obtained; the human body identifier corresponding to the target human features is obtained from the temporary classroom library, the facial identifier corresponding to the human body identifier is determined according to the mapping relationship, and the target facial features are obtained according to the facial identifier; it is determined whether there are teacher facial features matching the target facial features in the teacher information library; if so, the target person is a teacher; if not, the target person is a student.

[0101] In one embodiment, if the target person is a teacher, in the teacher information database, the time tag of the matched teacher's facial features is updated to the current timestamp, wherein the time tag is used to represent the most recent time when the teacher's facial features are identified as a teacher. In actual use, the teacher information database can be updated according to the time tag.

[0102] In one embodiment, a temporary classroom library is initialized for each class, and the temporary classroom library is used to store the human body features and facial features identified in this class, so as to find the corresponding facial features based on the human body features, and thus deal with the situation where the facial features of the target person cannot be obtained.

[0103] Step 50: Output the target image of the target person whose character is a student and the corresponding classroom behavior.

[0104] In this embodiment, target persons whose characters are teachers are filtered out, and only target persons whose characters are students and corresponding classroom behaviors are retained, so as to conduct classroom teaching analysis.

[0105] In the embodiment of the present invention, the teacher information database does not directly store teacher face pictures, but stores teacher face features to avoid the risk of information leakage. When the face features cannot be obtained, the role is identified based on the body features, which can deal with the situation where the person is facing away from the collection device or the face of the person is blocked, and can effectively identify the role of the person to whom the classroom behavior belongs, and then effectively count the classroom behavior in the classroom.

[0106] See also Figure 4 , before step 20, the following step 10 is also included. Step 10 will be executed in each class to obtain the facial features of the teacher in this class, and then continuously enrich it.

[0107] Step 10: Obtain a real-time classroom video stream, track and identify the classroom video stream, obtain the facial features of the teacher in this class, and update the teacher information database based on the teacher's facial features.

[0108] In this embodiment, the teacher information database does not directly store teacher face images to avoid information leakage risks. The teacher information database stores teacher facial features, but when it is first used, the teacher information database is empty. Each class will determine which person is the teacher of the class according to the teacher's judgment rules, and obtain the teacher's facial features and body features. After multiple classroom iterations, the facial features of people who have been judged as teachers multiple times will be added to the teacher information database, so that the teacher information database is constantly filled with the actual classroom teaching situation.

[0109] Among them, the facial feature represents the entire facial information, is output by the face recognition model, and can be a tensor of dimension (512, 1).

[0110] In this embodiment, the teacher's facial features are obtained based on the real-time classroom video stream, and the teacher's facial features will be added to the teacher information database. In this way, the teacher information database is continuously filled with the actual classroom teaching situation, without the need for pre-collection, which is easy to execute.

[0111] In one embodiment, see Figure 5 In step 40, the process of constructing the mapping relationship includes:

[0112] Step 401: obtain a real-time classroom video stream, track and identify the classroom video stream, assign a tracking identifier to the tracked person, and obtain the facial features and body features of the tracked person.

[0113] In one embodiment, at the beginning of each class, the classroom video stream is tracked and identified, a tracking identifier is assigned to the tracked person, and the facial features and body features of the tracked person are obtained. In target detection, the tracking identifier is usually used to identify and track the target in the video sequence. When the target moves in consecutive frames, the tracking algorithm attempts to match and track the target between different frames and assigns a unique tracking ID to each target.

[0114] In one embodiment, due to factors such as the target's movement speed, occlusion, and camera angle change, the target's tracking mark may be misaligned or offset between different frames. In teaching scenes such as elementary schools and middle schools, there is a large difference between the teacher's dress and the student's dress, and there is also a large difference between the teacher's height and the student's height. In a class, the difference between the teacher's body features and the student's body features is generally large. When performing feature recognition, the teacher's body features can be determined. In order to ensure the accuracy of teacher face recognition, the teacher's face features can be corrected based on the teacher's body features.

[0115] Specifically, if the same teacher's body feature corresponds to two different facial features, the subsequent frame images are tracked and identified to obtain the facial features corresponding to the teacher's body feature, and the facial features that appear more times are used as the teacher's facial features.

[0116] Specifically, for Figure 2 The scene shown, combined with Figure 6 The algorithm module analyzes two classroom video streams at the same time, identifies the human body frame, behavior category, and head frame (yolox multi-category target detection) of the characters from the two classroom video streams, and calculates the IOU (full name: Intersection over Union) of the previous and next frames based on the coordinate information of the human body frame. When IOU> the set threshold (for example, 0.8 or other values), the characters in the human body frames of the previous and next frames are regarded as the same person and assigned the same tracking identifier; since the teacher is standing on the podium, the height at which he is located will be relatively high. Therefore, for the human body detected by the second acquisition device, only the three highest human body frames above the image can be focused on, and the others can be ignored. The detected face and human body frames need to undergo quality inspection to obtain a frame of image with the highest quality of a face or human body with a certain tracking identifier within a specified time range for feature extraction.

[0117] In one embodiment, the head orientation component is detected, the frontal part of the face is retained, low-quality faces are initially filtered out, the current face quality score (0-1.0) is detected through a face quality model, and low-quality faces such as occlusion, blur or incomplete facial key points are further filtered out to obtain the highest quality frame within a specified time range for extracting facial features.

[0118] In one embodiment, the ratio of the human body frame and the detection frame is calculated, the human body frames with a ratio less than 0.5 are filtered, the key points of the human body are detected, and the key human bodies including the left shoulder, right shoulder, left elbow, and right elbow are retained to obtain human body features.

[0119] Step 402: assigning a face identifier to the face feature and assigning a body identifier to the body feature.

[0120] In one embodiment, for the human body corresponding to the teacher, a face identifier is assigned to the facial features, and a human body identifier is assigned to the human body features.

[0121] In one embodiment, for the human body corresponding to the student, a face identifier may be assigned to the face feature, and a human body identifier may be assigned to the human body feature. Alternatively, a face identifier may be assigned only to the face feature.

[0122] Among them, the face identifier Facei,i can be gradually increased from 1, and the body identifier Bodyj,j can also be gradually increased from 1.

[0123] Step 403: Associating the face identifier and the body identifier belonging to the same tracking identifier to obtain a mapping relationship.

[0124] For example, for a human body with tracking identifier T1, its corresponding face identifier is Face1, and its body identifier is Body1. Face1-Body1 is associated to obtain a mapping relationship. When the acquired face features do not meet the preset conditions, the face features can be obtained based on the body features and the mapping relationship, and then the role can be determined through the teacher information database.

[0125] Step 404: Add the facial features carrying the facial identifier, the body features carrying the body identifier, and the mapping relationship to the temporary classroom library.

[0126] In the temporary classroom library, not only the teacher's facial features, body features and corresponding mapping relationships are stored, but also the students' facial features, body features and corresponding mapping relationships are stored.

[0127] Under normal circumstances, in a class, most students sit in their seats and do not move around at will, so the tracking identifier corresponding to each student remains unchanged. Therefore, after recognizing that a student has performed a preset classroom behavior, the student's facial features can be obtained from the temporary classroom library based on the tracking identifier corresponding to the student, without the need to recognize the student's facial features again, which can save the time required for facial feature recognition and improve analysis efficiency.

[0128] In one embodiment, the tracking identifier, the face identifier, and the body identifier may be associated to obtain a mapping relationship. In this way, when performing classroom behavior analysis, if a target person who has performed a preset classroom behavior is identified, the body features or face features of the target person are not identified again, but the tracking identifier of the target person is obtained, and the face features corresponding to the tracking identifier are obtained based on the tracking identifier and the mapping relationship, and then the target face features of the target person are obtained, and the role of the target person is determined based on the target face features and the teacher information database.

[0129] In actual application scenarios, different judgment rules need to be set according to the deployment of the collection device to determine which person is the teacher. The following is an explanation based on different scenarios.

[0130] Judgment rule 1:

[0131] for Figure 2 In the scenario, the teacher can still collect the teacher's facial features when he / she is standing on the podium. Generally, at the beginning of a class, the teacher will stand on the podium. Figure 7 In step 10, the real-time classroom video stream is obtained, and the classroom video stream is tracked and identified to obtain the teacher's facial features of the class, including:

[0132] Step 101a: Acquire a real-time classroom video stream, track the characters appearing in the classroom video stream, and acquire facial features of the tracked characters.

[0133] Step 102a: Obtain the facial features that appear in the podium area for the longest time in the first N minutes of the classroom video stream and whose face quality score exceeds a set threshold, and determine whether to use the facial features as teacher facial features in combination with the human body features corresponding to the facial features.

[0134] There is no specific limit for the first N minutes, which depends on the actual situation.

[0135] Judgment rule 2:

[0136] In a class, the teacher's standing time is the longest, so the person who stands the longest in the first N minutes can be regarded as the teacher. Figure 8In step 10, the real-time classroom video stream is obtained, and the classroom video stream is tracked and identified to obtain the teacher's facial features of the class, including:

[0137] Step 101b: obtain a real-time classroom video stream, track the characters appearing in the classroom video stream, and obtain facial features of the tracked characters.

[0138] Step 102b: Obtain the facial features with the longest standing time in the classroom video stream and whose face quality score exceeds a set threshold, and determine whether to use the facial features as teacher facial features in combination with the body features corresponding to the facial features.

[0139] Judgment rule three:

[0140] In a class, the teacher's position moves most frequently, so the person who moves most frequently in the first N minutes can be regarded as the teacher. Fig. 9 In step 10, the real-time classroom video stream is obtained, and the classroom video stream is tracked and identified to obtain the teacher's facial features of the class, including:

[0141] Step 101c: obtain a real-time classroom video stream, track the characters appearing in the classroom video stream, and obtain facial features of the tracked characters.

[0142] Step 102c: Obtain the facial features of the human body that moves most frequently in the classroom video stream and whose face quality scores exceed a set threshold, and determine whether to use the facial features as teacher facial features in combination with the human body features corresponding to the facial features.

[0143] Usually, there are significant differences between teachers' attire and students' attire. Therefore, in the aforementioned judgment rules one to three, the judgment of whether to use the facial features as teacher facial features in combination with the body features corresponding to the facial features includes: obtaining the body features corresponding to the facial features; judging whether the body features meet the height requirements and / or dress requirements of teachers; if they do not meet the height requirements and / or dress requirements of teachers, then the facial features are not considered as teacher facial features; if they meet the height requirements and / or dress requirements of teachers, then the facial features are considered as teacher facial features.

[0144] In actual application scenarios, the above-mentioned judgment rules 1 to 3 are applicable to Figure 2 In the scenario, both judgment rules 2 and 3 are applicable Figure 1 scene.

[0145] In this embodiment, on the one hand, it is necessary to add newly recognized teachers to the teacher information database according to the actual teaching situation; on the other hand, it is also necessary to delete expired or inapplicable facial features in the teacher information database according to the actual teaching situation. The following specifically introduces the updating method of the teacher information database.

[0146] Update method 1:

[0147] See also Fig.10 In step 10, updating the teacher information database according to the teacher's facial features includes:

[0148] Step 101d: Determine whether the teacher's facial features exist in the teacher information database.

[0149] If it does not exist in the teacher information database, it means that it is a newly identified teacher facial feature or a feature that exists in the temporary teacher database, and step 102d needs to be executed. If it exists in the teacher information database, it means that it is a teacher facial feature that has been identified before, and no corresponding operation is required.

[0150] Step 102d: If it does not exist in the teacher information database, determine whether the teacher's facial features exist in the temporary teacher database.

[0151] Step 103d: If the teacher's facial feature does not exist in the temporary teacher database, then the teacher's facial feature is added to the temporary teacher database, and the number of records of the teacher's facial feature is initialized to 1.

[0152] Step 104d: If it exists in the temporary teacher database, the number of records of the teacher's facial features is increased by 1, and it is determined whether the number of records is greater than the set number threshold. If it is greater than the set number threshold, the teacher's facial features are added to the teacher information database.

[0153] There is no limit to the number of times the threshold is set, for example, it can be 5 times.

[0154] In this embodiment, the teacher's facial features that are recognized for the first time are first added to the temporary teacher database, and the teacher's facial features are added to the teacher information database only when the number of records of the teacher's facial features is greater than a set threshold. This can prevent misjudged teacher facial features from being added to the teacher information database, thereby improving the accuracy of role determination through the teacher information database.

[0155] Update method 2:

[0156] In one embodiment, the teacher information database stores teacher facial features and time tags corresponding to the teacher facial features, and the teacher information database can be updated according to the time tags.

[0157] Since the teacher information database is automatically generated by the algorithm, there is a certain probability of false detection. Therefore, an expiration mechanism for teacher facial features is added. For example, when a teacher's facial features have not been matched for a long time, it may be caused by the algorithm misjudging it as a teacher. Therefore, it is necessary to remove the teacher's facial features to reduce the possibility of false judgment.

[0158] See also Fig.11 , the updating process of the teacher information database further includes:

[0159] Step 101e: For the teacher facial features existing in the teacher information database, determine whether the time difference between the current time and the time tag of the teacher facial features is greater than a set time threshold.

[0160] Step 102e: If it is greater than the set time threshold, the corresponding teacher facial features are deleted from the teacher information database.

[0161] As can be seen from the above description, in the teacher information database, the time tag of the currently matched teacher's facial features will be updated to the current timestamp, and a time threshold can be set to delete the teacher's facial features that have not been matched for a long time. For example, the time threshold is 1 month or longer, which can be determined according to the actual situation.

[0162] In this way, not only can the misjudged teacher facial features be eliminated, but it can also ensure that the teacher facial features stored in the teacher information database are teachers who teach more frequently, while teachers who teach less frequently or have left are deleted to free up memory.

[0163] Update method three:

[0164] In one embodiment, the teacher information database also stores the algorithm version number corresponding to the teacher's facial features.

[0165] In actual use, when the algorithm is updated, the eigenvalues ​​output by the model are inconsistent with the eigenvalues ​​output by the old algorithm. The meanings represented by the eigenvalues ​​are different, and correct matching cannot be achieved. They also need to be eliminated to reduce the possibility of misjudgment.

[0166] See also Fig.12 , the updating process of the teacher information database further includes:

[0167] Step 101f: Obtain the current algorithm version number for performing facial feature recognition on the classroom video stream.

[0168] Step 102f: For the teacher facial features existing in the teacher information database, determine whether the current algorithm version number is compatible with the algorithm version number of the teacher facial features.

[0169] Step 103f: If they are not compatible, the corresponding teacher facial features are deleted from the teacher information database.

[0170] In one embodiment, if the current algorithm version number is equal to the algorithm version number of the teacher's facial features, it is determined to be incompatible, and the corresponding teacher's facial features are deleted. Then, the teacher's facial features are re-acquired according to the aforementioned method for acquiring the teacher's facial features, thereby achieving the effect of automatically updating the teacher information database.

[0171] Embodiment 2:

[0172] Based on the above-mentioned embodiment 1, this embodiment provides a student behavior recognition device based on a classroom scene, including:

[0173] The acquisition module is used to obtain real-time classroom video streams; it is also used to track and identify the classroom video streams, obtain the facial features of the teacher in this class, and update the teacher information database based on the teacher's facial features.

[0174] The analysis module is used to analyze the real-time classroom video stream to obtain a target image of a target person who exhibits preset classroom behavior; and to analyze the facial information of the target image. If the facial features analyzed from the target image meet preset conditions, the analyzed facial features are used as target facial features. If the facial features analyzed from the target image do not meet the preset conditions, the human body information of the target image is analyzed to obtain the target human body features of the target person.

[0175] The determination module is used to determine the character role of the target person according to the target person's facial features and the teacher information database; or, to determine the character role of the target person according to the target person's body features;

[0176] The output module is used to output the target image of the target person whose character is a student and the corresponding classroom behavior.

[0177] For the specific implementation process of each module, please refer to Example 1, which will not be repeated here.

[0178] In an embodiment of the present invention, the teacher information database does not directly store teacher face images, but stores teacher face features to avoid the risk of information leakage. The teacher's face features are obtained according to the real-time classroom video stream, and the teacher's face features will be added to the teacher information database, so that the teacher information database is continuously filled with the actual classroom teaching situation, without the need for pre-collection, which is easy to execute. In addition, when the facial features cannot be obtained, the role is identified according to the body features, which can cope with the situation where the person's back is facing the collection device or the person's face is blocked, and the role of the person to whom the classroom behavior belongs can be effectively identified, and then the classroom behavior in the classroom can be effectively counted.

[0179] Embodiment 3:

[0180] Based on the above-mentioned embodiments 1 to 2, Fig.13 As shown, this embodiment also provides a student behavior identification device, including: at least one processor 21; and a memory 22 communicatively connected to the at least one processor 21; wherein the memory 22 stores instructions executable by the at least one processor 21, and the instructions are executed by the at least one processor 21 so that the at least one processor 21 can execute the student behavior identification method based on the classroom scenario described in the aforementioned embodiment.

[0181] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.13 The example of connecting through bus is taken in the following.

[0182] The memory 22 is a non-volatile computer-readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the student behavior recognition method based on classroom scenes in the embodiment of the present invention. The processor 21 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, the student behavior recognition method based on classroom scenes in the above method embodiment is realized. Specifically, it includes: analyzing the real-time classroom video stream to obtain a target image of a target person who has a preset classroom behavior; analyzing the facial information of the target image, if the facial features analyzed from the target image meet the preset conditions, then using the analyzed facial features as the target facial features, and determining the character role of the target person according to the target facial features and the teacher information library; if the facial features analyzed from the target image do not meet the preset conditions, then analyzing the human body information of the target image, obtaining the target human body features of the target person, and determining the character role of the target person according to the target human body features; outputting the target image of the target person whose character is a student and the corresponding classroom behavior.

[0183] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the student behavior recognition device, etc. In addition, the memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the student behavior recognition device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0184] The one or more modules are stored in the memory 22, and when executed by the one or more processors 21, execute the student behavior identification method based on the classroom scene in any of the above-mentioned method embodiments, for example, execute the method steps of the student behavior identification method based on the classroom scene described above.

[0185] The above product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0186] Furthermore, the student behavior recognition device may also include: Wi-Fi device 23, display screen 24, Bluetooth device 25, audio circuit 26, power system 27, peripheral interface 28, sensor module 29, data conversion module 30 and other components. These components can communicate through one or more communication buses or signal lines. Those skilled in the art will understand that Fig.14 The hardware structure shown in the figure does not constitute a limitation on the student behavior identification device. The student behavior identification device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0187] The processor 21 is the control center of the student behavior recognition device, and uses various interfaces and lines to connect various parts of the student behavior recognition device. By running or executing the application stored in the memory 22, and calling the data and instructions stored in the memory 22, the processor 21 performs various functions of the student behavior recognition device and processes data. In some embodiments, the processor 21 may include one or more processing units; the processor 21 may also integrate an application processor 21 and a modem processor 21; wherein the application processor 21 mainly processes the operating system, user interface, and application programs, and the modem processor 21 mainly processes wireless communications. It is understandable that the above-mentioned modem processor 21 may not be integrated into the processor 21.

[0188] In some other embodiments of the present invention, the processor 21 may also include an artificial intelligence (AI) chip. The learning and processing capabilities of the artificial intelligence chip include image understanding capabilities, natural language understanding capabilities, and speech recognition capabilities. The artificial intelligence chip can enable the student behavior recognition device to have better performance, longer battery life, and better security and privacy. For example, if the student behavior recognition device processes data through the cloud, the data needs to be uploaded and processed before returning the result, which is very inefficient under existing technical conditions. If the local end of the student behavior recognition device has strong AI learning capabilities, then the student behavior recognition device does not need to upload the data to the cloud, and can be directly processed on the local end, thereby improving the security and privacy of the data while improving the processing efficiency.

[0189] The memory 22 is used to store applications and data. The processor 21 executes various functions and data processing of the student behavior recognition device by running the applications and data stored in the memory 2212. The memory 22 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system and an application required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created when using the student behavior recognition device (such as audio data, video data, etc.). In addition, the memory 22 can include a high-speed random access memory 22, and can also include a non-volatile memory 22, such as a disk memory 22, a flash memory device or other non-volatile solid-state memory 22. The memory 22 can store various operating systems, such as an operating system developed by Apple, an operating system developed by Microsoft, etc.

[0190] The display screen 24 is used to display images, videos, etc. The display screen 24 can be a touch screen. In some embodiments, the student behavior recognition device may include 1 or N display screens 24, where N is a positive integer greater than 1. The processor 21 may include one or more graphics processing units (GPUs) that execute program instructions to generate or change display information. The student behavior recognition device implements a display function through the GPU and the display screen 24. The GPU is used to perform mathematical and geometric calculations and is used for graphics rendering.

[0191] The Wi-Fi device 23 is used to provide the student behavior identification device with network access that complies with Wi-Fi related standard protocols. The student behavior identification device can access the Wi-Fi access point through the Wi-Fi device 23, thereby helping to browse the web and access streaming media, etc. It provides users with wireless broadband Internet access. The student behavior identification device can also establish a Wi-Fi connection with a terminal device connected to the Wi-Fi access point through the Wi-Fi device 23 and the Wi-Fi access point for mutual data transmission. In some other embodiments, the Wi-Fi device 23 can also serve as a Wi-Fi wireless access point, which can provide Wi-Fi network access for other electronic devices. Exemplarily, the Wi-Fi device 23 includes at least one wireless network card.

[0192] The Bluetooth device 25 is used to realize data exchange between the student behavior recognition device and other short-range electronic devices (such as terminals, smart watches, etc.). The Bluetooth device 25 in the embodiment of the present invention can be an integrated circuit or a Bluetooth chip.

[0193] The audio circuit 26, the speaker, and the microphone can provide an audio interface between the user and the student behavior recognition device. The audio circuit 26 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 26 and converted into audio data, and then the audio data is sent to the terminal via the Internet, Wi-Fi network, or Bluetooth, or the audio data is output to the memory 22 for further processing.

[0194] The power system 27 is used to charge various components of the student behavior recognition device. The power system 27 may include a battery and a power management module. The battery may be logically connected to the processor 21 through a power management chip, so that the power system 27 can manage charging, discharging, and power consumption.

[0195] The peripheral interface 28 is used to provide various interfaces for external input / output devices (external display, external memory 22, user identification module card, etc.). For example, the storage capacity of the student behavior recognition device can be expanded by connecting to the external memory 22 through the external memory 22 interface, such as a MicroSD card. The peripheral interface 28 can be used to couple the above-mentioned external input / output peripheral devices to the processor 21 and the memory 22.

[0196] The sensor module 29 may include at least one sensor. For example, a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor. Among them, the ambient light sensor can adjust the brightness of the display screen 24 according to the brightness of the ambient light. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the student behavior recognition device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), etc. Of course, according to actual needs, the sensor module 29 can also include any other feasible sensors.

[0197] The data conversion module 30 may include a digital-to-analog converter and an analog-to-digital converter. The relevant functions of the digital-to-analog converter and the analog-to-digital converter can be referred to the corresponding explanations in the above-mentioned related technical terms, which will not be repeated here.

[0198] The embodiment of the present invention also provides a non-volatile computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the method steps of the student behavior recognition method based on the classroom scene described above. Specifically, it includes: analyzing the real-time classroom video stream to obtain a target image of a target person who has a preset classroom behavior; analyzing the facial information of the target image, if the facial features analyzed from the target image meet the preset conditions, then using the analyzed facial features as target facial features, and determining the character role of the target person according to the target facial features and the teacher information library; if the facial features analyzed from the target image do not meet the preset conditions, then analyzing the human body information of the target image, obtaining the target human body features of the target person, and determining the character role of the target person according to the target human body features; outputting the target image of the target person whose character is a student and the corresponding classroom behavior.

[0199] An embodiment of the present invention also provides a computer program product, including a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the student behavior identification method based on classroom scenarios in any of the above-mentioned method embodiments, for example, executing the method steps of the student behavior identification method based on classroom scenarios described above.

[0200] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0201] Through the description of the above embodiments, it can be clearly understood by those skilled in the art that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (Read-OnlyMemory, abbreviated as ROM) or a random access memory (Random Access Memory, abbreviated as RAM), etc.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A student behavior recognition method based on classroom scenes, characterized in that: include: Analyze the real-time classroom video stream to obtain the target image of the target person who exhibits the preset classroom behavior; Analyze the facial information of the target image, and if the facial features analyzed from the target image meet the preset conditions, use the analyzed facial features as the target facial features, and determine the character of the target person according to the target facial features and the teacher information database; If the facial features analyzed from the target image do not meet the preset conditions, the human body information of the target image is analyzed to obtain the target human body features of the target person, and the character role of the target person is determined according to the target human body features; Output the target image of the target person whose character is the student and the corresponding classroom behavior.

2. The method for identifying student behavior based on classroom scenarios according to claim 1 is characterized in that: If the facial features analyzed from the target image do not meet the preset conditions, analyzing the human body information of the target image to obtain the target human body features of the target person, and determining the character role of the target person according to the target human body features includes: If the facial features analyzed from the target image do not meet the preset conditions, obtaining the target body features of the target person; Obtaining a human body identifier corresponding to the target human body feature from a temporary classroom library, determining a face identifier corresponding to the human body identifier according to a mapping relationship, and obtaining the target face feature according to the face identifier; Determine whether there is a teacher's facial feature matching the target facial feature in the teacher information database; If it exists, the target person is a teacher; If not present, the target person is a student.

3. The method for identifying student behavior based on classroom scenarios according to claim 2 is characterized in that: The process of constructing the mapping relationship includes: Acquire a real-time classroom video stream, track and identify the classroom video stream, assign a tracking identifier to the tracked person, and acquire facial features and body features of the tracked person; Assigning a face identifier to the face feature and assigning a body identifier to the body feature; Associating the face identifier and the body identifier belonging to the same tracking identifier to obtain a mapping relationship; The facial features carrying the facial identifier, the body features carrying the body identifier, and the mapping relationship are added to the temporary classroom library.

4. The method for identifying student behavior based on classroom scenarios according to claim 2 is characterized in that: After determining whether there is a teacher's facial feature matching the target facial feature in the teacher information database, the following further comprises: If the target person is a teacher, in the teacher information database, the time tag of the matched teacher's facial feature is updated to the current timestamp, wherein the time tag is used to represent the most recent time when the teacher's facial feature is identified as a teacher.

5. The method for identifying student behavior based on classroom scenarios according to claim 1 is characterized in that: The student behavior recognition method based on classroom scenarios also includes: Acquire real-time classroom video stream, track and identify the classroom video stream, obtain the facial features of the teacher in this class, and update the teacher information database according to the teacher facial features.

6. The method for identifying student behavior based on classroom scenarios according to claim 5 is characterized in that: The updating of the teacher information database according to the teacher's facial features comprises: Determine whether the teacher's facial features exist in the teacher information database; If it does not exist in the teacher information database, then determine whether the teacher's facial features exist in the temporary teacher database; If it does not exist in the temporary teacher database, the teacher's facial features are added to the temporary teacher database, and the number of records of the teacher's facial features is initialized to 1; If it exists in the temporary teacher database, the number of records of the teacher's facial features will be increased by 1, and it will be determined whether the number of records is greater than the set number threshold. If it is greater than the set number threshold, the teacher's facial features will be added to the teacher information database.

7. The method for identifying student behavior based on classroom scenarios according to claim 5 is characterized in that: The real-time classroom video stream is obtained, and the classroom video stream is tracked and identified to obtain the teacher's face features of the class, including: Acquire a real-time classroom video stream, track the characters appearing in the classroom video stream, and acquire facial features of the tracked characters; Obtain the facial features that appear in the podium area for the longest time in the first N minutes of the classroom video stream and whose face quality score exceeds a set threshold, and determine whether to use the facial features as teacher facial features in combination with the human features corresponding to the facial features; and / or, Obtaining the facial features of the longest standing time in the classroom video stream and whose facial quality score exceeds a set threshold, and determining whether to use the facial features as teacher facial features in combination with the human features corresponding to the facial features; and / or, Acquire the facial features of the human body that moves most frequently in the classroom video stream and whose face quality scores exceed a set threshold, and determine whether to use the facial features as teacher facial features in combination with the human body features corresponding to the facial features.

8. The method for identifying student behavior based on classroom scenarios according to claim 7 is characterized in that: The judging whether to use the facial feature as the teacher's facial feature in combination with the human body feature corresponding to the facial feature includes: Acquire the human body features corresponding to the facial features; Determine whether the physical characteristics meet the teacher's height requirements and / or dress requirements; If the height requirement and / or dress requirement of a teacher is not met, the facial features are not the facial features of a teacher; If the height requirement and / or dress requirement for teachers are met, the facial features are used as teacher facial features.

9. The method for identifying student behavior based on classroom scenarios according to claim 5 is characterized in that: The teacher information database stores teacher facial features and time tags corresponding to the teacher facial features; The updating process of the teacher information database also includes: For the teacher facial features existing in the teacher information database, determine whether the time difference between the current time and the time tag of the teacher facial features is greater than a set time threshold; If it is greater than the set time threshold, the corresponding teacher facial features will be deleted from the teacher information database.

10. The method for identifying student behavior based on classroom scenarios according to claim 5, characterized in that: The teacher information database also stores the algorithm version number corresponding to the teacher's facial features; The updating process of the teacher information database also includes: Obtaining the current algorithm version number for performing facial feature recognition on the classroom video stream; For the teacher facial features existing in the teacher information database, determining whether the current algorithm version number is compatible with the algorithm version number of the teacher facial features; If they are incompatible, the corresponding teacher facial features will be deleted from the teacher information database.

11. The method for identifying student behavior based on classroom scenarios according to claim 1 is characterized in that: The preset classroom behaviors include: one or more of: raising hands, standing up, interacting on the podium, and lying on the table.

12. A student behavior recognition device based on classroom scenes, characterized in that: include: The acquisition module is used to obtain real-time classroom video stream; The analysis module is used to analyze the real-time classroom video stream to obtain a target image of a target person who has a preset classroom behavior; and analyze the facial information of the target image. If the facial features analyzed from the target image meet the preset conditions, the analyzed facial features are used as the target facial features. If the facial features analyzed from the target image do not meet the preset conditions, the human body information of the target image is analyzed to obtain the target human body features of the target person; The determination module is used to determine the character role of the target person according to the target person's facial features and the teacher information database; or, to determine the character role of the target person according to the target person's body features; The output module is used to output the target image of the target person whose character is a student and the corresponding classroom behavior.

13. A student behavior recognition device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the student behavior identification method based on classroom scenarios as described in any one of claims 1-11.

Images