Methods, devices, equipment and storage media for blackboard writing behavior recognition
By filtering target individuals facing the blackboard and utilizing depth information and distance judgment, the problem of pose dependence and occlusion in existing blackboard writing behavior recognition methods is solved, achieving high accuracy and low computational load in blackboard writing behavior recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU AVA ELECTRONICS TECH CO LTD
- Filing Date
- 2022-12-16
- Publication Date
- 2026-05-26
AI Technical Summary
Existing methods for recognizing blackboard writing behavior require pre-setting the specific posture of the teacher/student writing on the blackboard, and are easily affected by occlusion in classroom settings, leading to reduced recognition accuracy and misidentification.
By acquiring the person detection results from video frames, target persons facing the blackboard are filtered out, and the orientation of the face and the distance from the blackboard are used to determine whether there is writing behavior. This avoids pre-setting the writing posture and uses depth information detection and distance judgment.
It improves the accuracy of blackboard writing behavior recognition, reduces the amount of machine processing, and avoids misrecognition of similar blackboard writing actions.
Smart Images
Figure CN116311494B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of video image recognition, and more specifically, to a method, apparatus, device, and storage medium for recognizing blackboard writing behavior. Background Technology
[0002] In teaching scenarios, it is often necessary to identify various teaching-related behaviors, one of which is blackboard writing. Currently, common solutions for detecting blackboard writing involve two-stage or even three-stage approaches. First, human detection is performed on the area where the writing takes place, obtaining the bounding box of the human body. Then, keypoint detection is performed on the human body within the bounding box. Some algorithms further classify the hand image based on the position of hand keypoints to distinguish whether the action is writing on the blackboard. These multi-stage solutions have the following drawbacks: First, they require pre-setting the specific posture of the teacher / student writing on the blackboard, and the richness of the pre-set postures determines the accuracy of this approach. Second, this approach identifies postures, but teachers / students may be making similar actions even when not writing on the blackboard, leading to misidentification. Finally, in classroom scenarios, there may be significant occlusion, such as the podium obscuring the body, reducing the accuracy of the first-stage human detection module. Summary of the Invention
[0003] To overcome the shortcomings of current methods for recognizing blackboard writing behavior, this invention provides a method, apparatus, device, and storage medium for blackboard writing behavior recognition. The technical solution adopted by this invention is as follows.
[0004] In a first aspect, the present invention provides a method for recognizing blackboard writing behavior, comprising the following steps:
[0005] Obtain video frames of the video to be identified and the person detection results of the video frames;
[0006] The face orientation of a person is determined based on the person detection results, and target persons that meet the orientation requirements are selected based on the face orientation.
[0007] The system detects the distance between the target person and the blackboard. When the distance between the target person and the blackboard is within a preset threshold range, it determines that there is writing behavior on the blackboard.
[0008] In one implementation, the person detection results include: head detection results and face detection results;
[0009] The process of determining the facial orientation of a person based on the person detection results, and then filtering out target persons that meet the orientation requirements based on the facial orientation, includes the following steps:
[0010] Filter out objects that can detect faces from the head detection results to obtain the remaining objects;
[0011] The remaining objects are identified as the target individuals.
[0012] In one implementation, the process of detecting the distance between the target person and the blackboard, and determining that there is writing behavior when the distance between the target person and the blackboard is within a preset threshold range, includes the following steps:
[0013] Perform full-image depth information detection on video frames;
[0014] Extract the depth information of the target person and the depth information of the blackboard area, wherein the blackboard area is pre-defined;
[0015] Based on the depth information of the target person, the depth information of the blackboard area, and the preset depth threshold, it is determined whether there is any writing behavior on the blackboard.
[0016] In one implementation, the person detection result includes: head detection result;
[0017] The process of determining whether there is writing activity on the blackboard based on the depth information of the target person, the depth information of the blackboard area, and a preset depth threshold includes the following steps:
[0018] Calculate the average depth of the head region;
[0019] Based on the depth information of the blackboard area and the preset depth threshold, the maximum and minimum depth values of the blackboard writing depth area are obtained, and the mapping relationship between these two values and the interval [0,1] is calculated.
[0020] The average depth of the head region is mapped and transformed using the mapping relationship to obtain the relative depth value of the head region in the blackboard region.
[0021] Determine whether there is blackboard writing based on the relative depth value.
[0022] In one implementation, the method further includes the step of filtering out the detection results of people outside the preset whiteboard area.
[0023] In one implementation, the person detection results include: face detection results and / or head detection results.
[0024] Secondly, the present invention provides a device for recognizing blackboard writing behavior, comprising:
[0025] The acquisition module is used to acquire video frames of the video to be identified and the person detection results of the video frames;
[0026] The filtering module is used to determine the face orientation of a person based on the person detection results, and then filter out target people that meet the orientation requirements based on the face orientation.
[0027] The judgment module is used to detect the distance between the target person and the blackboard. When the distance between the target person and the blackboard is within a preset threshold range, it is determined that there is writing behavior on the blackboard.
[0028] In one implementation, it further includes:
[0029] The filtering module is used to filter out the detection results of people who are outside the preset whiteboard area.
[0030] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any of the above embodiments.
[0031] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method of any of the above embodiments.
[0032] This invention utilizes the characteristic that when a person writes on the blackboard, they need to face the blackboard and are relatively close to it, within a certain distance. It identifies a person's blackboard writing behavior by filtering based on their facial orientation and distance from the blackboard. Compared to traditional methods, it does not require pre-setting specific writing postures, reducing preliminary workload. It also avoids mistakenly identifying actions similar to blackboard writing as actual writing, resulting in high accuracy and low computational load. Attached Figure Description
[0033] Figure 1 This is a flowchart illustrating one of the embodiments of the present invention.
[0034] Figure 2 This is a schematic diagram of the image captured by a panoramic camera.
[0035] Figure 3 This is a flowchart illustrating another embodiment of the present invention.
[0036] Figure 4 This is a schematic diagram of the overall structure of Embodiment 2 of the present invention. Detailed Implementation
[0037] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0038] It should be noted that the terms "first, second, ..." used in the embodiments of the present invention are merely used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, ..." can be interchanged in a specific order or sequence where permissible. It should be understood that the objects distinguished by "first, second, ..." can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.
[0039] Example 1
[0040] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for recognizing whiteboard writing behavior according to Embodiment 1 of the present invention. The method includes steps S110, S120, and S130. It should be noted that steps S110, S120, and S130 are merely reference numerals in the accompanying drawings and are used to clearly explain the embodiment and the accompanying drawings. Figure 1 The correspondence is not intended to limit the order of steps in this embodiment.
[0041] Step S110: Obtain the video frames of the video to be identified and the person detection results of the video frames.
[0042] Step S120: Determine the face orientation of the person based on the person detection results, and filter out target people that meet the orientation requirements based on the face orientation.
[0043] Step S130: Detect the distance between the target person and the blackboard. When the distance between the target person and the blackboard is within a preset threshold range, determine that there is writing behavior on the blackboard.
[0044] The process involves acquiring video footage captured by a camera, typically a panoramic camera, analyzing each frame of the video, and obtaining the person detection results for those frames.
[0045] Preferably, the person detection results include: face detection results and / or head detection results.
[0046] In classroom settings, there are often significant obstructions, such as the podium blocking the view of the body. In such cases, using the body for analysis would be ineffective. However, in most situations, the probability of a person's head being obscured is low, so it is preferable to use the head or face for analysis.
[0047] Teachers or students usually face the blackboard when writing, so in step S120, it is necessary to further filter the people in the video frame to filter out the people whose faces are facing the blackboard and define these people as target people.
[0048] For calculating the orientation of a person's face, the coordinates of the key points of the face predicted in two dimensions and the coordinates of the key points of the standard face in three-dimensional space can be used. Based on the perspective multi-point transformation method (solvePNP), the Euler angles (pitch angle, yaw angle, and roll angle) of the face result can be calculated. The yaw angle of the face is filtered out, and the face results that are not within the set threshold range (i.e., the face that is not facing the blackboard) are removed to obtain the face of the target person.
[0049] In one implementation, the person detection results include: head detection results and face detection results;
[0050] The process of step S120 includes steps S121 and S122.
[0051] Step S121: Filter out objects that can detect faces from the objects in the head detection results to obtain the remaining objects;
[0052] Step S122: Identify the remaining objects as the target individuals.
[0053] This is a scene captured using a panoramic camera. The image captured by the panoramic camera is as follows: Figure 2 As shown, if a teacher or student is writing on the blackboard, they will inevitably be facing away from the panoramic camera. Head detection can detect both heads facing the camera (i.e., facing away from the blackboard) and heads facing away from the camera (i.e., facing the blackboard), while face detection can only detect heads facing the camera (i.e., facing away from the blackboard). Therefore, by filtering out the head detection objects that can detect faces from the head detection objects, we can obtain the target person facing the blackboard.
[0054] This implementation method allows for the simple, quick, and low-processing filtering of target individuals.
[0055] like Figure 2As shown, if simply using facing the blackboard as the criterion for judging whether someone is writing on the board, students sitting and listening might be included as participants. Therefore, further filtering is required in step S130. Typically, when writing on the board, the person writing is relatively close to it, within a certain distance, such as an arm's length. Using this principle, after identifying the person facing the blackboard, the distance between them and the board can be used to determine who is writing. Therefore, step S130 detects the distance between the target person and the blackboard, and determines whether the currently detected video frame contains writing activity based on this distance.
[0056] The distance between the target person and the blackboard can be detected by means of infrared devices or by means of binocular cameras.
[0057] It should be noted that in step S130, the judgment can be made by first detecting the distance between the target person and the blackboard, and then comparing this distance with a preset threshold to determine whether there is writing behavior. Alternatively, the absolute distance between the target person and the blackboard can be detected first, and then the preset threshold can be used to calculate the corresponding distance range for the other person. Then, the absolute distance of the other person can be measured to see if it falls within the expected range, thus determining whether there is writing behavior. For example, if the detection device detects a distance of 3 meters from the blackboard, and the preset threshold is 0.5 meters, then when the person is within the range of 2.5-3 meters, there is writing behavior. The detection device then checks the distance to the person again. If the detected distance is 2 meters, there is no writing behavior; if the detected distance is 2.6 meters, there is writing behavior.
[0058] In one implementation, a monocular depth estimation module is used to perform full-image depth information detection on the current video frame to obtain the distance between the target person and the blackboard. This monocular depth estimation module mainly includes two sub-modules: a neural network-based monocular depth estimation model and a post-processing module for the model's output.
[0059] When using the monocular depth estimation module, step S130 includes:
[0060] Step S131: Perform full-image depth information detection on the video frames;
[0061] Step S132: Extract the depth information of the target person and the depth information of the blackboard area, wherein the blackboard area is pre-defined;
[0062] Step S133: Based on the depth information of the target person, the depth information of the blackboard area, and the preset depth threshold, determine whether there is any writing behavior on the blackboard.
[0063] To determine the distance between the target person and the pre-defined blackboard area, a monocular depth estimation module is used to estimate the target's depth. This estimation only needs to be performed once on the entire image. The output of the monocular depth estimation module is a floating-point depth map, where the size of the elements represents their distance from the camera itself; larger elements indicate greater distance, and smaller elements indicate closer distance.
[0064] Once the depth map is obtained, the depth information of the target person's location area and the blackboard area is extracted from it. Additionally, as mentioned earlier, a person writing on the blackboard will be within a certain distance; this distance is the depth threshold, which can be preset using empirical values. After obtaining the depth information of the target person, the depth information of the blackboard area, and the preset depth threshold, it can be determined whether there is blackboard writing behavior in the currently detected video frame.
[0065] In one implementation, the person detection result includes: face detection result and / or head detection result, and the process of step S133 includes: step S310, step S320 and step S330.
[0066] Step S310: Calculate the average depth of the head region;
[0067] Step S320: Based on the depth information of the blackboard area and the preset depth threshold, the maximum depth value and minimum depth value of the blackboard writing depth area are obtained, and the mapping relationship between these two values and the interval [0,1] is calculated.
[0068] Step S330: Use the mapping relationship to map and transform the average depth of the head region to obtain the relative depth value of the head region in the blackboard region;
[0069] Step S340: Determine whether there is any whiteboard writing behavior based on the relative depth value.
[0070] Calculate the average depth Hmean of the head region. Based on the depth information of the blackboard region and the preset depth threshold, obtain the maximum and minimum depth values Tmax and Tmin of the blackboard depth region, and then obtain the scale Tscale for converting the depth of the blackboard depth region into a normalized region [0, 1].
[0071] Tscale=(Tmax-Tmin) / (1-0)=(Tmax-Tmin);
[0072] Furthermore, we can obtain the proportion of the average depth Hmean of the head region under normalized scaling, Hratio:
[0073] Hratio=(Hmean-Tmin)*Tscale=(Hmean-Tmin)*(Tmax-Tmin)
[0074] Hratio will be used as the relative depth value of the head target in the whiteboard depth region to determine whether there is whiteboard writing behavior in the currently detected video frame. When the relative depth value of the head target in the current whiteboard depth region is greater than the preset threshold, the behavior of the target person will be identified as whiteboard writing behavior.
[0075] This method leverages the characteristic that when a person writes on the blackboard, they need to face the blackboard and be relatively close to it, within a certain distance. It identifies a person's blackboard writing behavior by filtering based on their facial orientation and distance from the blackboard. Compared to traditional methods, it does not require pre-setting specific writing postures, reducing preliminary workload. It also avoids mistakenly identifying actions resembling blackboard writing as actual writing, resulting in high accuracy and low computational load.
[0076] In one implementation, such as Figure 3 As shown, it also includes step S210.
[0077] Step S210: Filter out the character detection results that are outside the preset whiteboard area.
[0078] like Figure 2 As shown, when teachers / students write on the blackboard, they generally do so in or around the blackboard. It can be determined that anyone outside this area is not writing on the blackboard. Therefore, in this embodiment, this area is pre-defined as the blackboard writing area. Figure 2 The black box in the image (in the image) is used to filter out the people within the area of the whiteboard. This reduces the number of objects to be detected in subsequent steps, thus reducing the amount of computation. It also eliminates false detections of people outside the whiteboard area.
[0079] It should be noted that, since this step is relatively simple and can effectively filter out noise, it is preferable to use this step before step S120, filtering out some people through this step and then further filtering by the direction of their faces.
[0080] It should also be noted that the preset blackboard area here can be the same area as the blackboard area mentioned earlier, or they can be set separately according to the actual situation.
[0081] Example 2
[0082] Corresponding to the method in Example 1, such as Figure 4 As shown, the present invention also provides a device 4 for whiteboard behavior recognition, including: an acquisition module 401, a filtering module 402, and a judgment module 403.
[0083] The acquisition module 401 is used to acquire video frames of the video to be identified and the person detection results of the video frames;
[0084] The filtering module 402 is used to determine the face orientation of a person based on the person detection results, and to filter out target people that meet the orientation requirements based on the face orientation.
[0085] The judgment module 403 is used to detect the distance between the target person and the blackboard. When the distance between the target person and the blackboard is within a preset threshold range, it is determined that there is writing behavior on the blackboard.
[0086] In one implementation, the person detection results include: head detection results and face detection results;
[0087] The filtering module performs the following steps in the process of determining the facial orientation of a person based on the person detection results and filtering out target persons that meet the orientation requirements:
[0088] Filter out objects that can detect faces from the head detection results to obtain the remaining objects;
[0089] The remaining objects are identified as the target individuals.
[0090] In one implementation, the judgment module, when performing the detection of the distance between the target person and the blackboard, determines that there is writing behavior when the distance between the target person and the blackboard is within a preset threshold range, includes the following steps:
[0091] Perform full-image depth information detection on video frames;
[0092] Extract the depth information of the target person and the depth information of the blackboard area, wherein the blackboard area is pre-defined;
[0093] Based on the depth information of the target person, the depth information of the blackboard area, and the preset depth threshold, it is determined whether there is any writing behavior on the blackboard.
[0094] In one implementation, the person detection result includes: head detection result;
[0095] The judgment module, in performing the process of determining whether there is writing behavior on the blackboard based on the depth information of the target person, the depth information of the blackboard area, and a preset depth threshold, includes the following steps:
[0096] Calculate the average depth of the head region;
[0097] Based on the depth information of the blackboard area and the preset depth threshold, the maximum and minimum depth values of the blackboard writing depth area are obtained, and the mapping relationship between these two values and the interval [0,1] is calculated.
[0098] The average depth of the head region is mapped and transformed using the mapping relationship to obtain the relative depth value of the head region in the blackboard region.
[0099] Determine whether there is blackboard writing based on the relative depth value.
[0100] In one embodiment, the device for recognizing blackboard writing behavior further includes a filtering module;
[0101] The filtering module is used to filter out the detection results of people who are outside the preset whiteboard area.
[0102] In one implementation, the person detection results include: face detection results and / or head detection results.
[0103] This device identifies a person's blackboard writing behavior by analyzing their facial orientation and distance from the blackboard, based on the characteristic that people face the blackboard and are relatively close to it. Compared to traditional methods, it eliminates the need to pre-define the specific posture for writing, reducing preliminary work. It also avoids mistakenly identifying similar actions as writing, resulting in high accuracy and low computational load.
[0104] Example 3
[0105] This invention also provides a storage medium storing computer instructions that, when executed by a processor, implement the whiteboard behavior recognition method of any of the above embodiments.
[0106] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, random access memory (RAM), read-only memory (ROM), magnetic disks, or optical disks.
[0107] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, terminal, or network device, etc.) to execute all or part of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, RAM, ROM, magnetic disks, or optical disks.
[0108] Corresponding to the computer storage medium described above, one embodiment also provides a computer device, which includes a memory, an encoder, and a computer program stored in the memory and executable on the encoder, wherein the encoder executes the program to implement any of the methods for whiteboard behavior recognition as described in the above embodiments.
[0109] The aforementioned computer equipment, based on the characteristic that when a person writes on the blackboard, they need to face the blackboard and be relatively close to it, within a certain distance, identifies the person's blackboard writing behavior by filtering out the direction of their face and the distance between the person and the blackboard. Compared with traditional methods, it does not require pre-setting the specific posture of blackboard writing, reducing the amount of preliminary work, and it will not mistakenly identify similar actions as blackboard writing behavior. The action recognition accuracy is high, and the machine's computational load is low.
[0110] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0111] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A method for recognizing blackboard writing behavior, characterized in that, Including the following steps: Obtain video frames of the video to be identified and the person detection results of the video frames; wherein, the person detection results include: head detection results; The face orientation of a person is determined based on the person detection results, and target persons that meet the orientation requirements are selected based on the face orientation. The system detects the distance between the target person and the blackboard. When the distance between the target person and the blackboard is within a preset threshold range, it determines that there is writing behavior on the blackboard. The process of detecting the distance between the target person and the blackboard, and determining that there is writing behavior when the distance between the target person and the blackboard is within a preset threshold range, includes the following steps: Perform full-image depth information detection on video frames; Extract the depth information of the target person and the depth information of the blackboard area, wherein the blackboard area is pre-defined; Based on the depth information of the target person, the depth information of the blackboard area, and the preset depth threshold, determine whether there is any writing behavior on the blackboard; The method is based on the depth information of the target person, the depth information of the blackboard area, and a preset depth threshold. The process of determining whether there is blackboard writing includes the following steps: Calculate the average depth of the head region; Based on the depth information of the blackboard area and the preset depth threshold, the maximum and minimum depth values of the blackboard writing depth area are obtained, and the mapping relationship between these two values and the interval [0,1] is calculated. The average depth of the head region is mapped and transformed using the mapping relationship to obtain the relative depth value of the head region in the blackboard region. Determine whether there is blackboard writing based on the relative depth value.
2. The method for recognizing blackboard writing behavior according to claim 1, characterized in that, The person detection results include: head detection results and face detection results; The process involves determining the face orientation of a person based on the character detection results, and then filtering out those that match the orientation. The process of identifying the target person includes the following steps: filtering out objects that can detect faces from the objects in the head detection results to obtain the remaining objects; and identifying the remaining objects as the target person.
3. The method for recognizing blackboard writing behavior according to claim 1 or 2, characterized in that, Also includes Step: Filter out the character detection results that are outside the preset whiteboard area.
4. A device for recognizing blackboard writing behavior, characterized in that, include: The acquisition module is used to acquire video frames of the video to be identified and the person detection results of the video frames; The person detection results include: head detection results; and a filtering module, used to determine the face orientation of the person based on the person detection results, and to filter based on the face orientation. Select target individuals that meet the orientation requirements; the judgment module is used to detect the distance between the target individual and the blackboard. When the distance between the target person and the blackboard is within a preset threshold range, it is determined that there is writing activity on the blackboard; the judgment module performs the detection of the distance between the target person and the blackboard, and when the target person and the blackboard are within the preset threshold range, it determines that there is writing activity on the blackboard. The process of determining whether there is writing behavior when the distance between blackboards is within a preset threshold range includes the following steps: performing full-image depth information detection on the video frame; Extract the depth information of the target person and the depth information of the blackboard area, wherein the blackboard area is pre-defined; Based on the depth information of the target person, the depth information of the blackboard area, and the preset depth threshold, determine whether there is any writing behavior on the blackboard; The judgment module executes the judgment based on the depth information of the target person and the depth information of the blackboard area. The process of determining whether there is whiteboard writing behavior, based on a preset depth threshold, includes the following steps: Calculate the average depth of the head region; Based on the depth information of the blackboard area and the preset depth threshold, the maximum and minimum depth values of the blackboard writing depth area are obtained, and the mapping relationship between these two values and the interval [0,1] is calculated. The average depth of the head region is mapped and transformed using the mapping relationship to obtain the relative depth value of the head region in the blackboard region. Determine whether there is blackboard writing based on the relative depth value.
5. The device for recognizing blackboard writing behavior according to claim 4, characterized in that, Also includes: The filtering module is used to filter out the detection results of people who are outside the preset whiteboard area.
6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-3.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-3.