Posture recognition apparatus, method, electronic device, and storage medium

Abstract

The embodiment of the present disclosure provides a posture recognition device, method, electronic equipment and storage medium, and relates to the technical field of image processing and posture recognition. The posture recognition device comprises: an extraction unit configured to extract N image frames from a to-be-processed video stream, N being an integer greater than or equal to 2; a detection unit configured to detect, for each image frame in the N image frames, a target object and object position information in the image frame, the object position information indicating position information of an object with a predetermined posture in the target object, the predetermined posture comprising a valid predetermined posture and an invalid predetermined posture; and a determination unit configured to determine, according to the object position information, valid object position information of the target object with the valid predetermined posture.

Description

Technical Field

[0001] This disclosure relates to the fields of image processing and posture recognition technology, and more specifically, to a posture recognition device, method, electronic device, storage medium, and computer program product. Background Technology

[0002] Human posture recognition has wide applications in video surveillance, security, and video retrieval. For example, standing / sitting posture recognition, as a type of human posture recognition, studies the dynamic process of a person sitting down / standing up and standing up / sitting down, and it has direct applications in smart classrooms, intelligent recording and broadcasting, and other scenarios.

[0003] Currently, in closed environments such as smart classrooms and intelligent recording studios, the detection of human movement such as standing up primarily involves installing or deploying cameras in student areas to monitor student behavior. When a student meets certain behavioral conditions, such as standing up, the camera focuses on the area where the student is standing, generating interaction between the student and the teacher. Recognizing this interaction helps assess the quality of classroom teaching and student learning, and is of great significance for improving overall teaching standards. Summary of the Invention

[0004] This disclosure provides an attitude recognition device, method, electronic device, storage medium, and computer program product.

[0005] According to a first aspect of the present disclosure, a pose recognition device is provided, comprising: an extraction unit configured to extract N image frames from a video stream to be processed, where N is an integer greater than or equal to 2; a detection unit configured to detect, for each of the N image frames, a target object and object position information in that image frame, wherein the object position information indicates the position information of an object with a predetermined pose in the target object, the predetermined pose including a valid predetermined pose and an invalid predetermined pose; and a determination unit configured to determine, based on the object position information, valid object position information of an object with a valid predetermined pose in the target object.

[0006] According to embodiments of the present disclosure, the determining unit is further configured to: for each of the N image frames, in response to the detection unit detecting that the image frame has object position information, determine the image frame as a target image frame; and add valid object position information in the target image frame, wherein the pose of at least one target object is a valid predetermined pose, to the valid data set.

[0007] According to embodiments of this disclosure, an invalid predetermined pose includes a first invalid pose; the extraction unit is further configured to: extract a reference image frame from the video stream every m frames, where m is an integer greater than or equal to 2; the determination unit is further configured to: continuously extract u image frames after the target image frame from the video stream, where u is a positive integer less than m; in response to the detection unit detecting that at least one of the u image frames has object position information, determine the average moving distance of the object with the predetermined pose in the u image frames based on the object position information; and if the average moving distance is determined to be greater than a distance threshold, determine the pose of the object with the predetermined pose in the target image frame as the first invalid pose.

[0008] According to embodiments of this disclosure, the determining unit is further configured to: when the average moving distance is determined to be less than or equal to a distance threshold, acquire at least one valid object location information in the current valid data set; for each valid object location information, calculate the intersection-over-union ratio (IoU) between the valid object location information and the object location information of the target image frame to obtain at least one IoU; determine the maximum IoU from the at least one IoU; and when the maximum IoU is determined to be greater than or equal to the IoU threshold, determine an object with a predetermined posture in the target image frame as a valid object.

[0009] According to embodiments of this disclosure, the invalid predetermined pose further includes a second invalid pose; the determining unit is further configured to: when the maximum intersection-union ratio is determined to be less than the intersection-union ratio threshold, update the outline of the object with the predetermined pose in the target image frame according to the object position information of the target image frame and a preset ratio; extract the target image block and the reference image block from the target image frame and the reference image frame respectively according to the outline; calculate the difference between the target image block and the reference image block; and when the difference is determined to be less than or equal to the difference threshold, determine the pose of the object with the predetermined pose in the target image frame as the second invalid pose.

[0010] According to embodiments of this disclosure, the determining unit is further configured to: for each of the N image frames, add the number of target objects in the image frame to a target object data set; if the difference is determined to be greater than a difference threshold, determine the number of target objects that appear most frequently in the current target object data set as a target object number threshold; and if the number of target objects in the target image frame is determined to be greater than the target object number threshold, determine the pose of an object with a predetermined pose in the target image frame as a second invalid pose.

[0011] According to embodiments of this disclosure, the invalid predetermined posture further includes a second invalid posture; the determining unit is further configured to: when determining that the average moving distance is less than or equal to a distance threshold, in response to obtaining that the current valid data set is an empty set, update the outline of the object with the predetermined posture in the target image frame according to the object position information of the target image frame and a preset ratio; extract the target image block and the reference image block from the target image frame and the reference image frame respectively according to the outline; calculate the difference between the target image block and the reference image block; and when determining that the difference is less than or equal to a difference threshold, determine the posture of the object with the predetermined posture in the target image frame as the second invalid posture.

[0012] According to embodiments of this disclosure, the determining unit is further configured to: for each of the N image frames, add the number of target objects in the image frame to a target object data set; if the difference is determined to be greater than a difference threshold, determine the number of target objects that appear most frequently in the current target object data set as a target object number threshold; and if the number of target objects in the target image frame is determined to be greater than the target object number threshold, determine the pose of an object with a predetermined pose in the target image frame as a second invalid pose.

[0013] According to embodiments of this disclosure, the determining unit is further configured to: for each of the N image frames, in response to the detection unit not detecting object location information for that image frame, clear the current valid data set.

[0014] According to embodiments of this disclosure, the determining unit is further configured to: delete object location information from the target image frame from the current valid data set if it is determined that the maximum cross-union ratio is less than the cross-union ratio threshold.

[0015] According to a second aspect of the present disclosure, an electronic device is provided, including the posture recognition device of the above embodiments of the present disclosure.

[0016] According to a third aspect of the present disclosure, a pose recognition method is provided, comprising: extracting N image frames from a video stream to be processed, where N is an integer greater than or equal to 2; for each of the N image frames, detecting a target object and object position information in the image frame, wherein the object position information indicates the position information of an object with a predetermined pose in the target object, the predetermined pose including a valid predetermined pose and an invalid predetermined pose; and determining, based on the object position information, valid object position information with a valid predetermined pose in the target object.

[0017] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described attitude recognition method.

[0018] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided that stores executable instructions thereon, which, when executed by a processor, cause the processor to implement the above-described attitude recognition method.

[0019] According to a sixth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described posture recognition method.

[0020] The posture recognition device, method, electronic device, storage medium, and computer program products provided in this disclosure can process video streams acquired in closed scenarios such as smart classrooms and intelligent recording and broadcasting frame by frame to obtain the number of target objects in the current image frame and the position information of target objects with predetermined postures. This allows for the rapid and accurate identification of the position information of valid objects with valid predetermined postures, while eliminating invalid predetermined postures that cause interference. This helps to evaluate the quality of classroom teaching and student learning. Attached Figure Description

[0021] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments in conjunction with the accompanying drawings. It should be noted that throughout the drawings, the same elements are indicated by the same or similar reference numerals. In the figures:

[0022] Figure 1 A schematic diagram of the structure of an attitude recognition device according to an embodiment of the present disclosure is shown;

[0023] Figure 2 A schematic diagram illustrating the principle of the determining unit excluding a first invalid attitude from invalid predetermined attitudes according to an embodiment of the present disclosure is shown.

[0024] Figure 3 A schematic diagram illustrating the principle of the determining unit excluding a second invalid attitude from invalid predetermined attitudes according to an embodiment of the present disclosure is shown.

[0025] Figure 4A A comparison diagram of a target image block, a reference image block, and their differing regions is shown according to an embodiment of the present disclosure when no one is standing.

[0026] Figure 4B A comparison diagram of a target image block, a reference image block, and their differing regions when a person is standing is shown according to an embodiment of the present disclosure.

[0027] Figure 5A schematic diagram of the structure of an electronic device suitable for a posture recognition device according to an embodiment of the present disclosure is shown;

[0028] Figure 6 A flowchart of a pose recognition method according to an embodiment of the present disclosure is shown;

[0029] Figure 7 A schematic diagram illustrating the principle of a pose recognition method according to an embodiment of the present disclosure is shown;

[0030] Figure 8 A block diagram of an electronic device suitable for implementing a posture recognition method according to an embodiment of the present disclosure is shown. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the described embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure. In the following description, some specific embodiments are for illustrative purposes only and should not be construed as limiting this disclosure in any way, but are merely examples of embodiments of this disclosure.

[0032] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure shall have the ordinary meaning as understood by those skilled in the art. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components.

[0033] It is understood that in the specific embodiments of this disclosure, data related to image processing is involved. When the following embodiments of this disclosure are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0034] In realizing the present invention, the inventors discovered that the main difficulties in detecting human standing behavior in closed scenarios such as smart classrooms and intelligent recording studios are as follows: on the one hand, fast computing speed and low response latency are required to meet the needs of edge devices; on the other hand, in actual classroom scenarios, due to the inconsistent distribution of desks, high student density, individual and posture variations, and teachers entering the students' field of vision, detection is often difficult, making it hard to accurately identify effective standing behavior.

[0035] In closed environments such as smart classrooms and intelligent recording studios, cameras can be installed or deployed in student areas to capture or record students standing up, creating a video stream for processing and detection of student standing behavior.

[0036] Furthermore, to identify valid standing-up behaviors, the inventors have provided a clear definition of a student-side standing-up event: a student-side standing-up event refers to the process of a student standing up from their seat during class, upon hearing a teacher's question or raising their hand to answer a question. Specifically, this posture change is from a sitting posture to a standing posture. Moreover, after standing up, maintaining the standing posture in the same position for a period of time is uniformly referred to as a valid standing posture in this embodiment. Other standing postures, such as when a teacher appears in the student's frame, when a teacher walks among students in an aisle, or when someone is already standing in the same position at the beginning of the video feed after the smart classroom or recording system is activated, do not constitute a student-side standing-up event. Therefore, these are considered interference information, and the detection of human standing-up behavior should exclude this interference information as much as possible. In this embodiment, these standing postures that cause interference information are uniformly referred to as invalid standing postures. Invalid standing postures mainly include two types: invalid moving standing postures and invalid stationary standing postures. Invalid moving standing postures refer to standing postures while walking, such as a teacher walking in an aisle. An invalid static standing posture could be, for example, when a smart classroom or recording system is turned on, a person is seen standing in the same position from the very beginning of the video feed.

[0037] In view of this, embodiments of this disclosure provide a posture recognition device, method, electronic device, storage medium, and computer program product, relating to the fields of image processing and posture recognition technology. According to embodiments of this disclosure, based on the development of deep learning and object detection, video streams acquired in closed scenarios such as smart classrooms and intelligent recording studios are processed to detect the standing behavior of multiple target objects (e.g., all students in a classroom), quickly and accurately identifying valid objects (e.g., students with valid standing postures) with valid predetermined postures in the shot.

[0038] The various embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings.

[0039] Figure 1 A schematic diagram of the structure of an attitude recognition device according to an embodiment of the present disclosure is shown.

[0040] like Figure 1 As shown, the posture recognition device 100 of this embodiment includes an extraction unit 110, a detection unit 120, and a determination unit 130.

[0041] The extraction unit 110 is configured to extract N image frames from the video stream to be processed, where N is an integer greater than or equal to 2.

[0042] For example, in closed environments such as smart classrooms and intelligent recording studios, cameras can be installed or deployed in student areas to capture or record students standing up, creating a video stream to be processed.

[0043] The detection unit 120 is configured to detect the target object and object position information in each of the N image frames. The object position information indicates the position information of the object with a predetermined posture in the target object. The predetermined posture includes a valid predetermined posture and an invalid predetermined posture.

[0044] For example, in closed environments such as smart classrooms and intelligent recording studios, standing behavior detection can be performed on students in the classroom. The target group can be all students in the classroom, and the number of target groups is the total number of students in the classroom. The predetermined posture can be a standing posture, a valid predetermined posture can be a valid standing posture, and an invalid predetermined posture can be an invalid standing posture. The object location information can be the location of the student with the standing posture.

[0045] It should be noted that the object position information detected by the detection unit 120 only refers to the positions of students who are standing. If the detection unit 120 does not detect the object position information, it means that no students are standing in the current image frame, that is, all students in the classroom are sitting. At this time, the detection unit 120 can proceed to the detection of the next image frame.

[0046] For example, N image frames can be input frame by frame into a trained multi-class detection model to obtain the number and location information of target objects in the current image frame. For instance, this multi-class detection model can output two types of detection information: one is the total number of students in the image frame, and the other is the location of students in the image frame who are standing. Here, "standing posture" refers to a near-perfect standard standing posture, excluding the transition from sitting to standing or vice versa.

[0047] The determining unit 130 is configured to determine the position information of a valid object with a valid predetermined posture in the target object based on the object position information.

[0048] A valid object refers to a target object with a valid predetermined posture, such as a group of students in a classroom who have a valid standing posture. The valid object location information is the location information of these students.

[0049] The posture recognition device provided in this disclosure can process video streams acquired in closed scenarios such as smart classrooms and intelligent recording and broadcasting frame by frame to obtain the number of target objects in the current image frame and the position information of target objects with predetermined postures. This allows for the rapid and accurate identification of the position information of valid objects with valid predetermined postures, while eliminating invalid predetermined postures that cause interference. This helps to evaluate the quality of classroom teaching and student learning.

[0050] In this embodiment of the disclosure, the determining unit 130 is further configured to add the number of target objects in each of the N image frames to the target object data set.

[0051] When the detection unit 120 detects the number of target objects in each image frame, it adds the number of target objects in that image frame to the target object data set Pk = {p1, p2, p3, ... pk}. n In the diagram, p1 refers to the number of target objects in the first image frame, and n represents the length of the target object data set. Similarly, the target object data set Pk is updated according to the detection progress of the detection unit 120.

[0052] The contents of set Pk are updated with each frame. For example, when the length of set Pk is less than or equal to n, the length of set Pk increases with the results of image frame detection. When the length of set Pk is greater than n, that is, when the number of image frames detected by detection unit 120 exceeds n, the contents of set Pk... n The information will then be replaced by the detection result of the current image frame, and the p of the current image frame. n-1 Then use p from the previous image frame n Then replace P in the current image frame. n-2 Use the p from the previous image frame n-1 The replacement continues in this manner until p1 of the current image frame is replaced by p2 of the previous frame. For ease of understanding, Table 1 below provides an example of a target object data set Pk, where, when the number of image frames reaches n+1, Pk = {p2, p3, ..., p...} n P n+1}

[0053] Table 1

[0054] Frames <![CDATA[p1]]> <![CDATA[p2]]> …… <![CDATA[p n-1 ]]> <![CDATA[p n ]]> 1 1 …… 2 1 2 …… …… …… …… …… …… …… n 1 2 …… n-1 n n+1 2 3 …… n n+1 …… …… …… …… …… ……

[0055] It is understandable that limiting the length n of the target object data set Pk is to save hardware costs, avoid the need for excessively large storage capacity caused by a large number of video streams (with many frames), thereby improving computing speed, reducing response latency, and meeting the needs of deployment on edge devices.

[0056] In this embodiment of the disclosure, the determining unit 130 is further configured to: for each of the N image frames, in response to the detection unit 120 detecting that the image frame has object position information, determine the image frame as a target image frame; and add the valid object position information of at least one target object in the target image frame whose pose is a valid predetermined pose to the valid data set.

[0057] For example, when the detection unit 120 detects that the image frame has object location information, it indicates that the image frame contains a standing human body, and the standing posture of the human body can be recognized. At this time, the object location information s0 of the current image frame is recorded, and the current target image frame Fcur (raw frame data) is saved. Furthermore, when the detection unit 120 detects a valid object in the image frame, the location information of the valid object is saved to the valid data set Valid_S. The valid object location information of the video stream is determined using the current valid data set Valid_S. In this embodiment, the object location information s0 can show the outline of the object, and the shape of the outline is not limited in this disclosure. In some embodiments, when the outline is rectangular, the object location information s0 includes the lower left corner coordinates, width, and height of the outline.

[0058] In this embodiment of the disclosure, the determining unit 130 is further configured to: for each of the N image frames, in response to the detection unit 120 not detecting the object position information of the image frame, clear the current valid data set. When the detection unit 120 does not detect the object position information of the image frame, it indicates that there is no standing human body in the image frame, so Valid_S is cleared and the next image frame is processed.

[0059] In this embodiment of the disclosure, the extraction unit 110 is further configured to extract a reference image frame from the video stream every m frames, where m is an integer greater than or equal to 2.

[0060] In this embodiment of the disclosure, the invalid predetermined posture includes a first invalid posture, which may be, for example, an invalid moving standing posture. Based on this, the determining unit 130 is further configured to: continuously extract u image frames from the target image frame of the video stream, where u is a positive integer less than m; in response to the detection unit 120 detecting that at least one of the u image frames has object position information, determine the average moving distance of the object with the predetermined posture in the u image frames based on the object position information; and if the average moving distance is determined to be greater than a distance threshold, determine the posture of the object with the predetermined posture in the target image frame as the first invalid posture.

[0061] Figure 2 A schematic diagram illustrating the principle of the determining unit excluding a first invalid posture from invalid predetermined postures according to an embodiment of the present disclosure is shown.

[0062] As shown Figure 2 in the figure, on the time axis t of the video stream, the reference image frame Fbase (original frame data) is saved every m frames. For example, when the first reference image frame Fbase1 is saved, since the detection unit 120 does not detect the object position information of the first reference image frame Fbase1, the detection unit 120 continues to detect the next image frame until it detects the object position information of a certain image frame, and determines this image frame as the target image frame Fcur. Assuming this image frame is the first one, it is also called the first target image frame Fcur1. At this time, u image frames are continuously extracted after the target image frame Fcur in the video stream, and u < m, then the u image frames have not reached the time of the second reference image frame Fbase2.

[0063] Next, the detection unit 120 performs frame-by-frame detection on the u image frames. When the detection unit 120 detects that at least one of the u image frames has object position information, according to this object position information, the average moving distance of the object with a predetermined posture among the u image frames is determined.

[0064] For example, when the detection unit 120 detects that at least one of the u image frames has object position information, the determination unit 130 adds the object position information of each of the u image frames to the standing human body data set S = {s1, s2, s3,..., s u}, where s1 refers to the object position information of the first image frame among the u image frames, and so on. According to the detection progress of the detection unit 120, the standing human body data set S is updated.

[0065] It should be noted that within the continuous u image frames, the update of the target object data set Pk and the reference image frame Fbase needs to be stopped until the posture of the object on s0 in the target image frame Fcur is determined, and then the update continues.

[0066] The determining unit 130 uses the standing human body data set S to calculate the average movement distance delta_d of objects (e.g., standing human bodies) with a predetermined posture over u consecutive image frames. Based on a comparison of delta_d and a set distance threshold, it determines whether the posture of the object on s0 with the predetermined posture in the target image frame Fcur is a first invalid posture. If the average movement distance delta_d is greater than the distance threshold, it indicates that the movement amplitude of the standing human body in the target image frame Fcur is too large, such as a standing posture while walking. Therefore, the posture of the object on s0 with the predetermined posture in the target image frame Fcur is determined as a first invalid posture (e.g., an invalid moving standing posture). When the average movement distance delta_d is greater than the distance threshold, the object on s0 can be determined as an invalid standing human body, and the determining unit 130 can proceed to the processing of the next image frame.

[0067] The posture recognition device provided by the embodiments of this disclosure can avoid recording human bodies that maintain a standing posture in different positions as valid standing human bodies, excluding, for example, teachers walking in a corridor, and further improve the recognition accuracy of valid objects with valid predetermined postures in the lens.

[0068] In this embodiment of the disclosure, the determining unit 130 is further configured to: when the average moving distance is determined to be less than or equal to a distance threshold, acquire at least one valid object location information in the current valid data set; for each valid object location information, calculate the intersection-over-union ratio (IoU) between the valid object location information and the object location information of the target image frame to obtain at least one IoU; determine the maximum IoU from the at least one IoU; and when the maximum IoU is determined to be greater than or equal to the IoU threshold, determine the object with a predetermined posture in the target image frame as a valid object.

[0069] The pose recognition device provided in this embodiment can directly determine whether the object on s0 in Fcur is a valid object when the current valid data set Vaild_S is not empty, that is, when Vaild_S contains at least one valid object position information. This effectively improves the speed of pose recognition.

[0070] For example, to determine whether the object position information s0 of the current target image frame Fcur exists in Vaild_S, at least one valid object position information in the current valid data set Vaild_S can be traversed, and the intersection-over-union ratio (IoU) between each valid object position information and the object position information s0 of the target image frame Fcur can be calculated. After the traversal, the maximum IoU is determined, and then it is determined whether the maximum IoU is greater than or equal to the set IoU threshold (e.g., 0.8). If it is, it indicates that the object with the predetermined pose in the target image frame Fcur has become a valid object. At this point, the pose recognition of the target image frame Fcur ends, and the processing of the next image frame can proceed.

[0071] In this embodiment of the disclosure, the determining unit 130 is further configured to: delete object location information from the target image frame from the current valid data set when the maximum intersection-union ratio (CIU) is determined to be less than the CIU threshold. Since the valid object location information in the current valid data set Valid_S must exist in the object location information s0 detected in the current target image frame, if it does not exist, it indicates that the object on s0 has already been placed, and in this case, the object location information s0 in Valid_S needs to be deleted.

[0072] In this embodiment of the disclosure, the invalid predetermined posture further includes a second invalid posture, which may be, for example, an invalid static standing posture. Based on this, the determining unit 130 is further configured to: when the maximum intersection-over-union ratio (IoU) is determined to be less than an IoU threshold, update the bounding boxes of objects with predetermined postures in the target image frame according to the object position information of the target image frame and a preset ratio; extract target image blocks and reference image blocks from the target image frame and the reference image frame respectively based on the bounding boxes; calculate the difference between the target image blocks and the reference image blocks; and when the difference is determined to be less than or equal to a difference threshold, determine the posture of the object with predetermined posture in the target image frame as the second invalid posture.

[0073] For example, the difference can be the area of ​​the difference region between the target image block and the reference image block, and the difference threshold can be the area threshold. If the area of ​​the difference region between the target image block and the reference image block is less than or equal to the area threshold, the pose of the object with a predetermined pose in the target image frame is determined as the second invalid pose.

[0074] Since the maximum cross-union ratio (CUNR) is less than the CUNR threshold, it can be determined that the object at the object location information s0 in the target image frame Fcur belongs to a stationary standing human body and does not exist in the set Vaild_S. Based on the object location information s0 in the target image frame Fcur, the outline of the object with a predetermined posture (i.e., a stationary standing human body) can be displayed; then, the outline is updated by a preset ratio to obtain the updated outline.

[0075] For example, the preset ratio can be set between 0 and 1, equal to 1, or greater than 1. That is, updating the outline can include shrinking, scaling up, or enlarging. In some embodiments, when the preset ratio is greater than 1 and the outline is rectangular, the outline shown by the object position information s0 of the target image frame is used as a base, and the outline is updated (i.e., enlarged) according to the preset ratio with the center point of the outline as the center, resulting in the updated outline (x0, y0, w, h), where (x0, y0) represents the coordinates of the lower left corner of the updated outline, and w and h represent the width and height of the updated outline, respectively. In some embodiments, when the preset ratio is equal to 1, that is, the outline of the object is obtained proportionally based on the object position information s0 of the target image frame; when the preset ratio is between 0 and 1, that is, the outline shown by the object position information s0 of the target image frame is used as a base, and the outline is updated (i.e., shrunk) according to the preset ratio with the center point of the outline as the center, resulting in the updated outline. It should be noted that the preset ratio can be adjusted according to actual needs, and this disclosure does not impose any restrictions on it.

[0076] Figure 3 A schematic diagram illustrating the principle of the determining unit excluding a second invalid posture from invalid predetermined postures according to an embodiment of the present disclosure is shown.

[0077] like Figure 3 As shown, based on the updated bounding box (x0, y0, w, h), the first reference image block Ifbase1 and the first target image block Ifcur1 are extracted from the first reference image frame Fbase1 and the first target image frame Fcur1, respectively. Since the first reference image frame Fbase1 and the first target image frame Fcur1 are at different times in the video stream, the extracted first reference image block Ifbase1 and the first target image frame Fcur1 are also image blocks of the same size at different times, which facilitates the analysis of the pose change process of the object on s0 in the target image frame Fcur at different times in the same region.

[0078] Figure 4A A comparison diagram of a target image block, a reference image block, and their differing regions is shown according to an embodiment of the present disclosure when no one is standing. Figure 4BA comparison diagram of a target image block, a reference image block, and their differing regions when a person is standing is shown according to an embodiment of the present disclosure.

[0079] For example, when the predetermined posture is a standing posture, Figure 4B In the image, 420, 421, and 422 represent objects with standing postures, i.e., scenes where someone is standing. These correspond to the reference image block Ifbase, the target image block Ifcur, and the difference between them. Significant differences are observed. When the difference exceeds a threshold, it indicates that the standing person's movement is large, and the standing person satisfies the posture transition from sitting to standing. Therefore, the standing person in the target image frame can be determined as validly standing.

[0080] For ease of understanding, Figure 4A Figures 410, 411, and 412 respectively show the reference image block Ifbase, the test image block Ifcur′, and the difference region between them in a scene where no one is standing. Since the test image block Ifcur′ does not contain a standing human body, it cannot be called the target image block Ifcur; therefore, it is referred to as Ifcur′ for distinction. It can be seen that the difference is small. When the difference is less than or equal to the difference threshold, that is, the target object's movement amplitude at different times in the same position is small, and there is basically no posture transition from sitting to standing, such as a human body maintaining a standing posture after starting a smart classroom or recording system. Therefore, the posture of the standing human body in the current target image frame can be determined as an invalid static standing posture.

[0081] The posture recognition device provided in this embodiment calculates the image differences at different times at the same location, avoiding the misclassification of a person who maintains a standing posture but has a small range of motion as a valid standing person, reducing interference from invalid standing behavior, and further improving the accuracy of valid object recognition.

[0082] In an optional implementation, the determining unit 130 is further configured to: for each of the N image frames, add the number of target objects in the image frame to the target object data set; if the difference is determined to be greater than the difference threshold, determine the number of target objects that appear most frequently in the current target object data set as the target object number threshold; and if the number of target objects in the target image frame is determined to be greater than the target object number threshold, determine the pose of an object with a predetermined pose in the target image frame as a second invalid pose.

[0083] This optional implementation addresses situations where standing is not considered valid even if the difference exceeds a threshold. For example, in a real classroom setting, a teacher might walk from the podium or classroom door to the student area and remain standing when detected as a standing person. This situation needs to be excluded in the broadcasting or recording system. To exclude invalid standing in this case, the number of the most frequently occurring target objects can be counted based on the real-time updated target object data set Pk, denoted as the target object number threshold Pnum. When the number of target objects Pnum′ > Pnum in a target image frame is detected, it indicates that someone else has entered the student's view, and their standing is also considered invalid.

[0084] The posture recognition device provided in this embodiment improves the accuracy and reliability of posture recognition by combining the number of target objects in each frame with the dimension of object position information. Furthermore, it effectively avoids issues such as missed detections and false positives during student posture detection. For example, when a teacher appears in the student's frame and is standing, it excludes interference information from standing events that do not occur on the student's end.

[0085] In an optional implementation, the invalid predetermined posture further includes a second invalid posture; the determining unit 130 is further configured to: when the average moving distance is determined to be less than or equal to a distance threshold, in response to the acquisition that the current valid data set is an empty set, update the outline of the object with the predetermined posture in the target image frame according to the object position information of the target image frame and a preset ratio; extract the target image block and the reference image block from the target image frame and the reference image frame respectively according to the outline; calculate the difference between the target image block and the reference image block; and when the difference is determined to be less than or equal to a difference threshold, determine the posture of the object with the predetermined posture in the target image frame as the second invalid posture.

[0086] In the embodiments of this disclosure, if the current valid data set Valid_S is empty, that is, it is impossible to determine first whether the s0 of the current target image frame Fcur exists in Valid_S, then the process of eliminating the second invalid pose is directly entered.

[0087] Furthermore, the determining unit 130 is also configured to: for each of the N image frames, add the number of target objects in the image frame to the target object data set; if the difference is determined to be greater than the difference threshold, determine the number of target objects that appear most frequently in the current target object data set as the target object number threshold; and if the number of target objects in the target image frame is determined to be greater than the target object number threshold, determine the pose of the object with a predetermined pose in the target image frame as the second invalid pose.

[0088] This operation uses the target object dataset to exclude the second invalid posture, which is the same as the case mentioned above where the posture is still not considered valid even if the difference is greater than the difference threshold, so it will not be repeated here.

[0089] Figure 5 A schematic diagram of the structure of an electronic device suitable for a posture recognition device according to an embodiment of the present disclosure is shown.

[0090] like Figure 5 As shown, the electronic device 500 includes an image compression apparatus according to any embodiment of this disclosure. For example, the electronic device 500 may include the pose recognition apparatus 100 provided in the foregoing embodiments.

[0091] In some embodiments, the electronic device 500 may be a smartphone, camera, camcorder, tablet computer, laptop computer, monitoring equipment, etc.

[0092] It should be noted that the division of units in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be physically stored separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0093] For example, any plurality of modules in the extraction unit 110, detection unit 120, and determination unit 130 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the extraction unit 110, detection unit 120, and determination unit 130 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the extraction unit 110, detection unit 120, and determination unit 130 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0094] Embodiments of this disclosure also provide a posture recognition method performed by a posture recognition device, which will be described below in conjunction with... Figure 6 and Figure 7The method is described in detail.

[0095] Figure 6 A flowchart of a pose recognition method according to an embodiment of the present disclosure is shown.

[0096] like Figure 6 As shown, the posture recognition method of this embodiment can be applied to, for example, the posture recognition device 100 according to the foregoing embodiments of this disclosure. The posture recognition method of this embodiment includes operations S610 to S630.

[0097] In operation S610, N image frames are extracted from the video stream to be processed, where N is an integer greater than or equal to 2.

[0098] In this embodiment of the disclosure, operation S610 is performed by extraction unit 110, which corresponds to the operation performed by extraction unit 110 described above. For the sake of simplicity, it will not be described again here.

[0099] In operation S620, for each of the N image frames, the target object and object position information in the image frame are detected respectively. The object position information indicates the position information of the object with a predetermined posture in the target object. The predetermined posture includes a valid predetermined posture and an invalid predetermined posture.

[0100] In this embodiment of the disclosure, operation S620 is performed by the detection unit 120, and the operation performed by the detection unit 120 is the same as the operation performed by the detection unit 120 described above. For the sake of simplicity, it will not be described again here.

[0101] In operation S630, based on the object position information, the effective object position information with a valid predetermined posture is determined among the target objects.

[0102] In this embodiment of the disclosure, operation S630 is performed by the determining unit 130, and the operation performed by the determining unit 130 is corresponding to the operation performed by the determining unit 130 described above, and will not be described again here for the sake of simplicity.

[0103] The posture recognition method provided in this disclosure can process video streams acquired in closed scenarios such as smart classrooms and intelligent recording and broadcasting frame by frame to obtain the number of target objects in the current image frame and the position information of target objects with predetermined postures. This allows for the rapid and accurate identification of the position information of valid objects with valid predetermined postures, while eliminating invalid predetermined postures that cause interference. This helps to evaluate the quality of classroom teaching and student learning.

[0104] Figure 7 A schematic diagram illustrating the principle of a pose recognition method according to an embodiment of the present disclosure is shown.

[0105] like Figure 7 As shown, the posture recognition method of this embodiment includes operations S710 to S730.

[0106] In operation of S710, N image frames are extracted from the video stream to be processed.

[0107] During operation of S720, the input image frame is read, and the number of targets and object location information in the image frame are detected.

[0108] In operation S730, based on the object position information, the effective object position information with a valid predetermined posture is determined among the target objects.

[0109] In this embodiment of the disclosure, for each of the N image frames, the above-mentioned operation S730 may further include operations S731 to S734.

[0110] In operation S731, the valid data set Valid_S is initialized, and the first invalid pose is excluded from the invalid pre-defined poses.

[0111] In operation S732, it is determined whether the object position information s0 of the image frame exists in Valid_S. If it does, the operation S720 is returned to the above operation to process the next image frame; otherwise, the operation S733 is entered.

[0112] In operation S733, the second invalid attitude is excluded from the invalid pre-defined attitudes.

[0113] In operation S734, the valid data set Valid_S is updated, and the operation S720 is returned to proceed with the processing of the next image frame.

[0114] It should be noted that the steps of the above method embodiments correspond to the actions performed by each unit in the aforementioned posture recognition device embodiments. For the specific implementation process of each step in the method embodiments of this disclosure, please refer to the detailed functional description of each unit in the aforementioned posture recognition device embodiments, which will not be repeated here.

[0115] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a posture recognition method according to an embodiment of the present disclosure.

[0116] like Figure 8As shown, an electronic device 800 according to an embodiment of this disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0117] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0118] According to embodiments of this disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0119] Embodiments of this disclosure also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the attitude recognition method according to embodiments of this disclosure.

[0120] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.

[0121] Embodiments of this disclosure also provide a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the gesture recognition method of the embodiments of this disclosure.

[0122] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0123] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0124] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by processor 801, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0125] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module sub-circuit, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0128] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A posture recognition device, comprising: The extraction unit is configured to extract N image frames from the video stream to be processed, and extract a reference image frame from the video stream every m frames, where m and N are both integers greater than or equal to 2. The detection unit is configured to detect the target object and object position information in each of the N image frames. The object position information indicates the position information of an object with a predetermined posture in the target object. The predetermined posture includes a valid predetermined posture and an invalid predetermined posture. The invalid predetermined posture includes a first invalid posture. as well as The determining unit is configured to determine the effective object position information with a valid predetermined posture in the target object based on the object position information; The determining unit is further configured to: For each of the N image frames, in response to the detection unit detecting that the image frame has object location information, the image frame is determined as the target image frame; as well as Add the effective object location information of at least one target object in the target image frame whose pose is a valid predetermined pose to the effective data set. The determining unit is further configured to: Extract u consecutive image frames from the target image frame of the video stream, where u is a positive integer less than m; In response to the detection unit detecting that at least one of the u image frames has object position information, the average moving distance of the object with a predetermined posture in the u image frames is determined based on the object position information. as well as If the average moving distance is determined to be greater than the distance threshold, the pose of the object with the predetermined pose in the target image frame is determined as the first invalid pose; The determining unit is further configured to: If the average moving distance is determined to be less than or equal to the distance threshold, at least one valid object location information in the current valid data set is obtained; For each valid object location information in the at least one valid object location information, calculate the intersection-union ratio (IU / R) between the valid object location information and the object location information of the target image frame to obtain at least one IU / R. From the at least one crossover-union ratio, determine the maximum crossover-union ratio; as well as If the maximum cross-union ratio is determined to be greater than or equal to the cross-union ratio threshold, objects with a predetermined pose in the target image frame are determined to be valid objects.

2. The apparatus according to claim 1, wherein, The invalid predetermined posture also includes a second invalid posture; The determining unit is further configured to: If the maximum cross-union ratio is determined to be less than the cross-union ratio threshold, the outline of the object with a predetermined pose in the target image frame is updated according to the object position information of the target image frame and the preset ratio. Based on the bounding box, target image blocks and reference image blocks are extracted from the target image frame and the reference image frame, respectively. Calculate the difference between the target image patch and the reference image patch; as well as If the difference is determined to be less than or equal to the difference threshold, the pose of an object with a predetermined pose in the target image frame is determined as the second invalid pose.

3. The apparatus according to claim 2, wherein, The determining unit is further configured to: For each of the N image frames, the number of target objects in that image frame is added to the target object data set; If the difference is determined to be greater than the difference threshold, the number of target objects that appear most frequently in the current target object data set is determined as the target object number threshold. as well as If the number of target objects in the target image frame is determined to be greater than the target object number threshold, the pose of the object with a predetermined pose in the target image frame is determined as the second invalid pose.

4. The apparatus according to claim 1, wherein, The invalid predetermined posture also includes a second invalid posture; The determining unit is further configured to: If the average moving distance is determined to be less than or equal to the distance threshold, in response to the current valid data set being found to be empty, the outline of the object with a predetermined posture in the target image frame is updated according to the object position information of the target image frame and the preset ratio. Based on the bounding box, target image blocks and reference image blocks are extracted from the target image frame and the reference image frame, respectively. Calculate the difference between the target image patch and the reference image patch; as well as If the difference is determined to be less than or equal to the difference threshold, the pose of an object with a predetermined pose in the target image frame is determined as the second invalid pose.

5. The apparatus according to claim 4, wherein, The determining unit is further configured to: For each of the N image frames, the number of target objects in that image frame is added to the target object data set; If the difference is determined to be greater than the difference threshold, the number of target objects that appear most frequently in the current target object data set is determined as the target object number threshold. as well as If the number of target objects in the target image frame is determined to be greater than the target object number threshold, the pose of the object with a predetermined pose in the target image frame is determined as the second invalid pose.

6. The apparatus according to claim 1, wherein, The determining unit is further configured to: For each of the N image frames, in response to the detection unit not detecting the object location information of that image frame, the current valid data set is cleared.

7. The apparatus according to claim 1, wherein, The determining unit is further configured to: If the maximum cross-union ratio is determined to be less than the cross-union ratio threshold, the object location information in the target image frame is deleted from the current valid data set.

8. An electronic device, comprising: The posture recognition device according to any one of claims 1 to 7.

9. A pose recognition method, comprising: Extract N image frames from the video stream to be processed. Every m frames, extract a reference image frame from the video stream. Both m and N are integers greater than or equal to 2. For each of the N image frames, the target object and object position information in the image frame are detected respectively. The object position information indicates the position information of the object with a predetermined posture in the target object. The predetermined posture includes a valid predetermined posture and an invalid predetermined posture. The invalid predetermined posture includes a first invalid posture. as well as Based on the object location information, determine the valid object location information with a valid predetermined posture among the target objects; For each of the N image frames, in response to the detection unit detecting that the image frame has object location information, the image frame is determined as the target image frame; as well as Add the effective object location information of at least one target object in the target image frame whose pose is a valid predetermined pose to the effective data set. Extract u consecutive image frames from the target image frame of the video stream, where u is a positive integer less than m; In response to the detection unit detecting that at least one of the u image frames has object position information, the average moving distance of the object with a predetermined posture in the u image frames is determined based on the object position information. as well as If the average moving distance is determined to be greater than the distance threshold, the pose of the object with the predetermined pose in the target image frame is determined as the first invalid pose; If the average moving distance is determined to be less than or equal to the distance threshold, at least one valid object location information in the current valid data set is obtained; For each valid object location information in the at least one valid object location information, calculate the intersection-union ratio (IU / R) between the valid object location information and the object location information of the target image frame to obtain at least one IU / R. From the at least one crossover-union ratio, determine the maximum crossover-union ratio; as well as If the maximum cross-union ratio is determined to be greater than or equal to the cross-union ratio threshold, objects with a predetermined pose in the target image frame are determined to be valid objects.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to claim 9.

11. A computer program product comprising a computer program that, when executed by a processor, implements the method according to claim 9.

Images