A method and device for detecting bad postures

Abstract

The present application discloses a method for detecting bad postures. The method includes: obtaining an image frame, performing object detection on the image frame, detecting a first object, screening out a second object that meets prior conditions from the first object, retrieving a third object from the second object that matches the feature data of a specified object, performing posture estimation on the third object to obtain a posture estimation result of the third object, comparing the posture estimation result with the standard posture data of the specified object, and when there is a difference in the comparison result, determining it as a bad posture; wherein the feature data of the specified object and the standard posture data are pre-stored. The present application enables the exclusion of the posture estimation of non-specified objects during the posture detection process, while also taking into account the differences in individual skeletal structures, reducing the false detection of bad postures, and improving the accuracy and reliability of bad posture detection.

Description

Technical Field

[0001] The present invention relates to the field of image detection, and in particular, to a method for detecting bad postures. Background Art

[0002] Currently, the main method of posture recognition based on computer vision is to obtain an image, detect the targets in the image, and judge whether there are bad postures based on the postures detected and recognized. The existing posture detection cannot make a judgment on the interference scene when there are other targets besides the specified target in the image, cannot distinguish the specified target from the non-specified target, and is prone to false detection of target detection. In addition, during the detection of bad postures, due to the differences in limb structures of each target, there are relatively large false detections for some bad postures. Summary of the Invention

[0003] The present invention provides a method for detecting bad postures to improve the reliability of posture detection of specified targets.

[0004] A method for detecting bad postures provided by the present invention includes:

[0005] Obtain an image frame,

[0006] Perform target detection on the image frame and detect a first target,

[0007] Screen out a second target that meets the prior conditions from the first target,

[0008] Retrieve a third target from the second target whose feature data matches the specified target,

[0009] Perform posture estimation on the third target to obtain a posture estimation result of the third target,

[0010] Compare the posture estimation result with the standard posture data of the specified target, and when there is a difference in the comparison result, determine it as a bad posture;

[0011] Among them,

[0012] The feature data of the specified target and the standard posture data are stored in advance.

[0013] Preferably, performing posture estimation on the third target includes: estimating the posture in front of the table within a set space range relative to the support surface of the table,

[0014] The step of comparing the posture estimation result with the standard posture data of the specified target and determining it as a bad posture when there is a difference in the comparison result includes:

[0015] When there is a difference in the comparison result, record the current image frame,

[0016] Count the image frames recorded within the first time period of the statistical setting, and use the counted image frames as an image frame group.

[0017] Determine whether the image frame group is continuous and reaches the set quantity threshold. If so, it is determined as a bad pre-desk posture.

[0018] Preferably, the pre-desk posture estimation within the set space range relative to the desk support surface includes:

[0019] Obtain the skeleton point information of the third target. Among them, the skeleton points include the left and right eye skeleton points, left and right ear skeleton points, nose skeleton point, and left and right shoulder skeleton points, left and right elbow skeleton points, and left and right wrist skeleton points on the face; the skeleton point information includes the pixel coordinate information and confidence level of each skeleton point.

[0020] Obtain the segmentation information of the desk in the image frame. The segmentation information includes the pixel coordinate information of the upper edge of the desk.

[0021] According to the pixel coordinate information in the segmentation information and the pixel coordinate information in the skeleton point information, calculate the distance from each skeleton point to the desk.

[0022] The comparison of the posture estimation result with the standard posture data of the specified target includes:

[0023] Match the calculated distance from each skeleton point to the desk with the distance from each skeleton point to the desk in the standard posture data. If the match is successful, it is determined that the current posture is the standard posture; otherwise, it is determined that the current posture is a bad pre-desk posture.

[0024] Preferably, when the current posture is a bad pre-desk posture, it further includes:

[0025] Detect the type of the bad pre-desk posture. The bad pre-desk posture includes one of the following postures:

[0026] Bending-over-the-desk posture,

[0027] Supporting posture with the elbows on the desk support surface,

[0028] Head-down posture with the distance from the head to the desk support surface less than the set first distance threshold,

[0029] Head-tilting posture with the head at a non-parallel angle in the height direction relative to the desk support surface,

[0030] Uneven-shoulder posture with different heights of the left and right shoulders relative to the desk support surface,

[0031] Front-back shoulder posture with different front-back distances of the left and right shoulders in the plane where the height direction of the left and right shoulders relative to the desk support surface is located;

[0032] For each detected pre-desk bad posture,

[0033] Record the image frames including this type of pre-desk bad posture,

[0034] Count the image frames recorded within the first time period of this type of pre-desk bad posture, and use the counted image frames as an image frame group,

[0035] Determine whether the image frame group is continuous and reaches the quantity threshold for the continuity of this type of pre-desk bad posture. If so, it is determined as a pre-desk bad posture;

[0036] Among them,

[0037] The first time period for each type of pre-desk bad posture is different, and the quantity threshold for each type of pre-desk bad posture is different.

[0038] Preferably, the desk posture, head-down posture, head-tilted posture, front-back shoulder posture, and support state are obtained through the following detection methods:

[0039] For each posture,

[0040] Calculate the description parameters between the key points associated with this posture. When the calculated description parameters reach the set parameter threshold, perform counting,

[0041] Count the counting results. When the counting results are greater than the set counting threshold, determine this posture as the corresponding bad posture;

[0042] The uneven shoulder posture is detected in the following manner:

[0043] Calculate the deviation in the height direction between the left and right shoulder bone points. When this deviation is greater than the set height deviation threshold, it is determined as an uneven shoulder state.

[0044] Preferably, calculating the description parameters between the key points associated with this posture and performing counting when the calculated description parameters reach the set parameter threshold includes:

[0045] For the image frames,

[0046] Respectively calculate the first distances from each bone point located on the face to the upper edge of the desk, and perform weighted averaging on the calculated first distance values to obtain a first weighted average result; respectively calculate the second distances from each bone point located on the face in the standard posture data to the upper edge of the desk, and perform weighted averaging on the calculated second distance values to obtain a second weighted average result. If the first weighted average result is less than the second weighted average result, perform counting,

[0047] Calculate the third distance between any bone point located on the face and a shoulder bone point. If the third distance is less than the set first distance threshold, perform counting,

[0048] Calculate the position offset between any bone point on the face and the corresponding bone point in the standard pose data. When the offset exceeds the set offset threshold, count is performed. The offset is described by the angle between the line connecting any bone point on the face in the current image frame and the corresponding bone point in the standard pose data and the set direction.

[0049] When the counted result is greater than the set first count threshold, it is determined as a head-down pose.

[0050] Preferably, when calculating the description parameters between the key points associated with this pose and the calculated description parameters reach the set parameter threshold, count is performed, including:

[0051] Calculate the horizontal angle between the lines where the bone points of the left and right eyes are located in the image frame, the horizontal angle between the lines where the bone points of the left and right ears are located, and the horizontal angle between the lines where the bone points of the left and right shoulders are located respectively. Whenever the angle is greater than the set respective second angle threshold, count is performed.

[0052] If the counted result is greater than the set second count threshold, it is determined as a head-tilted pose.

[0053] Preferably, when calculating the description parameters between the key points associated with this pose and the calculated description parameters reach the set parameter threshold, count is performed, including:

[0054] For the image frame,

[0055] Calculate the first distance from each bone point on the face to the upper edge of the table respectively, perform weighted averaging on the calculated first distance values to obtain the first weighted average result; calculate the second distance from each bone point on the face in the standard pose data to the upper edge of the table respectively, perform weighted averaging on the calculated second distance values to obtain the second weighted average result. If the first weighted average result is less than the second weighted average result, count is performed.

[0056] Calculate the third distance between any bone point on the face and a shoulder bone point. If the third distance is less than the set second distance threshold, count is performed.

[0057] For the bone points on the same side, calculate the angle between the first line where the shoulder bone point and the elbow bone point are located and the second line where the elbow bone point and the wrist bone point are located. When the angle is less than the set third angle threshold, count is performed.

[0058] If the counted result is greater than the set third count threshold, it is determined as a desk-leaning pose.

[0059] Preferably, when calculating the description parameters between the key points associated with the posture, counting is performed when the calculated description parameters reach the set parameter threshold, including:

[0060] Calculate the confidence levels of the left and right shoulder bone points, left and right eye bone points, and left and right ear bone points in the image frame respectively. If any confidence level exceeds the set confidence threshold, then perform counting.

[0061] If the counted result is greater than the set fourth counting threshold, it is determined as the front and back shoulder posture.

[0062] Preferably, when calculating the description parameters between the key points associated with the posture, counting is performed when the calculated description parameters reach the set parameter threshold, including:

[0063] Calculate the fourth distance from the wrist bone point to the upper edge of the table and the fifth distance between the wrist bone point and the shoulder bone point in the image frame respectively. If the fourth distance is greater than the set third distance threshold and the fifth distance is less than the set fourth distance threshold, then perform counting.

[0064] When the counted result is greater than the set fifth counting threshold, it is determined as the supporting posture.

[0065] Preferably, the characteristic data of the specified target and the standard posture data are obtained through the following methods:

[0066] Obtain the image frame of the specified target in the standard posture, detect the target in the image frame, and obtain the segmentation information of the table in the image frame. The segmentation information includes the pixel coordinate information of the upper edge of the table.

[0067] Perform re-identification on the detected target, retrieve the specified target, and extract the characteristic data of the specified target.

[0068] Obtain the bone point information of the specified target, which includes the left and right eye bone points, left and right ear bone points, nose bone point, and left and right shoulder bone points, left and right elbow bone points, left and right wrist bone points located on the face; the bone point information includes the pixel coordinate information and confidence level of each bone point.

[0069] According to the pixel coordinate information in the segmentation information and the pixel coordinate information in the bone point information of the specified target, calculate the distance from each bone point to the upper edge of the table as the standard posture data of the specified target.

[0070] The present invention also provides a bad posture detection device, which includes an image acquisition device and a detection device.

[0071] [[ID=,36]]The image acquisition device is used to obtain an image frame.

[0072] The detection device is used for

[0073] performing object detection on the image frames obtained by the image acquisition device, detecting a first object,

[0074] screening out a second object that meets the prior conditions from the first object,

[0075] retrieving a third object that matches the feature data of the specified object from the second object,

[0076] performing pose estimation on the third object to obtain a pose estimation result of the third object,

[0077] comparing the pose estimation result with the standard pose data of the specified object, and when there are differences in the comparison result, determining it as a bad pose;

[0078] wherein,

[0079] the feature data of the specified object and the standard pose data are pre-stored.

[0080] The present invention further provides a bad pose detection device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the steps of any of the bad pose detection methods are implemented.

[0081] A bad pose detection method provided by the present invention screens out the objects in the image frames that match the specified object through the pre-stored feature data of the specified object, avoiding the interference caused by non-specified objects in the pose estimation process. Through the pre-stored standard pose data of the specified object, the pose detection process not only excludes the pose estimation of non-specified objects but also takes into account the differences in individual bone structures, reducing the false detection of bad poses, improving the accuracy and reliability of bad pose detection, facilitating use in public places, and enhancing universality. For various bad poses, multiple strategies are adopted for identification, improving the accuracy of bad pose detection. Brief Description of the Drawings

[0082] Figure 1 is a schematic flow chart of a bad pose detection method of the present application

[0083] Figure 2 is a schematic flow chart for obtaining the standard pose data of the specified object.

[0084] Figure 3 is a schematic diagram of the distribution of 11 bone points.

[0085] Figure 4 is a schematic flow chart of a bad pose detection method before the case.

[0086] Figure 5 It is a schematic diagram of the timing for detecting bad postures.

[0087] Figure 6 It is a schematic diagram for detecting the types of bad postures.

[0088] Figure 7 It is a side view schematic diagram of the bone points in the head-down posture and the non-head-down posture.

[0089] Figure 8 It is a side view schematic diagram of the bone points in the head-down posture and the non-head-down posture.

[0090] Figure 9 It is a side view schematic diagram of the bone points in the head-down posture.

[0091] Figure 10 It is a front view schematic diagram of the bone points in the head-tilted posture.

[0092] Figure 11 It is a side view schematic diagram of the bone points in the desk-bending posture.

[0093] Figure 12 It is a front view schematic diagram of the bone points in the uneven shoulder posture.

[0094] Figure 13 It is a side view schematic diagram of the bone points in the supporting posture.

[0095] Figure 14 It is a schematic diagram for the timing of detecting each bad posture.

[0096] Figure 15 It is a schematic diagram of the bad posture detection device of the present application.

[0097] Figure 16 It is a schematic diagram of the detection device of the present application. Detailed implementation manners

[0098] In order to make the purpose, technical means and advantages of the present application clearer and more understandable, the following further elaborates on the present application in conjunction with the accompanying drawings.

[0099] The present application pre-gets the feature data of a specified target and the standard posture data of the specified target, performs target detection on the acquired image frames and retrieves through the feature data of the specified target, so as to screen out the specified target in the image frames, performs posture estimation on the screened target, and compares the posture estimation result with the standard posture data of the specified target, thereby detecting bad postures.

[0100] See Figure 1 as shown Figure 1It is a schematic flowchart of a method for detecting bad postures in this application. The method includes:

[0101] Step 101, obtain an image frame.

[0102] Step 102, perform object detection on the image frame and detect a first object.

[0103] Step 103, screen out a second object that meets the prior conditions from the first object.

[0104] Step 104, retrieve a third object from the second object that matches the feature data of a specified object stored in advance.

[0105] Step 105, perform posture estimation on the third object to obtain the posture estimation result of the third object.

[0106] Step 106, compare the posture estimation result with the standard posture data of the specified object stored in advance. When there are differences in the comparison result, it is determined as a bad posture.

[0107] This application avoids misdetection caused by the bone differences of the detected object and improves the robustness of bad posture detection.

[0108] In practical applications, it is often necessary to remind teenagers among family members of bad postures to avoid affecting the development of teenagers due to bad postures. Given that teenagers spend a lot of time studying, therefore, the detection of bad postures in front of the desk has become one of the key points of bad postures.

[0109] The following takes the detection of the posture in front of the desk with the human body as the detected object as an example to illustrate the posture detection method of this application. The "in front of the desk" refers to the set spatial range of the target relative to the support surface of the desk. The desk includes but is not limited to, tables, operating tables, platforms, etc. Bad postures in front of the desk include but are not limited to, bending over the desk posture, hand-supporting posture with the elbow supported on the desk support surface, head-down posture with the distance between the head and the desk support surface less than the set first distance threshold, head-tilting posture with a non-parallel angle in the height direction of the head relative to the desk support surface, high-low shoulder posture with different heights of the left and right shoulders relative to the desk support surface, and front-back shoulder posture with different front-back distances of the left and right shoulders in the plane where the height direction of the desk support surface is located.

[0110] In order to implement posture detection on the specified object and avoid the interference of non-specified objects, in this embodiment, the specified object and its standard posture data in front of the desk are collected and stored in advance to obtain the feature data of the specified object and its standard posture data.

[0111] See Figure 2 as shown Figure 2A schematic flow chart for obtaining the standard pose data of a specified target. It includes:

[0112] Step 201, obtain an image frame of the specified target in the standard pose,

[0113] Step 202, use an object detection algorithm to detect the target in the image frame, and save the size of the detection box where the detected target is located, its position in the image frame, and the corresponding confidence level.

[0114] Among them, the object detection algorithm can be the CENTERNET algorithm (Objects as Points), and this algorithm can quickly and accurately identify human targets in the scene.

[0115] Step 203, use an image retrieval algorithm to re-identify the detected target to determine whether the detected target is the specified target.

[0116] Among them, the image retrieval algorithm can be the REID (Person re-identification) algorithm. This algorithm determines whether the specified target exists in the image by extracting the matching degree between the features of the specified target and the set features. It can also use a trained first deep learning model for feature extraction to extract the features of the detected target, and then based on the matching degree between the extracted target features and the set features, to identify whether the detected target is the specified target. Among them, the set features are the features of the specified target.

[0117] When the detected target is the specified target, then execute step 204. Otherwise, obtain the next image frame and return to step 202.

[0118] Step 204, use an image segmentation network algorithm to obtain the segmentation information of the table in the image frame. For example, segment the table edge from the image to obtain the pixel coordinate information of the upper edge of the table as the segmentation information and save it.

[0119] Among them, the image segmentation network algorithm can be a trained second deep learning model for image segmentation.

[0120] Step 205, use a pose estimation algorithm to obtain the upper body bone point information of the specified target. Each bone point includes its pixel coordinates (u coordinate, v coordinate) and confidence level.

[0121] The bone points include 11 bone points: left eye bone point, right eye bone point, left ear bone point, right ear bone point, nose bone point, left shoulder bone point, right shoulder bone point, left elbow bone point, right elbow bone point, left wrist bone point, and right wrist bone point. See Figure 3 as shown. Figure 3It is a schematic diagram of the distribution of 11 skeleton points. The information of each skeleton point can be used as the key point information of the standard pose.

[0122] Among them, the pose estimation algorithm can be the HRNET algorithm (High-Resolution Net), which can detect the key skeleton points of the human body through the detected human detection frame.

[0123] As a variation, it is also possible to obtain the skeleton point information of the specified target located at the upper edge by using the pose estimation algorithm according to the segmentation information in step 204.

[0124] Step 206: According to the skeleton point information and the segmentation information of the table, calculate the distance of each skeleton point relative to the edge of the table (for example, the upper edge) to obtain the standard pose data of the specified target and save it.

[0125] In this way, the standard pose data represents the height of the key points of the specified target in the standard pose relative to the table, and can be used as a judgment benchmark for bad postures.

[0126] In the above steps, there is no sequential relationship between step 204 and steps 202 and 203.

[0127] See Figure 4 as shown Figure 4 It is a schematic flowchart of a method for detecting bad postures in front of the table. The detection method includes the following processing based on the acquired current image frame:

[0128] Step 401: Use the target detection algorithm to detect the target in the image frame, and save the size of the detection frame where the detected first target is located, its position in the image frame, and the corresponding confidence level.

[0129] Step 402: Perform a prior condition judgment on the detected first target. The judgment method can specifically be:

[0130] For each detected first target, weight the size of the detection frame of the first target, its position in the image frame, and the confidence level to obtain a weighted result, and store it for use as the detection information of the image frame.

[0131] Compare the weighted result with the weighted result of the previous image frame. If the comparison result exceeds the set comparison threshold, it is determined that the target does not meet the prior conditions, and thus the first target that does not meet the prior conditions is excluded. Otherwise, it is determined that the first target meets the prior conditions, and the first target is retained to obtain the second target.

[0132] Through this step, the human target in the image frame can be detected.

[0133] Step 403: Using an image retrieval algorithm or a trained first deep learning model for feature extraction, extract features of the first target (second target) that meets the prior conditions, and match the extracted features with the feature data of the specified target stored in advance, so as to determine whether the first target (second target) that meets the prior conditions is the specified target according to the matching result.

[0134] Given that there may be multiple second targets, each second target can be separately subjected to feature extraction and feature matching to retrieve the specified target, thereby screening out the third target. In this way, the specified target can be retrieved from the first targets.

[0135] Step 404: Using a pose estimation algorithm, perform pose estimation on the screened third target to obtain the skeletal point information of the third target, that is, the skeletal point information of the specified target. Among them, each skeletal point includes its pixel coordinates (u coordinate, v coordinate) and confidence level. There can be a total of 11 skeletal points, and the information of each skeletal point is used as the key point information of the standard pose.

[0136] Preferably, perform pose estimation on the upper body of the screened third target.

[0137] Step 405: Using an image segmentation network algorithm, obtain the segmentation information of the table in the image frame. This segmentation information can be the pixel coordinate information of the upper edge of the table.

[0138] Preferably, since the spatial position of the table usually remains unchanged in practical applications, the segmentation information of the table can be saved when it is obtained for the first time.

[0139] Step 406: According to the segmentation information of the table and the skeletal point information of the specified target, calculate the distance from each skeletal point to the table.

[0140] In this step, the distance from each skeletal point to the table can be calculated according to the pixel coordinate information of the upper edge of the table and the pixel coordinate information of the skeletal points of the specified target.

[0141] Step 407: Compare the distance from the skeletal points of the specified target to the table with the standard pose data of the specified target, and determine whether there is a bad pose according to the comparison result. When there is a difference in the comparison result, it is determined as a bad pose.

[0142] As an example, when there is a bad pose, mark the current image frame with a first mark. For example, record the timestamp or frame number of the current image frame, and execute Step 408. Otherwise, end the pose detection in the current image frame and return to Step 401 to process the next image frame.

[0143] Step 408. To improve the reliability of the detection results and eliminate false detections, refer to Figure 5 as shown in Figure 5 FIG. Figure 5 is a schematic diagram of the timing of bad posture detection. Statistically record the image frames within a set first time period, and use the statistically recorded image frames as an image frame group. Determine whether the image frame group is continuous and reaches a set quantity threshold. If so, it is determined as a bad posture before the case.

[0144] As an example, take the image frames with a first mark within each set first time period t as a group, perform timing marking to obtain an image frame group with a second mark, and determine whether the image frame group with the second mark reaches M consecutively. If so, it is determined that the current posture belongs to a bad posture; otherwise, it is determined that the current posture belongs to a standard posture.

[0145] Among them, the image frames with the first mark are usually 3 to 5 frames, and M is usually 30 to 50. In this way, it is equivalent to distinguishing between bad postures with a longer duration and bad postures with a shorter timing time, thereby avoiding misjudging the postures in specific actions (such as occasional lowering of the head, etc.) as bad postures.

[0146] Preferably, for the image frames with the first mark, a voting method is further used to detect the types of bad postures. That is, for each posture, calculate the description parameters between the key points associated with the posture. When the calculated description parameters reach the set parameter threshold, count. Statistically record the counting result. When the counting result is greater than the set counting threshold, it is determined that the posture is the corresponding bad posture. Among them, the counting threshold can be set according to the detection sensitivity of the bad posture. The smaller the counting threshold, the easier it is to determine as a bad posture, and the higher the sensitivity.

[0147] Among them, refer to Figure 6 as shown in Figure 6 FIG. Figure 6 is a schematic diagram of detecting the types of bad postures. Step 407 further includes detecting the types of bad postures:

[0148] Step 4071, refer to Figure 7 as shown in Figure 7 FIG. Figure 7 is a side view schematic diagram of the bone points in the head-down posture and non-head-down posture. For the bone points located on the face, calculate the first distances from each bone point on the face to the upper edge of the desk respectively, and perform weighted averaging on the calculated first distance values to obtain a first weighted average result; calculate the second distances from each bone point on the face in the standard posture data to the upper edge of the desk respectively, and perform weighted averaging on the calculated second distance values to obtain a second weighted average result. If the first weighted average result is less than the second weighted average result, count as 1 as a voting result.

[0149] Secondly, calculate the third distance between a bone point on the face and a bone point on the shoulder. If the third distance is less than the set first distance threshold, count it as 1 as a voting result.

[0150] Among them, the bone points on the face and the shoulder can be on the same side or on different sides. The first distance threshold can be set differently based on whether the bone points are on the same side or different sides, and can be specifically set according to the accuracy and reliability of such pose detection.

[0151] See Figure 8 shown in Figure 8 Fig. is a side view schematic diagram of bone points in the head-down posture and non-head-down posture. When the posture is the head-down state, the third distance is not equal to the third distance in the standard posture. Therefore, when the third distance is less than the first distance threshold, counting can be performed.

[0152] Finally, calculate the position offset between any bone point on the face in the current image frame and the corresponding bone point in the standard posture data. When the offset exceeds the set offset threshold, count it as 1 as a voting result. The offset can be described by the angle between the straight line passing through any bone point on the face in the current image frame and the corresponding bone point in the standard posture data and the set direction.

[0153] See Figure 9 shown in Figure 9 Fig. is a side view schematic diagram of bone points in the head-down posture. In the figure, if the angle between the straight line passing through the left ear bone point in the current image frame and the left ear bone point in the standard posture data and the horizontal direction is greater than the set first angle threshold 1, count it as 1, and / or

[0154] if the angle between the straight line passing through the right ear bone point in the current image frame and the right ear bone point in the standard posture data and the horizontal direction is greater than the set first angle threshold 2, count it as 1, and / or

[0155] if the angle between the straight line passing through the left eye bone point in the current image frame and the left eye bone point in the standard posture data and the horizontal direction is greater than the set first angle threshold 3, count it as 1, and / or

[0156] if the angle between the straight line passing through the right eye bone point in the current image frame and the right eye bone point in the standard posture data and the horizontal direction is greater than the set first angle threshold 4, count it as 1, and / or

[0157] if the angle between the straight line passing through the nose bone point in the current image frame and the nose bone point in the standard posture data and the horizontal direction is greater than the set first angle threshold 5, count it as 1;

[0158] The first included angle thresholds 1, 2, 3, 4, and 5 may be the same or different, and may be specifically set according to the accuracy and reliability of such posture detection. The horizontal direction is the direction perpendicular to the plane where the height direction is located.

[0159] Count all the counting results. If the counting result is greater than the set first counting threshold, it is determined as a lowered head posture.

[0160] Step 4072, see Figure 10 as shown in Figure 10 is a front view schematic diagram of the bone points in a tilted head posture. Calculate the horizontal included angles of the left and right eyes, the horizontal included angles of the left and right ears, and the horizontal included angles of the left and right shoulders in the current image frame respectively. Whenever the included angle is greater than the set respective second included angle threshold, count 1.

[0161] For example,

[0162] When the horizontal included angle of the left and right eyes is greater than the second included angle threshold 1, count 1, and / or

[0163] When the horizontal included angle of the left and right ears is greater than the second included angle threshold 2, count 1, and / or

[0164] When the horizontal included angle of the left and right shoulders is greater than the second included angle threshold 3, count 1.

[0165] The second included angle thresholds 1, 2, and 3 may be the same or different, and may be specifically set according to the accuracy and reliability of such posture detection.

[0166] Count all the counting results. If the counting result is greater than the set second counting threshold, it is determined as a tilted head posture.

[0167] Step 4073, for each bone point located on the face, calculate the distance from the face bone point to the upper edge of the desk respectively, perform weighted averaging on the calculated first distance values to obtain a first weighted average result; calculate the distances from each face bone point in the standard posture data to the upper edge of the desk respectively, perform weighted averaging on the calculated second distance values to obtain a second weighted average result. If the first weighted average result is less than the second weighted average result, count 1 as a voting result.

[0168] Secondly, calculate the third distance between the bone points located on the face and the left and right bone points located on the shoulders respectively. If any one of the third distances is less than the set second distance threshold, count 1 as a voting result.

[0169] Finally, see Figure 11 as shown in Figure 11A side view schematic diagram of skeletal points in a desk-leaning posture. For skeletal points on the same side, calculate the angle between the first straight line passing through the shoulder skeletal point and the elbow skeletal point and the second straight line passing through the elbow skeletal point and the wrist skeletal point. When the angle is less than the set third angle threshold, count.

[0170] Specifically,

[0171] For the first straight line passing through the left shoulder skeletal point and the left elbow skeletal point and the second straight line passing through the left elbow skeletal point and the left wrist skeletal point, calculate the angle between the first straight line and the second straight line. When the angle is less than the set third angle threshold 1, count as 1, and / or, for the first straight line passing through the right shoulder skeletal point and the right elbow skeletal point and the second straight line passing through the right elbow skeletal point and the right wrist skeletal point, calculate the angle between the first straight line and the second straight line. When the angle is less than the set third angle threshold 2, count as 1. Among them, the third angle threshold 1 and the third angle threshold 2 can be the same or different.

[0172] Statistically analyze all the counting results. If the counting result is greater than the set third counting threshold, it is determined to be in a desk-leaning posture.

[0173] Step 4074, see Figure 12 as shown Figure 12 A front view schematic diagram of skeletal points in a high-low shoulder posture. Calculate the deviation in the height direction between the left and right shoulder skeletal points. When the deviation is greater than the set height deviation threshold, it is determined to be in a high-low shoulder state;

[0174] Step 4075, in view of the fact that the front and back shoulders are usually in a turning state, at this time, the left or right side is not fully presented in the image. Therefore, calculate the confidence levels of the left and right shoulders, the left and right eyes, and the left and right ears respectively. If any confidence level exceeds the set confidence level threshold, count as 1 as a voting result,

[0175] Statistically analyze all the counting results. If the counting result is greater than the set fourth counting threshold, it is determined to be in a front-back shoulder posture.

[0176] Step 4076, see Figure 13 as shown Figure 13 A side view schematic diagram of skeletal points in a supporting posture. Calculate the fourth distance from the wrist skeletal point to the upper edge of the desk and the fifth distance between the wrist skeletal point and the shoulder skeletal point respectively. If the fourth distance from the wrist skeletal point to the upper edge of the desk is greater than the set third distance threshold and the fifth distance between the wrist skeletal point and the shoulder skeletal point is less than the set fourth distance threshold, count.

[0177] As an example,

[0178] Calculate the fourth distance from the left wrist bone point to the upper edge of the table, and the fifth distance between the left wrist bone point and the left shoulder bone point. If the fourth distance is greater than the set third distance threshold 1, and the fifth distance is less than the set fourth distance threshold 1, then the count is 1.

[0179] Calculate the fourth distance from the right wrist bone point to the upper edge of the table, and the fifth distance between the right wrist bone point and the right shoulder bone point. If the fourth distance is greater than the set third distance threshold 2, and the fifth distance is less than the set fourth distance threshold 2, then the count is 1.

[0180] Among them, the third distance threshold 1 and the third distance threshold 2 can be the same or different, and the fourth distance threshold 1 and the fourth distance threshold 2 can be the same or different. Specifically, they can be set according to the accuracy and reliability of this type of posture detection.

[0181] Statistical all the counting results. If the counting result is greater than the set fifth counting threshold, it is determined as the hand support state.

[0182] The above steps 4071 to 4076 are for the detection of each pre-table bad posture, and there is no strict time sequence relationship. The first counting threshold, the second counting threshold, the third counting threshold, the fourth counting threshold, and the fifth counting threshold can be different, and can be specifically set according to the detection sensitivity of each posture.

[0183] Similar to step 408, in order to improve the reliability of the detection results of each bad posture and eliminate false detections, see Figure 14 as shown Figure 14 is a schematic diagram of the detection time sequence for each bad posture. For each posture, the image frames with the first mark within each first time period t i are taken as a group for time sequence marking to obtain a group of image frames with the second mark, and it is judged whether the group of image frames with the second mark reaches continuous M i If so, it is determined that the current posture belongs to this type of bad posture; otherwise, it is determined that the current posture does not belong to this type of bad posture.

[0184] It should be understood that the first time period t i for each bad posture can be different, and the continuous quantity M i of the image frame group for each bad posture can be different.

[0185] See Figure 15 as shown Figure 15 is a schematic diagram of a bad posture detection device of this application. The device includes,

[0186] A target detection module for performing target detection on the image frame and detecting the first target.

[0187] A prior detection module, configured to screen out a second target that meets the prior conditions from the first target.

[0188] A specified target screening module, configured to retrieve a third target that matches the feature data of the specified target from the second target.

[0189] An attitude estimation module, configured to perform attitude estimation on the third target to obtain an attitude estimation result of the third target.

[0190] An attitude detection module, configured to compare the attitude estimation result with the standard attitude data of the specified target, and when there is a difference in the comparison result, determine it as a bad attitude.

[0191] Preferably, the device further includes

[0192] An identification module, configured to identify the type of bad attitude.

[0193] The identification module includes

[0194] A head-down attitude recognition sub-module, configured to respectively calculate the first distance from each bone point on the face to the upper edge of the desk, perform weighted averaging on the calculated first distance values to obtain a first weighted average result; respectively calculate the second distance from each bone point on the face in the standard attitude data to the upper edge of the desk, perform weighted averaging on the calculated second distance values to obtain a second weighted average result, and if the first weighted average result is less than the second weighted average result, then perform counting.

[0195] Calculate the third distance between any bone point on the face and a shoulder bone point, and if the third distance is less than a set first distance threshold, then perform counting.

[0196] Calculate the position offset between any bone point on the face and the corresponding bone point in the standard attitude data. When the offset exceeds a set offset threshold, then perform counting. The offset is described by the angle between the straight line where any bone point on the face and the corresponding bone point in the standard attitude data in the current image frame and a set direction; when the counted result is greater than a set first counting threshold, then determine it as a head-down attitude.

[0197] A head-tilting attitude recognition sub-module, configured to respectively calculate the horizontal angle between the straight lines where the left and right eye bone points are located in the image frame, the horizontal angle between the straight lines where the left and right ear bone points are located, and the horizontal angle between the straight lines where the left and right shoulder bone points are located. Whenever the angle is greater than a set respective second angle threshold, then perform counting. If the counted result is greater than a set second counting threshold, then determine it as a head-tilting attitude.

[0198] The desk posture recognition sub-module is used to calculate the first distances from each bone point on the face to the upper edge of the desk respectively, perform weighted averaging on the calculated first distance values to obtain a first weighted average result; calculate the second distances from each bone point on the face in the standard posture data to the upper edge of the desk respectively, perform weighted averaging on the calculated second distance values to obtain a second weighted average result. If the first weighted average result is less than the second weighted average result, then count.

[0199] Calculate the third distance between any bone point on the face and a shoulder bone point. If the third distance is less than the set second distance threshold, then count.

[0200] For the bone points on the same side, calculate the angle between the first straight line where the shoulder bone point and the elbow bone point are located and the second straight line where the elbow bone point and the wrist bone point are located. When the angle is less than the set third angle threshold, then count.

[0201] If the counted result is greater than the set third counting threshold, then it is determined as a desk posture.

[0202] The high and low shoulder posture recognition sub-module is used to calculate the deviation in the height direction between the left and right shoulder bone points. When the deviation is greater than the set height deviation threshold, then it is determined as a high and low shoulder state.

[0203] The front and back shoulder posture recognition sub-module is used to calculate the confidence levels of the left and right shoulder bone points, the left and right eye bone points, and the left and right ear bone points in the image frame respectively. If any confidence level exceeds the set confidence level threshold, then count. If the counted result is greater than the set fourth counting threshold, then it is determined as a front and back shoulder posture.

[0204] The supporting posture recognition sub-module is used to calculate the fourth distance from the wrist bone point to the upper edge of the desk and the fifth distance between the wrist bone point and the shoulder bone point in the image frame respectively. If the fourth distance is greater than the set third distance threshold and the fifth distance is less than the set fourth distance threshold, then count. When the counted result is greater than the set fifth counting threshold, then it is determined as a supporting posture.

[0205] The recognition module further includes

[0206] The statistics sub-module is used to record the image frames including each type of bad posture for each detected bad posture, count the image frames recorded within the first time period of this type of bad pre-desk posture, use the counted image frames as an image frame group, and determine whether the image frame group is continuous and reaches the quantity threshold for the continuity of this type of bad pre-desk posture. If so, then it is determined as a bad posture.

[0207] The device further includes

[0208] A specified target standard pose data acquisition module is used to acquire an image frame of a specified target in the standard pose, detect the target in the image frame, and acquire the segmentation information of the table in the image frame. The segmentation information includes the pixel coordinate information of the upper edge of the table.

[0209] Re-identify the detected target, retrieve the specified target, and extract the feature data of the specified target.

[0210] Acquire the skeletal point information of the specified target.

[0211] According to the pixel coordinate information in the segmentation information and the pixel coordinate information in the skeletal point information of the specified target, calculate the distance from each skeletal point to the upper edge of the table as the standard pose data of the specified target.

[0212] An image segmentation module is used to acquire the segmentation information of the table in the image frame and provide the segmentation information to the pose estimation module.

[0213] See Figure 16 as shown Figure 16 is a schematic diagram of a detection device of this application. It includes an image acquisition device and a detection device. Among them, the detection device includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of any of the above-mentioned bad pose detection methods.

[0214] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0215] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0216] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above-mentioned bad posture detection methods are implemented.

[0217] For the embodiments of the device / network-side device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, please refer to the partial description of the method embodiments.

[0218] In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0219] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for detecting bad postures, characterized in that This method is applied to public places and includes: Obtaining an image frame, Performing object detection on the image frame to detect a first object, Screening out multiple second objects that meet the prior conditions from each detected first object to detect human objects in the image frame. Here, the prior conditions are judged in the following way: for each detected first object, the size of the detection box of the first object, its position in the image frame, and the confidence level are weighted to obtain the weighted result of the first object. The weighted result is compared with the weighted result of the first object in the previous image frame. If the comparison result exceeds the set comparison threshold, it is determined that the first object does not meet the prior conditions; otherwise, it is determined that the first object meets the prior conditions. Retrieving a third object from the second objects that matches the feature data of the specified object, Performing pose estimation on the third object to obtain the pose estimation result of the third object, Comparing the pose estimation result with the standard pose data of the specified object. When there are differences in the comparison result, it is determined as an improper pre-pose, and the type of the improper pre-pose is detected. The improper pre-pose includes one of the following poses: Bending-over-the-desk pose, Supporting pose with the elbows on the support surface of the desk, Head-down pose with the distance from the head to the support surface of the desk less than the set first distance threshold, Head-tilting pose with the head at a non-parallel angle in the height direction relative to the support surface of the desk, Uneven-shoulder pose with different heights of the left and right shoulders relative to the support surface of the desk, Front-and-back-shoulder pose with different front-to-back distances of the left and right shoulders in the plane of the height direction relative to the support surface of the desk, The bending-over-the-desk pose is detected in the following way: For the image frame, Respectively calculate the first distances from each bone point on the face to the upper edge of the desk, and perform weighted averaging on the calculated first distance values to obtain the first weighted average result; respectively calculate the second distances from each bone point on the face in the standard pose data to the upper edge of the desk, and perform weighted averaging on the calculated second distance values to obtain the second weighted average result. If the first weighted average result is less than the second weighted average result, a count is made. Calculate the third distance between any bone point on the face and a shoulder bone point. If the third distance is less than the set second distance threshold, a count is made. For bone points on the same side, calculate the angle between the first straight line where the shoulder bone point and the elbow bone point are located and the second straight line where the elbow bone point and the wrist bone point are located. When the angle is less than the set third angle threshold, a count is made; If the counted result is greater than the set third count threshold, it is determined as the bending-over-the-desk pose; The supporting pose is detected in the following way: Respectively calculate the fourth distance from the wrist bone point in the image frame to the upper edge of the desk and the fifth distance between the wrist bone point and the shoulder bone point. If the fourth distance is greater than the set third distance threshold and the fifth distance is less than the set fourth distance threshold, a count is made. When the counted result is greater than the set fifth count threshold, it is determined as the supporting pose; The head-down pose is detected in the following way: For the image frame, Calculate the first distance from each bone point on the face to the upper edge of the table respectively, perform weighted averaging on the calculated first distance values to obtain a first weighted average result; calculate the second distance from each bone point on the face in the standard pose data to the upper edge of the table respectively, perform weighted averaging on the calculated second distance values to obtain a second weighted average result. If the first weighted average result is less than the second weighted average result, then count. Calculate the third distance between any bone point on the face and a shoulder bone point. If the third distance is less than a set first distance threshold, then count. Calculate the position offset between any bone point on the face and the corresponding bone point in the standard pose data. When the offset exceeds a set offset threshold, then count. The offset is described by the angle between the straight line passing through any bone point on the face and the corresponding bone point in the standard pose data in the current image frame and a set direction. When the counted result is greater than a set first count threshold, then it is determined as a head-down pose. The head-tilt pose is detected in the following way: Calculate the horizontal angles of the straight lines where the left and right eye bone points are located, the horizontal angles of the straight lines where the left and right ear bone points are located, and the horizontal angles of the straight lines where the left and right shoulder bone points are located in the image frame respectively. Whenever the angle is greater than a set respective second angle threshold, then count. If the counted result is greater than a set second count threshold, then it is determined as a head-tilt pose. The front and back shoulder pose is detected in the following way: Calculate the confidence levels of the left and right shoulder bone points, the left and right eye bone points, and the left and right ear bone points in the image frame respectively. If any confidence level exceeds a set confidence threshold, then count. If the counted result is greater than a set fourth count threshold, then it is determined as a front and back shoulder pose. For each type of bad pre-table pose detected, Record the image frame including this type of bad pre-table pose. Count the image frames recorded within a first time period of this type of bad pre-table pose, and take the counted image frames as an image frame group. Judge whether the image frame group is continuous and reaches the quantity threshold for the continuity of this type of bad pre-table pose. If so, then it is determined as a bad pre-table pose. Among them, The first time periods for each type of bad pre-table pose are different, and the quantity thresholds for each type of bad pre-table pose are different. The characteristic data of the specified target and the standard pose data are pre-stored.

2. The method according to claim 1, wherein Performing pose estimation on the third target includes: pre-table pose estimation within a set space range relative to the table support surface. Comparing the pose estimation result with the standard pose data of the specified target. When there are differences in the comparison result, it is determined as a bad pre-table pose, including: When there are differences in the comparison result, record the current image frame. Count the image frames recorded within a set first time period, and take the counted image frames as an image frame group. Judge whether the image frame group is continuous and reaches the set quantity threshold. If so, then it is determined as a bad pre-table pose.

3. The method according to claim 2, wherein The pre-table pose estimation within a set space range relative to the table support surface includes, Obtain the skeletal point information of the third target, where the skeletal points include the left and right eye skeletal points, left and right ear skeletal points, nose skeletal point on the face, as well as the left and right shoulder skeletal points, left and right elbow skeletal points, and left and right wrist skeletal points; the skeletal point information includes the pixel coordinate information and confidence of each skeletal point. Obtain the segmentation information of the table in the image frame, and the segmentation information includes the pixel coordinate information of the upper edge of the table. Calculate the distance from each skeletal point to the table according to the pixel coordinate information in the segmentation information and the pixel coordinate information in the skeletal point information. The comparison of the pose estimation result with the standard pose data of the specified target includes: Match the calculated distance from each skeletal point to the table with the distance from each skeletal point to the table in the standard pose data. If the match is successful, determine that the current pose is the standard pose; otherwise, determine that the current pose is a bad pose in front of the table.

4. The method according to claim 1, characterized in that The high and low shoulder pose is detected in the following way: Calculate the deviation in the height direction between the left and right shoulder skeletal points. When the deviation is greater than the set height deviation threshold, it is determined that it is in a high and low shoulder state.

5. The method according to claim 3, wherein The characteristic data and standard pose data of the specified target are obtained in the following way: Obtain the image frame of the specified target in the standard pose, detect the target in the image frame, and obtain the segmentation information of the table in the image frame. The segmentation information includes the pixel coordinate information of the upper edge of the table. Re-identify the detected target, retrieve the specified target, and extract the characteristic data of the specified target. Obtain the skeletal point information of the specified target, where the skeletal points include the left and right eye skeletal points, left and right ear skeletal points, nose skeletal point on the face, as well as the left and right shoulder skeletal points, left and right elbow skeletal points, and left and right wrist skeletal points; the skeletal point information includes the pixel coordinate information and confidence of each skeletal point. Calculate the distance from each skeletal point to the upper edge of the table according to the pixel coordinate information in the segmentation information and the pixel coordinate information in the skeletal point information of the specified target, and use it as the standard pose data of the specified target.

6. A bad posture detection device, characterized in that, The bad pose detection device is applied to public places and includes an image acquisition device and a detection device. The image acquisition device is used to obtain an image frame. The detection device is used to Perform target detection on the image frame obtained by the image acquisition device and detect the first target. Screen out the second target that meets the prior conditions from the first targets, where the prior conditions are judged in the following way: for each detected first target, weight the size of the detection frame of the first target, its position in the image frame, and the confidence to obtain the weighted result of the first target. Compare the weighted result with the weighted result of the first target in the previous image frame. If the comparison result exceeds the set comparison threshold, it is determined that the first target does not meet the prior conditions; otherwise, it is determined that the first target meets the prior conditions. Retrieve the third target that matches the characteristic data of the specified target from the second targets. Perform pose estimation on the third target to obtain the pose estimation result of the third target. Compare the pose estimation result with the standard pose data of the specified target. When there are differences in the comparison result, it is determined as an incorrect pre-pose, and the type of the incorrect pre-pose is detected. The incorrect pre-pose includes one of the following poses: Bending-over-the-desk pose, Supporting pose with the elbow supported on the support surface of the desk, Head-down pose where the distance between the head and the support surface of the desk is less than the set first distance threshold, Head-tilting pose where the head is at a non-parallel angle in the height direction relative to the support surface of the desk, Uneven-shoulder pose where the left and right shoulders have different heights relative to the support surface of the desk, Front-back shoulder pose where the left and right shoulders have different front-back distances in the plane of the height direction relative to the support surface of the desk, The bending-over-the-desk pose is detected in the following way: For an image frame, Calculate the first distance from each bone point on the face to the upper edge of the desk respectively, and perform weighted averaging on the calculated first distance values to obtain the first weighted average result; calculate the second distance from each bone point on the face in the standard pose data to the upper edge of the desk respectively, and perform weighted averaging on the calculated second distance values to obtain the second weighted average result. If the first weighted average result is less than the second weighted average result, count it, Calculate the third distance between any bone point on the face and a shoulder bone point. If the third distance is less than the set second distance threshold, count it, For the bone points on the same side, calculate the angle between the first straight line passing through the shoulder bone point and the elbow bone point and the second straight line passing through the elbow bone point and the wrist bone point. When the angle is less than the set third angle threshold, count it; If the counted result is greater than the set third counting threshold, it is determined as the bending-over-the-desk pose; The supporting pose is detected in the following way: Calculate the fourth distance from the wrist bone point to the upper edge of the desk and the fifth distance between the wrist bone point and the shoulder bone point in the image frame respectively. If the fourth distance is greater than the set third distance threshold and the fifth distance is less than the set fourth distance threshold, count it, When the counted result is greater than the set fifth counting threshold, it is determined as the supporting pose; The head-down pose is detected in the following way: For an image frame, Calculate the first distance from each bone point on the face to the upper edge of the desk respectively, and perform weighted averaging on the calculated first distance values to obtain the first weighted average result; calculate the second distance from each bone point on the face in the standard pose data to the upper edge of the desk respectively, and perform weighted averaging on the calculated second distance values to obtain the second weighted average result. If the first weighted average result is less than the second weighted average result, count it, Calculate the third distance between any bone point on the face and a shoulder bone point. If the third distance is less than the set first distance threshold, count it, Calculate the position offset between any bone point on the face and the corresponding bone point in the standard pose data. When the offset exceeds the set offset threshold, count it. The offset is described by the angle between the straight line passing through any bone point on the face in the current image frame and the corresponding bone point in the standard pose data and the set direction. When the counted result is greater than the set first counting threshold, it is determined as a head-down posture; The tilted-head posture is detected in the following manner: Respectively calculate the horizontal angles of the straight lines where the bone points of the left and right eyes, the left and right ears, and the left and right shoulders are located in the image frame. Whenever the angle is greater than the set respective second angle threshold, a count is performed. If the counted result is greater than the set second counting threshold, it is determined as a tilted-head posture; The front-back shoulder posture is detected in the following manner: Respectively calculate the confidence levels of the bone points of the left and right shoulders, the left and right eyes, and the left and right ears in the image frame. If any confidence level exceeds the set confidence threshold, a count is performed. If the counted result is greater than the set fourth counting threshold, it is determined as a front-back shoulder posture; For each type of detected pre-bad gesture, Record the image frame including this type of pre-bad gesture. Count the image frames recorded within the first time period of this type of pre-bad gesture, and use the counted image frames as an image frame group. Determine whether the image frame group is continuous and reaches the quantity threshold for the continuity of this type of pre-bad gesture. If so, it is determined as a pre-bad gesture. Among them, The first time periods of each type of pre-bad gesture are different, and the quantity thresholds of each type of pre-bad gesture are different; The feature data of the specified target and the standard gesture data are pre-stored.

7. An improper posture detection device, characterized in that, The device includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, it implements the steps of the pre-bad gesture detection method as described in any one of claims 1 to 5.

Images