Posture recognition method and device

Abstract

The invention provides a posture recognition method and device, and relates to the technical field of computer vision, and the method comprises the steps: obtaining continuous multi-frame RGB images and depth images containing a to-be-detected human body; key points of multiple to-be-detected targets on the to-be-detected human body in the RGB image are detected, and depth values corresponding to the key points are calculated based on the RGB image and the depth image; determining a current detection mode based on the depth values corresponding to the key points of the to-be-detected targets; and determining a current to-be-detected target corresponding to the current detection mode from the to-be-detected targets, and identifying the posture of the current to-be-detected target based on the key point of the current to-be-detected target and the corresponding depth value thereof. According to the method, the postures of different to-be-detected targets are detected in different detection modes, targeted posture detection of the to-be-detected targets is achieved, posture recognition of the different to-be-detected targets is achieved through combination of key point information and depth information in different detection modes, and the accuracy, real-time performance and stability of posture recognition are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a gesture recognition method and device. BACKGROUND

[0002] At present, most of the traditional gesture recognition technologies are based on 2D images (RGB images), which are easily affected by background mixing or light changes, resulting in low gesture recognition accuracy. Although some gesture recognition technologies combine TOF cameras to obtain depth information, it is still difficult to recognize gestures in occluded or low-contrast scenes relying only on depth information, that is, the traditional gesture recognition technology has the problem of low gesture recognition accuracy in complex background, multi-light source interference and insufficient depth information. Moreover, the traditional gesture recognition technology is for single-part gesture recognition, such as face, hand or trunk, and cannot flexibly determine the current recognition part and the switching of each part gesture recognition mode according to the state of multiple parts in the image collected at different shooting distances.

[0003] Therefore, it is urgent for those skilled in the art to provide a solution to the above technical problems. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a gesture recognition method and device to improve the accuracy of gesture recognition.

[0005] The technical solution adopted by the present application to solve the technical problem is as follows: In a first aspect, a gesture recognition method is provided, wherein the method comprises: obtaining continuous multiple frames of RGB images and depth images containing a to-be-detected human body; detecting key points of multiple to-be-detected targets on the to-be-detected human body in the RGB images, and calculating depth values corresponding to the key points of each to-be-detected target based on the RGB images and the depth images; determining a current detection mode based on the depth values corresponding to the key points of each to-be-detected target; determining a current to-be-detected target corresponding to the current detection mode from each to-be-detected target, and recognizing the gesture of the current to-be-detected target based on the key points of the current to-be-detected target and the depth values corresponding to the key points, to obtain a corresponding gesture recognition result.

[0006] Optionally, the detection of the key points of the multiple to-be-detected targets on the to-be-detected human body in the RGB images comprises: using depth data in the depth images to strengthen the contour features and edge features of the to-be-detected human body in the RGB images; inputting a plurality of continuous RGB images into a pre-trained key point detection model to detect key points of a plurality of to-be-detected targets on a human body in the RGB images; In the training process of the pre-trained key point detection model, further comprising: performing spatial transformation or photometric transformation on the image samples for training, and scaling or normalizing pixel coordinates of the labeled key points on the image samples according to scaling of the image samples to obtain processed image samples; constructing a training set using the processed image samples, and inputting the processed image samples in the training set into an initial key point detection model constructed based on a Heatmap regression coding and decoding architecture in sequence to obtain corresponding prediction outputs; generating corresponding two-dimensional Gaussian distributions according to pixel coordinates of the labeled key points on the processed image samples to obtain GT Gaussian Heatmaps; calculating a loss according to the prediction outputs and the GT Gaussian Heatmaps, and calculating gradients of the loss with respect to model parameters; updating the model parameters using the gradients, repeating traversal of the training set until a current trained model meets a preset convergence condition, and obtaining the pre-trained key point detection model.

[0007] Optionally, the determining of the current detection mode based on the depth values corresponding to the key points of the to-be-detected targets comprises: determining average depth values corresponding to the to-be-detected targets according to the depth values corresponding to the key points of the to-be-detected targets; determining the current detection mode based on the average depth values corresponding to the to-be-detected targets.

[0008] Optionally, when the plurality of to-be-detected targets include a human hand and a human torso, the determining of the current detection mode based on the average depth values corresponding to the to-be-detected targets comprises: determining whether average depth values of the human hand in each frame of the RGB images meet a first preset distance condition, and determining whether average depth values of the human torso meet a second preset distance condition to obtain a corresponding first determination result; determining the current detection mode as a hand detection mode for detecting the human hand or a torso detection mode for detecting the human torso based on the first determination result corresponding to the plurality of continuous RGB images.

[0009] Optionally, the determining of the current detection mode as the hand detection mode for detecting the human hand or the torso detection mode for detecting the human torso based on the first determination result corresponding to the plurality of continuous RGB images comprises: In a case where the first determination result corresponding to the continuous multiple frames of RGB images indicates that the average depth value of the human hand satisfies the first preset distance condition, and the average depth value of the human torso does not satisfy the second preset distance condition, the current detection mode is determined as a hand detection mode for detecting the human hand. In a case where the first determination result corresponding to the continuous multiple frames of images indicates that the average depth value of the human hand does not satisfy the first preset distance condition, and the average depth value of the human torso satisfies the second preset distance condition, the current detection mode is determined as a torso detection mode for detecting the human torso.

[0010] Optionally, in a case where the current detection mode is the hand detection mode for detecting the human hand, the current detection target corresponding to the current detection mode is determined from the detection targets, and the posture of the current detection target is recognized based on the key points of the current detection target and the depth values corresponding to the key points, including: The current detection target corresponding to the current detection mode is determined as the human hand from the detection targets, and the joint angles of each finger and the three-dimensional distances between each finger are determined based on the key points of the human hand in each frame of image and the depth values corresponding to the key points. Whether the current gesture of the human hand satisfies a preset gesture determination logic is determined based on the joint angles of each finger and the three-dimensional distances between each finger, to obtain a corresponding second determination result. The static gesture of the human hand is determined based on the second determination result. Or, the current detection target corresponding to the current detection mode is determined as the human hand from the detection targets, and the three-dimensional position change and the angle rotation change of the center point of the human hand in the three-dimensional space are determined based on the key points of the human hand in the continuous multiple frames of images and the depth values corresponding to the key points. The dynamic gesture of the human hand is determined based on the three-dimensional position change and the angle rotation change of the center point of the human hand.

[0011] Optionally, in a case where the current detection mode is the torso detection mode for detecting the human torso, the current detection target corresponding to the current detection mode is determined from the detection targets, and the posture of the current detection target is recognized based on the key points of the current detection target and the depth values corresponding to the key points, including: The current detection target corresponding to the current detection mode is determined as the human torso from the detection targets, and the bending angles of the arm limbs and / or the knee limbs are determined based on the key points of the human torso in each frame of image and the depth values corresponding to the key points. The postures of the arm limbs and / or the knee limbs are determined based on the bending angles of the arm limbs and / or the knee limbs. or, determining, from each of the to-be-detected targets, that a current to-be-detected target corresponding to a current detection mode is a human body trunk, and determining a change trend of a body center of gravity height based on the key points of the human body trunk and the depth values corresponding to the key points in the continuous multiple frames of images; judging whether a fall occurs based on the change trend of the body center of gravity height.

[0012] Optionally, the calculating, based on the RGB image and the depth image, of the depth values corresponding to the key points of each of the to-be-detected targets comprises: aligning the RGB image and the depth image by using a preset image registration algorithm; mapping the coordinates of the key points to corresponding aligned depth images to obtain the depth values corresponding to the key points; or, determining the depth values corresponding to the key points of each of the to-be-detected targets by using a bilinear difference value method; or, converting the aligned depth image into a 3D point cloud; based on the 3D point cloud, and determining the depth values corresponding to the key points of each of the to-be-detected targets by using a nearest neighbor search method.

[0013] Optionally, the determining, based on the depth values corresponding to the key points of each of the to-be-detected targets, of the current detection mode further comprises: judging whether the depth values corresponding to the key points are greater than a preset depth threshold value; when the depth values corresponding to the key points are greater than the preset depth threshold value, filtering out the key points.

[0014] In a second aspect, the present application further discloses a posture recognition device, wherein the device comprises: an image acquisition module, configured to acquire continuous multiple frames of RGB images and depth images containing a to-be-detected human body; a key point determination module, configured to detect key points of multiple to-be-detected targets on the to-be-detected human body in the RGB image; a depth value calculation module, configured to calculate, based on the RGB image and the depth image, depth values corresponding to the key points of each of the to-be-detected targets; a mode determination module, configured to determine a current detection mode based on the depth values corresponding to the key points of each of the to-be-detected targets; a posture recognition module, configured to determine, from each of the to-be-detected targets, a current to-be-detected target corresponding to the current detection mode, and recognize a posture of the current to-be-detected target based on the key points of the current to-be-detected target and the depth values corresponding to the key points, to obtain a corresponding posture recognition result.

[0015] In a third aspect, the present application discloses a terminal, comprising: a memory, a processor, and a posture recognition program stored in the memory and executable on the processor, and when the posture recognition program is executed by the processor, the steps of the posture recognition method are implemented.

[0016] In a fourth aspect, the present application discloses a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the posture recognition method.

[0017] The posture recognition method and device provided by the present application, the posture recognition method comprises: acquiring continuous multiple frames of RGB images and depth images containing a to-be-detected human body; detecting key points of multiple to-be-detected targets on the to-be-detected human body in the RGB images, and calculating depth values corresponding to the key points of each to-be-detected target based on the RGB images and the depth images; determining a current detection mode based on the depth values corresponding to the key points of each to-be-detected target; determining a current to-be-detected target corresponding to the current detection mode from each to-be-detected target, and recognizing a posture of the current to-be-detected target based on the key points of the current to-be-detected target and the depth values corresponding to the key points, to obtain a corresponding posture recognition result. It can be seen that, by determining the key points of multiple to-be-detected targets on the to-be-detected human body, calculating the depth values corresponding to the key points, determining the current detection mode based on the depth values corresponding to the key points of each to-be-detected target, and determining the current to-be-detected target in the current detection mode, the postures of different to-be-detected targets in different detection modes are detected, the targeted posture detection of each to-be-detected target is realized, in the process of recognizing the posture of the current to-be-detected target in the current detection mode, the posture recognition of the current to-be-detected target is realized by combining the key point information and the depth information, the problem of low posture recognition accuracy of the traditional posture recognition in a complex background, multi-light source interference and insufficient depth information can be solved, and therefore the accuracy, real-time performance and stability of the posture recognition can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a traditional gesture recognition comparison diagram disclosed by the present application; Figure 2 is a flowchart of a preferred embodiment of the posture recognition method in the present application; Figure 3 is a specific hand key point detection diagram disclosed by the present application; Figure 4 is a specific posture recognition system hardware diagram disclosed by the present application; Figure 5 is a functional principle block diagram of a preferred embodiment of the posture recognition device in the present application; Figure 6 is a function principle block diagram of a preferred embodiment of the terminal in the present application. DETAILED DESCRIPTION

[0019] In order to make the objects, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0020] Referring to Figure 1 As shown in the figure, the conventional gesture recognition technology is mostly based on 2D image (only RGB image), which is easily affected by background mixture or light change, resulting in low gesture recognition accuracy. Although some gesture recognition technologies combine TOF cameras to obtain depth information, it is still difficult to recognize gestures in shielding or low-contrast scenes only relying on depth information, that is, the conventional gesture recognition technology has the problem of low gesture recognition accuracy in complex background, multi-light source interference and insufficient depth information. Moreover, the conventional gesture recognition technology is for single part gesture recognition, such as single part of face, hand or torso, and cannot flexibly determine the current to-be-recognized part and the switching of each part gesture recognition mode according to the state of multiple parts in the image collected at different shooting distances. Therefore, the present application provides a gesture recognition scheme, which can improve the accuracy, real-time performance and stability of gesture recognition.

[0021] Referring to Figure 2 , Figure 2 is a flowchart of the gesture recognition method in the present application. As shown in the figure, Figure 2 The gesture recognition method described in the embodiment of the present application comprises: Step S11, acquiring continuous multiple frames of RGB images and depth images containing a to-be-detected human body.

[0022] In this embodiment, the RGB image (color image) and the depth image can be collected by an RGBD camera, that is, the to-be-detected human body is continuously photographed by the RGBD camera, and continuous multiple frames of RGB images and depth images containing the to-be-detected human body output by the RGBD camera are obtained. It should be noted that the shooting distance can be close-up shooting or long-distance shooting. When the to-be-detected human body is close-up photographed, the image contains the to-be-detected part of the upper body or arm of the human body; when the to-be-detected human body is long-distance photographed, the image can contain the whole human body.

[0023] In this embodiment, after acquiring the continuous multiple frames of RGB images and depth images containing the human body to be detected, the specific process can further include: pre-processing the RGB images and the depth images to determine the noise and outliers existing in the RGB images and the depth images, such as the noise and outliers existing in the RGB images and the depth images due to the rapid movement or reflection of the human body to be detected, and then correcting the noise and outliers, so as to ensure that the subsequent mode switching decision is made by analyzing the images in a stable state for a period of time. In the image preprocessing, the depth image can also be subjected to range clipping and linear normalization processing. The range clipping of the depth image can improve the data quality and focus on the key area in the image, thereby reducing the invalid calculation amount. The linear normalization processing of the depth image can improve the stability of the depth data in the depth image.

[0024] In step S12, the key points of the multiple detection targets on the human body to be detected in the RGB image are detected, and the depth values corresponding to the key points of each detection target are calculated based on the RGB image and the depth image.

[0025] The multiple detection targets on the human body to be detected can be the hand, face, body trunk, and other parts that need to be recognized in a gesture, and the key point is a two-dimensional key point.

[0026] In this embodiment, for the human body to be detected, the key points of the multiple detection targets thereon are determined to provide a data basis for subsequent gesture recognition. Specifically, the contour features and edge features of the human body to be detected in the RGB image are enhanced using the depth data in the depth image, and the continuous multiple frames of RGB images are input into a pre-trained key point detection model to detect the key points of the multiple detection targets on the human body to be detected in the RGB image, that is, the pre-trained key point detection model is used to find the key points of the multiple detection targets on the human body to be detected in the enhanced RGB image. For example, the AI pre-trained gesture detection model (pre-trained key point detection model) detects the key points of the hand and the body trunk of the human body to be detected in the RGB image, such as 21 key points of the hand and 33 key points of the body part. As shown in FIG. 11, if only the hand of the human body is included in the RGB image, the key points of the hand are output by the AI pre-trained gesture detection model when detecting the key points. Figure 3

[0027] ​It can be understood that before the continuous multiple frames of RGB images are input into the pre-trained key point detection model, the contour features and edge features of the human body to be detected in the RGB image are enhanced through the depth data in the depth image, which can improve the key point detection accuracy, especially in the scene with complex background, low texture object, occlusion or light challenge, the use of geometric information to make up for the deficiency of pure RGB vision is most significant for the improvement of key point detection accuracy. Specifically, the original depth image is preprocessed (such as filtering denoising, hole filling, normalization), the contour or edge can be extracted from the preprocessed depth image through depth gradient calculation, Canny edge detection, edge detection based on normal change, wherein the edge represents the geometric edge, then the extracted depth edge map (such as binary or weighted) is directly added to the edge map or a certain feature channel of the RGB image, the depth contour or edge information is used to generate a spatial attention map, which highlights the weight of the corresponding object edge region in the RGB feature map, the depth edge map is used as an additional input channel, which is spliced with the original RGB image and input into the network for learning fusion, the RGB information and the depth edge information are processed respectively through the network branch, and the fusion is performed in the high-level features of the network.

[0028] It should be further pointed out that in the training process of the pre-trained key point detection model, specifically, the image samples for training can be subjected to spatial transformation or photometric transformation, and the pixel coordinates of the labeled key points on the image samples are scaled or normalized according to the scaling of the image samples to obtain processed image samples; the processed image samples are used to construct a training set, and the processed image samples in the training set are input into an initial key point detection model constructed based on a Heatmap regression coding and decoding architecture in sequence to obtain corresponding prediction outputs; according to the pixel coordinates of the labeled key points on the processed image samples, corresponding two-dimensional Gaussian distribution is generated to obtain GT Gaussian Heatmap; the loss is calculated according to the Heatmap of the prediction output and the GT Gaussian Heatmap, and the gradient of the loss relative to the model parameters is calculated; the model parameters are updated using the gradient, and the training set is repeatedly traversed until the current trained model meets the preset convergence condition, and the pre-trained key point detection model is obtained.

[0029] For example, the training of a keypoint detection model involves multiple steps such as data preparation, model architecture selection, loss function design, training strategy formulation, and evaluation optimization. First, images or video sequences containing target objects (such as people, faces, hands) are obtained, and the data should cover the performance of target objects in various poses, scales, perspectives, lighting conditions, and occlusion situations, as well as the diversity of backgrounds. Then, the key points to be detected (such as 33 key points of the body, 68 points of the face, 21 joints of the hand, etc.) are defined, and then the accurate pixel coordinates (x, y) of all defined key points in each image can be manually labeled using labeling tools (such as LabelMe, CVAT, Labelbox, etc.). The consistency and accuracy of labeling directly affect the performance of the model, and usually multiple people are required to label and check for consistency and correct (such as removing outliers, smoothing trajectories). Common labeling formats include COCO JSON (COCO annotation JSON file), MPII JSON (MPII human pose JSON annotation file), CSV (comma-separated value file), etc., which contain image paths, key point coordinates, visibility / occlusion flags, target bounding boxes, etc. COCO JSON represents the JSON format annotation file used by the COCO (Common Objects in Context) dataset, which stores the bounding box, class label, segmentation mask, and key points of the target in the image. The JSON format annotation file of the MPII Human Pose Dataset is specifically used for human key point detection tasks and contains joint coordinates, visibility markers, and human box size information. CSV is a pure text table data format that separates fields by commas, with each row representing a record, widely used for data exchange and simple database storage. Further preprocessing of the image includes standardization or normalization, size adjustment, denoising, etc. Key points can also be preprocessed, such as scaling pixel coordinates of key points according to image scaling or normalizing relative to the target bounding box. To increase data diversity and improve model generalization and prevent overfitting, data augmentation can be performed, such as spatial transformation or photometric transformation of the image. During the execution of the enhancement operation, the image and the corresponding key point coordinates must be applied synchronously and consistently. A coding-decoding architecture based on Heatmap regression can be used as the model framework for key point detection models, which outputs a K-channel feature map (Heatmap) with the same spatial resolution as the input image (or after downsampling). Each channel corresponds to a key point.In training, according to the labeled (x, y) coordinates, a two-dimensional Gaussian distribution is generated at each key point position as the Ground Truth Gaussian Heatmap (GT Gaussian Heatmap), which is a kind of probability distribution map used to represent key point positions in computer vision. Its core is to convert discrete key point coordinates into continuous two-dimensional probability distribution through Gaussian function. The occluded points can generate a low confidence Heatmap or be ignored. In order to minimize the difference between the predicted Heatmap and the GT Gaussian Heatmap, Heatmap regression loss can be used during training. In training, the dataset can be divided into small batches, and then the batch data is input, the prediction output is calculated, and then the loss is calculated, such as calculating the loss value according to the prediction output and the GT, calculating the gradient of the loss relative to the model parameters, and then updating the model parameters using the optimizer using the gradient. Repeat the entire training data set multiple times until the training is completed. In model validation, a part of the training data can be separated as a validation set. After each Epoch or fixed iteration step, the model performance is evaluated on the validation set to select the best model as the final key point detection model.

[0030] In this embodiment, the depth value corresponding to each key point of the target to be detected is calculated. The specific implementation can be: using a preset image registration algorithm to align the RGB image and the depth image, and after completing the registration and alignment of the RGB image and the depth image, the depth value corresponding to the key point can be directly obtained by mapping the coordinates of the key point to the corresponding depth image, so that subsequent detection mode determination can be realized based on the depth value corresponding to the key point, and the key point can also be reconstructed in three-dimensional pose based on the depth value, such as the coordinates of a two-dimensional key point being (x, y), and the three-dimensional coordinates (x, y, z) of the key point can be obtained by combining the depth value. After completing the registration and alignment of the RGB image and the depth image, the depth value corresponding to each key point of the target to be detected can also be determined by using a bilinear difference value, or the aligned depth image can be converted into a 3D point cloud, and the depth value corresponding to each key point of the target to be detected can be determined by using a nearest neighbor search.

[0031] Among them, the registration and alignment of the RGB image and the depth image can ensure high-precision alignment of the RGB image and the depth image, for example, the registration and alignment of the RGB image and the depth image based on camera internal parameter geometric correction and depth image compensation technology, that is, each pixel of the depth image corresponds to a pixel of the color image (or is converted through the internal parameter), and it is necessary to ensure that the depth image coordinate system used and the color image coordinate system used for key point detection are accurately aligned. Then, for each detected 2D key point pixel coordinate, the depth value stored at the coordinate position in the registered and aligned depth image is directly queried.

[0032] The specific registration process of the RGB image and the depth image can be: first, determining the intrinsic parameters and extrinsic parameters of the camera; the intrinsic parameters represent the focal length, principal point position, and distortion parameters of the camera, and the extrinsic parameters represent the positional relationship (such as a rotation matrix R and a translation vector T) between the RGB camera and the depth camera.

[0033] For each depth image pixel point and its corresponding depth value , the corresponding spatial coordinates are calculated using the intrinsic parameters of the depth camera, that is: ; ; wherein, is the focal length, is the principal point position (i.e., the optical center of the camera).

[0034] The three-dimensional point of the depth camera is converted to the RGB camera coordinate system, that is: ; The point in the RGB camera coordinate system is projected onto the RGB image plane, that is: ; ; Thus, the pixel position in the corresponding RGB image is determined, and the depth value is placed in the corresponding RGB pixel, thereby completing the registration.

[0035] In step S13, the current detection mode is determined based on the depth values corresponding to the key points of each target to be detected.

[0036] In this embodiment, in order to realize posture recognition of the target to be detected, the detection mode corresponding to each target to be detected is pre-configured, and then the current detection mode can be determined based on the depth values corresponding to the key points of each target to be detected. Specifically, according to the depth values corresponding to the key points of each target to be detected, the average depth value corresponding to each target to be detected is determined, and the current detection mode is determined based on the average depth value corresponding to each target to be detected, so as to adapt to the posture recognition of the target to be detected in the current image scene. After switching the detection mode to the current detection mode, all historical data related to other targets to be detected can be cleared.

[0037] In the embodiment, before determining the current detection mode based on the depth value corresponding to the key point of each target to be detected, it can further include: judging whether the depth value corresponding to the key point is greater than a preset depth threshold; when the depth value corresponding to the key point is greater than the preset depth threshold, filtering out the key point; when the depth value corresponding to the key point is not greater than the preset depth threshold, not filtering out the key point. That is, by using the depth threshold to remove the misdetected points beyond the specified distance, the detection accuracy and anti-interference ability can be further improved.

[0038] When the multiple targets to be detected include a human hand and a human torso, the current detection mode is determined based on the average depth value corresponding to each target to be detected. Specifically, it can include: judging whether the average depth value of the human hand in each frame of RGB image satisfies a first preset distance condition, and judging whether the average depth value of the human torso satisfies a second preset distance condition to obtain a first judgment result; based on the first judgment result corresponding to the continuous multiple frames of RGB images, the current detection mode is determined as a hand detection mode for detecting the human hand or a torso detection mode for detecting the human torso.

[0039] When the first judgment result corresponding to the continuous multiple frames of RGB images indicates that the average depth value of the human hand satisfies the first preset distance condition, and the average depth value of the human torso does not satisfy the second preset distance condition, the current detection mode is determined as the hand detection mode for detecting the human hand; when the first judgment result corresponding to the continuous multiple frames of RGB images indicates that the average depth value of the human hand does not satisfy the first preset distance condition, and the average depth value of the human torso satisfies the second preset distance condition, the current detection mode is determined as the torso detection mode for detecting the human torso. It can be understood that, based on the average depth value of the hand and the average depth value of the torso, the current detection mode is dynamically switched to the hand detection mode and the body detection mode.

[0040] For example, based on the average depth value of the hand and the average depth value of the torso, the mode is automatically switched between the "hand" and the "body" mode. When the average depth value of the hand does not satisfy the condition of being less than 1000 mm (the first preset distance condition), and the average depth value of the torso satisfies the condition of being greater than 1000 mm (the second preset distance condition), it is determined that the target person to be detected is located far away, and it is more suitable to detect the posture of the human torso at this time. If the current is not the body mode, the average depth value of the hand does not satisfy the first preset distance condition, and the average depth value of the human torso satisfies the second preset distance condition in the continuous 3 frames of images, the current detection mode will be formally switched to the body mode, and all historical data related to gesture recognition will be cleared.

[0041] In step S14, a current detection target corresponding to the current detection mode is determined from the detection targets, and a pose of the current detection target is recognized based on the key points of the current detection target and the depth values corresponding to the key points, to obtain a corresponding pose recognition result.

[0042] It can be understood that, in the process of recognizing the pose of the current detection target in the current detection mode, the pose recognition of the current detection target is realized by combining the key point information and the depth information, the pose of the current detection target corresponding to the current detection mode can be recognized, and thus the accuracy, real-time performance and stability of the pose recognition can be improved.

[0043] In one specific embodiment, when the current detection mode is a hand detection mode for detecting a human hand, the current detection target corresponding to the current detection mode is determined to be a human hand from the detection targets, and the joint angles of each finger and the three-dimensional distances between the fingers are determined based on the key points of the human hand in each frame of image and the depth values corresponding to the key points; whether the current gesture of the human hand satisfies a preset gesture determination logic is determined based on the joint angles of each finger and the three-dimensional distances between the fingers, to obtain a corresponding second determination result; a static gesture of the human hand is determined based on the second determination result; or, the current detection target corresponding to the current detection mode is determined to be a human hand from the detection targets, and the three-dimensional position changes and angle rotation changes of a center point of the human hand in a three-dimensional space are determined based on the key points of the human hand in a plurality of continuous frames of image and the depth values corresponding to the key points; a dynamic gesture of the human hand is determined based on the three-dimensional position changes and angle rotation changes of the center point of the human hand. It can be understood that, the dynamic gesture and the static gesture are determined based on the key points and the depth values corresponding to the key points.

[0044] For example, the five-finger open determination logic: determine whether the five fingers are in the straightened state, that is, only when the is_extended determination results of the thumb, index finger, middle finger, ring finger and little finger are all True, the function will return True, and then the hand gesture can be identified as five-finger open. The fist determination logic, that is, the opposite of the five-finger open (Open Palm), that is, when the is_extended determination results of the thumb, index finger, middle finger, ring finger and little finger are all False, the hand gesture is identified as a fist. The scissors hand determination logic: the index finger and the middle finger must be straightened, and the other three fingers must be bent. The index finger and the middle finger must have sufficient spacing, such as the spacing between the index finger and the middle finger being greater than a preset spacing value. The index finger and the middle finger must have sufficient spacing to distinguish between a scissors gesture and a two-finger pointing gesture. If the three conditions are met, the hand gesture can be identified as a scissors gesture. Similarly, the determination logic of other gestures can also be set in the above manner. Or, the current hand gesture of the hand can be identified according to the positional relationship of the finger key points in a single frame image relative to the palm or other key points, such as a fist, scissors, five-finger open, OK, etc.

[0045] It can be understood that the recognition of all static gestures is based on the auxiliary function is_extended, which functions to determine whether a single finger is in a straightened state. For example, the angles of two main joints of a finger, such as the first and second joints of the index finger, can be calculated. If the angles of the two joints are both greater than 150 degrees, it is determined that the finger is straightened, otherwise it is considered to be bent.

[0046] In addition, for dynamic gestures, the position and angle rotation change of the hand center point between consecutive frames can be compared to determine and identify waving, sliding or rotating actions, etc.

[0047] In another specific embodiment, when the current detection mode is a torso detection mode for detecting a human torso, a current detection target corresponding to the current detection mode is determined to be a human torso from each detection target, and a bending angle of an arm limb and / or a knee limb is determined based on key points of the human torso in each frame of image and depth values corresponding to the key points. The posture of the arm limb and / or the knee limb is determined based on the bending angle of the arm limb and / or the knee limb. Or, a current detection target corresponding to the current detection mode is determined to be a human torso from each detection target, and a change trend of the body center of gravity height is determined based on key points of the human torso and depth values corresponding to the key points in consecutive multiple frames of image, and then it is determined whether a fall occurs based on the change trend of the body center of gravity height.

[0048] For example, when the left arm bending angle is less than 90 degrees and the right arm straightening angle is greater than 150 degrees, it is identified as waving to the left; when the right arm bending angle is less than 90 degrees and the left arm straightening angle is greater than 150 degrees, it is identified as waving to the right; and when the angles of the left and right knees are both less than 120 degrees, it is identified as deep squatting. In addition, the change trend of the body center of gravity height can be quickly realized by analyzing the change trend of the body center of gravity height through the key points of the trunk in multiple frames of images, and the detection of falling can be realized, that is, in the body mode, the detection of falling can be realized.

[0049] In the embodiment, after the posture recognition is completed, the hand or body skeleton markers and the corresponding depth information can be superimposed in the original RGB image, and the depth image can be displayed in real time through an independent depth window, and the key screenshot and data recording function can be set on the hardware device implementing the posture recognition scheme of the application.

[0050] It can be seen that, in the embodiment of the application, the key points of a plurality of to-be-detected targets on a to-be-detected human body are determined, the depth values corresponding to the key points are calculated, the current detection mode is determined based on the depth values corresponding to the key points of each to-be-detected target, and the current to-be-detected target in the current detection mode is determined, so as to detect the postures of different to-be-detected targets in different detection modes, realize the targeted posture detection of each to-be-detected target, and realize the posture recognition of the current to-be-detected target by combining the key point information and the depth information in the process of recognizing the posture of the current to-be-detected target in the current detection mode, so as to solve the problem of low posture recognition accuracy of the traditional posture recognition in a complex background, multi-light source interference and insufficient depth information, thereby improving the accuracy, real-time performance and stability of the posture recognition.

[0051] For example, as shown in Figure 4 The technical scheme of the application is executed by using a system composed of a central controller (CPU), a universal serial bus (USB), an RGB camera and a TOF camera, wherein the CPU can transmit relevant processing data to the universal serial bus for storage, and a sending signal is synchronized to the RGB camera and the TOF camera through the CPU control to control the synchronization acquisition of the RGB image and the depth image by the RGB camera and the TOF camera, wherein the RGB camera and the TOF camera are fixed to form an RGBD camera through a rigid support.

[0052] In one embodiment, as shown in Figure 5 Based on the above posture recognition method, the application also correspondingly provides a posture recognition device, which comprises: An image acquisition module 11 is configured to acquire continuous multiple frames of RGB images and depth images containing a to-be-detected human body; A key point determination module 12 is configured to detect key points of a plurality of to-be-detected targets on the to-be-detected human body in the RGB image; The depth value calculation module 13 is configured to calculate a depth value corresponding to each key point of each target to be detected based on the RGB image and the depth image. The mode determination module 14 is configured to determine a current detection mode based on the depth value corresponding to each key point of each target to be detected. The posture recognition module 15 is configured to determine a current target to be detected corresponding to the current detection mode from each target to be detected, and recognize a posture of the current target to be detected based on the key point of the current target to be detected and the depth value corresponding to the key point, to obtain a corresponding posture recognition result.

[0053] In addition, it is worth noting that the working process of the posture recognition device provided in the embodiment is the same as the working process of the posture recognition method described above, and will not be described here again. For details, please refer to the working process of the posture recognition method described above.

[0054] Figure 6 The terminal provided in the embodiment of the present application has a structure diagram. The terminal can include: The memory 501, the processor 502, and the computer program stored in the memory 501 and executable on the processor 502.

[0055] The processor 502 implements the posture recognition method provided in the above embodiment when executing the program.

[0056] Further, the terminal further includes: The communication interface 503 is configured to communicate in the memory 501 and the processor 502.

[0057] The memory 501 is configured to store the computer program executable on the processor 502.

[0058] The memory 501 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0059] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0060] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can complete communication between each other through an internal interface.

[0061] The processor 502 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0062] The embodiment further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the posture recognition method.

[0063] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such steps that are apparent to those skilled in the art to which the application pertains. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the application are indicated by the claims.

[0064] In the description of the application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. The illustrative description of the above terms in the specification does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or N embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0065] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be specifically embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can read instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions execution systems, apparatus or devices.

[0066] It should be understood that parts of the application can be implemented in hardware, software, firmware or a combination thereof. In the above-described embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and as in another embodiment, it can be implemented by any one or a combination of the following technologies known in the art: discrete logic circuit with logic gates for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gates, programmable gate array (PGA), field programmable gate array (FPGA) and the like.

[0067] It should be understood that the application is not limited to the above examples, and that all improvements and modifications made by those skilled in the art based on the above description should be within the scope of the appended claims of the application.

Claims

1. A pose recognition method, characterized in that, The method includes: Acquire consecutive multi-frame RGB images and depth images containing the human body to be detected; Detect key points of multiple targets on the human body to be detected in the RGB image, and calculate the depth value corresponding to the key points of each target based on the RGB image and the depth image; The current detection mode is determined based on the depth value corresponding to the key point of each of the targets to be detected; The current target to be detected is determined from the targets to be detected according to the current detection mode, and the pose of the current target to be detected is identified based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, so as to obtain the corresponding pose recognition result.

2. The pose recognition method according to claim 1, characterized in that, The key points of multiple targets on the human body to be detected in the RGB image include: The contour and edge features of the human body to be detected in the RGB image are enhanced by using depth data in the depth image; Multiple consecutive frames of RGB images are input into a pre-trained keypoint detection model to detect keypoints of multiple targets on the human body in the RGB images. The training process of the pre-trained keypoint detection model also includes: The image samples used for training are subjected to spatial transformation or photometric transformation, and the pixel coordinates of the key points marked on the image samples are scaled or normalized according to the scaling of the image samples to obtain the processed image samples. A training set is constructed using the processed image samples, and the processed image samples in the training set are sequentially input into an initial keypoint detection model constructed based on a heatmap regression encoding and decoding architecture to obtain the corresponding prediction output. Based on the pixel coordinates of the key points marked on the processed image sample, a corresponding two-dimensional Gaussian distribution is generated to obtain the GT Gaussian Heatmap. The loss is calculated based on the predicted output heatmap and the GT Gaussian heatmap, and the gradient of the loss relative to the model parameters is calculated. The model parameters are updated using the gradient, and the training set is repeatedly traversed until the current trained model meets the preset convergence condition, thus obtaining the pre-trained keypoint detection model.

3. The pose recognition method according to claim 1, characterized in that, Determining the current detection mode based on the depth values ​​corresponding to the key points of each of the targets to be detected includes: Based on the depth values ​​corresponding to the key points of each of the targets to be detected, determine the average depth value corresponding to each of the targets to be detected; The current detection mode is determined based on the average depth value corresponding to each of the targets to be detected.

4. The pose recognition method according to claim 3, characterized in that, When multiple targets to be detected include human hands and human torsos, determining the current detection mode based on the average depth value corresponding to each of the targets to be detected includes: Determine whether the average depth value of the human hand in each frame of RGB image meets the first preset distance condition, and determine whether the average depth value of the human torso meets the second preset distance condition, and obtain the corresponding first judgment result; Based on the first judgment result corresponding to multiple consecutive RGB images, the current detection mode is determined to be either the hand detection mode for detecting human hands or the torso detection mode for detecting human torso.

5. The pose recognition method according to claim 4, characterized in that, The step of determining the current detection mode as a hand detection mode for detecting human hands or a torso detection mode for detecting human torsos based on the first judgment result corresponding to multiple consecutive RGB images includes: If the first judgment result corresponding to multiple consecutive RGB images shows that the average depth value of the human hand meets the first preset distance condition, and the average depth value of the human torso does not meet the second preset distance condition, then the current detection mode is determined as the hand detection mode for detecting the human hand. If the first judgment result corresponding to multiple consecutive frames of images indicates that the average depth value of the human hand does not meet the first preset distance condition, and the average depth value of the human torso meets the second preset distance condition, then the current detection mode is determined as the torso detection mode for detecting the human torso.

6. The pose recognition method according to claim 4, characterized in that, When the current detection mode is a hand detection mode for detecting human hands, the step of determining the current target to be detected corresponding to the current detection mode from among the targets to be detected, and identifying the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, includes: The current target to be detected corresponding to the current detection mode is determined to be the human hand from the various targets to be detected. Based on the key points of the human hand in each frame image and the depth value corresponding to the key points, the joint angle of each finger and the three-dimensional distance between each finger are determined. Based on the joint angles of each finger and the three-dimensional distance between each finger, it is determined whether the current hand gesture of the human hand satisfies the preset gesture determination logic, and the corresponding second determination result is obtained. The static hand gesture of the human hand is determined based on the second judgment result; Alternatively, the current target to be detected corresponding to the current detection mode is determined from each of the targets to be detected as a human hand, and the three-dimensional position change and angle rotation change of the center point of the human hand in three-dimensional space are determined based on the key points of the human hand and the depth values ​​corresponding to the key points in multiple consecutive frames of images. The dynamic gestures of the human hand are determined based on the three-dimensional position changes and angular rotation changes of the center point of the human hand.

7. The pose recognition method according to claim 4, characterized in that, When the current detection mode is a torso detection mode for detecting the human torso, the step of determining the current target to be detected corresponding to the current detection mode from among the targets to be detected, and identifying the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, includes: The current target to be detected corresponding to the current detection mode is determined from each of the targets to be detected as the human torso, and the bending angle of the arm limbs and / or knee limbs is determined based on the key points of the human torso in each frame image and the depth value corresponding to the key points. The posture of the arm and / or knee is determined based on the bending angle of the arm and / or knee. Alternatively, the current target to be detected corresponding to the current detection mode is determined from each of the targets to be detected as the human torso, and the trend of change of the height of the body's center of gravity is determined based on the key points of the human torso and the depth values ​​corresponding to the key points in multiple consecutive frames of images. Whether a fall has occurred is determined based on the changing trend of the body's center of gravity height.

8. The pose recognition method according to claim 1, characterized in that, The calculation of the depth value corresponding to the key point of each of the targets to be detected based on the RGB image and the depth image includes: Align RGB and depth images using a preset image registration algorithm; The coordinates of the key points are mapped to the corresponding aligned depth image to obtain the depth value corresponding to the key points; Alternatively, the depth value corresponding to the key point of each of the targets to be detected can be determined by using bilinear interpolation. Alternatively, the aligned depth image can be converted into a 3D point cloud; Based on the 3D point cloud, the depth value corresponding to the key point of each target to be detected is determined by using nearest neighbor search.

9. The pose recognition method according to any one of claims 1 to 8, characterized in that, Before determining the current detection mode based on the depth values ​​corresponding to the key points of each of the targets to be detected, the method further includes: Determine whether the depth value corresponding to the key point is greater than a preset depth threshold; If the depth value corresponding to the key point is greater than the preset depth threshold, then the key point is filtered out.

10. A posture recognition device, characterized in that, The device includes: The image acquisition module is used to acquire multiple consecutive frames of RGB images and depth images containing the human body to be detected; The key point determination module is used to detect key points of multiple targets on the human body to be detected in the RGB image; A depth value calculation module is used to calculate the depth value corresponding to the key points of each of the targets to be detected based on the RGB image and the depth image; The mode determination module is used to determine the current detection mode based on the depth value corresponding to the key point of each of the targets to be detected; The pose recognition module is used to determine the current target to be detected corresponding to the current detection mode from each of the targets to be detected, and to recognize the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, so as to obtain the corresponding pose recognition result.

Images