A gesture recognition method and apparatus

Abstract

The application provides a posture recognition method and device, and relates to the technical field of computer vision.The method comprises the following steps: acquiring continuous multiple frames of RGB images and depth images containing a human body to be detected; detecting key points of multiple targets to be detected on the human body to be detected in the RGB images, and calculating depth values corresponding to the key points based on the RGB images and the depth images; determining a current detection mode based on the depth values corresponding to the key points of each target to be detected; determining a current target to be detected corresponding to the current detection mode from each target to be detected, and recognizing a posture of the current target to be detected based on the key points of the current target to be detected and the corresponding depth values.The application realizes targeted posture detection of each target to be detected by detecting postures of different targets to be detected under different detection modes, and realizes posture recognition of different targets to be detected by combining key point information and depth information under different detection modes, so that the accuracy, real-time performance and stability of posture recognition are improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more particularly to a pose recognition method and apparatus. Background Technology

[0002] Currently, traditional gesture recognition technologies are mostly based on 2D images (RGB images), which are easily affected by background clutter or changes in lighting, resulting in low gesture recognition accuracy. Although some gesture recognition technologies combine TOF cameras to obtain depth information, relying solely on depth information still has difficulties in recognizing gestures in occluded or low-contrast scenes. In other words, traditional gesture recognition technologies suffer from low gesture recognition accuracy in complex backgrounds, multi-light source interference, and insufficient depth information. Moreover, traditional posture recognition technologies are all aimed at recognizing single parts of the body, such as the face, hands, or torso, and cannot flexibly determine the current part to be recognized and switch the posture recognition mode for each part based on the state of multiple parts in images captured at different shooting distances.

[0003] Therefore, providing a solution to the above-mentioned technical problems is something that those skilled in the art urgently need to focus on. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a posture recognition method and apparatus that can improve the accuracy of posture recognition, in view of the above-mentioned defects of the prior art.

[0005] The technical solution adopted by this invention to solve the technical problem is as follows:

[0006] In a first aspect, a pose recognition method, wherein the method includes:

[0007] Acquire consecutive multi-frame RGB images and depth images containing the human body to be detected;

[0008] Detect key points of multiple targets on the human body to be detected in the RGB image, and calculate the depth value corresponding to the key points of each target based on the RGB image and the depth image;

[0009] The current detection mode is determined based on the depth value corresponding to the key point of each of the targets to be detected;

[0010] The current target to be detected is determined from the targets to be detected according to the current detection mode, and the pose of the current target to be detected is identified based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, so as to obtain the corresponding pose recognition result.

[0011] Optionally, the detection of key points of multiple targets on the human body in the RGB image includes:

[0012] The contour and edge features of the human body to be detected in the RGB image are enhanced by using depth data in the depth image;

[0013] Multiple consecutive frames of RGB images are input into a pre-trained keypoint detection model to detect keypoints of multiple targets on the human body in the RGB images.

[0014] The training process of the pre-trained keypoint detection model also includes:

[0015] The image samples used for training are subjected to spatial transformation or photometric transformation, and the pixel coordinates of the key points marked on the image samples are scaled or normalized according to the scaling of the image samples to obtain the processed image samples.

[0016] A training set is constructed using the processed image samples, and the processed image samples in the training set are sequentially input into an initial keypoint detection model constructed based on a heatmap regression encoding and decoding architecture to obtain the corresponding prediction output.

[0017] Based on the pixel coordinates of the key points marked on the processed image sample, a corresponding two-dimensional Gaussian distribution is generated to obtain the GT Gaussian Heatmap.

[0018] The loss is calculated based on the predicted output heatmap and the GT Gaussian heatmap, and the gradient of the loss relative to the model parameters is calculated.

[0019] The model parameters are updated using the gradient, and the training set is repeatedly traversed until the current trained model meets the preset convergence condition, thus obtaining the pre-trained keypoint detection model.

[0020] Optionally, determining the current detection mode based on the depth values ​​corresponding to the key points of each of the targets to be detected includes:

[0021] Based on the depth values ​​corresponding to the key points of each of the targets to be detected, determine the average depth value corresponding to each of the targets to be detected;

[0022] The current detection mode is determined based on the average depth value corresponding to each of the targets to be detected.

[0023] Optionally, when multiple targets to be detected include a human hand and a human torso, determining the current detection mode based on the average depth value corresponding to each of the targets to be detected includes:

[0024] Determine whether the average depth value of the human hand in each frame of RGB image meets the first preset distance condition, and determine whether the average depth value of the human torso meets the second preset distance condition, and obtain the corresponding first judgment result;

[0025] Based on the first judgment result corresponding to multiple consecutive RGB images, the current detection mode is determined to be either the hand detection mode for detecting human hands or the torso detection mode for detecting human torso.

[0026] Optionally, determining the current detection mode as a hand detection mode for detecting human hands or a torso detection mode for detecting human torsos based on the first judgment result corresponding to multiple consecutive RGB images includes:

[0027] If the first judgment result corresponding to multiple consecutive RGB images shows that the average depth value of the human hand meets the first preset distance condition, and the average depth value of the human torso does not meet the second preset distance condition, then the current detection mode is determined as the hand detection mode for detecting the human hand.

[0028] If the first judgment result corresponding to multiple consecutive frames of images indicates that the average depth value of the human hand does not meet the first preset distance condition, and the average depth value of the human torso meets the second preset distance condition, then the current detection mode is determined as the torso detection mode for detecting the human torso.

[0029] Optionally, when the current detection mode is a hand detection mode for detecting human hands, the step of determining the current target to be detected corresponding to the current detection mode from among the targets to be detected, and identifying the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, includes:

[0030] The current target to be detected corresponding to the current detection mode is determined to be the human hand from the various targets to be detected. Based on the key points of the human hand in each frame image and the depth value corresponding to the key points, the joint angle of each finger and the three-dimensional distance between each finger are determined.

[0031] Based on the joint angles of each finger and the three-dimensional distance between each finger, it is determined whether the current hand gesture of the human hand satisfies the preset gesture judgment logic, and the corresponding second judgment result is obtained.

[0032] The static hand gesture of the human hand is determined based on the second judgment result;

[0033] Alternatively, the current target to be detected corresponding to the current detection mode is determined from each of the targets to be detected as a human hand, and the three-dimensional position change and angle rotation change of the center point of the human hand in three-dimensional space are determined based on the key points of the human hand and the depth values ​​corresponding to the key points in multiple consecutive frames of images.

[0034] The dynamic gestures of the human hand are determined based on the three-dimensional position changes and angular rotation changes of the center point of the human hand.

[0035] Optionally, when the current detection mode is a torso detection mode for detecting the human torso, the step of determining the current target to be detected corresponding to the current detection mode from among the targets to be detected, and identifying the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, includes:

[0036] The current target to be detected corresponding to the current detection mode is determined from each of the targets to be detected as the human torso, and the bending angle of the arm limbs and / or knee limbs is determined based on the key points of the human torso in each frame image and the depth value corresponding to the key points.

[0037] The posture of the arm and / or knee is determined based on the bending angle of the arm and / or knee.

[0038] Alternatively, the current target to be detected corresponding to the current detection mode is determined from each of the targets to be detected as the human torso, and the trend of change of the height of the body's center of gravity is determined based on the key points of the human torso and the depth values ​​corresponding to the key points in multiple consecutive frames of images.

[0039] Whether a fall has occurred is determined based on the changing trend of the body's center of gravity height.

[0040] Optionally, calculating the depth value corresponding to the key point of each of the targets to be detected based on the RGB image and the depth image includes:

[0041] Align RGB and depth images using a preset image registration algorithm;

[0042] The coordinates of the key points are mapped to the corresponding aligned depth image to obtain the depth value corresponding to the key points;

[0043] Alternatively, the depth value corresponding to the key point of each of the targets to be detected can be determined by using bilinear interpolation.

[0044] Alternatively, the aligned depth image can be converted into a 3D point cloud;

[0045] Based on the 3D point cloud, the depth value corresponding to the key point of each target to be detected is determined by using nearest neighbor search.

[0046] Optionally, before determining the current detection mode based on the depth values ​​corresponding to the key points of each of the targets to be detected, the method further includes:

[0047] Determine whether the depth value corresponding to the key point is greater than a preset depth threshold;

[0048] If the depth value corresponding to the key point is greater than the preset depth threshold, then the key point is filtered out.

[0049] Secondly, the present invention also discloses a posture recognition device, wherein the device comprises:

[0050] The image acquisition module is used to acquire multiple consecutive frames of RGB images and depth images containing the human body to be detected;

[0051] The key point determination module is used to detect key points of multiple targets on the human body to be detected in the RGB image;

[0052] A depth value calculation module is used to calculate the depth value corresponding to the key points of each of the targets to be detected based on the RGB image and the depth image;

[0053] The mode determination module is used to determine the current detection mode based on the depth value corresponding to the key point of each of the targets to be detected;

[0054] The pose recognition module is used to determine the current target to be detected corresponding to the current detection mode from each of the targets to be detected, and to recognize the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, so as to obtain the corresponding pose recognition result.

[0055] Thirdly, the present invention discloses a terminal, comprising: a memory, a processor, and a posture recognition program stored in the memory and executable on the processor, wherein the posture recognition program, when executed by the processor, implements the steps of the posture recognition method as described above.

[0056] Fourthly, the present invention discloses a computer-readable storage medium storing a computer program that can be executed to implement the steps of the posture recognition method as described above.

[0057] This invention provides a posture recognition method and apparatus. The posture recognition method includes: acquiring multiple consecutive frames of RGB images and depth images containing a human body to be detected; detecting multiple key points of a target to be detected on the human body in the RGB images, and calculating the depth value corresponding to the key point of each target to be detected based on the RGB images and the depth images; determining a current detection mode based on the depth value corresponding to the key point of each target to be detected; determining the current target to be detected corresponding to the current detection mode from among the targets to be detected, and recognizing the posture of the current target to be detected based on the key point and the depth value corresponding to the key point of the current target to be detected, thereby obtaining a corresponding posture recognition result. Therefore, this invention determines multiple key points of the target on the human body to be detected, calculates the depth values ​​corresponding to the key points, and then determines the current detection mode based on the depth values ​​corresponding to the key points of each target, and determines the current target under the current detection mode. This allows for the detection of the posture of different targets under different detection modes, achieving targeted posture detection for each target. In the process of recognizing the posture of the current target under the current detection mode, the posture recognition of the current target is achieved by combining key point information and depth information. This solves the problem of low posture recognition accuracy in traditional posture recognition under complex backgrounds, multi-light source interference, and insufficient depth information, thereby improving the accuracy, real-time performance, and stability of posture recognition. Attached Figure Description

[0058] Figure 1 This is a comparative schematic diagram of a traditional gesture recognition method disclosed in this invention;

[0059] Figure 2 This is a flowchart of a preferred embodiment of the pose recognition method in this invention;

[0060] Figure 3 This is a schematic diagram of a specific hand key point detection method disclosed in this invention;

[0061] Figure 4 This is a hardware schematic diagram of a specific posture recognition system disclosed in this invention;

[0062] Figure 5 This is a functional principle block diagram of a preferred embodiment of the posture recognition device in this invention;

[0063] Figure 6 This is a functional principle block diagram of a preferred embodiment of the terminal in this invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0065] See Figure 1 As shown, traditional gesture recognition technologies are mostly based on 2D images (RGB images only), which are easily affected by background clutter or changes in lighting, resulting in low gesture recognition accuracy. Although some gesture recognition technologies combine TOF cameras to obtain depth information, relying solely on depth information still has difficulties in recognizing gestures in occluded or low-contrast scenes. In other words, traditional gesture recognition technologies suffer from low gesture recognition accuracy in complex backgrounds, multi-light source interference, and insufficient depth information. Moreover, traditional posture recognition technologies are designed for single-part posture recognition, such as the face, hand, or torso, and cannot flexibly determine the current part to be recognized and switch the posture recognition mode for each part based on the states of multiple parts in images captured at different shooting distances. To address this, this application provides a posture recognition scheme that can improve the accuracy, real-time performance, and stability of posture recognition.

[0066] Please see Figure 2 , Figure 2 This is a flowchart of the pose recognition method in this invention. For example... Figure 2 As shown, the pose recognition method described in this embodiment of the invention includes:

[0067] Step S11: Obtain multiple consecutive RGB images and depth images containing the human body to be detected.

[0068] In this embodiment, RGB images (color images) and depth images can be acquired using an RGBD camera. Specifically, the RGBD camera continuously captures images of the human body to be detected, resulting in a series of multiple frames of RGB and depth images output by the camera, containing the human body. It should be noted that the shooting distance can be either close-up or long-distance. When shooting the human body at close range, the image includes the upper body or arms to be detected; when shooting the human body at a long distance, the image may include the entire human body.

[0069] In this embodiment, after acquiring multiple consecutive frames of RGB and depth images containing the human body to be detected, the process may further include: preprocessing the RGB and depth images to identify noise and outliers, such as noise and outliers caused by rapid movement or reflections of the human body. The noise and outliers are then corrected to ensure that subsequent mode switching decisions are based on images with stable states over a period of time. The image preprocessing may also include range cropping and linear normalization of the depth images. Range cropping improves data quality and focuses on key areas in the image, reducing unnecessary computation; linear normalization improves the stability of depth data in the depth images.

[0070] Step S12: Detect key points of multiple targets on the human body to be detected in the RGB image, and calculate the depth value corresponding to the key points of each target to be detected based on the RGB image and the depth image.

[0071] Among them, the multiple targets to be detected on the human body can be parts that need to be identified in terms of posture, such as hands, face, and torso, and the key point is a two-dimensional key point.

[0072] In this embodiment, for the human body to be detected, key points of multiple targets on it are determined to provide a data foundation for subsequent pose recognition. Specifically, depth data from the depth image is used to enhance the contour and edge features of the human body in the RGB image. Multiple consecutive frames of RGB images are input into a pre-trained keypoint detection model to detect key points of multiple targets on the human body in the RGB image. That is, the pre-trained keypoint detection model is used to find key points of multiple targets on the human body in the enhanced RGB image. For example, the AI ​​pre-trained pose detection model (the pre-trained keypoint detection model) detects key points of the hands and torso of the human body in the RGB image, such as 21 key points of the hands and 33 key points of the body parts. See also... Figure 3 As shown, if an RGB image contains only the human hand, when a keypoint is detected by an AI pre-trained pose detection model, the model outputs the keypoints of the hand.

[0073] It is understandable that enhancing the contour and edge features of the human body in the RGB image by using depth data from the depth image before inputting multiple consecutive frames of RGB images into a pre-trained keypoint detection model can improve keypoint detection accuracy. This is especially true in scenarios with complex backgrounds, low-texture objects, occlusion, or lighting challenges. Utilizing geometric information to compensate for the shortcomings of pure RGB vision significantly improves keypoint detection accuracy. Specifically, preprocessing the original depth map (such as filtering, denoising, hole filling, and normalization) allows for the extraction of contours or edges from the preprocessed depth map through depth gradient calculation, Canny edge detection, or edge detection based on normal changes. These edges represent geometric edges. The extracted depth edge map (e.g., binary or weighted) is then directly added to the edge map of the RGB image or a specific feature channel. A spatial attention map is generated using the depth contour or edge information, highlighting the weights of the corresponding object edge regions in the RGB feature map. The depth edge map is then used as an additional input channel, concatenated with the original RGB image, and fed into the network for learning and fusion. The network branches process the RGB and depth edge information separately, fusing them at the network's high-level features.

[0074] It should also be noted that the training process of the pre-trained keypoint detection model may specifically include: performing spatial or photometric transformations on the image samples used for training, and scaling or normalizing the pixel coordinates of the keypoints marked on the image samples according to the scaling of the image samples to obtain processed image samples; constructing a training set using the processed image samples, and sequentially inputting the processed image samples in the training set into the initial keypoint detection model constructed based on the heatmap regression encoding and decoding architecture to obtain the corresponding prediction output; generating the corresponding two-dimensional Gaussian distribution based on the pixel coordinates of the keypoints marked on the processed image samples to obtain the GT Gaussian Heatmap; calculating the loss based on the predicted output Heatmap and the GT Gaussian Heatmap, and calculating the gradient of the loss relative to the model parameters; updating the model parameters using the gradient, and repeatedly traversing the training set until the current trained model meets the preset convergence condition to obtain the pre-trained keypoint detection model.

[0075] For example, training a keypoint detection model involves multiple stages, including data preparation, model architecture selection, loss function design, training strategy formulation, and evaluation and optimization. First, acquire image or video sequences containing the target object (such as a person, face, or hand). The data should cover the target object's performance under various poses, scales, viewpoints, lighting conditions, occlusion, and background diversity. Then, clearly define the keypoints to be detected (e.g., 33 keypoints for the body, 68 keypoints for the face, 21 joints for the hand, etc.). Then, use annotation tools (such as LabelMe, CVAT, Labelbox, etc.) to manually annotate the precise pixel coordinates (x, y) of all defined keypoints in each image. The consistency and accuracy of the annotation directly affect model performance, usually requiring multiple people to annotate and perform consistency checks and corrections (e.g., removing outliers, smoothing trajectories). Common annotation formats include COCO JSON (i.e., COCO annotation JSON file), MPII JSON (i.e., MPII human pose JSON annotation file), and CSV (i.e., comma-separated value file), which contain image paths, keypoint coordinates, visibility / occlusion markers, target bounding boxes, etc. Among them, COCO JSON refers to the JSON format annotation file used by the COCO (Common Objects in Context) dataset, which stores information such as the bounding box, class label, segmentation mask, and keypoints of the target in the image. The JSON format annotation file of MPII Human PoseDataset is specifically used for human keypoint detection tasks and contains joint coordinates, visibility markers, and human bounding box scale information. CSV is a plain text table data format, with fields separated by commas, and each row representing a record. It is widely used for data exchange and simple database storage. Further preprocessing of the image includes standardization or normalization, resizing, and denoising. Keypoints can also be preprocessed, such as scaling their pixel coordinates according to image scaling or normalizing them relative to the target bounding box. To increase data diversity, improve model generalization, and prevent overfitting, data augmentation can be performed, such as spatial or photometric transformations. During augmentation, the augmentation operations must be applied synchronously and consistently to both the image and the corresponding keypoint coordinates. A heatmap regression-based encoding / decoding architecture can be used as the framework for the keypoint detection model. This architecture outputs a K-channel feature map (heatmap) with the same spatial resolution as the input image (or after downsampling). Each channel corresponds to one keypoint.During training, a two-dimensional Gaussian distribution, known as the Ground Truth Gaussian Heatmap (GT Gaussian Heatmap), is generated at each keypoint location based on the labeled (x, y) coordinates. The GT Gaussian Heatmap is a probability distribution map used in computer vision to represent keypoint locations; its core principle is to transform discrete keypoint coordinates into a continuous two-dimensional probability distribution using a Gaussian function. Occluded points may generate a low-confidence Heatmap or be ignored. To minimize the difference between the predicted Heatmap and the GT Gaussian Heatmap, a Heatmap regression loss can be used during training. During training, the dataset can be divided into mini-batches, and then batch data is input. The predicted output is calculated, and the loss is then calculated, such as by calculating the loss value based on the predicted output and the GT, then calculating the gradient of the loss relative to the model parameters, and finally using the optimizer to update the model parameters using the gradient. This process is repeated multiple times across the entire training dataset until training is complete. For model validation, a portion of the training data can be separated as a validation set. After each epoch or a fixed number of iterations, the model performance is evaluated on the validation set to select the best model as the final keypoint detection model.

[0076] In this embodiment, the depth values ​​corresponding to the key points of each target to be detected are calculated. Specifically, this can be achieved by aligning the RGB image and the depth image using a preset image registration algorithm. After the registration and alignment of the RGB and depth images are completed, the depth values ​​corresponding to the key points can be directly obtained by mapping the coordinates of the key points to the corresponding depth images. This allows for subsequent determination of the detection mode based on the depth values ​​of the key points. Furthermore, the 3D pose of the key points can be reconstructed based on the depth values. For example, if the coordinates of a 2D key point are (x, y), the 3D coordinates (x, y, z) of the key point can be obtained by combining the depth values. Alternatively, after the registration and alignment of the RGB and depth images are completed, the depth values ​​corresponding to the key points of each target to be detected can be determined using bilinear interpolation, or the aligned depth map can be converted into a 3D point cloud, and the depth values ​​corresponding to the key points of each target to be detected can be determined using nearest neighbor search.

[0077] The registration and alignment of the RGB and depth images ensures high-precision alignment. For example, geometric correction and depth image compensation techniques based on camera intrinsics align the RGB and depth images, ensuring a one-to-one correspondence between each pixel in the depth map and a pixel in the color image (or through intrinsic parameter conversion). This requires precise alignment between the depth map coordinate system and the color image coordinate system used for keypoint detection. Then, for each detected 2D keypoint pixel coordinate, the depth value stored at that coordinate location is directly queried from the registered and aligned depth map.

[0078] The specific registration process between the RGB image and the depth image can be as follows: First, determine the intrinsic and extrinsic parameters of the camera; where the intrinsic parameters represent the camera's focal length, principal point position, and distortion parameters, and the extrinsic parameters represent the positional relationship between the RGB camera and the depth camera (such as rotation matrix R and translation vector T).

[0079] For each depth image pixel and its corresponding depth value Calculate the corresponding spatial coordinates using the depth camera intrinsics. ,Right now:

[0080] ;

[0081] ;

[0082] in, It's the focal length. It is the principal point position (i.e., the optical center of the camera).

[0083] 3D points from the depth camera Transform to the RGB camera coordinate system, that is: ;

[0084] Projecting points in the RGB camera coordinate system onto the RGB image plane, i.e.:

[0085] ;

[0086] ;

[0087] This determines the pixel position in the corresponding RGB image, and then sets the depth value... Simply place it onto the corresponding RGB pixel to complete the registration.

[0088] Step S13: Determine the current detection mode based on the depth value corresponding to the key point of each of the targets to be detected.

[0089] In this embodiment, to achieve targeted pose recognition of the targets to be detected, a detection mode corresponding to each target is pre-configured. Then, the current detection mode can be determined based on the depth values ​​corresponding to the key points of each target. Specifically, the average depth value corresponding to each target is determined based on the depth values ​​corresponding to the key points of each target, and the current detection mode is determined based on the average depth value corresponding to each target to adapt to the pose recognition of the targets in the current image scene. Furthermore, after switching the detection mode to the current detection mode, all historical data related to other targets can be cleared.

[0090] In this embodiment, before determining the current detection mode based on the depth value corresponding to the key point of each target to be detected, the method may further include: determining whether the depth value corresponding to the key point is greater than a preset depth threshold; if the depth value corresponding to the key point is greater than the preset depth threshold, then the key point is filtered out; if the depth value corresponding to the key point is not greater than the preset depth threshold, then the key point does not need to be filtered out. In other words, by eliminating false detection points that exceed a specified distance through the depth threshold, the detection accuracy and anti-interference ability can be further improved.

[0091] Furthermore, when multiple targets to be detected include human hands and human torsos, the current detection mode is determined based on the average depth value corresponding to each target to be detected. Specifically, this may include: determining whether the average depth value of human hands in each frame of RGB image meets a first preset distance condition, and determining whether the average depth value of human torsos meets a second preset distance condition, to obtain a first judgment result; based on the first judgment result corresponding to multiple consecutive frames of RGB images, the current detection mode is determined as a hand detection mode for detecting human hands or a torso detection mode for detecting human torsos.

[0092] If the first judgment result corresponding to multiple consecutive RGB images indicates that the average depth value of the human hand meets the first preset distance condition, but the average depth value of the human torso does not meet the second preset distance condition, then the current detection mode is determined to be the hand detection mode for detecting the human hand. If the first judgment result corresponding to multiple consecutive RGB images indicates that the average depth value of the human hand does not meet the first preset distance condition, but the average depth value of the human torso meets the second preset distance condition, then the current detection mode is determined to be the torso detection mode for detecting the human torso. It can be understood that the current detection mode is dynamically switched between hand detection mode and body detection mode based on the average depth values ​​of the hand and torso.

[0093] For example, based on the average depth values ​​of the hands and torso, the system automatically switches between "hand" and "body" modes. When the average depth value of the hands does not meet the condition of being less than 1000mm (first preset distance condition), and the average depth value of the torso meets the condition of being greater than 1000mm (second preset distance condition), it is determined that the person to be detected is at a distance, and in this case, it is more suitable to detect the posture of the human torso. If the current non-body mode fails to meet the first preset distance condition for the average depth value of the hands in three consecutive frames, and the average depth value of the human torso meets the second preset distance condition, the current detection mode will be officially switched to body mode, and all historical data related to gesture recognition will be cleared.

[0094] Step S14: Determine the current target to be detected corresponding to the current detection mode from each of the targets to be detected, and identify the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, and obtain the corresponding pose recognition result.

[0095] Understandably, in the process of recognizing the pose of the target under the current detection mode, combining key point information and depth information to achieve pose recognition of the target under the current detection mode can specifically identify the pose of the target under the current detection mode, thereby improving the accuracy, real-time performance and stability of pose recognition.

[0096] In one specific implementation, when the current detection mode is a hand detection mode for detecting human hands, the current target to be detected is determined to be a human hand from among the targets to be detected. Based on the key points of the human hand in each frame of the image and the corresponding depth values, the joint angles of each finger and the three-dimensional distance between each finger are determined. Based on the joint angles of each finger and the three-dimensional distance between each finger, it is determined whether the current gesture of the human hand satisfies the preset gesture determination logic, and a corresponding second determination result is obtained. Based on the second determination result, the static gesture of the human hand is determined. Alternatively, the current target to be detected is determined to be a human hand from among the targets to be detected. Based on the key points of the human hand in multiple consecutive frames of the image and the corresponding depth values, the three-dimensional position change and angle rotation change of the center point of the human hand in three-dimensional space are determined. Based on the three-dimensional position change and angle rotation change of the center point of the human hand, the dynamic gesture of the human hand is determined. It can be understood that dynamic and static gesture determination is achieved based on key points and their corresponding depth values.

[0097] For example, the logic for determining an open palm gesture is as follows: It checks if all five fingers are straight. That is, the function only returns True if the `is_extended` check result for the thumb, index finger, middle finger, ring finger, and little finger is all True, thus recognizing the hand gesture as an open palm. The logic for a fist gesture is the opposite of the open palm gesture. That is, the hand gesture is recognized as a fist only if the `is_extended` check result for the thumb, index finger, middle finger, ring finger, and little finger is all False. The logic for a scissor gesture is as follows: the index and middle fingers must be straight, the other three fingers must be bent, and there must be sufficient distance between the index and middle fingers (e.g., greater than a preset distance value). There must also be sufficient distance between the index and middle fingers to distinguish between a scissor gesture and a pointing gesture with two fingers together. If these three conditions are met, the hand gesture can be recognized as a scissor gesture. Similarly, the logic for other gestures can be set in the same way. Alternatively, the current hand gesture can be identified based on the positional relationship between finger key points and the palm or other key points in a single frame image, such as a fist, scissors, fingers spread, OK, etc.

[0098] Understandably, the recognition of all static gestures relies on the helper function `is_extended`, which determines whether a single finger is straight. For example, it calculates the angles of the two main joints of a finger, such as the first and second joints of the index finger. If both angles are greater than 150 degrees, the finger is considered straight; otherwise, it is considered bent.

[0099] Furthermore, for dynamic gestures, the position and angle of the hand's center point can be compared between consecutive frames to identify waving, sliding, or rotating actions.

[0100] Another specific implementation involves determining the human torso as the current detection target from among the various targets to be detected, based on the key points of the human torso and the corresponding depth values ​​in each frame of the image, the bending angles of the arms and / or knees are determined; the postures of the arms and / or knees are determined based on the bending angles of the arms and / or knees; or, determining the human torso as the current detection target from among the various targets to be detected, based on the key points of the human torso and the corresponding depth values ​​in multiple consecutive frames of the image, the trend of change in the height of the body's center of gravity is determined, and then the trend of change in the height of the body's center of gravity is used to determine whether a fall has occurred.

[0101] For example, when the left arm is bent at an angle less than 90 degrees and the right arm is extended at an angle greater than 150 degrees, it is identified as waving to the left; when the right arm is bent at an angle less than 90 degrees and the left arm is extended at an angle greater than 150 degrees, it is identified as waving to the right; when the angles of both knees are less than 120 degrees, it is identified as a deep squat. Furthermore, by analyzing the changes in body center of gravity height through key points of the torso in multiple frames of images, it is possible to quickly detect whether a fall has occurred. For example, if the change in body center of gravity height shows a downward trend, a fall is identified, meaning that fall detection can be achieved in body mode.

[0102] In this embodiment, after completing the posture recognition, hand or body skeleton markers and corresponding depth information can be superimposed on the original RGB image, and the depth image can be displayed in real time through an independent depth window. The hardware device implementing the posture recognition scheme of this application is also equipped with button screenshot and data recording functions.

[0103] As can be seen, in this embodiment of the invention, by determining multiple key points of the target to be detected on the human body, and simultaneously calculating the depth values ​​corresponding to the key points, the current detection mode is determined based on the depth values ​​corresponding to the key points of each target to be detected, and the current target to be detected under the current detection mode is determined, so as to detect the posture of different targets to be detected under different detection modes, thereby achieving targeted posture detection of each target to be detected. In the process of recognizing the posture of the current target to be detected under the current detection mode, the posture recognition of the current target to be detected is achieved by combining key point information and depth information. This can solve the problem of low posture recognition accuracy in traditional posture recognition when there are complex backgrounds, multiple light source interferences and insufficient depth information, thereby improving the accuracy, real-time performance and stability of posture recognition.

[0104] For example, see Figure 4 As shown, the technical solution of this application is executed using a system consisting of a central controller (CPU), a universal serial bus (USB), an RGB camera, and a TOF camera. The CPU can transmit relevant processing data to the universal serial bus for storage, and control the transmission of signals to synchronize the RGB camera and the TOF camera to control their synchronous acquisition of RGB images and depth images. The RGB camera and the TOF camera are fixed together by a rigid bracket to form an RGBD camera.

[0105] In one embodiment, such as Figure 5 As shown, based on the above-described posture recognition method, the present invention also provides a posture recognition device, comprising:

[0106] Image acquisition module 11 is used to acquire a series of RGB images and depth images containing the human body to be detected;

[0107] Key point determination module 12 is used to detect key points of multiple targets on the human body to be detected in the RGB image;

[0108] The depth value calculation module 13 is used to calculate the depth value corresponding to the key point of each of the targets to be detected based on the RGB image and the depth image;

[0109] The mode determination module 14 is used to determine the current detection mode based on the depth value corresponding to the key point of each of the targets to be detected;

[0110] The pose recognition module 15 is used to determine the current target to be detected corresponding to the current detection mode from each of the targets to be detected, and to recognize the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, so as to obtain the corresponding pose recognition result.

[0111] Furthermore, it is worth noting that the working process of the posture recognition device provided in this embodiment is the same as that of the posture recognition method described above, so it will not be repeated here. For details, please refer to the working process of the posture recognition method described above.

[0112] Figure 6 A schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal may include:

[0113] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0114] When the processor 502 executes the program, it implements the attitude recognition method provided in the above embodiments.

[0115] Furthermore, the terminal also includes:

[0116] Communication interface 503 is used for communication between memory 501 and processor 502.

[0117] The memory 501 is used to store computer programs that can run on the processor 502.

[0118] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0119] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one line is used in the diagram, but this does not imply that there is only one bus or one type of bus.

[0120] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0121] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0122] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described posture recognition method.

[0123] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0124] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0125] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can read and execute instructions from and from an instruction execution system, apparatus or device).

[0126] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0127] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A pose recognition method, characterized in that, The method includes: Acquire consecutive multi-frame RGB images and depth images containing the human body to be detected; Detect key points of multiple targets on the human body to be detected in the RGB image, and calculate the depth value corresponding to the key points of each target based on the RGB image and the depth image; The current detection mode is determined based on the depth value corresponding to the key point of each of the targets to be detected; The current target to be detected is determined from the targets to be detected according to the current detection mode, and the pose of the current target to be detected is identified based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, so as to obtain the corresponding pose recognition result. The process of acquiring multiple consecutive frames of RGB images and depth images containing the human body to be detected further includes: The RGB and depth images are preprocessed to identify and correct noise and outliers. Specifically, the depth image is cropped and the RGB image is linearly normalized. The key points of multiple targets on the human body to be detected in the RGB image include: The contour and edge features of the human body to be detected in the RGB image are enhanced by using depth data in the depth image; Multiple consecutive frames of RGB images are input into a pre-trained keypoint detection model to detect keypoints of multiple targets on the human body in the RGB images; wherein, the pre-trained keypoint detection model is an AI pre-trained pose detection model. Determining the current detection mode based on the depth values ​​corresponding to the key points of each of the targets to be detected includes: Based on the depth values ​​corresponding to the key points of each of the targets to be detected, determine the average depth value corresponding to each of the targets to be detected; The current detection mode is determined based on the average depth value corresponding to each of the aforementioned targets to be detected; When multiple targets to be detected include human hands and human torsos, determining the current detection mode based on the average depth value corresponding to each of the targets to be detected includes: Determine whether the average depth value of the human hand in each frame of RGB image meets the first preset distance condition, and determine whether the average depth value of the human torso meets the second preset distance condition, and obtain the corresponding first judgment result; Based on the first judgment result corresponding to multiple consecutive RGB images, the current detection mode is determined to be either the hand detection mode for detecting human hands or the torso detection mode for detecting human torso.

2. The pose recognition method according to claim 1, characterized in that, The training process of the pre-trained keypoint detection model also includes: The image samples used for training are subjected to spatial transformation or photometric transformation, and the pixel coordinates of the key points marked on the image samples are scaled or normalized according to the scaling of the image samples to obtain the processed image samples. A training set is constructed using the processed image samples, and the processed image samples in the training set are sequentially input into an initial keypoint detection model constructed based on a heatmap regression encoding and decoding architecture to obtain the corresponding prediction output. Based on the pixel coordinates of the key points marked on the processed image sample, a corresponding two-dimensional Gaussian distribution is generated to obtain the GT Gaussian Heatmap. The loss is calculated based on the predicted output heatmap and the GT Gaussian heatmap, and the gradient of the loss relative to the model parameters is calculated. The model parameters are updated using the gradient, and the training set is repeatedly traversed until the current trained model meets the preset convergence condition, thus obtaining the pre-trained keypoint detection model.

3. The pose recognition method according to claim 1, characterized in that, The step of determining the current detection mode as a hand detection mode for detecting human hands or a torso detection mode for detecting human torsos based on the first judgment result corresponding to multiple consecutive RGB images includes: If the first judgment result corresponding to multiple consecutive RGB images shows that the average depth value of the human hand meets the first preset distance condition, and the average depth value of the human torso does not meet the second preset distance condition, then the current detection mode is determined as the hand detection mode for detecting the human hand. If the first judgment result corresponding to multiple consecutive frames of images indicates that the average depth value of the human hand does not meet the first preset distance condition, and the average depth value of the human torso meets the second preset distance condition, then the current detection mode is determined as the torso detection mode for detecting the human torso.

4. The pose recognition method according to claim 1, characterized in that, When the current detection mode is a hand detection mode for detecting human hands, the step of determining the current target to be detected corresponding to the current detection mode from among the targets to be detected, and identifying the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, includes: The current target to be detected corresponding to the current detection mode is determined to be the human hand from the various targets to be detected. Based on the key points of the human hand in each frame image and the depth value corresponding to the key points, the joint angle of each finger and the three-dimensional distance between each finger are determined. Based on the joint angles of each finger and the three-dimensional distance between each finger, it is determined whether the current hand gesture of the human hand satisfies the preset gesture determination logic, and the corresponding second determination result is obtained. The static hand gesture of the human hand is determined based on the second judgment result; Alternatively, the current target to be detected corresponding to the current detection mode is determined from each of the targets to be detected as a human hand, and the three-dimensional position change and angle rotation change of the center point of the human hand in three-dimensional space are determined based on the key points of the human hand and the depth values ​​corresponding to the key points in multiple consecutive frames of images. The dynamic gestures of the human hand are determined based on the three-dimensional position changes and angular rotation changes of the center point of the human hand.

5. The pose recognition method according to claim 1, characterized in that, When the current detection mode is a torso detection mode for detecting the human torso, the step of determining the current target to be detected corresponding to the current detection mode from among the targets to be detected, and identifying the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, includes: The current target to be detected corresponding to the current detection mode is determined to be the human torso from each of the targets to be detected. Based on the key points of the human torso in each frame image and the depth value corresponding to the key points, the bending angle of the arm limbs and / or knee limbs is determined. The posture of the arm and / or knee is determined based on the bending angle of the arm and / or knee. Alternatively, the current target to be detected corresponding to the current detection mode is determined from each of the targets to be detected as the human torso, and the trend of change of the height of the body's center of gravity is determined based on the key points of the human torso and the depth values ​​corresponding to the key points in multiple consecutive frames of images. Whether a fall has occurred is determined based on the changing trend of the body's center of gravity height.

6. The pose recognition method according to claim 1, characterized in that, The calculation of the depth value corresponding to the key point of each of the targets to be detected based on the RGB image and the depth image includes: Align RGB and depth images using a preset image registration algorithm; The coordinates of the key points are mapped to the corresponding aligned depth image to obtain the depth value corresponding to the key points; Alternatively, the depth value corresponding to the key point of each of the targets to be detected can be determined by using bilinear interpolation. Alternatively, the aligned depth image can be converted into a 3D point cloud; Based on the 3D point cloud, the depth value corresponding to the key point of each target to be detected is determined by using nearest neighbor search.

7. The pose recognition method according to any one of claims 1 to 6, characterized in that, Before determining the current detection mode based on the depth values ​​corresponding to the key points of each of the targets to be detected, the method further includes: Determine whether the depth value corresponding to the key point is greater than a preset depth threshold; If the depth value corresponding to the key point is greater than the preset depth threshold, then the key point is filtered out.

8. A posture recognition device, characterized in that, The device includes: The image acquisition module is used to acquire multiple consecutive frames of RGB images and depth images containing the human body to be detected; The key point determination module is used to detect key points of multiple targets on the human body to be detected in the RGB image; A depth value calculation module is used to calculate the depth value corresponding to the key points of each of the targets to be detected based on the RGB image and the depth image; The mode determination module is used to determine the current detection mode based on the depth value corresponding to the key point of each of the targets to be detected; The pose recognition module is used to determine the current target to be detected corresponding to the current detection mode from each of the targets to be detected, and to recognize the pose of the current target to be detected based on the key points of the current target to be detected and the depth values ​​corresponding to the key points, so as to obtain the corresponding pose recognition result. The posture recognition device further includes: This module is used to preprocess RGB and depth images to identify noise and outliers in the RGB and depth images and to correct the noise and outliers; wherein, in the image preprocessing, the depth image is cropped and the RGB image is linearly normalized. The key point determination module is specifically used for: The contour and edge features of the human body to be detected in the RGB image are enhanced by using depth data in the depth image; Multiple consecutive frames of RGB images are input into a pre-trained keypoint detection model to detect keypoints of multiple targets on the human body in the RGB images; wherein, the pre-trained keypoint detection model is an AI pre-trained pose detection model. The pattern determination module is specifically used for: Based on the depth values ​​corresponding to the key points of each of the targets to be detected, determine the average depth value corresponding to each of the targets to be detected; The current detection mode is determined based on the average depth value corresponding to each of the aforementioned targets to be detected; When multiple targets to be detected include human hands and human torsos, it is determined whether the average depth value of the human hands in each frame of RGB image meets the first preset distance condition, and whether the average depth value of the human torso meets the second preset distance condition, so as to obtain the corresponding first judgment result. Based on the first judgment result corresponding to multiple consecutive RGB images, the current detection mode is determined to be either the hand detection mode for detecting human hands or the torso detection mode for detecting human torso.

Images