A skeleton key point position detection method and a rehabilitation robot

Abstract

The application provides a skeleton key point position detection method and a rehabilitation robot, and the method comprises the following steps: obtaining a pre-trained deep learning network model; the deep learning network model is trained based on a depth image training set pre-labeled with labels of each grid and user postures, and the label comprises body part information included in the grid; collecting a depth image of the user, and performing image preprocessing on the depth image to obtain a local depth image; the local depth image is a local image of an active area of a target part of the user extracted from the depth image; inputting the local depth image into the trained deep learning network model to obtain the current posture of the user and label information of each grid in the local depth image; and determining the skeleton key point coordinates of the user based on the current posture of the user and the label information of each grid in the local depth image. The application can improve the detection efficiency of key point detection, has good detection real-time performance, and improves the detection accuracy of key points.

Description

Technical Field

[0001] This invention relates to the field of rehabilitation robot technology, and in particular to a method for detecting the location of key skeletal points and a rehabilitation robot. Background Technology

[0002] Human skeletal keypoint detection refers to extracting key joints from images or videos. These key joints can represent information such as posture and movement. Commonly used key joints in human skeletal keypoint extraction include those of the head, arms, legs, and feet. Human skeletal keypoint extraction is mainly applied in fields such as motion analysis, medical rehabilitation, security monitoring, virtual reality, and game interaction.

[0003] Existing human skeleton key point detection technologies typically use feature matching to identify key point locations. This requires image preprocessing, human detection, and key point localization of the acquired images, which involves a large amount of computation, a long computation time, poor real-time performance, and is easily affected by factors such as image quality, human posture, and clothing occlusion, resulting in low key point detection accuracy. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method for detecting the location of key points in bones and a rehabilitation robot, which can improve the computational efficiency of key point detection, speed up the detection response time, improve the real-time performance of detection, and enhance the detection accuracy of key points.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0006] In a first aspect, embodiments of the present invention provide a method for detecting the location of key points in a skeleton, including:

[0007] Obtain a pre-trained deep learning network model; wherein the deep learning network model is trained based on a deep image training set with pre-labeled grids and user poses, and the labels include body part information included in the grids;

[0008] A depth image of the user is acquired, and the depth image is preprocessed to obtain a local depth image; wherein, the local depth image is a local image of the active region of the user's target part extracted from the depth image;

[0009] The local depth image is input into the trained deep learning network model to obtain the user's current pose and the label information of each grid in the local depth image;

[0010] The coordinates of the user's skeletal key points are determined based on the user's current pose and the label information of each grid in the local depth image.

[0011] Furthermore, this embodiment of the invention provides a first possible implementation of the first aspect, wherein the user posture includes the user's upper arm movement posture, the target part includes the upper limb, and the step of determining the coordinates of the user's skeletal key points based on the user's current posture and the label information of each grid in the local depth image includes:

[0012] Based on the user's current posture and the label information of each grid region in the local depth image, determine the grid regions where each arm part of the user is located in the local depth image;

[0013] The coordinates of the key points of the user's arm skeleton are determined based on the intersection of the grid areas where two adjacent arm parts are located.

[0014] Furthermore, this embodiment of the invention provides a second possible implementation of the first aspect, wherein each of the arm parts includes: hand, forearm, upper arm, and shoulder; the step of determining the coordinates of the skeletal key points of the user's arm based on the intersection position of the grid regions where two adjacent arm parts are located includes:

[0015] The pixel coordinates of the shoulder joint point are determined based on the boundary between the grid region where the shoulder is located and the grid region where the upper arm is located.

[0016] The pixel coordinates of the elbow joint are determined based on the boundary between the grid region where the forearm is located and the grid region where the upper arm is located.

[0017] The pixel coordinates of the wrist joint are determined based on the boundary between the grid area where the forearm is located and the grid area where the hand is located;

[0018] The pixel coordinates of the center point of the palm are determined based on the center position of the grid area where the hand is located.

[0019] Furthermore, this embodiment of the invention provides a third possible implementation of the first aspect, wherein the skeletal keypoint coordinates of the user's arm include three-dimensional coordinates; and further includes:

[0020] The three-dimensional coordinates of the shoulder joint are determined based on the pixel coordinates of the shoulder joint and the corresponding depth value.

[0021] The three-dimensional coordinates of the elbow joint are determined based on the pixel coordinates of the elbow joint and the corresponding depth value.

[0022] The three-dimensional coordinates of the wrist joint are determined based on the pixel coordinates of the wrist joint and the corresponding depth value.

[0023] The three-dimensional coordinates of the center point of the palm are determined based on the pixel coordinates of the center point of the palm and the corresponding depth value.

[0024] Furthermore, this embodiment of the invention provides a fourth possible implementation of the first aspect, wherein the skeletal key point location detection method further includes:

[0025] Multiple depth images of the user in various poses are acquired in advance, and the depth images are preprocessed to obtain depth image samples; wherein, the image preprocessing includes mesh division, and the depth image samples include multiple local depth images of the active regions of the user's target parts extracted from each of the depth images;

[0026] Labeling and orientation annotation are performed on each grid in the depth image sample to obtain the depth image training set.

[0027] Furthermore, this embodiment of the invention provides a fifth possible implementation of the first aspect, wherein the target part includes the upper limb, the grid is labeled as a stack, and the elements in the stack include the arm parts corresponding to the grid; wherein, when the stack includes multiple arm parts, the arm parts are sorted in ascending order of depth value.

[0028] Furthermore, this embodiment of the invention provides a sixth possible implementation of the first aspect, wherein the image preprocessing step includes:

[0029] Obtain the background depth image of the acquisition area corresponding to the depth image;

[0030] A depth image of the user's body region is extracted from the depth image based on the difference in pixel depth values ​​between the depth image and the background depth image;

[0031] Obtain the maximum range of motion of the user's target body part, and extract the local depth image corresponding to the maximum range of motion from the depth image of the user's body region;

[0032] The local depth image is divided into grids to divide it into multiple rectangular grids of equal area.

[0033] Furthermore, this embodiment of the invention provides a seventh possible implementation of the first aspect, wherein the target body part includes the upper limb, and the step of obtaining the maximum range of motion of the user's target body part includes:

[0034] Acquire target depth images of the user's arm in both vertically and horizontally extended postures;

[0035] The user's fingertip pixel coordinates are obtained from the target depth image, and the maximum range of motion of the user's arm is determined based on the fingertip pixel coordinates.

[0036] Furthermore, this invention provides an eighth possible implementation of the first aspect, wherein the target region includes the upper limb, and the skeletal key point location detection method further includes:

[0037] Obtain the restricted movement regions corresponding to each arm part of the user, and determine all the label information that can be assigned to each restricted movement region based on the restricted movement regions corresponding to each arm part, and record it as the base label;

[0038] When the trained deep learning network model identifies the label information of each grid in the local depth image, it determines whether the label information of each grid in the local depth image is correct based on the reference label.

[0039] When there are incorrect labels in the label information of each grid in the local depth image, the incorrect labels are deleted.

[0040] Secondly, embodiments of the present invention also provide a rehabilitation robot, comprising: an image sensor, a processor, and a storage device; the image sensor is used to acquire depth images of the user;

[0041] The storage device stores a computer program that, when executed by the processor, performs the method described in any of the first aspects.

[0042] This invention provides a method for detecting skeletal keypoint locations and a rehabilitation robot. The method includes: acquiring a pre-trained deep learning network model; wherein the deep learning network model is trained based on a training set of depth images pre-labeled with tags for each grid and the user's pose, the tags including information about body parts included in the grid; acquiring the user's depth image and performing image preprocessing on the depth image to obtain a local depth image; wherein the local depth image is a local image of the active area of ​​the user's target body part extracted from the depth image; inputting the local depth image into the trained deep learning network model to obtain the user's current pose and the tag information of each grid in the local depth image; and determining the coordinates of the user's skeletal keypoints based on the user's current pose and the tag information of each grid in the local depth image. This invention preprocesses the acquired user depth image, extracting only the local depth image of the user's body activity area. This local depth image is then input into a trained deep learning network model to identify the user's current pose and the label information of each grid. This reduces the number of image pixels that need to be processed, improves the computational efficiency of key point detection, speeds up the detection response time, and provides good real-time performance. By determining the coordinates of the user's skeletal key points based on the identified user's current pose and the label information of each grid, the local depth image is subdivided into grids for key point detection, thus improving the accuracy of key point detection.

[0043] Other features and advantages of the embodiments of the present invention will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above in the embodiments of the present invention.

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0045] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 The flowchart of a skeletal key point location detection method provided by an embodiment of the present invention is shown;

[0047] Figure 2a This diagram illustrates a background depth image of a data acquisition area provided by an embodiment of the present invention.

[0048] Figure 2b This invention provides a depth image of a user in a horizontal arm abduction pose, as shown in an embodiment of the invention.

[0049] Figure 2c This invention illustrates a depth image of a user's body region provided by an embodiment of the invention;

[0050] Figure 3a This invention provides a schematic diagram of the background depth image of another acquisition area according to an embodiment of the invention.

[0051] Figure 3b This image illustrates a depth image of a user in an arm-upward gesture, as provided in an embodiment of the present invention.

[0052] Figure 3c This invention provides another depth image of a user's body region according to an embodiment of the invention.

[0053] Figure 4 This diagram illustrates the maximum range of motion of a user's arm as provided in an embodiment of the present invention.

[0054] Figure 5 This diagram illustrates a grid division of a local depth image according to an embodiment of the present invention.

[0055] Figure 6 This image shows a local depth image of a user with their shoulder joint flexed at 90 degrees and elbow joint extended, as provided in an embodiment of the present invention.

[0056] Figure 7a This invention provides a local depth image of a motion-limiting region segmentation diagram when a user's shoulder joint is flexed at 90 degrees and the elbow joint is extended, according to an embodiment of the present invention.

[0057] Figure 7b This invention provides a local depth image mapping of motion-limited regions when a user's arm is extended vertically upwards, according to an embodiment of the invention.

[0058] Figure 7c This invention provides a local depth image of a motion-limited region segmentation diagram when a user's arm is horizontally extended, according to an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0060] Human skeletal keypoint detection technology is a technique that uses computer vision to identify and analyze human posture and movement. Its technological background primarily stems from the continuous development of artificial intelligence and the widespread adoption of sensor technology. Its applications are becoming increasingly broad, including but not limited to intelligent video surveillance, human-computer interaction, virtual reality, motion analysis, and medical rehabilitation. In the development of human skeletal keypoint extraction technology, deep learning-based methods have gradually become mainstream. Common deep learning networks include Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), and Long Short-Term Memory Networks (LSTM).

[0061] Besides deep learning methods, traditional computer vision methods also play a crucial role in human pose recognition. Furthermore, with the continuous development of sensor technology, some sensor-based pose recognition methods have gradually become research hotspots. In short, research on human pose recognition technology has become an important direction in the field of computer vision, and its application scenarios are becoming increasingly widespread. In the future, with the continuous development of technology, human pose recognition will be applied in more fields, bringing greater convenience and safety to people's lives.

[0062] Currently, existing technologies for detecting key points in the human skeleton typically include computer vision-based, deep learning-based, and sensor-based methods. Computer vision-based methods usually use feature descriptors or models to describe the human skeletal structure, such as SIFT, SURF, and HOG models. They identify the location of key points through feature matching or model recognition, requiring multiple complex image processing steps and consuming a significant amount of time for location registration, resulting in poor real-time performance. Deep learning-based methods typically build neural network models to automatically learn the features of key points from the input image and predict the location coordinates of key points through regression, which involves a large amount of computation. Sensor-based methods use hardware devices such as inertial sensors (e.g., accelerometers and gyroscopes) and optical sensors (e.g., cameras) to acquire human motion information and achieve skeletal key point recognition through signal processing and machine learning. This method has high hardware requirements, leading to high equipment costs.

[0063] To address the aforementioned issues, this invention provides a method for detecting the location of key skeletal points and a rehabilitation robot. The following provides a detailed description of the embodiments of this invention.

[0064] This embodiment provides a method for detecting the location of key points in a skeleton. See [link to relevant documentation]. Figure 1 The flowchart shown is for a method to detect the location of key points on a skeleton. This method mainly includes the following steps:

[0065] Step S102: Obtain the pre-trained deep learning network model;

[0066] The aforementioned deep learning network model is trained on a deep image training set with pre-labeled grids and user poses. The labels include information about the body parts included in the grids.

[0067] Each depth image in the aforementioned depth image training set is labeled with a grid label and a user pose. The grid label corresponds to the image of the human body part for that grid. For example, for the hand region in a depth image, the label information for all grids within that hand region includes the hand. The user pose can include the user's upper limb pose, i.e., the arm movement pose.

[0068] Step S104: Acquire the user's depth image and perform image preprocessing on the depth image to obtain a local depth image;

[0069] The aforementioned local depth image is a partial image of the active region of the user's target body extracted from the depth image. The target body can be the upper limb, lower limb, or other parts of the human body.

[0070] Since the collected user images usually include background images of the collection area, removing the background depth images from the depth images and extracting only the depth images of the human body area reduces the computational load during image recognition and reduces interference factors. When detecting key points of the upper limb skeleton of a user, only a local image of the active area of ​​the user's target part can be cropped. For example, when detecting key points of the upper limb skeleton, only a local image of the upper limb active area can be cropped, and the lower limb depth images that are not in the user's upper limb active area can be deleted, thereby further reducing the computational load during image recognition and improving the real-time performance of user pose recognition.

[0071] Step S106: Input the local depth image into the trained deep learning network model to obtain the user's current pose and the label information of each grid in the local depth image;

[0072] The captured local depth image is input into the trained deep learning network model so that the deep learning network model can quickly identify the user's current pose in the local depth image. The local depth image input into the deep learning network model also includes a pre-divided grid. The deep learning model can also identify the label information of each grid in the local depth image. The label information is the image of the user's part in the grid. For example, if the label information is "shoulder", it means that the grid is located in the shoulder region of the image.

[0073] In one specific implementation, the aforementioned deep learning network model can be a convolutional neural network model.

[0074] Step S108: Determine the coordinates of the user's skeletal key points based on the user's current pose and the label information of each grid in the local depth image.

[0075] Since each grid in the local depth image carries label information, the positional information of various body parts of the user can be determined based on the label information and the corresponding grid position. The user's current posture can be identified to determine the user's body movement information. Furthermore, the positions of key joints can be determined based on the positional information of each body part and the user's body movement information. For example, when the user's current posture is standing, the position of the ankle joint can be determined based on the intersection of the thigh and calf positions; when the user's current posture is with an outstretched arm, the position of the elbow joint can be determined based on the intersection of the upper arm and forearm positions.

[0076] The coordinates of the aforementioned skeletal key points can be pixel coordinates in a local depth image or a depth image, or they can be three-dimensional coordinates obtained based on image depth information and image sensor intrinsic parameters.

[0077] The skeletal key point location detection method provided in this embodiment preprocesses the acquired user depth image, extracting only the local depth image of the user's body activity area. This local depth image is then input into the trained deep learning network model to identify the user's current pose and the label information of each grid. This reduces the number of image pixels that need to be processed, improves the computational efficiency of key point detection, speeds up the detection response time, and provides good real-time performance. By determining the coordinates of the user's skeletal key points based on the identified user's current pose and the label information of each grid, the local depth image is subdivided into grids for key point detection, thereby improving the detection accuracy of key points.

[0078] In one embodiment, the method provided in this embodiment further includes: acquiring multiple depth images of the user in various poses that have been pre-collected; performing image preprocessing on the depth images to obtain depth image samples; and labeling and pose-labeling each grid in the depth image samples to obtain a depth image training set.

[0079] The aforementioned depth image samples include multiple local depth images of the active regions of the user's target area extracted from each of the depth images. The aforementioned image preprocessing includes grid partitioning. By preprocessing the depth images under various pre-acquired action poses, the images used to train the deep learning network model are of the same type as the images input to the trained deep learning network model. This reduces the number of pixels in each sample image in the resulting depth image samples, shortening the training time of the deep learning network model and improving its training efficiency. Since each image in the depth image training set contains the depth value of each pixel, by labeling each image in the depth image training set with the corresponding pose and the label of each grid, the trained deep learning network model can establish a mapping relationship between the depth value distribution characteristics and the actual labels of each grid, as well as a mapping relationship between the actual labels of each grid and the user pose. This enables the trained deep learning network model to recognize the grid labels and user poses in the local depth images.

[0080] In one embodiment, this embodiment provides an implementation method for image preprocessing, which can be performed according to the following steps:

[0081] Step 1): Obtain the background depth image of the acquisition area corresponding to the depth image;

[0082] The image preprocessing implementation method in this embodiment can be applied to the step of preprocessing images to obtain local depth images before training a deep learning network model from a depth image input. It can also be applied to the step of preprocessing multiple depth images to obtain depth image samples.

[0083] A depth image sensor is installed at a designated location within the aforementioned data acquisition area to capture a static background depth image before the user enters the area; this image is denoted as the background depth image. For example, see... Figure 2a The image shown is a schematic diagram of the background depth of the acquisition area. Figure 2a The image shows a background depth image when the user has not entered the acquisition area. In one embodiment, the depth image sensor can continuously acquire depth images over a period of time and calculate the average value to obtain the final background depth image before the user enters the sensor's acquisition range. The depth value of each pixel in the background depth image represents the distance between the object point in the real environment corresponding to that pixel and the depth image sensor.

[0084] Let the resolution of the depth image sensor (i.e., the depth camera) be M×N, then the background depth image I b It can be represented as a two-dimensional floating-point array d M,N Each element d in the array i,j ∈[0,D], where D is the maximum depth value that the depth image sensor can detect, and i∈[0,M-1] and j∈[0,N-1] are the index values ​​of the horizontal and vertical axes of the image, respectively.

[0085] Step 2): Extract the depth image of the user's body region from the depth image based on the difference in pixel depth values ​​between the depth image and the background depth image;

[0086] Depth image I H Subtract background depth image I b This yields a depth image of the user's body region, calculated by subtracting the depth value L between each pixel in the depth image and the corresponding pixel in the background depth image.

[0087]

[0088] The depth value of a pixel in a depth image. Let d be the depth value of a pixel in the background depth image. If the difference in depth values ​​of pixels L ≤ ∈ , then the pixel is a background image pixel, and its depth value in the depth image is set to 0. If the difference in depth values ​​of pixels L > ∈ , then the pixel represents the location of the user's body area, and its depth value in the depth image is retained as d. I This yields a depth image of the user's body region, where ∈ is a preset depth threshold.

[0089] Typically, the depth image of the user's body region obtained above may still contain some isolated noise points. Median filtering algorithm can be used to filter out the noise points in the depth image of the user's body region, retaining only the largest connected region of the user's body region.

[0090] For example, see such as Figure 2b The user pose shown is a depth image of an arm in a horizontal abduction motion. This depth image includes a depth image of the user's body region and a background depth image. Pixels whose pixel depth values ​​differ from the background depth image by a preset depth threshold are retained, resulting in the following: Figure 2c The image shown is a depth image of the user's body region.

[0091] See also Figure 3b The user pose shown is a depth image of an arm moving vertically upwards. Based on this depth image and... Figure 3a The difference in pixel depth values ​​of the background depth image shown can be extracted from the depth image. Figure 3c The image shown is a depth image of the user's body area.

[0092] Step 3): Obtain the maximum range of motion of the user's target body part, and extract the local depth image corresponding to the maximum range of motion from the depth image of the user's body region;

[0093] Based on the pixel dimensions of the target area, the edge position that the target area can reach can be determined, i.e., the maximum range of motion of the target area. A rectangle containing the maximum range of motion is extracted from the depth image of the user's body region and denoted as the local depth image.

[0094] In one embodiment, this embodiment provides a specific implementation method for obtaining the maximum range of motion of a user's target body part: obtaining target depth images of the user's arm in vertical and horizontal extended postures; obtaining the user's fingertip pixel coordinates from the target depth images; and determining the maximum range of motion of the user's arm body part based on the fingertip pixel coordinates.

[0095] When the target area mentioned above is one of the user's upper limbs, see as follows: Figure 4 The diagram shows the maximum range of motion of the user's arm. Point A is the fingertip position when the user's arm is horizontally extended, and point B is the fingertip position when the user's arm is vertically extended upwards. Point O is the intersection of a horizontal line drawn through point A and a vertical line drawn through point B. Let the pixel coordinates of point A be (A...). x A y The pixel coordinates of point B are (b x ,b y The pixel coordinates of point O are (O x O y By extending line segments OA and OB to the left and lower part of the image, respectively, the maximum range of motion of the user's unilateral upper limb was obtained (i.e., Figure 4 The gray shaded area in the image), the rectangular area of ​​maximum motion range can be represented by P. max and P minThe two endpoints represent that P max The pixel coordinates are (A) x B y ), P min The pixel coordinates are (2O) x -A x ,2O y -B y By cropping the rectangular region corresponding to the maximum range of motion, the local depth image corresponding to the maximum range of motion is obtained.

[0096] Since the depth image size in the depth video stream acquired by the depth image sensor is M×N, by only cropping the local depth image that involves the maximum range of motion of the target part, the calculation can be performed only on the local pixels, reducing the computational load of key point detection.

[0097] Step 4): Divide the local depth image into a grid to divide it into multiple rectangular grids of equal area.

[0098] To further improve computational efficiency, the extracted local depth image is divided into a mesh. For example, see [example missing]. Figure 5 The diagram shown illustrates the mesh division of a local depth image, which divides the local depth image into multiple rectangular meshes of the same area with a preset size.

[0099] In one embodiment, the target body part provided in this embodiment includes the upper limb, the grid is labeled as a stack, and the elements in the stack include the arm parts corresponding to the grid; wherein, when the stack includes multiple arm parts, the arm parts are sorted in ascending order of depth value.

[0100] See also Figure 6 The image shown is a local depth image of a user with their shoulder flexed at 90 degrees and elbow extended. In this local depth image, all parts of the user's arm—shoulder, upper arm, forearm, and hand—are located in a circular region Z. The label stack of the grid inside the circular region Z is [shoulder, upper arm, forearm, hand], in the order from top to bottom of the stack from left to right. The label stacks of the grids in other regions outside the circular region Z are empty.

[0101] For example, due to Figure 5 In the image, the upper limb posture is an outstretched arm. Therefore, the grid labels for the hand region are "[Hand]", the grid labels for the upper arm region are "[Upper Arm]", and so on, to obtain the labels for each grid in the local depth image. To reduce computational load, only the grids within the arm region of the local depth image can be labeled or identified.

[0102] In one embodiment, the user's pose includes the user's upper arm movement pose, and the target area includes the upper limb. This embodiment provides an implementation method for determining the coordinates of the user's skeletal key points based on the user's current pose and the label information of each grid in the local depth image. The specific steps are as follows:

[0103] Step (1): Based on the user's current pose and the label information of each grid region in the local depth image, determine the grid regions where each arm part of the user is located in the local depth image;

[0104] After the deep learning network model identifies the user's current pose and the label information of each grid region in the local depth image, when the user's current pose is an arm extended or bent, the grids with the same top element in the label stack are merged to obtain the grid region where the arm part of the top element of the stack is located.

[0105] When the user's current posture is with the shoulder joint flexed at 90 degrees and the elbow joint extended, the grids corresponding to each arm part have a high overlap rate. Merging the grids with the top element being the hand will give the grid area where the hand is located. Merging the grids with the second element being the forearm in the label stack will give the grid area where the forearm is located. Similarly, merging the grids with the third element being the upper arm in the label stack will give the grid area where the upper arm is located.

[0106] Step (2): Determine the coordinates of the key points of the user's arm skeleton based on the intersection of the grid areas where two adjacent arm parts are located.

[0107] When the user's current posture is an extended or bent arm, determining the midpoint of the boundary line between any two adjacent arm segments in the grid area yields the coordinates of the user's arm's skeletal key points. These key points include the shoulder joint, elbow joint, wrist joint, and the center of the palm.

[0108] In one specific embodiment, the aforementioned arm parts include: hand, forearm, upper arm, and shoulder;

[0109] The pixel coordinates of the shoulder joint are determined based on the boundary between the grid region containing the shoulder and the grid region containing the upper arm. The center point of the boundary line between the grid region containing the shoulder and the grid region containing the upper arm can be determined as the pixel coordinates of the shoulder joint. If there is an overlapping area between the grid region containing the shoulder and the grid region containing the upper arm, the center point of the overlapping area can be determined as the pixel coordinates of the shoulder joint.

[0110] The pixel coordinates of the elbow joint are determined based on the boundary between the grid regions containing the forearm and the grid regions containing the upper arm. The center point of the boundary line between the grid regions containing the forearm and the grid regions containing the upper arm can be determined as the pixel coordinates of the elbow joint. If there is an overlapping area between the grid regions containing the forearm and the grid regions containing the upper arm, the center point of the overlapping area can be determined as the pixel coordinates of the elbow joint.

[0111] The pixel coordinates of the wrist joint are determined based on the boundary between the grid regions containing the forearm and the grid regions containing the hand. The center point of the boundary line between the grid regions containing the forearm and the grid regions containing the hand can be determined as the pixel coordinates of the wrist joint. If there is an overlapping area between the grid regions containing the forearm and the grid regions containing the hand, the center point of the overlapping area can be determined as the pixel coordinates of the wrist joint.

[0112] The pixel coordinates of the center point of the palm are determined based on the center position of the grid area where the hand is located.

[0113] In one embodiment, the method provided in this embodiment further includes: determining the three-dimensional coordinates of the shoulder joint point based on the pixel coordinates of the shoulder joint point and the corresponding depth value;

[0114] The three-dimensional coordinates of the elbow joint are determined based on the pixel coordinates of the elbow joint and the corresponding depth value.

[0115] The three-dimensional coordinates of the wrist joint are determined based on the pixel coordinates of the wrist joint and the corresponding depth values.

[0116] The three-dimensional coordinates of the center point of the palm are determined based on the pixel coordinates of the center point and the corresponding depth value.

[0117] Beforehand, the depth image sensor that acquires the user's depth image is calibrated to obtain its intrinsic and extrinsic parameters. When the coordinates of the key points of the user's arm skeleton are pixel coordinates in the local depth image, these coordinates are first converted to pixel coordinates in the depth image. Since the depth image sensor's position is fixed, based on the pixel coordinates of the key points of the user's arm skeleton in the depth image and the sensor's intrinsic and extrinsic parameters, the pixel coordinates of the key points of the user's arm skeleton can be converted into three-dimensional coordinates with the depth image sensor as the origin.

[0118] In one embodiment, the method provided in this embodiment further includes:

[0119] Obtain the restricted movement areas corresponding to each part of the user's arms, and determine all the label information that can be assigned to each restricted movement area based on the restricted movement areas corresponding to each arm, which is recorded as the baseline label;

[0120] When the trained deep learning network model recognizes the label information of each grid in the local depth image, it judges whether the label information of each grid in the local depth image is correct based on the benchmark label.

[0121] When erroneous labels are found in the label information of each grid in the local depth image, these erroneous labels are deleted. Specifically, it is determined whether the label information of each identified grid contains label content that is not present in the reference label of the corresponding restricted motion region; any label content that is not present in the reference label of the corresponding restricted motion region is identified as an erroneous label.

[0122] by Figure 6 Taking the local depth image shown as an example, each grid can be considered a processing unit. Each processing unit may be assigned one or more labels. Based on the movement trajectory of the human upper limb and joint constraints, with the shoulder joint O as the origin, the image is divided into four restricted motion regions by three concentric circles of different radii: including three circular regions X, Y, and Z, and region S outside the circular regions. Rectangular grids within region S can only be labeled "hand"; rectangular grids within region X may be labeled "forearm" or "hand"; rectangular grids within region Y may be labeled "forearm," "hand," or "upper arm"; and rectangular grids within region Z may be labeled "forearm," "hand," "upper arm," or "shoulder." That is, the closer the region is to point O, the more labels it may be covered and defined by.

[0123] All the label information that can be assigned to the grid within each restricted motion region is denoted as the baseline label; for example, the baseline label for each grid in region S is [hand], the baseline label for each grid in region X is [forearm, hand], the baseline label for each grid in region Y is [forearm, hand, upper arm], and the baseline label for each grid in region Z is [forearm, hand, upper arm, shoulder]. Each grid includes the corresponding pixel depth value, the baseline label, and the actual label information recognized by the deep learning network model.

[0124] For example, see such as Figure 7a The image shown is a local depth image of the restricted motion region when the user's shoulder joint is flexed at 90 degrees and the elbow joint is extended. Figure 7a The reference labels for the location of point M are [forearm, hand, upper arm, shoulder].

[0125] See also Figure 7b The image shown is a local depth image of the user's arm when it is extended vertically upwards, which limits the region of motion. Figure 7b The baseline labels for point M are [forearm, hand, upper arm, shoulder], for point S are [forearm, hand, upper arm], and for point T are [forearm, hand]. The trained deep learning network model identifies... Figure 7b When the grid label of point T in the local depth image is [shoulder, forearm], since the grid label includes the label information "shoulder" which does not exist in the reference label within the restricted motion area, the identified "shoulder" is determined to be an incorrect label, and the "shoulder" in the label is deleted to improve the recognition accuracy of key points.

[0126] And such as Figure 7c The image shown is a local depth image of a user with their arm horizontally extended, representing a region of motion that is restricted. Figure 7c The baseline labels for point M are [forearm, hand, upper arm, shoulder], for point N are [forearm, hand, upper arm], and for point R are [forearm, hand]. When the deep learning network model recognizes... Figure 7c If the label for point M is [shoulder], the label for point N is [upper arm], the label for point R is [forearm], and the label stacks for points S and T are empty, then the identification is correct.

[0127] In practical applications, the skeletal key point location detection method provided in this embodiment has relatively low hardware requirements compared to traditional deep learning-based algorithms. It can achieve a recognition and running efficiency of around 30 frames per second using a 10th generation Intel i5 CPU and integrated graphics card. The accuracy rate for key point detection, including the head, shoulder joint, elbow joint, wrist joint, and hand, is over 95%, and the proportion of time with correct detection is 96%.

[0128] The skeletal key point location detection method provided in this embodiment targets the key location detection of the human upper limb skeleton. By introducing restrictions on human joint movement to extract the active area of ​​the target upper limb for calculation, the number of image pixels that need to be processed is reduced, improving computational efficiency and response time, while also reducing hardware configuration requirements. By using depth video stream information to detect the key locations of the human upper limb skeleton, the need for color video stream for location registration is eliminated, resulting in faster speed and lower hardware requirements, thus reducing the cost of signal acquisition equipment. By orderly arranging the occlusion caused by limb coverage during human upper limb movement in the form of a label stack, the training efficiency of the deep learning model is increased, and the accuracy of the model prediction results is improved.

[0129] Corresponding to the skeletal key point location detection method provided in the above embodiments, this invention provides a rehabilitation robot, including: an image sensor, a processor, and a storage device; the image sensor is used to acquire depth images of the user; the storage device stores a computer program, which executes the skeletal key point location detection method provided in the above embodiments when run by the processor.

[0130] The rehabilitation robot provided in this embodiment has the same implementation principle and technical effects as the aforementioned embodiments. For the sake of brevity, any parts not mentioned in the rehabilitation robot embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0131] This invention provides a computer-readable medium storing computer-executable instructions. When these computer-executable instructions are invoked and executed by a processor, they cause the processor to implement the methods described in the above embodiments.

[0132] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing embodiments, and will not be repeated here.

[0133] The computer program product for the skeletal key point location detection method and rehabilitation robot provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0134] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0135] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0137] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting the location of key points in a skeleton, characterized in that, include: Obtain a pre-trained deep learning network model; wherein the deep learning network model is trained based on a deep image training set with pre-labeled grids and user poses, and the labels include body part information included in the grids; A depth image of the user is acquired, and the depth image is preprocessed to obtain a local depth image; wherein, the local depth image is a local image of the active region of the user's target part extracted from the depth image; The local depth image is input into the trained deep learning network model to obtain the user's current pose and the label information of each grid in the local depth image; The coordinates of the user's skeletal key points are determined based on the user's current pose and the label information of each grid in the local depth image.

2. The method according to claim 1, characterized in that, The user pose includes the user's upper arm movement pose, the target area includes the upper limb, and the step of determining the coordinates of the user's skeletal key points based on the user's current pose and the label information of each grid in the local depth image includes: Based on the user's current posture and the label information of each grid region in the local depth image, determine the grid regions where each arm part of the user is located in the local depth image; The coordinates of the key points of the user's arm skeleton are determined based on the intersection of the grid areas where two adjacent arm parts are located.

3. The method according to claim 2, characterized in that, Each of the aforementioned arm parts includes: hand, forearm, upper arm, and shoulder; the step of determining the coordinates of the skeletal key points of the user's arm based on the intersection of the grid regions of two adjacent arm parts includes: The pixel coordinates of the shoulder joint point are determined based on the boundary between the grid region where the shoulder is located and the grid region where the upper arm is located. The pixel coordinates of the elbow joint are determined based on the boundary between the grid region where the forearm is located and the grid region where the upper arm is located. The pixel coordinates of the wrist joint are determined based on the boundary between the grid area where the forearm is located and the grid area where the hand is located; The pixel coordinates of the center point of the palm are determined based on the center position of the grid area where the hand is located.

4. The method according to claim 3, characterized in that, The coordinates of the key points of the user's arm skeleton include three-dimensional coordinates; and also include: The three-dimensional coordinates of the shoulder joint are determined based on the pixel coordinates of the shoulder joint and the corresponding depth value. The three-dimensional coordinates of the elbow joint are determined based on the pixel coordinates of the elbow joint and the corresponding depth value. The three-dimensional coordinates of the wrist joint are determined based on the pixel coordinates of the wrist joint and the corresponding depth value. The three-dimensional coordinates of the center point of the palm are determined based on the pixel coordinates of the center point of the palm and the corresponding depth value.

5. The method according to claim 1, characterized in that, Also includes: Multiple depth images of the user in various poses are acquired in advance, and the depth images are preprocessed to obtain depth image samples; wherein, the image preprocessing includes mesh division, and the depth image samples include multiple local depth images of the active regions of the user's target parts extracted from each of the depth images; Labeling and pose annotation are performed on each grid in the depth image sample to obtain the depth image training set.

6. The method according to claim 1 or 5, characterized in that, The target area includes the upper limb, the grid is labeled as a stack, and the elements in the stack include the arm parts corresponding to the grid; wherein, when the stack includes multiple arm parts, the arm parts are sorted in ascending order of depth value.

7. The method according to claim 1 or 5, characterized in that, The image preprocessing steps include: Obtain the background depth image of the acquisition area corresponding to the depth image; A depth image of the user's body region is extracted from the depth image based on the difference in pixel depth values ​​between the depth image and the background depth image; Obtain the maximum range of motion of the user's target body part, and extract the local depth image corresponding to the maximum range of motion from the depth image of the user's body region; The local depth image is divided into grids to divide it into multiple rectangular grids of equal area.

8. The method according to claim 7, characterized in that, The target body part includes the upper limb, and the step of obtaining the maximum range of motion of the user's target body part includes: Acquire target depth images of the user's arm in both vertically and horizontally extended postures; The user's fingertip pixel coordinates are obtained from the target depth image, and the maximum range of motion of the user's arm is determined based on the fingertip pixel coordinates.

9. The method according to claim 1, characterized in that, The target area includes the upper limb, and the skeletal key point location detection method further includes: Obtain the restricted movement regions corresponding to each arm part of the user, and determine all the label information that can be assigned to each restricted movement region based on the restricted movement regions corresponding to each arm part, and record it as the base label; When the trained deep learning network model identifies the label information of each grid in the local depth image, it determines whether the label information of each grid in the local depth image is correct based on the reference label. When there are incorrect labels in the label information of each grid in the local depth image, the incorrect labels are deleted.

10. A rehabilitation robot, characterized in that, include: An image sensor, a processor, and a storage device; the image sensor is used to acquire depth images of the user; The storage device stores a computer program that, when executed by the processor, performs the method as described in any one of claims 1 to 9.

Images