Posture recognition method, device, system, equipment, medium, product and rehabilitation mirror

Abstract

The embodiment of the invention discloses a posture recognition method, device, system and equipment, a medium, a product and a rehabilitation mirror, and the method comprises the steps: obtaining the infrared image data and the depth image data of a target object, the infrared image data and the depth image data being a sequence image data pair synchronously collected in a preset time; inputting the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtaining an object structure segmentation map based on the object structure recognition result; and performing data fitting on the object structure segmentation image and the corresponding depth image data to obtain a three-dimensional skeleton sequence image of the target object. According to the technical scheme provided by the embodiment of the invention, the problems of inaccurate posture recognition and limitation of direction and angle recognition in the prior art are solved, the three-dimensional skeleton sequence diagram can be generated by combining the two-dimensional and three-dimensional image data and the neural network model, postures in any direction and angle can be recognized, and the accuracy and stability of posture recognition are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of computer vision, and in particular to a posture recognition method, device, system, equipment, medium, product and rehabilitation mirror. Background Art

[0002] With the development of computer technology, gesture recognition has played an important role in many fields such as motion analysis and sports, medical treatment and rehabilitation, and entertainment.

[0003] However, current posture recognition has limitations. It can only recognize in specified directions and angles. The recognition process is complicated and the recognition accuracy is not high. The acquired posture data is also more unstable. Summary of the invention

[0004] The embodiments of the present invention provide a posture recognition method, device, system, equipment, medium, product and rehabilitation mirror, which can generate a three-dimensional skeleton sequence diagram by combining two-dimensional and three-dimensional image data and a neural network model, can recognize postures in any direction and angle, and improve the accuracy and stability of posture recognition.

[0005] In a first aspect, an embodiment of the present invention provides a gesture recognition method, the method comprising:

[0006] Acquire infrared image data and depth image data of the target object, wherein the infrared image data and the depth image data are a pair of sequential image data acquired synchronously within a preset time period;

[0007] Inputting the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtaining an object structure segmentation map based on the object structure recognition result;

[0008] The object structure segmentation map is fitted with the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within a preset time length.

[0009] In a second aspect, an embodiment of the present invention provides a gesture recognition device, the device comprising:

[0010] An image acquisition module is used to acquire infrared image data and depth image data of a target object, wherein the infrared image data and the depth image data are a pair of sequential image data acquired synchronously within a preset time period;

[0011] An object structure segmentation map generation module is used to input infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtain an object structure segmentation map based on the object structure recognition result;

[0012] The three-dimensional skeleton sequence diagram generation module is used to perform data fitting between the object structure segmentation diagram and the corresponding depth image data to obtain a three-dimensional skeleton sequence diagram of the target object within a preset time length.

[0013] In a third aspect, an embodiment of the present invention provides a gesture recognition system, the system comprising:

[0014] 3D depth camera, infrared camera and computing processing platform;

[0015] Wherein, the three-dimensional depth camera is used to collect depth image data of the target object within a preset time;

[0016] An infrared camera, used to collect infrared image data of a target object within a preset time;

[0017] The computing and processing platform is used to perform gesture recognition based on the depth image data and the infrared image data according to the method provided by any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0019] one or more processors;

[0020] A memory for storing one or more programs;

[0021] When the one or more programs are executed by one or more processors, the one or more processors implement the gesture recognition method provided by any embodiment of the present invention.

[0022] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a gesture recognition method as provided in any embodiment of the present invention.

[0023] In a sixth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a gesture recognition method as provided in any embodiment of the present invention.

[0024] In a seventh aspect, an embodiment of the present invention further provides a rehabilitation mirror, characterized in that it comprises:

[0025] Image acquisition module, computing and processing module and display screen;

[0026] The image acquisition module is used to acquire depth image data and infrared image data of the target object within a preset time;

[0027] The computing and processing module is used to perform gesture recognition and generate a three-dimensional skeleton sequence graph and / or a three-dimensional skeleton graph based on the depth image data and the infrared image data according to the gesture recognition method provided by any embodiment of the present invention;

[0028] The display screen is used to display the three-dimensional skeleton sequence diagram and / or the three-dimensional skeleton diagram.

[0029] The embodiments of the above invention have the following advantages or beneficial effects:

[0030] The embodiment of the present invention obtains infrared image data and depth image data of the target object, wherein the infrared image data and the depth image data are a pair of sequence image data acquired synchronously within a preset time length; the infrared image data is input into a pre-trained object structure recognition model to obtain an object structure recognition result, and an object structure segmentation map is obtained based on the object structure recognition result; the object structure segmentation map is fitted with the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within the preset time length. The technical solution of the embodiment of the present invention solves the current problems of inaccurate posture recognition and limited recognition direction and angle. By combining two-dimensional and three-dimensional image data and a neural network model, a three-dimensional skeleton sequence map can be generated, which can recognize postures in any direction and angle, and improve the accuracy and stability of posture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of a gesture recognition method provided by an embodiment of the present invention;

[0032] Figure 2 is a flow chart of a gesture recognition method provided by an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of a joint space information storage provided by an embodiment of the present invention;

[0034] Figure 4 is a flow chart of a gesture recognition method provided by an embodiment of the present invention;

[0035] Figure 5 is a schematic diagram of target object determination provided by an embodiment of the present invention;

[0036] Figure 6 is a structural schematic diagram of a gesture recognition device provided by an embodiment of the present invention;

[0037] Figure 7 is a structural schematic diagram of a gesture recognition system provided by an embodiment of the present invention;

[0038] Figure 8 is a functional schematic diagram of a gesture recognition system provided by an embodiment of the present invention;

[0039] Fig. 9 is a structural schematic diagram of a rehabilitation mirror provided by an embodiment of the present invention;

[0040] Fig.10 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0042] Figure 1 The present invention provides a flowchart of a gesture recognition method, which can be applied to gesture recognition scenarios. The method can be performed by a gesture recognition device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0043] like Figure 1 As shown, the gesture recognition method of this embodiment includes the following steps:

[0044] S110: Acquire infrared image data and depth image data of a target object.

[0045] The infrared image data and the depth image data are a pair of sequential image data acquired synchronously within a preset time period.

[0046] The target object may be an object that needs rehabilitation training or whose body posture needs to be recognized, such as a human body or other livestock and pets.

[0047] Acquire a sequence of image data pairs of a target object acquired by the infrared camera and the depth camera synchronously within a preset time. The preset time can be set according to actual needs and can be determined by the actual acquisition time, which is not limited in this embodiment. A set of sequence image data pairs includes an infrared image and a depth image at the corresponding acquisition time.

[0048] In this embodiment, the depth camera may be composed of a three-dimensional depth sensor based on the time-of-flight ranging principle and a color camera, and has good depth image data acquisition capabilities.

[0049] S120, inputting the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtaining an object structure segmentation map based on the object structure recognition result.

[0050] The pre-trained object structure recognition model can be a neural network model based on deep learning. After being trained and optimized with a large amount of data, it has learned the characteristics of the joints of the target object and the various body parts of the target object in the object structure. When the infrared image data is input, the object structure recognition model can quickly identify the various joints of the target object, such as the elbow joint, knee joint and ankle joint. At the same time, the object structure recognition model can also perform fine segmentation on the various parts of the target object, dividing the target into different areas such as the head, neck, torso, arms, hands, legs and feet. According to the joint recognition results and the part recognition results, a two-dimensional segmentation map of the object joints and various parts of the object is obtained.

[0051] S130 , performing data fitting on the object structure segmentation map and the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within a preset time length.

[0052] The depth image data is fitted with the two-dimensional object joints and the segmentation maps of each part of the object to obtain the object joints with three-dimensional coordinates and form a skeleton, and a three-dimensional skeleton sequence diagram corresponding to each acquisition moment of the target object within a preset time length is obtained. The three-dimensional skeleton sequence diagram can reflect the body posture of the target object at each corresponding acquisition moment.

[0053] The technical solution of this embodiment is to obtain infrared image data and depth image data of the target object, wherein the infrared image data and the depth image data are a pair of sequence image data acquired synchronously within a preset time length; input the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtain an object structure segmentation map based on the object structure recognition result; perform data fitting on the object structure segmentation map and the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within the preset time length. The technical solution of the embodiment of the present invention solves the current problems of inaccurate posture recognition and limited recognition direction and angle. By combining two-dimensional and three-dimensional image data and a neural network model, a three-dimensional skeleton sequence map can be generated, which can recognize postures in any direction and angle, and improve the accuracy and stability of posture recognition.

[0054] Figure 2 This is a flowchart of a gesture recognition method provided by an embodiment of the present invention. This embodiment and the gesture recognition method in the above embodiment belong to the same inventive concept, and further describes the process of updating the three-dimensional skeleton graph. The method can be executed by a gesture recognition device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0055] like Figure 2 As shown, the gesture recognition method of this embodiment includes the following steps:

[0056] S210: Acquire infrared image data and depth image data of a target object.

[0057] The infrared image data and the depth image data are a pair of sequential image data acquired synchronously within a preset time period.

[0058] S220 , inputting the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtaining an object structure segmentation map based on the object structure recognition result.

[0059] S230 , performing data fitting on the object structure segmentation map and the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within a preset time length.

[0060] S240, obtaining the first coordinate observation value of each joint in each three-dimensional skeleton image in the three-dimensional skeleton sequence image in the camera coordinate system.

[0061] The first coordinate observation value of each joint in each 3D skeleton diagram in the camera coordinate system is read by the coordinate reading function and stored in a preset storage location. The first coordinate observation value may include a first position coordinate observation value and a first direction coordinate observation value.

[0062] The spatial position coordinates of the joints of a specific object, i.e., the first position coordinate observation value, and the joint direction, i.e., the first direction coordinate observation value, represented by the quaternion, can be obtained from the preset storage location according to the identification of the object. The joints form a skeleton according to the hierarchy and connection relationship. Taking human body tracking and skeleton recognition as an example, the skeleton includes a total of 32 joints. The joint hierarchy is distributed from the heel joint in the center of the human body to the limbs. Each connection (bone) links the parent joint to the child joint. The joint connection and spatial information format are shown in Table 1 below. Table 1 is a table of joint connection and joint space information. Among them, the first three elements x, y, z in the joint space information [x, y, z, w, x, y, z] represent the spatial position coordinates of the joint, i.e., the first position coordinate observation value, and the last four elements w, x, y, z are the joint directions, i.e., the first direction coordinate observation values, represented by normalized quaternions.

[0063] Table 1

[0064]

[0065]

[0066]

[0067] S250, transform the first coordinate observation value to obtain the second coordinate observation value of each joint in the object coordinate system.

[0068] The second coordinate observation value includes a second position coordinate observation value.

[0069] The machine vision technology used for rehabilitation training monitoring is mainly based on posture recognition technology. Machine vision technology can evaluate the patient's motor ability by analyzing the patient's motion trajectory and posture, and provide personalized sports training programs by monitoring the patient's motion state in real time; whether it is based on machine vision to evaluate the patient's motor ability or to monitor the patient's motion state, it is centered on the patient. However, the origin of the original coordinate system of a general three-dimensional depth camera is located at the camera focus. The human joint and skeleton coordinates in the original camera coordinate system are not suitable for functions such as rehabilitation training monitoring. Therefore, this embodiment converts the original camera coordinate system to the object coordinate system of the user or patient. Specifically, the coordinates of the specific joints of the object can be selected as the reference coordinate point, and the camera coordinate system with the origin of the coordinate system at the camera focus can be converted to the second coordinate observation value in the object coordinate system with the reference coordinate point as the origin.

[0070] In an optional embodiment, the first coordinate observation value is transformed to obtain the second coordinate observation value of each joint in the object coordinate system, which can be to determine the relative position relationship between the coordinate reference joint in each three-dimensional skeleton image and the origin in the object coordinate system; determine the coordinate system mapping relationship between the camera coordinate system and the object coordinate system based on the first coordinate observation value and the relative position relationship of the coordinate reference joint in the camera coordinate system; based on the coordinate system mapping relationship and the position relationship between the non-coordinate reference joint and the corresponding coordinate reference joint in each three-dimensional skeleton image, transform the first coordinate observation value of the non-coordinate reference joint to obtain the second coordinate observation value of each joint in each three-dimensional skeleton image in the object coordinate system.

[0071] The coordinate reference joint can be a joint located at the center of the object skeleton. According to the first coordinate observation value of the coordinate reference joint in the camera coordinate system and the relative position relationship with the origin in the object coordinate system, the coordinate system mapping relationship between the camera coordinate system and the object coordinate system is determined through algorithms such as matrix transformation.

[0072] Taking the target object as a human body, the coordinate reference joint can be the pelvic joint. Suppose the coordinates of any joint of the human body except the root joint, i.e., the pelvic joint, in the original camera coordinate system are t camera [x,y,z], direction is q camera [w,x,y,z]; the position coordinate of this joint in the second coordinate observation value in the human body coordinate system is t body [x,y,z], direction coordinate is q body [w,x,y,z], then the conversion method of this joint in the original camera coordinate system to the human body coordinate system is:

[0073] t body=t camera -t;

[0074] q body =q×q camera .

[0075] Among them, q×q camera Multiply the quaternion of the direction coordinates.

[0076] It is understandable that although posture recognition in the three-dimensional space coordinate system can obtain object joints with three-dimensional coordinates and form a skeleton, there are joint tracking errors caused by depth information noise, occlusion, low confidence, etc., which make the spatial information of the joints and skeleton unreasonable and inaccurate. Therefore, it is necessary to filter the joint spatial information, that is, the first coordinate observation value of the joint in the camera coordinate system.

[0077] Therefore, in an optional embodiment, before converting the first coordinate observation value, the first coordinate observation value is compared and analyzed with a predetermined sample coordinate observation value to obtain a comparison analysis result; based on the comparison analysis result, the outliers in the first coordinate observation value are determined and eliminated; wherein the sample coordinate observation value is obtained by synchronously collecting infrared image data and depth image data of the sample object in a stationary and moving state and performing coordinate reading within a preset sample coordinate collection time.

[0078] Since the random system error follows Gaussian distribution, we can experimentally obtain sample data of the spatial position information of human joint data before recognizing the posture, and fit it to the Gaussian distribution. For example, make the human body or human body model in the effective working space of the camera and keep it still. The system cyclically reads the position coordinates of any joint in the human coordinate system within a period of time and saves them to the data sequence of this joint. The human body is moved multiple times in the effective working space of the camera and the spatial position coordinates of the same joint are repeatedly read and added to the aforementioned data sequence. The samples of this data sequence generally conform to the Gaussian distribution, and the Gaussian distribution is fitted: N(μ,σ), and the mean μ and standard deviation σ of this distribution are saved.

[0079] When reading the spatial position coordinate information of the joint, first compare whether the deviation between the coordinate value and the average value μ of the Gaussian distribution exceeds 3 times the standard deviation σ, and the data points exceeding 3 times the standard deviation are filtered out as outliers.

[0080] S260, performing coordinate prediction based on the second coordinate observation value to obtain a second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton image.

[0081] like Figure 3As shown, since the model calculates the predicted value of the second position coordinate of the corresponding joint in the adjacent next three-dimensional skeleton image, the joint space information of the current three-dimensional skeleton image is required, so all joint space information of the human body is saved starting from the initial time t0; the present application uses a queue data structure to respectively save the current three-dimensional skeleton image corresponding to time t-1 and the adjacent next three-dimensional skeleton image corresponding to time t.

[0082] Based on the second coordinate observation value, the Kalman filter algorithm is used to perform coordinate prediction to obtain the second coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton image. The Kalman filter algorithm can calculate the second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton image, that is, the three-dimensional skeleton image at time t, based on the second coordinate observation value of the joint in the three-dimensional skeleton image at time (t-1).

[0083] In an optional embodiment, coordinate prediction is performed based on the second coordinate observation value to obtain the second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton graph. The joint connection vector between each joint and the parent joint of the corresponding joint can be determined based on the second coordinate observation value of each joint and the corresponding parent joint; the second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton graph is determined based on the second coordinate observation value of the joint connection vector and the parent joint associated with the joint connection vector in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which the joint is located.

[0084] The skeleton of human posture recognition includes several joints. The joint hierarchy is distributed from the heel joint in the center of the human body to the limbs. Each connection links the parent joint to the child joint to form a skeleton. The observed value of the spatial position of each joint can be read through the reading function, or the predicted value of the spatial position of each joint can be calculated through the spatial position and direction of the parent joint of each joint, and the connection vector from the parent joint position to the current joint position. For example, the observed value of the position of the left knee joint (KNEE_LEFT) can be directly read, and it can be calculated through the position and direction of its parent joint, the left hip joint (HIP_LEFT), and the vector from the left hip joint (HIP_LEFT) to the knee joint (KNEE_LEFT). Suppose at time t, the position of the current joint is The position of the parent joint of the current joint is The parent joint orientation expressed as a normalized quaternion is The connection vector from the parent joint position to the current joint position is Where t-1 represents the previous moment of moment t.

[0085] In an optional embodiment, the second coordinate observation value includes a second direction coordinate observation value, and the second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton graph is determined according to the second coordinate observation value of the joint connection vector and the parent joint associated with the joint connection vector in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which it is located. It can be to calculate the rotation matrix corresponding to the second direction coordinate observation value of the parent joint of the joint in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which it is located; determine the product of the rotation matrix and the joint connection vector; add the product and the second position coordinate observation value of the parent joint of the joint in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which it is located, and obtain the second position coordinate prediction value of the joint in the next adjacent three-dimensional skeleton graph.

[0086] The second direction coordinate observation value represented by the quaternion of the parent joint in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which it is located. For example, the quaternion q is converted into a rotation matrix R by the Rodrigues formula, that is, the parent joint direction represented by the quaternion The corresponding rotation matrix is

[0087] Defining the state transition model And calculate the predicted value of the second coordinate of the current joint Then we have:

[0088]

[0089] S270, coordinate fusion is performed according to the second position coordinate observation value and the second position coordinate prediction value of each joint in each three-dimensional skeleton image, and the corresponding three-dimensional skeleton image is updated based on the coordinate fusion result.

[0090] According to the second position coordinate observation value and the second position coordinate prediction value of each joint in each three-dimensional skeleton image, coordinate fusion is performed through the Kalman filter algorithm, and the corresponding three-dimensional skeleton image is updated based on the coordinate fusion result.

[0091] For example, the system observation equation and state transfer equation of the Kalman filter algorithm are established. Assume that the human joint position coordinate reading function is H(t), the observation noise is v(t), and the joint space position observation vector corresponding to the second position coordinate observation value is x(t); the observation equation is defined as:

[0092] x(t)=H(t)+v(t).

[0093] State transition model Assume that the process noise is w(t), and the joint space position state vector corresponding to the second position coordinate prediction value is y(t); the state transfer equation is defined as:

[0094] y(t)=F(x(t-1))+w(t).

[0095] According to the observation equation and state transfer equation of the above system, the Kalman filter algorithm is used to perform iterative optimization of prediction and update, and the system state is optimally estimated to obtain the coordinate fusion result. The Kalman filter algorithm can be an unscented Kalman filter algorithm and an extended Kalman filter algorithm, etc. This embodiment does not limit the type of the Kalman filter algorithm.

[0096] The technical solution of this embodiment is to obtain infrared image data and depth image data of the target object, wherein the infrared image data and the depth image data are a pair of sequence image data obtained by synchronous acquisition within a preset time length; input the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtain an object structure segmentation map based on the object structure recognition result; perform data fitting on the object structure segmentation map and the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within a preset time length; obtain the first coordinate observation value of each joint in each three-dimensional skeleton map in the three-dimensional skeleton sequence map in the camera coordinate system; transform the first coordinate observation value to obtain the second coordinate observation value of each joint in the object coordinate system, wherein the second coordinate observation value includes a second position coordinate observation value; perform coordinate prediction based on the second coordinate observation value to obtain a second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton map; perform coordinate fusion according to the second position coordinate observation value and the second position coordinate prediction value of each joint in each three-dimensional skeleton map, and update the corresponding three-dimensional skeleton map based on the coordinate fusion result. The technical solution of the embodiment of the present invention solves the current problems of inaccurate posture recognition and limited recognition direction and angle. By combining two-dimensional and three-dimensional image data and a neural network model, a three-dimensional skeleton sequence diagram can be generated, which can recognize postures in any direction and angle, thereby improving the accuracy and stability of posture recognition. By converting the original camera coordinate system to the object coordinate system, a system observation equation and a state transfer equation for human posture recognition are established. By fusing the coordinate observation values ​​and the predicted values, the three-dimensional skeleton diagram is updated, thereby improving the accuracy, stability, and real-time performance of the posture recognition technology, thereby improving the usability of the human posture recognition technology in auxiliary rehabilitation training.

[0097] Figure 4 This is a flowchart of a gesture recognition method provided by an embodiment of the present invention. This embodiment and the gesture recognition method in the above embodiment belong to the same inventive concept, and further describes the process of determining the target object. The method can be executed by a gesture recognition device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0098] like Figure 4As shown, the gesture recognition method of this embodiment includes the following steps:

[0099] S310: Identify the body movements of at least one object within the field of view of image data acquisition.

[0100] During the use of the system, there may be multiple objects in the camera's field of view. In the multi-person state, the system cannot identify a specific single user. Therefore, it is necessary to lock the specific user among multiple people by having the user perform specific body movements such as raising hands or legs, that is, the system obtains the body identification number (body_ID) of the specific user.

[0101] S320: Determine a target object from at least one object according to the movement posture, movement amplitude, and movement duration of the limb movement.

[0102] like Figure 5 As shown, an example is given in which the user raises his hand through an action gesture, the action amplitude is that the hand is raised beyond the preset angle, and the action lasts for 2 seconds. The locking method is that after entering the hand-raising recognition, the system first initializes the body identification record, that is, the body_ID record and the current time, and then enters the loop. First, it is determined whether there is a body_ID record. If there is a record, the human skeleton corresponding to the body_ID is calculated. If not, the human skeleton and body_ID are obtained, and then the arm lifting angle is calculated. If the arm lifting angle is greater than the preset angle, it is further determined whether the body_ID and time have been recorded. If the body_ID has been recorded, the duration of the hand-raising is calculated. If the body_ID has not been recorded, the current body_ID and time are recorded. On the contrary, if the arm raising angle does not exceed the preset angle such as 100 degrees, the current body_ID record and time are cleared. The system displays the current image to facilitate the user to adjust the posture. Finally, it is determined whether the hand-raising time exceeds 2 seconds. If it exceeds 2 seconds, the client is told that the body has been locked, and the locked body body_ID is returned to exit the loop. If the hand-raising time does not exceed two seconds, it means that the hand-raising time is not enough or the arm is not raised at an enough angle, and the hand-raising recognition will continue.

[0103] S330: Acquire infrared image data and depth image data of the target object.

[0104] The infrared image data and the depth image data are a pair of sequential image data acquired synchronously within a preset time period.

[0105] S340: Input the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtain an object structure segmentation map based on the object structure recognition result.

[0106] S350 , performing data fitting on the object structure segmentation map and the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within a preset time length.

[0107] S360, obtaining the first coordinate observation value of each joint in each three-dimensional skeleton image in the three-dimensional skeleton sequence image in the camera coordinate system.

[0108] S370. Transform the first coordinate observation value to obtain a second coordinate observation value of each joint in the object coordinate system.

[0109] The second coordinate observation value includes a second position coordinate observation value.

[0110] S380, performing coordinate prediction based on the second coordinate observation value to obtain a second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton image.

[0111] S390, coordinate fusion is performed according to the second position coordinate observation value and the second position coordinate prediction value of each joint in each three-dimensional skeleton image, and the corresponding three-dimensional skeleton image is updated based on the coordinate fusion result.

[0112] The technical solution of this embodiment is to identify the body movements of at least one object within the field of view of image data acquisition; determine the target object in at least one object according to the posture, amplitude and duration of the body movements; obtain infrared image data and depth image data of the target object, wherein the infrared image data and the depth image data are a pair of sequence image data obtained by synchronous acquisition within a preset time length; input the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtain an object structure segmentation map based on the object structure recognition result; perform data fitting on the object structure segmentation map and the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within the preset time length; obtain the first coordinate observation value of each joint in each three-dimensional skeleton map in the three-dimensional skeleton sequence map in the camera coordinate system; transform the first coordinate observation value to obtain the second coordinate observation value of each joint in the object coordinate system, wherein the second coordinate observation value includes a second position coordinate observation value; perform coordinate prediction based on the second coordinate observation value to obtain the second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton map; perform coordinate fusion based on the second position coordinate observation value and the second position coordinate prediction value of each joint in each three-dimensional skeleton map, and update the corresponding three-dimensional skeleton map based on the coordinate fusion result. The technical solution of the embodiment of the present invention solves the current problems of inaccurate posture recognition and limited recognition direction and angle. By combining two-dimensional and three-dimensional image data and a neural network model, a three-dimensional skeleton sequence diagram can be generated, which can recognize postures in any direction and angle, improve the accuracy and stability of posture recognition, and determine the target object in a multi-person scene by recognizing limb movements, thereby improving the usability of posture recognition in rehabilitation training.

[0113] Figure 6 The present invention provides a schematic diagram of a gesture recognition device according to an embodiment of the present invention. The present embodiment is applicable to gesture recognition scenarios. The gesture recognition device can be implemented by software and / or hardware and integrated into a computer terminal device with application development function.

[0114] like Figure 6 As shown, the posture recognition device includes: an image acquisition module 410 , an object structure segmentation graph generation module 420 and a three-dimensional skeleton sequence graph generation module 430 .

[0115] Among them, the image acquisition module 410 is used to acquire infrared image data and depth image data of the target object, wherein the infrared image data and the depth image data are a sequence image data pair acquired synchronously within a preset time length; the object structure segmentation map generation module 420 is used to input the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtain an object structure segmentation map based on the object structure recognition result; the three-dimensional skeleton sequence map generation module 430 is used to perform data fitting on the object structure segmentation map and the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within a preset time length.

[0116] The technical solution of this embodiment is to obtain infrared image data and depth image data of the target object, wherein the infrared image data and the depth image data are a pair of sequence image data acquired synchronously within a preset time length; input the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtain an object structure segmentation map based on the object structure recognition result; perform data fitting on the object structure segmentation map and the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within the preset time length. The technical solution of the embodiment of the present invention solves the current problems of inaccurate posture recognition and limited recognition direction and angle. By combining two-dimensional and three-dimensional image data and a neural network model, a three-dimensional skeleton sequence map can be generated, which can recognize postures in any direction and angle, and improve the accuracy and stability of posture recognition.

[0117] In an optional embodiment, the device further comprises:

[0118] A three-dimensional skeleton graph updating module is used to obtain the first coordinate observation value of each joint in each three-dimensional skeleton graph in the three-dimensional skeleton sequence graph in the camera coordinate system; transform the first coordinate observation value to obtain the second coordinate observation value of each joint in the object coordinate system, wherein the second coordinate observation value includes a second position coordinate observation value; perform coordinate prediction based on the second coordinate observation value to obtain the second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton graph; perform coordinate fusion according to the second position coordinate observation value and the second position coordinate prediction value of each joint in each three-dimensional skeleton graph, and update the corresponding three-dimensional skeleton graph based on the coordinate fusion result.

[0119] In an optional implementation, the 3D skeleton graph updating module is specifically used for:

[0120] Determine the relative position relationship between the coordinate reference joint in each three-dimensional skeleton image and the origin in the object coordinate system; determine the coordinate system mapping relationship between the camera coordinate system and the object coordinate system based on the first coordinate observation value and relative position relationship of the coordinate reference joint in the camera coordinate system; based on the coordinate system mapping relationship and the position relationship between the non-coordinate reference joint in each three-dimensional skeleton image and the corresponding coordinate reference joint, transform the first coordinate observation value of the non-coordinate reference joint to obtain the second coordinate observation value of each joint in each three-dimensional skeleton image in the object coordinate system.

[0121] In an optional implementation, the 3D skeleton graph updating module is further used for:

[0122] Based on the second coordinate observation value of each joint and the corresponding parent joint, the joint connection vector between each joint and the corresponding parent joint is determined; based on the second coordinate observation value of the joint connection vector and the parent joint associated with the joint connection vector in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which it is located, the second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton graph is determined.

[0123] In an optional implementation, the second coordinate observation value includes a second direction coordinate observation value, and the 3D skeleton graph updating module is further used for:

[0124] Calculate the rotation matrix corresponding to the second direction coordinate observation value of the parent joint of the joint in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which it is located; determine the product of the rotation matrix and the joint connection vector; add the product to the second position coordinate observation value of the parent joint of the joint in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which it is located, and obtain the second position coordinate prediction value of the joint in the next adjacent three-dimensional skeleton graph.

[0125] In an optional embodiment, the device further comprises:

[0126] The target object determination module is used to identify the body movements of at least one object within the field of view of image data acquisition; and determine the target object in at least one object based on the body movement posture, movement amplitude and movement duration.

[0127] In an optional embodiment, the device further comprises:

[0128] The outlier elimination module is used to compare and analyze the first coordinate observation value with the predetermined sample coordinate observation value to obtain a comparison analysis result; determine the outlier in the first coordinate observation value based on the comparison analysis result, and eliminate the outlier; wherein the sample coordinate observation value is obtained by synchronously collecting infrared image data and depth image data of the sample object in a static and moving state within a preset sample coordinate collection time and performing coordinate reading.

[0129] The gesture recognition device provided in the embodiment of the present invention can execute the gesture recognition method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0130] Figure 7 A schematic diagram of the structure of a gesture recognition system provided by an embodiment of the present invention.

[0131] like Figure 7 As shown, the gesture recognition system includes:

[0132] 3D depth camera 510, infrared camera 520 and computing processing platform 530;

[0133] Wherein, the three-dimensional depth camera 510 is used to collect depth image data of the target object within a preset time;

[0134] Infrared camera 520, used to collect infrared image data of the target object within a preset time;

[0135] The computing and processing platform 530 is used to perform gesture recognition based on the depth image data and the infrared image data according to the method provided by any embodiment of the present invention.

[0136] The technical solution of this embodiment uses a three-dimensional depth camera to collect depth image data of the target object within a preset time; an infrared camera is used to collect infrared image data of the target object within a preset time; and a computing processing platform is used to perform posture recognition based on the depth image data and the infrared image data according to the method provided in any embodiment of the present invention. The technical solution of the embodiment of the present invention solves the current problems of inaccurate posture recognition and limited recognition direction and angle. By combining two-dimensional and three-dimensional image data and a neural network model, a three-dimensional skeleton sequence diagram can be generated, which can recognize postures in any direction and angle, and improve the accuracy and stability of posture recognition.

[0137] To ensure the real-time and availability of the gesture recognition system, this embodiment deploys the gesture recognition method to a computing and processing platform, which may be an edge computing platform. This embodiment preferably runs the inference of the pre-trained object structure recognition model on a GPU (Graphics Processing Unit) for parallel computing and inference, thereby improving the speed and real-time performance of the system computing and processing. Taking the configuration of the edge computing platform shown in Table 2 as an example, the system computing and processing speed represented by the frame rate (FPS, Frame per Second) is not less than 25FPS, that is, the system computing and processing bandwidth is 25Hz.

[0138] Table 2

[0139] Configuration items describe operating system Windows 11 64 bit System Memory 16G harddisk 512G

[0140] In a preferred embodiment of the present invention, Figure 8 As shown, a three-dimensional depth camera composed of a three-dimensional depth sensor based on the time of flight (ToF) principle, a color camera and an infrared camera is used as a sensor, and an edge computing platform with a GPU (Graphics Processing Unit) is used as a computing and processing platform; the infrared image data and depth map data of the camera are first read through the three-dimensional depth camera software development interface (API, Application Programming Interface), and the infrared image data is used to obtain two-dimensional human joints and segmentation maps of various parts of the human body through a pre-trained human body part recognition model, and the depth map data is used to fit the two-dimensional human joints, so as to obtain human joints with three-dimensional coordinates and form a skeleton.

[0141] Fig. 9 A schematic structural diagram of a rehabilitation mirror provided in an embodiment of the present invention.

[0142] like Fig. 9 As shown, the rehabilitation mirror includes:

[0143] Image acquisition module 610 , calculation processing module 620 and display screen 630 .

[0144] Among them, the image acquisition module 610 is used to collect depth image data and infrared image data of the target object within a preset time; the computing and processing module 620 is used to perform posture recognition and generate a three-dimensional skeleton sequence diagram and / or a three-dimensional skeleton diagram based on the depth image data and the infrared image data according to the posture recognition method provided in any embodiment of the present invention; the display screen 630 is used to display the three-dimensional skeleton sequence diagram and / or the three-dimensional skeleton diagram.

[0145] The patient can observe the three-dimensional skeleton sequence diagram and / or the three-dimensional skeleton diagram on the display screen 630 to understand whether his or her movements are correct and where improvements are needed, thereby adjusting the movements in a timely manner and improving the effect of rehabilitation training.

[0146] The technical solution of this embodiment is to use an image acquisition module to collect depth image data and infrared image data of the target object within a preset time; the calculation processing module is used to perform posture recognition and generate a three-dimensional skeleton sequence diagram and / or a three-dimensional skeleton diagram based on the depth image data and infrared image data according to the posture recognition method provided by any embodiment of the present invention; the display screen is used to display the three-dimensional skeleton sequence diagram and / or the three-dimensional skeleton diagram. The technical solution of the embodiment of the present invention solves the problem that the posture recognition currently used in the rehabilitation mirror is inaccurate, and the recognition direction and angle are limited, which makes it more difficult for patients who already have posture problems to train with the rehabilitation mirror and has a poor user experience. By combining two-dimensional and three-dimensional image data and a neural network model, a three-dimensional skeleton sequence diagram can be generated, which can recognize postures in any direction and angle, reduce the movement requirements for patients during training, and improve the user experience of patient users.

[0147] After sports injuries are repaired, effective rehabilitation training is needed to ensure complete recovery of the injury. With the development of computer technology, rehabilitation mirrors are constantly being used in medical rehabilitation training, especially computer vision technology plays an important role in monitoring rehabilitation training. Computer vision technology used for rehabilitation training monitoring first needs to solve the problem of human posture recognition.

[0148] In this embodiment, the rehabilitation mirror can be a medical rehabilitation auxiliary device with posture recognition technology, which can provide efficient and accurate support for the patient's rehabilitation training.

[0149] Optionally, the above-mentioned rehabilitation mirror further includes:

[0150] The rehabilitation program customization module is used to determine the target rehabilitation training program based on the rehabilitation demand data of the target object.

[0151] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Fig.10 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Fig.10 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as an intelligent controller and server, a mobile phone and other terminal devices.

[0152] like Fig.10 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0153] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0154] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0155] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig.10 not shown, usually called a "hard drive"). Although Fig.10 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0156] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0157] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Fig.10 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RFID systems, tape drives, and data backup storage systems.

[0158] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing the gesture recognition method provided in the embodiment of the present invention, which includes:

[0159] Acquire infrared image data and depth image data of the target object, wherein the infrared image data and the depth image data are a pair of sequential image data acquired synchronously within a preset time period;

[0160] Inputting the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtaining an object structure segmentation map based on the object structure recognition result;

[0161] The object structure segmentation map is fitted with the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within a preset time length.

[0162] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a gesture recognition method as provided in any embodiment of the present invention is implemented. The method includes:

[0163] Acquire infrared image data and depth image data of the target object, wherein the infrared image data and the depth image data are a pair of sequential image data acquired synchronously within a preset time period;

[0164] Inputting the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtaining an object structure segmentation map based on the object structure recognition result;

[0165] The object structure segmentation map is fitted with the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within a preset time length.

[0166] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0167] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0168] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0169] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, Python, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0170] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the gesture recognition method provided in any embodiment of the present application.

[0171] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, Python, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0172] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0173] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A gesture recognition method, characterized in that: include: Acquire infrared image data and depth image data of a target object, wherein the infrared image data and the depth image data are a pair of sequential image data acquired synchronously within a preset time period; Inputting the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtaining an object structure segmentation map based on the object structure recognition result; The object structure segmentation map is data-fitted with the corresponding depth image data to obtain a three-dimensional skeleton sequence map of the target object within the preset time length.

2. The method according to claim 1, characterized in that The method also includes: Obtaining a first coordinate observation value of each joint in each three-dimensional skeleton image in the three-dimensional skeleton sequence image in a camera coordinate system; Converting the first coordinate observation value to obtain a second coordinate observation value of each joint in the object coordinate system, wherein the second coordinate observation value includes a second position coordinate observation value; Performing coordinate prediction based on the second coordinate observation value to obtain a second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton image; Coordinate fusion is performed according to the second position coordinate observation value and the second position coordinate prediction value of each joint in each of the three-dimensional skeleton images, and the corresponding three-dimensional skeleton image is updated based on the coordinate fusion result.

3. The method according to claim 2, characterized in that The converting the first coordinate observation value to obtain a second coordinate observation value of each joint in the object coordinate system includes: Determine the relative position relationship between the coordinate reference joint in each three-dimensional skeleton image and the origin in the object coordinate system; Determine a coordinate system mapping relationship between the camera coordinate system and the object coordinate system according to a first coordinate observation value of the coordinate reference joint in the camera coordinate system and the relative position relationship; According to the coordinate system mapping relationship and the positional relationship between the non-coordinate reference joint and the corresponding coordinate reference joint in each three-dimensional skeleton image, the first coordinate observation value of the non-coordinate reference joint is transformed to obtain the second coordinate observation value of each joint in each three-dimensional skeleton image in the object coordinate system.

4. The method according to claim 2, characterized in that: The step of performing coordinate prediction based on the second coordinate observation value to obtain a second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton image includes: Determine a joint connection vector between each joint and the parent joint of the corresponding joint according to the second coordinate observation value of each joint and the corresponding parent joint; According to the joint connection vector and the second coordinate observation value of the parent joint associated with the joint connection vector in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph where the joint connection vector is located, the second position coordinate prediction value of the corresponding joint in the next adjacent three-dimensional skeleton graph is determined.

5. The method according to claim 4, characterized in that The second coordinate observation value includes a second direction coordinate observation value, and the second coordinate observation value of the joint connection vector and the parent joint associated with the joint connection vector in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which the joint connection vector is located is determined to predict the second position coordinate of the corresponding joint in the next adjacent three-dimensional skeleton graph, including: Calculate the rotation matrix corresponding to the second direction coordinate observation value of the parent joint of the joint in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph where the parent joint is located; Determining the product of the rotation matrix and the joint connection vector; The product is added to the second position coordinate observation value of the parent joint of the joint in the next three-dimensional skeleton graph adjacent to the three-dimensional skeleton graph in which the joint is located, to obtain the second position coordinate prediction value of the joint in the next adjacent three-dimensional skeleton graph.

6. The method according to claim 1, characterized in that Before acquiring infrared image data and depth image data of the target object, it also includes: identifying a body movement of at least one subject within a field of view of image data acquisition; A target object is determined in the at least one object according to the movement posture, movement amplitude and movement duration of the limb movement.

7. The method according to claim 2, characterized in that Before converting the first coordinate observation value, the method further includes: Comparing and analyzing the first coordinate observation value with a predetermined sample coordinate observation value to obtain a comparative analysis result; Determine outliers in the first coordinate observation values ​​according to the comparative analysis results, and remove the outliers; The sample coordinate observation values ​​are obtained by synchronously collecting infrared image data and depth image data of the sample object in a static and moving state and reading the coordinates within a preset sample coordinate collection time.

8. A gesture recognition device, characterized in that: include: An image acquisition module, used to acquire infrared image data and depth image data of a target object, wherein the infrared image data and the depth image data are a pair of sequential image data acquired synchronously within a preset time period; An object structure segmentation map generation module is used to input the infrared image data into a pre-trained object structure recognition model to obtain an object structure recognition result, and obtain an object structure segmentation map based on the object structure recognition result; The three-dimensional skeleton sequence diagram generation module is used to perform data fitting on the object structure segmentation diagram and the corresponding depth image data to obtain a three-dimensional skeleton sequence diagram of the target object within the preset time length.

9. A gesture recognition system, characterized in that: include: 3D depth camera, infrared camera and computing processing platform; Wherein, the three-dimensional depth camera is used to collect depth image data of the target object within a preset time; The infrared camera is used to collect infrared image data of the target object within a preset time; The computing and processing platform is used to perform gesture recognition based on the depth image data and the infrared image data according to the method described in any one of claims 1-7.

10. A computer device, characterized in that: The computer device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the gesture recognition method as described in any one of claims 1 to 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the gesture recognition method as described in any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the gesture recognition method as described in any one of claims 1 to 7.

13. A rehabilitation mirror, characterized in that: include: Image acquisition module, computing and processing module and display screen; Wherein, the image acquisition module is used to acquire depth image data and infrared image data of the target object within a preset time; The computing and processing module is used to perform posture recognition and generate a three-dimensional skeleton sequence graph and / or a three-dimensional skeleton graph based on the depth image data and the infrared image data according to the method described in any one of claims 1 to 7; The display screen is used to display the three-dimensional skeleton sequence diagram and / or the three-dimensional skeleton diagram.

Images