Pose Detection Method, Device, Electronic Device and Computer Readable Storage Medium

Abstract

An embodiment of the present application provides a posture detection method, device, electronic device, and computer-readable storage medium, which relate to the field of computer processing. The method includes: obtaining an image sequence corresponding to a target object and motion data of a target part of the target object, and determining first three-dimensional coordinate information of each key point of the target part in any image in the image sequence. According to the first three-dimensional coordinate information of each key point, a first unit vector representing the direction of the target part is determined. Based on the motion data of the target part, a second unit vector representing the direction of the target part is determined. According to the similarity between the first unit vector and the second unit vector, and the first three-dimensional coordinate information of each key point, the posture of the target part is determined. By combining the first three-dimensional coordinate information of the target part with the motion data of the target part, the present application can accurately determine the posture of the target part and improve user perception.

Description

Technical Field

[0001] The present application relates to the field of computer processing. Specifically, the present application relates to a method, device, electronic device, and computer-readable storage medium for gesture detection. Background Art

[0002] With the development of deep learning, the accuracy of human pose estimation is getting higher and higher, and more applications have emerged. For example, some terminal devices support somatosensory games, AI (Artificial Intelligence) fitness, etc., and some live broadcast software provides the driving of 3D (3-dimension) virtual characters.

[0003] In the related art, deep learning algorithms based on monocular vision can adapt to most scenarios. However, in cases where certain parts of the organism are occluded or the image is blurred due to the rapid movement of the limbs, the accuracy of the human pose obtained by the deep learning algorithm is poor, and there may even be obvious jitter. Although the use of inertial sensors can solve the problem of poor human pose accuracy caused by the occlusion of certain parts of the organism, the measured values of the inertial sensors will have errors, and when accumulating the measured values of the inertial sensors to calculate the displacement of the organism, obvious offsets may occur. It can be seen that the above methods cannot detect the relatively accurate pose of the organism. Summary of the Invention

[0004] Embodiments of the present application provide a method, device, electronic device, and computer-readable storage medium for gesture detection, which can accurately determine the pose of the target part. The specific technical solutions are as follows:

[0005] According to one aspect of the embodiments of the present application, a method for gesture detection is provided, and the method includes:

[0006] Obtain an image sequence corresponding to a target object and motion data of a target part of the target object;

[0007] Determine the first three-dimensional coordinate information of each key point of the target part in any image in the image sequence;

[0008] According to the first three-dimensional coordinate information of each key point, determine a first unit vector representing the direction of the target part;

[0009] Based on the motion data of the target part, determine a second unit vector representing the direction of the target part;

[0010] According to the similarity between the first unit vector and the second unit vector, and the first three-dimensional coordinate information of each key point, determine the pose of the target part.

[0011] According to another aspect of the embodiments of the present application, there is provided a posture detection device, which includes:

[0012] An image and motion data acquisition module, configured to acquire an image sequence corresponding to a target object and motion data of a target part of the target object;

[0013] A three-dimensional coordinate information determination module, configured to determine first three-dimensional coordinate information of each key point of the target part in any image in the image sequence;

[0014] A first unit vector determination module, configured to determine a first unit vector representing the direction of the target part according to the first three-dimensional coordinate information of each key point;

[0015] A second unit vector determination module, configured to determine a second unit vector representing the direction of the target part based on the motion data of the target part;

[0016] A posture determination module, configured to determine the posture of the target part according to the similarity between the first unit vector and the second unit vector, and the first three-dimensional coordinate information of each key point.

[0017] According to still another aspect of the embodiments of the present application, there is provided an electronic device, which includes a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the steps of the above method.

[0018] According to still another aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0019] The beneficial effects brought by the technical solution provided by the embodiments of the present application are:

[0020] Based on the pose detection method of the target object provided by this application, when detecting the pose of the target object, the first three-dimensional coordinate information of each key point of the target part in any image in the image sequence is determined. By combining the first three-dimensional coordinate information of the target part with the motion data of the target part, the similarity between the first unit vector determined based on the first three-dimensional coordinate information of the target part and the second unit vector determined based on the motion data of the target part is considered. Considering that the real-time nature of the direction of the target part represented by the second unit vector is stronger, based on this similarity, it can be determined whether the pose information of the target part cannot be accurately obtained based on the processing of the image due to reasons such as the target part being occluded in the acquired image or the rapid movement of the target object. At the same time, considering that the position information of each key point obtained by processing the image has better stability, based on the result of whether the pose information of the target part can be accurately obtained based on the processing of the image and the first three-dimensional coordinate information of each key point, the pose of the target part can be accurately determined, improving the user experience and avoiding affecting the user perception in cases where the target part is occluded or moves rapidly. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of this application, the following will briefly introduce the drawings required for the description of the embodiments of this application.

[0022] Figure 1 The flowchart of a pose detection method provided by an embodiment of this application is shown;

[0023] Figure 2 The schematic diagram of an application scenario applicable to an embodiment of this application is shown;

[0024] Figure 3 The schematic diagram of another application scenario applicable to an embodiment of this application is shown;

[0025] Figure 4 The schematic structural diagram of a pose detection device provided by an embodiment of this application;

[0026] Figure 5 The schematic structural diagram of an electronic device applicable to an embodiment of this application is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The embodiments of this application will be described below with reference to the drawings in this application. It should be understood that the embodiments described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application and do not constitute limitations on the technical solutions of the embodiments of this application.

[0028] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the terms "comprising" and "including" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude being implemented as other features, information, data, steps, operations, elements, components and / or their combinations supported by the technical field of the present application, etc. It should be understood that when we say an element is "connected" or "coupled" to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or implemented as "B", or implemented as "A and B".

[0029] To make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0030] Based on the foregoing description, in the related art, a relatively accurate human posture cannot be detected. And in some related arts, multiple cameras are required to obtain depth information, and the key points of visual detection and the wearing positions of the inertial measurement unit (IMU) must strictly correspond to obtain the exact position information of the IMU. The placement requirements for the cameras in these methods limit the usage scenarios. And in some related arts, it is necessary to preset the internal parameters and external parameters for calibration, which is relatively complex, resulting in limited usage scenarios and being inapplicable to scenarios such as digital games and fitness applications.

[0031] Based on this, in view of at least one of the above technical problems or areas for improvement existing in the related art, the present application proposes a posture detection method, device, electronic device and computer-readable storage medium. This solution determines the first three-dimensional coordinate information of each key point of the target part of the target object and the first unit vector representing the direction of the target part based on the image corresponding to the target object. Based on the motion data of the target part, a second unit vector representing the direction of the target part is determined. Thus, according to the first unit vector, the second unit vector and the first three-dimensional coordinate information of each key point, the image detection result and the obtained motion data are combined to accurately determine the posture of the target part and improve the user experience.

[0032] The posture detection method can be implemented by a posture detection device, which can be a terminal or a server. Among them, the terminal can be any electronic device, such as a fitness terminal or a game terminal, etc. The server can be a local server, a cloud server, or a server cluster composed of at least one of the local server and the cloud server. The embodiments of the present application do not limit this.

[0033] The posture detection method can be applied to different application scenarios. For example, the posture detection method can be applied during fitness or a motion-sensing game. By accurately collecting the user's posture and displaying the user's posture on the corresponding display of the fitness terminal, the user can adjust the motion state in real time according to the displayed posture on the display, so as to better improve the fitness effect or game perception and enhance the user experience. For another example, the posture detection method can also be applied during game production. By collecting the postures of the action personnel through this posture method, the art designers can better control the postures of the game characters in the game scene according to the collected postures of the action personnel, thereby making the action performance of the game characters in the game scene more realistic and enhancing the game experience of the game players during the game process.

[0034] Next, through the description of several exemplary embodiments, the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application will be described. It should be noted that the following embodiments can refer to, draw on, or combine with each other. For the same terms, similar features, and similar implementation steps in different embodiments, they will not be described repeatedly.

[0035] Figure 1 The flowchart of a posture detection method provided by the embodiments of the present application is shown. As Figure 1 shown, this method can be applied to any electronic device. Specifically, this method includes step S110 to step S150.

[0036] Step S110: Obtain an image sequence corresponding to the target object and motion data of the target part of the target object.

[0037] Step S120: Determine the first three-dimensional coordinate information of each key point of the target part in any image in the image sequence.

[0038] Step S130: Determine a first unit vector representing the direction of the target part according to the first three-dimensional coordinate information of each key point.

[0039] Step S140: Based on the motion data of the target part, determine a second unit vector representing the direction of the target part.

[0040] Step S150: Determine the posture of the target part according to the similarity between the first unit vector and the second unit vector, and the first three-dimensional coordinate information of each key point.

[0041] Based on the posture detection method provided in the embodiments of the present application, the first unit vector is obtained by processing an image including the target object, and the second unit vector is directly obtained by processing the acquired motion data of the target part. Considering that the direction of the target part represented by the second unit vector is more real-time, by determining the similarity between the first unit vector and the second unit vector, it can be determined whether the posture of the target part can be accurately obtained based on the processing of the acquired image. At the same time, considering that the position information of each key point obtained by processing the image is more stable, based on the result of whether the posture of the target part can be accurately obtained based on the processing of the acquired image, and the first three-dimensional coordinate information of each key point, the posture of the target part can be accurately determined, improving the user experience and avoiding the situation where the posture of the target part cannot be accurately obtained based on the processing of the acquired image due to occlusion or rapid movement of the target part, affecting the user perception.

[0042] Optionally, the target object can be any movable object. For example, the target object can be a person, an animal, etc. In the embodiments of the present application, a person can be used as the target object to illustrate this posture detection method. Taking the target object being a person as an example, the target part of the target object can be the hand, arm, leg, foot, etc. of the target object. Of course, the target part can also be a part of the arm, such as the upper arm or the lower arm, or the entire arm. Among them, when the target part is the entire arm, the target part can include two sub-parts, namely the upper arm and the lower arm.

[0043] In the embodiments of the present application, any image acquisition device can be used to acquire images of the target object to obtain an image sequence corresponding to the target object. The embodiments of the present application do not limit this. Among them, the image acquisition device can include but is not limited to cameras, video cameras, cameras, or other devices with image acquisition functions (such as mobile phones, tablet computers, etc.). Among them, the image acquisition device can be a monocular image acquisition device. Optionally, the image acquisition device can be a monocular RGB camera (color camera, RGB means red - green - blue).

[0044] Regarding the acquisition method of the motion data of the target part, the embodiments of the present application do not limit this. Optionally, an inertial sensor, such as an IMU bracelet, can be worn on the target part of the target object, and the data measured by the IMU bracelet worn on the target part can be used as the motion data of the target part.

[0045] The embodiments of the present application do not limit the specific performance of the inertial sensor, which can be determined according to the actual situation. In order to obtain more accurate data corresponding to the target part, the worn inertial sensor can be set as a nine-axis IMU, and the data of the three directions X, Y, and Z corresponding to the magnetometer in the IMU, the data of the three directions X, Y, and Z corresponding to the accelerometer, and the data of the three directions X, Y, and Z corresponding to the gyroscope, a total of nine degrees of freedom of data, are used as the motion data of the target part. Among them, the data of the accelerometer is used to indicate acceleration, the data of the gyroscope is used to indicate angular velocity, and the data of the magnetometer is used to indicate direction.

[0046] Before obtaining the motion data of the target part, the coordinate system corresponding to the inertial sensor can be calibrated based on the position information of the image acquisition device, so that the coordinate system corresponding to the inertial sensor is aligned with the coordinate system corresponding to the image acquisition device.

[0047] Figure 2 A schematic diagram of an application scenario applicable to the embodiments of the present application is shown. As Figure 2 shown, the image acquisition device can be placed vertically, and the vertical plane where the image acquisition device is located is determined as the X, Y plane. Control the target object to stand facing the image acquisition device and pose in a T shape. The direction in which the target object faces the image acquisition device is the positive direction. Initialize the inertial sensor and record the initial data corresponding to the inertial sensor, so that in the subsequent process of obtaining the real-time data of the target part through the inertial sensor, the difference between the real-time data of the target part obtained and the initial data is determined as the motion data of the target part, so as to complete the calibration of the coordinate system corresponding to the inertial sensor and ensure that the coordinate system corresponding to the inertial sensor is aligned with the coordinate system corresponding to the image acquisition device.

[0048] The embodiments of the present application do not limit the number and wearing position of the inertial sensors, which can be determined according to the actual situation. For example, for a target part, an inertial sensor can be worn on the target part, and the inertial sensor is worn in the exact middle of the target part, that is, at a position where the distances from the key points at the two end positions of the target part are the same. Of course, in the case where the target part includes multiple sub-parts, an inertial sensor can be worn on each sub-part of the target part.

[0049] For any image in the image sequence, the first three-dimensional coordinate information of each key point in the image can be determined through any three-dimensional pose detection model. It is also possible to first determine the two-dimensional coordinate information of each key point in the image through any two-dimensional pose detection algorithm, and then based on the images in the image sequence that have a temporal relationship with this image, determine the first three-dimensional coordinate information of each key point in this image. The embodiments of the present application do not limit this.

[0050] It should be understood that the coordinate information of all key points of the target object (i.e., including two-dimensional coordinate information and three-dimensional coordinate information) can be determined by the method provided in the embodiments of the present application. In the embodiments of the present application, only the key points of the target part of the target object are taken as examples for illustration. The coordinate information of each key point can be the coordinate information relative to the root key point (root). Among them, the present application does not limit the selection method of the root key point, which can be one of all the key points of the target object or a specified point in the coordinate system corresponding to the image acquisition device. For example, the waist key point of the target object can be used as the root key point.

[0051] Optionally, the image sequence includes at least two frames of images. Determining the first three-dimensional coordinate information of each key point of the target part in any image in the image sequence includes:

[0052] For any image, determining the first two-dimensional coordinate information of each key point;

[0053] For any image, based on the first two-dimensional coordinate information of each key point in at least one image adjacent to the image in the image sequence and the first two-dimensional coordinate information of each key point corresponding to the image, determining the first three-dimensional coordinate information of each key point in the image.

[0054] In this implementation manner, the first two-dimensional coordinate information of each key point relative to the root key point can be determined based on any two-dimensional pose detection model. Among them, the two-dimensional pose detection model can be HRNet (a neural network model).

[0055] At least one image adjacent to the image, that is, an adjacent image having a timing relationship with the image. Among them, if the method is applied to a real-time scenario, at least one image adjacent to the image is the 27 frames of images obtained before obtaining the image.

[0056] For any image, the VideoPose algorithm can be used to select the waist key point of the target object as the root key point, first determine at least one image adjacent to the image in the image sequence, and based on the timing relationship between the adjacent image and the image, the two-dimensional coordinate information of each key point in the adjacent image, and the first two-dimensional coordinate information of each key point corresponding to the image, determine the first three-dimensional coordinate information of each key point in the image relative to the waist key point.

[0057] Through the above method, the first three-dimensional coordinate information of each key point in each frame of the image sequence can be accurately determined, and the position capture of each key point of the target object can be realized in real time.

[0058] Optionally, the initial three-dimensional coordinate information of each key point corresponding to the target part is determined by the following method:

[0059] Obtain an initial image sequence corresponding to the target object in a specified pose.

[0060] For any initial image in the initial image sequence, determine the initial two-dimensional coordinate information of each key point.

[0061] For any initial image, based on the initial two-dimensional coordinate information of each key point in at least one image adjacent to the initial image in the initial image sequence and the initial two-dimensional coordinate information of each key point corresponding to the initial image, determine the initial three-dimensional coordinate information of each key point in the initial image.

[0062] Based on the initial three-dimensional coordinate information of each key point corresponding to the target part, determine the length of the target part, including:

[0063] Determine two target key points among the key points of the target part that are at the end positions of the target part.

[0064] Based on the initial three-dimensional coordinate information of the two target key points, determine the distance between the two target key points, and determine the distance between the two target key points as the length of the target part.

[0065] In this implementation manner, the initial image sequence is the reference image sequence, and the three-dimensional coordinate information of the target object in any pose can be calibrated according to the initial three-dimensional coordinate information determined based on the initial image sequence.

[0066] Considering that when the target object is in a T-pose, there will be no situations such as occlusion of the target part of the target object. Therefore, the specified pose can be set as the T-pose. By obtaining the initial image sequence of the target object when the target object is in the T-pose, the initial three-dimensional coordinate information of each key point of the target part can be further determined according to the initial image sequence of the target object. In the embodiments of the present application, the initial three-dimensional coordinate information of each key point can be determined while calibrating the inertial sensor.

[0067] The initial three-dimensional coordinate information of each key point can be determined based on the above method for determining the first three-dimensional coordinate information of each key point. Specifically as follows:

[0068] Optionally, when the target object is in a T pose, an initial image sequence including 27 images corresponding to the target object can be collected by an image acquisition device, and each image in the initial image sequence is an initial image of the target object in a T pose. For example, the target object can be controlled to maintain a preset time period in the T pose, and within this preset time period, the initial image sequence corresponding to the target object is obtained. The embodiment of the present application does not limit the preset time period, which can be determined according to the actual situation. For example, when the image acquisition frequency of the image acquisition device is 30 times per second, the preset time period can be set to 5 seconds.

[0069] For any initial image in the initial image sequence, the initial two-dimensional coordinate information of each key point can be determined based on HRNet. For any image in the initial image sequence, through the VideoPose algorithm, the waist key point of the target object is selected as the root key point, 27 images adjacent to the initial image in the initial image sequence are determined, and based on the adjacent images, the temporal relationship between the images, the initial two-dimensional coordinate information of each key point in the adjacent images, and the initial two-dimensional coordinate information of each key point in the image, the initial three-dimensional coordinate information of each key point in the image relative to the waist key point is determined.

[0070] For a target part, there may be multiple key points. In the embodiment of the present application, after determining the initial three-dimensional coordinate information of each key point of the target part, it can be described by taking two target key points at the end positions of each key point of a target part as an example. The distance L0 = |ki1 - ki2| between the initial three-dimensional coordinate information ki1 and ki2 of the two target key points of the target part can be determined, and this distance L0 is determined as the length of the target part.

[0071] When the target object is in any pose, the displacement between the first three-dimensional coordinate information k1 and k2 of two key points of the target part can be determined The displacement between the first three-dimensional coordinate information of the two key points of the target part The ratio to the length L0 of the target part Is determined as the first unit vector representing the direction of the target part (that is, the first unit vector corresponding to the target part, the unit vector corresponding to the target part obtained based on image processing).

[0072] In this implementation manner, for example, based on an algorithm such as AHRS (attitude heading reference system), combined with the motion data of the target part, the second unit vector representing the direction of the target part can be determined (That is, the second unit vector corresponding to the target part, which is the unit vector corresponding to the target part obtained based on the processing of the motion data).

[0073] It should be understood that, in an ideal situation, the first unit vector and the second unit vector corresponding to the target part should be the same, and the similarity between the two is 1. That is, no matter by what method the direction of the target part is determined, it should be consistent. In practical applications, since the target part may be blocked in the acquired image or a clear image including the target object cannot be obtained due to the rapid movement of the target object, the unit vector corresponding to the target part obtained based on the image processing is inaccurate. At this time, if the posture of the target part is still determined based on the first three-dimensional coordinate information of the target part obtained from the image processing, it may be inaccurate.

[0074] To avoid the above situation, in the embodiments of the present application, after determining the first unit vector, the second unit vector, and the first three-dimensional coordinate information of each key point of the target part, it is possible to determine whether, due to reasons such as the target part being blocked in the acquired image or the rapid movement of the target object, the posture information of the target part cannot be accurately obtained based on the image processing, so that the posture of the target part can be further determined based on the first three-dimensional coordinate information of each key point.

[0075] Among them, if the similarity between the first unit vector and the second unit vector is high, that is, the direction of the target part obtained based on the motion data of the target part is consistent with the direction of the target part obtained based on the motion data of the target part, the target part is not blocked in the acquired image; if the similarity between the first unit vector and the second unit vector is low, that is, the direction of the target part obtained based on the motion data of the target part is inconsistent with the direction of the target part obtained based on the motion data of the target part, the target part is blocked in the acquired image.

[0076] Since, in the process of processing the image including the target object, even if some key points of the target part are blocked in the acquired image, the first three-dimensional coordinate information of the blocked key points can be predicted based on the first three-dimensional coordinate information of other unblocked key points. That is, in the case where some key points of the target part are blocked, the obtained first three-dimensional coordinate information of the blocked key points may not be accurate enough. Considering that the length of the target part does not change and the real-time nature of the second unit vector representing the direction of the target part determined based on the motion data of the target part is stronger, the embodiments of the present application also provide the following specific methods for determining the posture of the target part:

[0077] Optionally, according to the similarity between the first unit vector and the second unit vector, and the first three-dimensional coordinate information of each key point, determine the pose of the target part, including:

[0078] If the similarity is greater than or equal to a preset threshold, determine the pose of the target part based on the first three-dimensional coordinate information of each key point corresponding to the target part;

[0079] If the similarity is less than the preset threshold, determine the length of the target part based on the initial three-dimensional coordinate information of each key point corresponding to the target part, determine the second three-dimensional coordinate information of other key points based on the length of the target part and the second unit vector; determine the pose of the target part based on the first three-dimensional coordinate information of the first specified key point and the second three-dimensional coordinate information of other key points, where the other key points are the key points except the first specified key point among each key point.

[0080] In this implementation manner, the specific value of the preset threshold is not limited, and it can be an experimental value or an empirical value, or it can be determined according to the actual situation. For example, the preset threshold can be set to 0.7. In the embodiments of the present application, since the modulus of the first unit vector and the modulus of the second unit vector are both 1, the inner product of the first unit vector and the second unit vector, that is, the cosine value of the included angle between the first unit vector and the second unit vector, can be directly used as the similarity between the first unit vector and the second vector. For example, when the first unit vector is The second unit vector is In this case, the similarity between the second unit vector and the second vector is

[0081] If the similarity is greater than or equal to the preset threshold, that is, as described above, the similarity between the first unit vector and the second unit vector is relatively high, the direction of the target part obtained based on the motion data of the target part is consistent with the direction of the target part obtained based on the motion data of the target part, and the target part is not blocked in the acquired image.

[0082] Considering that the stability of the position information of each key point obtained by processing the image is better, the first three-dimensional coordinate information of each key point obtained by processing the image can be used as the three-dimensional coordinate information of the first specified key point required to determine the pose of the target part, that is, directly determine the pose of the target part based on the first three-dimensional coordinate information of each key point corresponding to the target part.

[0083] Optionally, since the direction of the target part obtained based on image processing is consistent with the direction of the target part obtained based on the motion data of the target part, and the real-time performance of the direction of the target part obtained based on the motion data of the target part is stronger. To more accurately determine the pose of the target part, it is also possible to determine the dynamic target part based on the first three-dimensional coordinate information of each key point corresponding to the target part and the direction of the target part obtained based on the motion data of the target part, that is, the complete action performance of the target part.

[0084] If the similarity is less than the preset threshold, that is, as described above, the similarity between the first unit vector and the second unit vector is low. The direction of the target part obtained based on the motion data of the target part is inconsistent with the direction of the target part obtained based on the motion data of the target part. The target part may be blocked in the acquired image, or due to the rapid movement of the target object, it is impossible to obtain accurate information of the target part through the image acquisition device.

[0085] Considering that regardless of whether the target part is blocked or the target object has a rapid movement, the length of the target part remains unchanged, and the direction of the target part determined based on the motion data of the target part can be measured in real time, that is, the real-time performance of the second unit vector representing the direction of the target part determined based on the motion data of the target part is stronger. Therefore, the first three-dimensional coordinate information of each key point of the target part can be corrected based on the length of the target part and the motion data of the target part to obtain the corrected first three-dimensional coordinate information of each key point, that is, the second three-dimensional coordinate information of each key point (that is, the three-dimensional coordinate information of the first specified key point required to finally determine the pose of the target part).

[0086] Optionally, key points with relatively small change possibilities in the target part can be selected as the first specified key points, and the first three-dimensional coordinate information of the first specified key points obtained based on image processing can be determined as the second three-dimensional coordinate information of the first specified key points. Then, based on the length of the target part and the second unit vector determined based on the motion data of the target part, the second three-dimensional coordinate information of other key points can be determined.

[0087] Taking the lower leg (regardless of left or right direction) as an example of the target part, the key points at the positions of the two endpoints corresponding to the lower leg are the ankle key point and the knee key point respectively. The first three-dimensional coordinate information of the knee key point is k3, and the first three-dimensional coordinate information of the ankle key point is k4. Considering that usually, the first three-dimensional coordinate information k3 of the knee key point will be relatively fixed, the knee key point can be set as the first specified key point. Assuming that the initial three-dimensional coordinate information of the knee key point is ki3 and the initial three-dimensional coordinate information of the ankle key point is ki4, and the length of the lower leg is L' = |ki4 - ki3|, based on the direction of the target part obtained by image processing The unit vector of the calf (pointing from the knee key point to the ankle key point) determined based on the movement data of the calf is If is less than the preset threshold, the second three-dimensional coordinate information of the ankle key point can be determined as

[0088] After determining the second three-dimensional coordinate information of other key points of the target part, the posture of the target part can be determined based on the first three-dimensional coordinate information of the first specified key point and the second three-dimensional coordinate information of other key points.

[0089] In the above manner, based on the similarity between the first unit vector and the second unit vector, when the similarity is high, the posture of the target part is directly determined based on the first three-dimensional coordinate information of each key point obtained by processing the image. When the similarity is low, considering that the length of the target part remains unchanged, the posture of the target part is determined based on the first three-dimensional coordinate information of the first specified key point, the second unit vector of the target part, and the length of the target part. That is, in the above manner, the posture of the target part can be accurately determined when the posture of the target part cannot be accurately obtained based on the acquired image of the target object.

[0090] Optionally, the target part includes at least two sub-parts with a connection relationship, the first three-dimensional coordinate information of each key point includes the first three-dimensional coordinate information of each key point of each sub-part, and the movement data of the target part includes the movement data corresponding to each sub-part;

[0091] Determining the first unit vector corresponding to the target part according to the first three-dimensional coordinate information of each key point includes:

[0092] For each sub-part, determining the first unit vector corresponding to the sub-part according to the first three-dimensional coordinate information of the key points of the sub-part;

[0093] Determining the second unit vector of the target part based on the movement data of the target part includes:

[0094] For each sub-part, determining the second unit vector corresponding to the sub-part according to the movement data of the sub-part;

[0095] Determining the posture of the target part according to the similarity between the first unit vector and the second unit vector, and the first three-dimensional coordinate information of each key point includes:

[0096] For each sub-part, determining the posture of the sub-part according to the similarity between the first unit vector corresponding to the sub-part and the second unit vector corresponding to the sub-part, and the first three-dimensional coordinate information of the key points of the sub-part;

[0097] Determine the pose of the target part based on the poses of the sub - parts of the target part.

[0098] In this implementation, the target part can be divided into at least two sub - parts with a connection relationship based on the connection key points among the key points of the target part.

[0099] Such as Figure 2 and Figure 3 shown, taking the right - hand arm (the arm corresponding to the negative x - axis direction) with the target part as the target object as an example, in this target part, the key point corresponding to the elbow of the right hand should be the connection key point of this target part. Through this connection key point, this target part can be divided into 2 sub - parts, namely the upper arm of the right - hand arm of the target object (hereinafter referred to as "right upper arm") and the lower arm of the right - hand arm (hereinafter referred to as "right lower arm"). Correspondingly, if the target part is the left - hand arm of the target object, the key point corresponding to the elbow of the left hand should be the connection key point of this target part. Through this connection key point, this target part can be divided into 2 sub - parts, namely the upper arm of the left - hand arm of the target object (hereinafter referred to as "left upper arm") and the lower arm of the left - hand arm (hereinafter referred to as "left lower arm"). And as Figure 2 and Figure 3 shown, the worn inertial sensors can include four, and each inertial sensor can be worn at any position on the right upper arm, right lower arm, left upper arm, and left lower arm of the target object respectively.

[0100] Referring to the method for determining the initial three - dimensional coordinate information and the first three - dimensional coordinate information of the key points of the target part through the images in the image sequence described above, as Figure 2 shown, the initial three - dimensional coordinate information of the key points of the right upper arm can be determined as ki r1 , ki r2 , the initial three - dimensional coordinate information of the key points of the right lower arm can be determined as ki r2 , ki r3 . Among them, the initial three - dimensional coordinate information of the key points of the left upper arm can be determined as ki l1 , ki l2 , the initial three - dimensional coordinate information of the key points of the left lower arm can be determined as ki l2 , ki l3 . As Figure 3 shown, the first three - dimensional coordinate information of the key points of the right upper arm can be determined as k r1 , ki r2 , the first three - dimensional coordinate information of the key points of the right lower arm can be determined as k r2 , k r3 . Among them, the first three - dimensional coordinate information of the key points of the left upper arm can be determined as k l1 , k l2, the first three-dimensional coordinate information of each key point on the left forearm is k l2 , k l3 .

[0101] Referring to the method for determining the length of the target part based on the initial three-dimensional coordinate information of each key point of the target part described above, and the method for determining the first unit vector corresponding to the target part based on the first three-dimensional coordinate information of each key point of the target part, the length of the right upper arm can be determined as L1 = |ki r2 - ki r1 |, and the first unit vector corresponding to the right upper arm is The length of the right forearm is L2 = |ki r3 - ki r2 |, and the first unit vector corresponding to the right forearm is The length of the left upper arm is L3 = |ki l2 - ki l1 |, and the first unit vector corresponding to the left upper arm is The length of the left upper arm is L4 = |ki l3 - ki l2 |, and the first unit vector corresponding to the left forearm is

[0102] Referring to the method for determining the second unit vector corresponding to the target part based on the motion data of the target part described above, the second unit vector corresponding to the right upper arm can be determined based on the AHRS algorithm in combination with the motion data of each sub-part of the target part as The second unit vector corresponding to the right forearm is The second unit vector corresponding to the left upper arm is The second unit vector corresponding to the left forearm is

[0103] Referring to the method for determining the similarity between the first unit vector corresponding to the target part and the second unit vector corresponding to the target part described above, the similarity corresponding to the right upper arm (i.e., the similarity between the first unit vector corresponding to the right upper arm and the second unit vector corresponding to this sub-part) can be determined as The similarity corresponding to the right forearm is The similarity corresponding to the left upper arm is The similarity corresponding to the left forearm is

[0104] Since there are connection relationships among the sub-parts of the target part, by sequentially determining the first unit vector, the second unit vector, and the similarity between them corresponding to the sub-parts with connection relationships in the target part, for each sub-part, the posture of the sub-part can be determined according to the similarity corresponding to the sub-part and the first three-dimensional coordinate information of each key point of the sub-part. After determining the postures of the sub-parts, based on the postures of the sub-parts of the target part and the connection relationships among the sub-parts, the posture of the target part can be determined, which can make the determined posture of the target part more accurate.

[0105] As described above, since in the process of processing an image including a target object, when some key points of the target part are occluded, the obtained first three-dimensional coordinate information of the occluded key points may not be accurate enough, that is, the first three-dimensional coordinate information of the key points in each sub-part of the target part may also not be accurate enough. For the same consideration, that is, the lengths of the sub-parts will not change and the real-time performance of the second unit vector representing the direction of each sub-part determined based on the motion data of each sub-part is stronger, the embodiments of the present application also provide the following specific ways to determine the posture of the target part:

[0106] For each sub-part, according to the similarity between the first unit vector corresponding to the sub-part and the second unit vector corresponding to the sub-part, and the first three-dimensional coordinate information of each key point of the sub-part, determining the posture of the sub-part includes:

[0107] For the first sub-part whose similarity is greater than or equal to the corresponding preset threshold, based on the first three-dimensional coordinate information of each key point of the sub-part, determine the posture of the sub-part;

[0108] For the second sub-part whose similarity is less than the corresponding preset threshold, based on the first three-dimensional coordinate information of each key point of the sub-part and the second unit vector corresponding to the sub-part, determine the second three-dimensional coordinate information of each key point of the sub-part; according to the second three-dimensional coordinate information of each key point of the sub-part, determine the posture of the sub-part.

[0109] In this implementation, the preset threshold corresponding to each sub-part can be the same or different, and the embodiments of the present application do not limit this. Taking the target part described above as the right arm as an example, considering that the activity range of the right upper arm is smaller than that of the right lower arm, the preset threshold corresponding to the right upper arm can be set greater than the preset threshold corresponding to the right lower arm For example, it can be set to 0.8, and

[0110] In the same way, for a sub - part, if the corresponding similarity is greater than or equal to the corresponding preset threshold, it means that the similarity between the first unit vector of the sub - part (i.e., the first sub - part) obtained based on image processing and the second unit vector of the sub - part obtained based on the motion data of the sub - part is relatively high. The posture of the sub - part can be directly determined based on the first three - dimensional coordinate information of each key point of the sub - part obtained by image processing.

[0111] As Figure 3 shown, taking the target part described above as the right - hand arm as an example, the similarity corresponding to the right upper arm the similarity corresponding to the right lower arm After that, if a > 0.8 and b > 0.6, then the posture of the right upper arm can be determined based on k r1 k r2 and the posture of the right lower arm can be determined based on k r2 k r3 .

[0112] If the similarity corresponding to the sub - part is less than the corresponding preset threshold, it means that the similarity between the first unit vector of the sub - part (i.e., the second sub - part) obtained based on image processing and the second unit vector of the sub - part obtained based on the motion data of the sub - part is relatively low. It is necessary to correct the first three - dimensional coordinate information of each key point of the sub - part, that is, it is necessary to further determine the second three - dimensional coordinate information of each key point of the sub - part to accurately determine the posture of the sub - part.

[0113] Based on the above - mentioned method, the postures of each sub - part can be accurately determined.

[0114] As recorded above, considering that the length of the sub - part will not change, the length of the sub - part can be first determined based on the initial three - dimensional coordinate information of each key point of the sub - part. Further, based on the length of the sub - part and the second unit vector corresponding to the sub - part, the second three - dimensional coordinate information of each key point of the sub - part is determined. Thus, according to the second three - dimensional coordinate information of each key point of the sub - part, the posture of the sub - part is determined. The specific implementation method is as follows:

[0115] Optionally, the initial three - dimensional coordinate information of each key point includes the initial three - dimensional coordinate information of each key point of each sub - part. For the second sub - part, determining the second three - dimensional coordinate information of each key point of the sub - part based on the first three - dimensional coordinate information of each key point of the sub - part and the second unit vector corresponding to the sub - part includes:

[0116] Determine the length of the sub - part based on the initial three - dimensional coordinate information of each key point of the sub - part;

[0117] If at least two sub - parts include a first sub - part and a second sub - part, the first three - dimensional coordinate information of the first connection key point among the key points of the target first sub - part having a connection relationship with the sub - part is used as the second three - dimensional coordinate information of the first connection key point in the sub - part; according to the length of the sub - part and the second unit vector corresponding to the sub - part, the second coordinate information of the other key points in the sub - part except the first connection key point is determined. The target first sub - part is the first sub - part for which the second three - dimensional coordinate information of the corresponding key points has been determined, and the first connection key point is the common key point between the sub - part and the target first sub - part.

[0118] If each sub - part among at least two sub - parts is a second sub - part, determine the second specified key point among the key points, and determine the second three - dimensional coordinate information of the key points in the sub - part through the following method:

[0119] For the specified second sub - part to which the second specified key point belongs, the first three - dimensional coordinate information of the second specified key point is used as the second three - dimensional coordinate information of the second specified key point, and according to the length of the sub - part and the second unit vector corresponding to the sub - part, the second three - dimensional coordinate information of the other key points in the sub - part except the second specified key point is determined;

[0120] For the other sub - parts among at least two sub - parts except the specified second sub - part, according to the length of the sub - part and the second unit vector corresponding to the sub - part, the second three - dimensional coordinate information of the other key points in the sub - part except the second connection key point is determined. The target second sub - part is the second sub - part for which the second three - dimensional coordinate information of the corresponding key points has been determined, and the second connection key point is the common key point between the sub - part and the target second sub - part having a connection relationship with the sub - part.

[0121] As recorded above, the pose of the first sub - part can be directly determined based on the first three - dimensional coordinate information of the key points of the first sub - part, that is, the first three - dimensional coordinate information of the key points of the first sub - part can be directly determined as the three - dimensional coordinate information of the corresponding key points for determining the pose of the first sub - part.

[0122] Since there are connection relationships among the sub - parts in the target part, there are common key points between the mutually connected sub - parts (that is, there is a key point that is the same in two sub - parts, and this same key point is a key point at the end - point position of their respective corresponding sub - parts). If the target part includes both the first sub - part and the second sub - part at the same time, for the second sub - part that has a connection relationship with the first sub - part in the target part, the second three - dimensional coordinate information of the common key point between the second sub - part and the first sub - part can be determined according to the first three - dimensional coordinate information of each key point of the first sub - part (that is, the three - dimensional coordinate information of the common key point required to determine the posture of the second sub - part, and its essence is the first three - dimensional coordinate information of this common key point). Taking an example where a second sub - part includes key points at two end - point positions, after obtaining the second three - dimensional coordinate information of a key point at one end - point position of the second sub - part, the second three - dimensional coordinate information of the other key point of the second sub - part can be determined based on the length of the second sub - part (that is, the distance between the key points at the two end - point positions of the second sub - part determined based on the initial three - dimensional coordinate information of the key points at the two end - point positions of the second sub - part) and the second unit vector of this sub - part.

[0123] As Figure 3 shown, taking the target part described above as the right - hand arm as an example, if a > 0.8 and b < 0.6, then the posture of the right upper arm can be determined based on k r1 and k r2 , and further, according to the length of the right lower arm and the second unit vector corresponding to the right lower arm, the second three - dimensional coordinate information of the key point in the right lower arm that is not connected to the right upper arm can be determined. Specifically: And based on k r2 and k r3 ’, the posture of the right lower arm is determined.

[0124] If a < 0.8 and b > 0.6, then the posture of the right lower arm can be determined based on k r2 and k r3 , and further, according to the length of the right upper arm and the second unit vector corresponding to the right upper arm, the second three - dimensional coordinate information of the key point in the right upper arm that is not connected to the right lower arm can be determined. Specifically: And based on k r1 ’ and k r2 , the posture of the right upper arm is determined.

[0125] If all sub - parts in the target part are second - type sub - parts, that is, for any sub - part in the target part, the similarity between the first unit vector corresponding to this sub - part obtained based on image processing and the second unit vector corresponding to this sub - part obtained based on the motion data of this sub - part is relatively low, it is possible that the target part is completely occluded in the image. At this time, in order to minimize errors as much as possible, key points with relatively small possible offsets among all key points can be selected as the second specified key points, and the first three - dimensional coordinate information of this second specified key point is determined as the second three - dimensional coordinate information of this second specified key point (i.e., the three - dimensional coordinate information of this second specified key point required to determine the specified second sub - part to which this second specified key point belongs). Further, according to the length of this sub - part and the second unit vector corresponding to this sub - part, the second three - dimensional coordinate information of other key points in this sub - part except the second specified key point is determined.

[0126] Since the second three - dimensional coordinate information of each key point of the specified second sub - part has been determined, referring to the above - mentioned method of determining the second three - dimensional coordinate information of each key point of the second sub - part that has a connection relationship with the first sub - part based on the first three - dimensional coordinate information of each key point of the first sub - part and the second unit vector of the second sub - part that has a connection relationship with the first sub - part, the second three - dimensional coordinate information of each key point of all sub - parts in the target part can be determined, so as to determine the postures of all sub - parts in the target part.

[0127] As Figure 3 shown, taking the right - hand arm described above as an example of the target part, if a < 0.8 and b < 0.6, it can be determined that the right - hand arm may be completely occluded. Since the common key point between the right upper arm and the right shoulder belongs to the shoulder joint and generally does not have a large offset, the common key point between the right upper arm and the right shoulder can be used as the second specified key point, and k r1 is used as the second three - dimensional coordinate information of this second specified key point, and further the second three - dimensional coordinate information of another key point of the right upper arm is determined as Based on k r1 and k r2 ’, the posture of the right upper arm is determined. Further, the second three - dimensional coordinate information of another key point of the right lower arm that is not connected to the right upper arm is determined as And k r2 ’ and k r3 ’ are used to determine the posture of the right lower arm.

[0128] Through the above - mentioned method, the postures of all sub - parts in the target part can be accurately determined, so as to better determine the posture of the target part.

[0129] It should be noted that although the above examples are all based on the target part including two key points or the sub - part including two key points, and each key point is the key point at the end position of the corresponding part, in actual applications, there may be multiple key points for a target part or a sub - part. The above method can be adopted to wear inertial sensors between every two key points. Based on the first three - dimensional coordinate information of each key point, the length of the part between every two adjacent key points, the first unit vector and the second unit vector corresponding to the part between every two adjacent key points, and the connection relationship between the part between every two adjacent key points and other parts, the second three - dimensional coordinate information of all key points in the target part is determined, and based on the second three - dimensional coordinate information of all key points in the target part, the posture of the target part is accurately determined.

[0130] Based on the same principle as the posture detection method provided in the embodiments of the present application, an embodiment of the present application provides a posture detection device. Figure 4 As shown in the structure schematic diagram of a posture detection device provided in an embodiment of the present application, Figure 4 as shown, the device 40 may include:

[0131] An image and motion data acquisition module 401, configured to acquire an image sequence corresponding to a target object and motion data of a target part of the target object;

[0132] A three - dimensional coordinate information determination module 402, configured to determine the first three - dimensional coordinate information of each key point of the target part in any image of the image sequence;

[0133] A first unit vector determination module 403, configured to determine a first unit vector representing the direction of the target part according to the first three - dimensional coordinate information of each key point;

[0134] A second unit vector determination module 404, configured to determine a second unit vector representing the direction of the target part based on the motion data of the target part;

[0135] A posture determination module 405, configured to determine the posture of the target part according to the similarity between the first unit vector and the second unit vector, and the first three - dimensional coordinate information of each key point.

[0136] Optionally, when the posture determination module 405 determines the posture of the target part according to the similarity between the first unit vector and the second unit vector, and the first three - dimensional coordinate information of each key point, it is specifically configured to:

[0137] If the similarity is greater than or equal to a preset threshold, determine the posture of the target part based on the first three - dimensional coordinate information of each key point corresponding to the target part;

[0138] If the similarity is less than the preset threshold, based on the initial three-dimensional coordinate information of each key point corresponding to the target part, determine the length of the target part, and based on the length of the target part and the second unit vector, determine the second three-dimensional coordinate information of other key points; based on the first three-dimensional coordinate information of the first specified key point and the second three-dimensional coordinate information of other key points, determine the pose of the target part, where the other key points are the key points other than the first specified key point among all key points.

[0139] Optionally, the target part includes at least two sub-parts with a connection relationship, the first three-dimensional coordinate information of each key point includes the first three-dimensional coordinate information of each key point of each sub-part, and the motion data of the target part includes the motion data corresponding to each sub-part;

[0140] When the first unit vector determination module 403 determines the first unit vector corresponding to the target part according to the first three-dimensional coordinate information of each key point, it specifically is used for:

[0141] For each sub-part, determine the first unit vector corresponding to the sub-part according to the first three-dimensional coordinate information of the key points of the sub-part;

[0142] When the second unit vector determination module 404 determines the second unit vector of the target part based on the motion data of the target part, it specifically is used for:

[0143] For each sub-part, determine the second unit vector of the sub-part according to the motion data of the sub-part;

[0144] When the pose determination module 405 determines the pose of the target part according to the similarity between the first unit vector and the second unit vector, and the first three-dimensional coordinate information of each key point, it specifically is used for:

[0145] For each sub-part, determine the pose of the sub-part according to the similarity between the first unit vector corresponding to the sub-part and the second unit vector corresponding to the sub-part, and the first three-dimensional coordinate information of each key point of the sub-part;

[0146] Based on the poses of the sub-parts of the target part, determine the pose of the target part.

[0147] Optionally, for each sub-part, when the pose determination module 405 determines the pose of the sub-part according to the similarity between the first unit vector corresponding to the sub-part and the second unit vector corresponding to the sub-part, and the first three-dimensional coordinate information of each key point of the sub-part, it specifically is used for:

[0148] For the first sub-part whose similarity is greater than or equal to the corresponding preset threshold, determine the pose of the sub-part based on the first three-dimensional coordinate information of each key point of the sub-part;

[0149] For a second sub - part whose similarity is less than the corresponding preset threshold, based on the first three - dimensional coordinate information of each key point of the sub - part and the second unit vector corresponding to the sub - part, determine the second three - dimensional coordinate information of each key point of the sub - part; according to the second three - dimensional coordinate information of each key point of the sub - part, determine the pose of the sub - part.

[0150] Optionally, the initial three - dimensional coordinate information of each key point includes the initial three - dimensional coordinate information of each key point of each sub - part. For the second sub - part, when the pose determination module 405 determines the second three - dimensional coordinate information of each key point of the sub - part based on the first three - dimensional coordinate information of each key point of the sub - part and the second unit vector corresponding to the sub - part, it specifically is used for:

[0151] Based on the initial three - dimensional coordinate information of each key point of the sub - part, determine the length of the sub - part;

[0152] If at least two sub - parts include a first sub - part and a second sub - part, use the first three - dimensional coordinate information of the first connection key point among the key points of the target first sub - part that has a connection relationship with the sub - part as the second three - dimensional coordinate information of the first connection key point in this sub - part; according to the length of the sub - part and the second unit vector corresponding to the sub - part, determine the second coordinate information of the other key points in the sub - part except the first connection key point. The target first sub - part is the first sub - part for which the second three - dimensional coordinate information of the corresponding key points has been determined, and the first connection key point is the common key point between this sub - part and the target first sub - part;

[0153] If all sub - parts among at least two sub - parts are second sub - parts, determine the second specified key point among the key points, and determine the second three - dimensional coordinate information of each key point in this sub - part through the following method:

[0154] For the specified second sub - part to which the second specified key point belongs, use the first three - dimensional coordinate information of the second specified key point as the second three - dimensional coordinate information of the second specified key point, and according to the length of the sub - part and the second unit vector corresponding to the sub - part, determine the second three - dimensional coordinate information of the other key points in the sub - part except the second specified key point;

[0155] For the other sub - parts among at least two sub - parts except the specified second sub - part, according to the length of the sub - part and the second unit vector corresponding to the sub - part, determine the second three - dimensional coordinate information of the other key points in the sub - part except the second connection key point. The target second sub - part is the second sub - part for which the second three - dimensional coordinate information of the corresponding key points has been determined, and the second connection key point is the common key point between this sub - part and the target second sub - part that has a connection relationship with this sub - part.

[0156] Optionally, the image sequence includes at least two frames of images. When determining the first three-dimensional coordinate information of each key point of the target part in any image in the image sequence, the three-dimensional coordinate information determination module 402 is specifically configured to:

[0157] For any image, determine the first two-dimensional coordinate information of each key point;

[0158] For any image, based on the first two-dimensional coordinate information of each key point in at least one image adjacent to the image in the image sequence and the first two-dimensional coordinate information of each key point corresponding to the image, determine the first three-dimensional coordinate information of each key point in the image.

[0159] Optionally, the initial three-dimensional coordinate information of each key point corresponding to the target part is determined by the following method:

[0160] Obtain the initial image sequence corresponding to the target object in the specified pose,

[0161] For any initial image in the initial image sequence, determine the initial two-dimensional coordinate information of each key point;

[0162] For any initial image, based on the initial two-dimensional coordinate information of each key point in at least one image adjacent to the initial image in the initial image sequence and the initial two-dimensional coordinate information of each key point corresponding to the initial image, determine the initial three-dimensional coordinate information of each key point in the initial image;

[0163] When determining the length of the target part based on the initial three-dimensional coordinate information of each key point corresponding to the target part, the pose determination module 405 is specifically configured to:

[0164] Determine two target key points at the end positions of the target part among the key points of the target part;

[0165] Based on the initial three-dimensional coordinate information of the two target key points, determine the distance between the two target key points, and determine the distance between the two target key points as the length of the target part.

[0166] The device according to the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device according to the embodiments of the present application correspond to the steps in the method according to the embodiments of the present application. For the detailed function description of each module of the device, reference can be specifically made to the description in the corresponding method shown above, and details are not described herein again.

[0167] Based on the same principle as the gesture detection method and device provided in the embodiments of the present application, an electronic device (such as a server) is also provided in the embodiments of the present application. The electronic device may include a memory, a processor, and a computer program stored on the memory. The processor executes the above computer program to implement the steps of the method provided in any optional embodiment of the present application.

[0168] Optionally, Figure 5 FIG. shows a schematic structural diagram of an electronic device applicable to the embodiments of the present application, such as Figure 5 shown, Figure 5 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. Among them, the processor 4001 and the memory 4003 are connected, such as connected through a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, and the transceiver 4004 may be used for data interaction between the electronic device and other electronic devices, such as data sending and / or data receiving, etc. It should be noted that in practical applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation to the embodiments of the present application.

[0169] The processor 4001 may be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in connection with the disclosure of the present application. The processor 4001 may also be a combination that implements a computing function, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0170] The bus 4002 may include a path for transmitting information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture, extended industry standard structure) bus, etc. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 5 only a thick line is shown in, but it does not mean that there is only one bus or one type of bus.

[0171] The memory 4003 may be a ROM (Read Only Memory), or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory), or other types of dynamic storage devices that can store information and instructions. It may also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, which is not limited herein.

[0172] The memory 4003 is used to store the computer program for implementing the embodiments of the present application and is controlled by the processor 4001 to execute. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0173] The embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0174] The embodiments of the present application also provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0175] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification, claims and drawings of the present application are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than that shown in the drawings or described in words.

[0176] It should be understood that although the flowcharts in the embodiments of the present application indicate each operation step by arrows, the execution order of these steps is not limited to the order indicated by the arrows. Unless there is a clear description in this article, in some implementation scenarios of the embodiments of the present application, the implementation steps in each flowchart can be executed in other orders according to requirements. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage among these sub-steps or stages can also be executed at different times respectively. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and the embodiments of the present application do not limit this.

[0177] The above are only optional implementation manners of some implementation scenarios of the present application. It should be noted that for those of ordinary skill in the art, without departing from the technical concept of the solution of the present application, using other similar implementation means based on the technical idea of the present application also belongs to the protection scope of the embodiments of the present application.

Claims

1. A posture detection method, characterized in that, Including: Obtaining an image sequence corresponding to a target object and motion data of a target part of the target object; Determining first three-dimensional coordinate information of each key point of the target part in any image in the image sequence; Determining a first unit vector representing the direction of the target part according to the first three-dimensional coordinate information of each key point; Determining a second unit vector representing the direction of the target part based on the motion data of the target part; Determining the pose of the target part according to the similarity between the first unit vector and the second unit vector, and the first three-dimensional coordinate information of each key point; Wherein, determining the pose of the target part according to the similarity between the first unit vector and the second unit vector, and the first three-dimensional coordinate information of each key point includes: If the similarity is greater than or equal to a preset threshold, determining the pose of the target part based on the first three-dimensional coordinate information of each key point corresponding to the target part; If the similarity is less than the preset threshold, determining the length of the target part based on the initial three-dimensional coordinate information of each key point corresponding to the target part, determining second three-dimensional coordinate information of other key points based on the length of the target part and the second unit vector; determining the pose of the target part based on the first three-dimensional coordinate information of a first specified key point and the second three-dimensional coordinate information of the other key points, where the other key points are key points other than the first specified key point among each key point.

2. The method according to claim 1, characterized in that, The target part includes at least two sub-parts having a connection relationship, the first three-dimensional coordinate information of each key point includes the first three-dimensional coordinate information of each key point of each sub-part, and the motion data of the target part includes the motion data corresponding to each sub-part; The determining the first unit vector corresponding to the target part according to the first three-dimensional coordinate information of each key point includes: For each sub-part, determining the first unit vector corresponding to the sub-part according to the first three-dimensional coordinate information of each key point of the sub-part; The determining the second unit vector corresponding to the target part based on the motion data of the target part includes: For each sub-part, determining the second unit vector corresponding to the sub-part based on the motion data corresponding to the sub-part; The determining the pose of the target part according to the first unit vector, the second unit vector and the first three-dimensional coordinate information of each key point includes: For each sub-part, determining the pose of the sub-part according to the similarity between the first unit vector corresponding to the sub-part and the second unit vector corresponding to the sub-part, and the first three-dimensional coordinate information of each key point of the sub-part; Determining the pose of the target part based on the poses of the sub-parts of the target part.

3. The method according to claim 2, characterized in that, For each sub-part, the determining the pose of the sub-part according to the similarity between the first unit vector corresponding to the sub-part and the second unit vector corresponding to the sub-part, and the first three-dimensional coordinate information of each key point of the sub-part includes: For a first sub - part with a similarity greater than or equal to the corresponding preset threshold, based on the first three - dimensional coordinate information of each key point of the sub - part, determine the pose of the sub - part; For a second sub - part with a similarity less than the corresponding preset threshold, based on the first three - dimensional coordinate information of each key point of the sub - part and the second unit vector corresponding to the sub - part, determine the second three - dimensional coordinate information of each key point of the sub - part; according to the second three - dimensional coordinate information of each key point of the sub - part, determine the pose of the sub - part.

4. The method according to claim 3, characterized in that, The initial three - dimensional coordinate information of each key point includes the initial three - dimensional coordinate information of each key point of each sub - part. For the second sub - part, the determining of the second three - dimensional coordinate information of each key point of the sub - part based on the first three - dimensional coordinate information of each key point of the sub - part and the second unit vector corresponding to the sub - part includes: Based on the initial three - dimensional coordinate information of each key point of the sub - part, determine the length of the sub - part; If the at least two sub - parts include a first sub - part and a second sub - part, use the first three - dimensional coordinate information of the first connection key point among the key points of the target first sub - part that has a connection relationship with the sub - part as the second three - dimensional coordinate information of the first connection key point in the sub - part; according to the length of the sub - part and the second unit vector corresponding to the sub - part, determine the second coordinate information of the other key points in the sub - part except the first connection key point. The target first sub - part is a first sub - part for which the second three - dimensional coordinate information of the corresponding key points has been determined, and the first connection key point is the common key point between the sub - part and the target first sub - part; If all the sub - parts among the at least two sub - parts are second sub - parts, determine the second designated key point among the key points, and determine the second three - dimensional coordinate information of each key point of the sub - part in the following way: For the designated second sub - part to which the second designated key point belongs, use the first three - dimensional coordinate information of the second designated key point as the second three - dimensional coordinate information of the second designated key point, and according to the length of the sub - part and the second unit vector corresponding to the sub - part, determine the second three - dimensional coordinate information of the other key points in the sub - part except the second designated key point; For the other sub - parts among the at least two sub - parts except the designated second sub - part, according to the length of the sub - part and the second unit vector corresponding to the sub - part, determine the second three - dimensional coordinate information of the other key points in the sub - part except the second connection key point. The second connection key point is the common key point between the sub - part and the target second sub - part that has a connection relationship with the sub - part, and the target second sub - part is a second sub - part for which the second three - dimensional coordinate information of the corresponding key points has been determined.

5. The method according to claim 1, characterized in that, The image sequence includes at least two frames of images. The determining of the first three - dimensional coordinate information of each key point of the target part in any image in the image sequence includes: For any image, determine the first two - dimensional coordinate information of each key point; For any of the images, based on the first two-dimensional coordinate information of each key point in at least one image adjacent to the image in the image sequence and the first two-dimensional coordinate information of each key point corresponding to the image, determine the first three-dimensional coordinate information of each key point in the image.

6. The method according to claim 1, wherein The initial three-dimensional coordinate information of each key point corresponding to the target part is determined by the following method: Obtain the initial image sequence corresponding to the target object in a specified posture, For any initial image in the initial image sequence, determine the initial two-dimensional coordinate information of each key point; For any of the initial images, based on the initial two-dimensional coordinate information of each key point in at least one image adjacent to the initial image in the initial image sequence and the initial two-dimensional coordinate information of each key point corresponding to the initial image, determine the initial three-dimensional coordinate information of each key point in the initial image; The determining of the length of the target part based on the initial three-dimensional coordinate information of each key point corresponding to the target part includes: Determine two target key points among the key points of the target part that are at the end positions of the target part; Based on the initial three-dimensional coordinate information of the two target key points, determine the distance between the two target key points, and determine the distance between the two target key points as the length of the target part.

7. A posture detection device, characterized in that, It includes: An image and motion data acquisition module, configured to acquire an image sequence corresponding to a target object and motion data of a target part of the target object; A three-dimensional coordinate information determination module, configured to determine the first three-dimensional coordinate information of each key point of the target part in any image in the image sequence; A first unit vector determination module, configured to determine a first unit vector representing the direction of the target part according to the first three-dimensional coordinate information of each key point; A second unit vector determination module, configured to determine a second unit vector representing the direction of the target part based on the motion data of the target part; A posture determination module, configured to determine the posture of the target part according to the first unit vector, the second unit vector, and the first three-dimensional coordinate information of each key point; Wherein, when the posture determination module determines the posture of the target part according to the similarity between the first unit vector and the second unit vector and the first three-dimensional coordinate information of each key point, it is configured to: If the similarity is greater than or equal to a preset threshold, determine the posture of the target part based on the first three-dimensional coordinate information of each key point corresponding to the target part; If the similarity is less than the preset threshold, determine the length of the target part based on the initial three-dimensional coordinate information of each key point corresponding to the target part, determine the second three-dimensional coordinate information of other key points based on the length of the target part and the second unit vector; determine the posture of the target part based on the first three-dimensional coordinate information of a first specified key point and the second three-dimensional coordinate information of the other key points, where the other key points are the key points other than the first specified key point among each key point.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1-6 are implemented.

Images