Three-dimensional pose estimation method and device based on fine-tuned standard model and storage medium

Abstract

The present disclosure relates to the field of computer vision, and particularly relates to a three-dimensional pose estimation method and device based on fine-tuning a standard model and a storage medium. The method comprises: obtaining target data, the target data comprising synchronized video data and IMU data; obtaining a two-dimensional skeleton point sequence by two-dimensional pose estimation according to the video data, the two-dimensional skeleton point sequence comprising two-dimensional coordinate data of a plurality of skeleton key points of a target object; obtaining a standard skeleton point sequence by kinematics calculation according to the IMU data, the standard skeleton point sequence comprising three-dimensional coordinate data of a plurality of skeleton key points of a standard target object; and determining a three-dimensional skeleton point sequence according to the two-dimensional skeleton point sequence and the standard skeleton point sequence, the three-dimensional skeleton point sequence comprising predicted three-dimensional coordinate data of a plurality of skeleton key points of the target object. The present disclosure combines video data and IMU data to perform three-dimensional pose estimation, thereby improving the accuracy of three-dimensional pose estimation.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and in particular to a method, apparatus and storage medium for three-dimensional pose estimation based on a fine-tuned standard model. Background Technology

[0002] Understanding human behavior in video information is a crucial research area in computer vision, with significant implications for applications such as human-computer interaction, intelligent surveillance, and virtual reality. To guide machines in better understanding human behavior, researchers have proposed various methods, including classifying human behavior, 3D pose estimation (including estimating human skeletal keypoints), and estimating human surface skin. Among these, 3D skeletal keypoint estimation is a key step, as it provides more accurate guidance for surface skin estimation and behavior classification.

[0003] Three-dimensional skeletal keypoints are generally estimated using deep neural networks. In past research, researchers have proposed many structurally specific deep neural network models to estimate 3D skeletal keypoints in videos, such as Long Short-Term Memory Networks (LSTM), Convolutional Neural Networks (CNN), diffusion models, and attention-based Transformer networks. These methods have achieved good estimation results at the qualitative level, exhibiting good spatiotemporal consistency and conforming to general human behavioral logic.

[0004] However, due to the limitations of monocular video input information, current methods often fall short of achieving measurement-level accuracy in quantitative terms. This is because monocular information does not contain sufficient spatial information, making it difficult for neural networks to accurately predict spatial information in images solely through estimation. Therefore, a reasonable and effective method for achieving more accurate 3D pose estimation has not yet been provided in related technologies. Summary of the Invention

[0005] In view of this, this disclosure proposes a three-dimensional attitude estimation method, device and storage medium based on a fine-tuned standard model.

[0006] According to one aspect of this disclosure, a three-dimensional pose estimation method based on a fine-tuned standard model is provided, the method comprising:

[0007] Acquire target data, which includes synchronized video data and inertial measurement unit (IMU) data. The video data includes visual information of the target object, and the IMU data includes inertial information of the target object obtained by IMU measurement.

[0008] Based on the video data, a two-dimensional skeleton point sequence is obtained through two-dimensional pose estimation. The two-dimensional skeleton point sequence includes the two-dimensional coordinate data of multiple skeletal key points of the target object.

[0009] Based on the IMU data, a standard skeletal point sequence is obtained through kinematic calculations. The standard skeletal point sequence includes the three-dimensional coordinate data of multiple skeletal key points of the target object.

[0010] A three-dimensional skeleton point sequence is determined based on the two-dimensional skeleton point sequence and the standard skeleton point sequence. The three-dimensional skeleton point sequence includes the predicted three-dimensional coordinate data of multiple skeletal key points of the target object.

[0011] In one possible implementation, determining the three-dimensional skeleton point sequence based on the two-dimensional skeleton point sequence and the standard skeleton point sequence includes:

[0012] Extract the first bone feature corresponding to the two-dimensional bone point sequence and the second bone feature corresponding to the standard bone point sequence;

[0013] The first and second bone point features are fused to obtain the target bone point features.

[0014] The three-dimensional skeleton point sequence is determined based on the target skeleton point features.

[0015] In another possible implementation, the feature fusion of the first skeletal point features and the second skeletal point features to obtain the target skeletal point features includes:

[0016] The first skeletal point feature and the second skeletal point feature are concatenated to obtain the concatenated feature;

[0017] The spliced ​​features are weighted and calculated and mapped to a two-dimensional space to obtain the target skeleton point features;

[0018] Determining the three-dimensional skeleton point sequence based on the target skeleton point features includes:

[0019] The target skeletal point features are regressed into three-dimensional space using a multilayer perceptron (MLP) to obtain the three-dimensional skeletal point sequence.

[0020] In another possible implementation, the IMU data includes multiple frames of IMU data, and the step of obtaining a standard skeleton point sequence through kinematic calculation based on the IMU data includes:

[0021] Based on the IMU data of each frame, standard skeletal point information of each frame is obtained through kinematic calculations. The standard skeletal point information of each frame includes the three-dimensional coordinate data of multiple skeletal key points of the target object in each frame.

[0022] The standard skeleton point information of each frame is combined to obtain the standard skeleton point sequence.

[0023] In another possible implementation, obtaining standard skeleton point information for each frame through kinematic calculations based on the IMU data of each frame includes:

[0024] Based on the IMU data of each frame, determine the joint angles of multiple skeletal key points of the target object in each frame;

[0025] The standard bone point information for each frame is determined based on the joint angles of multiple skeletal key points of the target object in each frame.

[0026] In another possible implementation, obtaining a two-dimensional skeleton point sequence based on the video data through two-dimensional pose estimation includes:

[0027] The video data is converted into multiple frames of two-dimensional images;

[0028] Based on the two-dimensional image of each frame, two-dimensional skeletal point information of each frame is obtained through two-dimensional pose estimation. The two-dimensional skeletal point information of each frame includes the two-dimensional coordinate data of multiple skeletal key points of the target object in each frame.

[0029] The two-dimensional skeleton point information of each frame is combined to obtain the two-dimensional skeleton point sequence.

[0030] In another possible implementation, the method further includes:

[0031] A loss value is determined based on the three-dimensional skeleton point sequence and the labeled skeleton point sequence. The labeled skeleton point sequence includes three-dimensional coordinate data of multiple skeletal key points of the benchmark target object. The loss value is used to indicate the difference between the three-dimensional skeleton point sequence and the lost labeled skeleton point sequence.

[0032] In another possible implementation, the target data further includes motion capture data synchronized with the video data and the IMU data, and before determining the loss value based on the 3D skeleton point sequence and the labeled skeleton point sequence, the method further includes:

[0033] The label skeletal point sequence is determined based on the motion capture data, which includes motion information of the target object captured by a motion capture device.

[0034] According to another aspect of this disclosure, a three-dimensional pose estimation apparatus based on a fine-tuned standard model is provided, the apparatus comprising:

[0035] An acquisition module is used to acquire target data, which includes synchronized video data and IMU data. The video data includes visual information of the target object, and the IMU data includes inertial information of the target object obtained by IMU measurement.

[0036] The estimation module is used to obtain a two-dimensional skeleton point sequence by two-dimensional pose estimation based on the video data. The two-dimensional skeleton point sequence includes two-dimensional coordinate data of multiple skeleton key points of the target object.

[0037] The calculation module is used to obtain a standard skeleton point sequence through kinematic calculation based on the IMU data. The standard skeleton point sequence includes the three-dimensional coordinate data of multiple skeletal key points of the target object.

[0038] The determination module is used to determine a three-dimensional skeleton point sequence based on the two-dimensional skeleton point sequence and the standard skeleton point sequence, wherein the three-dimensional skeleton point sequence includes the three-dimensional coordinate data of multiple skeletal key points of the target object predicted.

[0039] In one possible implementation, the determining module is further configured to:

[0040] Extract the first bone feature corresponding to the two-dimensional bone point sequence and the second bone feature corresponding to the standard bone point sequence;

[0041] The first and second bone point features are fused to obtain the target bone point features.

[0042] The three-dimensional skeleton point sequence is determined based on the target skeleton point features.

[0043] In another possible implementation, the determining module is further configured to:

[0044] The first skeletal point feature and the second skeletal point feature are concatenated to obtain the concatenated feature;

[0045] The spliced ​​features are weighted and calculated and mapped to a two-dimensional space to obtain the target skeleton point features;

[0046] The target skeletal point features are regressed into a three-dimensional space using MLP to obtain the three-dimensional skeletal point sequence.

[0047] In another possible implementation, the computing module is further configured to:

[0048] Based on the IMU data of each frame, standard skeletal point information of each frame is obtained through kinematic calculations. The standard skeletal point information of each frame includes the three-dimensional coordinate data of multiple skeletal key points of the target object in each frame.

[0049] The standard skeleton point information of each frame is combined to obtain the standard skeleton point sequence.

[0050] In another possible implementation, the computing module is further configured to:

[0051] Based on the IMU data of each frame, determine the joint angles of multiple skeletal key points of the target object in each frame;

[0052] The standard bone point information for each frame is determined based on the joint angles of multiple skeletal key points of the target object in each frame.

[0053] In another possible implementation, the estimation module is further configured to:

[0054] The video data is converted into multiple frames of two-dimensional images;

[0055] Based on the two-dimensional image of each frame, two-dimensional skeletal point information of each frame is obtained through two-dimensional pose estimation. The two-dimensional skeletal point information of each frame includes the two-dimensional coordinate data of multiple skeletal key points of the target object in each frame.

[0056] The two-dimensional skeleton point information of each frame is combined to obtain the two-dimensional skeleton point sequence.

[0057] In another possible implementation, the apparatus further includes: a loss determination module, configured to:

[0058] A loss value is determined based on the three-dimensional skeleton point sequence and the labeled skeleton point sequence. The labeled skeleton point sequence includes three-dimensional coordinate data of multiple skeletal key points of the benchmark target object. The loss value is used to indicate the difference between the three-dimensional skeleton point sequence and the lost labeled skeleton point sequence.

[0059] In another possible implementation, the target data further includes motion capture data synchronized with the video data and the IMU data, and the device further includes: a tag determination module, used for:

[0060] The label skeletal point sequence is determined based on the motion capture data, which includes motion information of the target object captured by a motion capture device.

[0061] According to another aspect of this disclosure, a three-dimensional pose estimation apparatus based on a fine-tuned standard model is provided, comprising:

[0062] processor;

[0063] Memory used to store processor-executable instructions;

[0064] The processor is configured to implement the above method when executing instructions stored in the memory.

[0065] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the method described above.

[0066] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of a computing device, the processor in the computing device performs the above-described method.

[0067] This disclosure provides a three-dimensional attitude estimation method based on a fine-tuned standard model. It acquires synchronized video data and inertial measurement unit (IMU) data. Based on the video data, a two-dimensional skeleton point sequence is obtained through two-dimensional attitude estimation. Based on the IMU data, a standard skeleton point sequence is obtained through kinematic calculations. Finally, based on the two-dimensional skeleton point sequence and the standard skeleton point sequence, a predicted three-dimensional skeleton point sequence is determined. In other words, when the input information only provides limited spatial information, it introduces prior human skeletal constraints, transforming simple IMU data into a standard skeleton point sequence with clear physical meaning and higher relevance to the target. This supplements the model with spatial information without increasing the network inference burden or significantly increasing the number of network parameters, thereby improving the accuracy of three-dimensional attitude estimation by combining video data and IMU data.

[0068] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0069] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0070] Figure 1 A schematic diagram of the structure of a computing device provided in an exemplary embodiment of the present disclosure is shown.

[0071] Figure 2A flowchart of a three-dimensional pose estimation method based on a fine-tuned standard model provided in an exemplary embodiment of this disclosure is shown.

[0072] Figure 3 A flowchart of a three-dimensional pose estimation method based on a fine-tuned standard model provided in another exemplary embodiment of this disclosure is shown.

[0073] Figure 4 A schematic diagram of multiple skeletal key points provided in an exemplary embodiment of this disclosure is shown.

[0074] Figure 5 A flowchart of a three-dimensional pose estimation method based on a fine-tuned standard model provided in another exemplary embodiment of this disclosure is shown.

[0075] Figure 6 The normalized coordinates of the x, y, and z axes of a head keypoint provided in an exemplary embodiment of this disclosure are shown.

[0076] Figure 7 The normalized coordinates of the x, y, and z axes of a key point on the right ankle provided in an exemplary embodiment of this disclosure are shown.

[0077] Figure 8 A schematic diagram of the structure of a three-dimensional attitude estimation device based on a fine-tuning standard model provided in an exemplary embodiment of this disclosure is shown.

[0078] Figure 9 This is a block diagram illustrating an apparatus according to an exemplary embodiment. Detailed Implementation

[0079] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0080] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0081] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0082] First, the application scenarios involved in this disclosure will be introduced. Please refer to... Figure 1The diagram illustrates a structural schematic of a computing device provided in an exemplary embodiment of this disclosure.

[0083] Computing devices can be servers or terminals. Terminals include mobile terminals or fixed terminals, such as mobile phones, tablets, laptops, and desktop computers. A server can be a single server, a server cluster consisting of several servers, or a cloud computing service center.

[0084] Optionally, when performing 3D pose estimation tasks, computing devices typically need to be equipped with multiple sensors and corresponding processing units. The following are some hardware architectures that may be involved in 3D pose estimation:

[0085] 1. Inertial Measurement Unit (IMU): An IMU is a device capable of measuring the acceleration and angular velocity of a carrier without external reference. An IMU can be an inertial sensor, typically including an accelerometer, gyroscope, etc., used to capture inertial information of the human body, such as angular velocity and acceleration. A basic IMU typically includes an accelerometer and a gyroscope, used to measure acceleration and angular velocity along three spatial axes, forming a so-called 6-axis or 6-degree-of-freedom (6DOF) configuration. In embodiments of this disclosure, IMU data includes inertial information of the target object measured by the IMU, which may include angular velocity and acceleration. Optionally, the inertial information may include acceleration along three spatial axes and angular velocity along three spatial axes.

[0086] 2. Camera: A monocular or multi-view camera can provide video data (including multiple frames of 2D images), which is one of the key inputs for 3D pose estimation. Through image processing and computer vision algorithms, 2D coordinate data of multiple skeletal key points can be extracted from it. In this embodiment of the disclosure, the video data includes visual information of the target object, and the video data may include visual information captured by the camera.

[0087] 3. Motion capture equipment: This is a technical device used to capture and record the motion of an object. It captures the motion trajectory of the target object, converting the actual action into a digital signal, thereby achieving precise replication and simulation of the action. The main components of motion capture equipment include sensors, data acquisition units, and computer software. Sensors are the core components of motion capture equipment, responsible for capturing the motion of the target object. Depending on the type of sensor, motion capture equipment can be divided into various types, such as optical, inertial, and mechanical. In this embodiment, the motion capture data includes motion information of the target object captured by the motion capture equipment. This data can record in detail the movement and posture changes of multiple skeletal key points in space.

[0088] 4. Processing unit: Used to process data from the above sensors and perform complex computational tasks, such as the inference process of deep learning models.

[0089] In addition to the hardware architecture, specialized software and algorithms running on computing devices are required to process the data and perform 3D pose estimation. This includes neural network models that are trained to predict accurate 3D human poses from various sensor data.

[0090] It should be noted that, when performing 3D pose estimation, the computing device does not necessarily need to integrate all sensors, but it needs to be able to receive and process data from these sensors. Furthermore, depending on the application requirements, other auxiliary devices or specially designed hardware may be needed to meet specific performance requirements. This disclosure limits these aspects.

[0091] Optionally, the computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will understand that... Figure 1 The structure shown does not constitute a limitation on the computing device and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein:

[0092] The processor 10 is the control center of the computing device. It connects various parts of the computing device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 20, and by calling data stored in the memory 20, it performs various functions of the computing device and processes data, thereby controlling the computing device as a whole. The processor 10 can be implemented by a CPU or a GPU.

[0093] The memory 20 can be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. The program storage area may store the operating system 21, the acquisition module 22, the estimation module 23, the calculation module 24, the determination module 25, and the application program 26 required for at least one function, etc.; the data storage area may store data created according to the use of the computing device, etc. The memory 20 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Accordingly, memory 20 may also include a memory controller to provide processor 10 with access to memory 20.

[0094] The processor 10 performs the following functions by running the acquisition module 22: acquiring target data, which includes synchronized video data and IMU data. The video data includes visual information of the target object, and the IMU data includes inertial information of the target object measured by the IMU. The processor 10 performs the following functions by running the estimation module 23: obtaining a two-dimensional skeleton point sequence based on the video data through two-dimensional pose estimation. The two-dimensional skeleton point sequence includes two-dimensional coordinate data of multiple skeletal key points of the target object. The processor 10 performs the following functions by running the calculation module 24: obtaining a standard skeleton point sequence based on the IMU data through kinematic calculation. The standard skeleton point sequence includes three-dimensional coordinate data of multiple skeletal key points of the standard target object. The processor 10 performs the following functions by running the determination module 25: determining a three-dimensional skeleton point sequence based on the two-dimensional skeleton point sequence and the standard skeleton point sequence. The three-dimensional skeleton point sequence includes the predicted three-dimensional coordinate data of multiple skeletal key points of the target object.

[0095] It should be noted that the three-dimensional pose estimation provided in this disclosure includes three-dimensional skeletal keypoint estimation. This method has a wide range of applications, including human-computer interaction, motion analysis, rehabilitation training, human motion capture, and virtual reality and augmented reality. This disclosure does not limit its application in these areas.

[0096] The following describes the three-dimensional pose estimation method based on a fine-tuned standard model provided in this disclosure using several exemplary embodiments.

[0097] Please refer to Figure 2 This document illustrates a flowchart of a three-dimensional pose estimation method based on a fine-tuned standard model provided in an exemplary embodiment of this disclosure. This embodiment uses this method for... Figure 1 The following example uses a computing device as an illustration. The method includes the following steps.

[0098] Step 201: Acquire target data. The target data includes synchronized video data and IMU data. The video data includes visual information of the target object, and the IMU data includes inertial information of the target object obtained by IMU measurement.

[0099] Optionally, synchronized video data and IMU data can be obtained through a synchronization signal. The video data includes multiple frames of two-dimensional images, and the IMU data includes multiple frames of IMU data. The term "synchronization" indicates that there is a one-to-one correspondence between the multiple frames of two-dimensional images and the multiple frames of IMU data.

[0100] For video data, it can be extracted frame by frame and converted into a multi-frame two-dimensional image. If the original video data is in color, the converted two-dimensional image will be an RGB image, containing information from the red, green, and blue color channels.

[0101] For IMU data, due to its typically high sampling frequency (above 100Hz), it can capture rapid motion over short periods. During processing, the time synchronization between IMU data and video data needs to be considered to ensure temporal consistency. Optionally, "synchronization" of video and IMU data can include frame number alignment and sampling rate alignment. Frame number alignment: For video data, it is generally necessary to align it with IMU data in terms of frame number. This means that each frame of the video data corresponds to one frame of IMU data to maintain temporal consistency. Sampling rate alignment: In addition to frame number alignment, it is also necessary to ensure that the sampling rates of the video and IMU data match. If their sampling rates differ, resampling or discarding some data may be necessary to ensure temporal consistency.

[0102] Step 202: Based on the video data, a two-dimensional skeleton point sequence is obtained through two-dimensional pose estimation. The two-dimensional skeleton point sequence includes the two-dimensional coordinate data of multiple skeleton key points of the target object.

[0103] The target object can be the person to be estimated. Multiple skeletal key points can also be called multiple human skeletal key points. Multiple skeletal key points include key points of multiple body parts of the person. For example, multiple skeletal key points include the pelvic key point, the left thigh key point, the right thigh key point, the left calf key point, the right calf key point, the left ankle key point, and the right ankle key point.

[0104] In one possible implementation, after converting video data into multi-frame two-dimensional images, two-dimensional skeleton point information for each frame is obtained through two-dimensional pose estimation. The two-dimensional skeleton point information for each frame includes the two-dimensional coordinate data of multiple skeletal keypoints of the target object in that frame. The two-dimensional skeleton point information of each frame is then combined to obtain a two-dimensional skeleton point sequence, which includes the two-dimensional skeleton point information from multiple frames. The two-dimensional skeleton point sequence includes a first consecutive frame sequence corresponding to each of the multiple skeletal keypoints. The first consecutive frame sequence for each skeletal keypoint includes the two-dimensional coordinate data of each skeletal keypoint across the multiple frames. The two-dimensional coordinate data is positional information defined in two-dimensional space, consisting of x and y coordinate values.

[0105] Optionally, for each frame of a 2D image, the corresponding 2D skeleton point information can be obtained through a 2D pose estimation network. Illustratively, each frame of a 2D image is input into the 2D pose estimation network, and the output is the 2D skeleton point information for each frame. In this implementation, the 2D pose estimation network is a pre-defined neural network for 2D pose estimation based on 2D images, such as a Vision Transformer (ViT) neural network. A ViT neural network is a self-attention-based neural network primarily used for processing image data.

[0106] In another possible implementation, a two-dimensional skeleton point sequence is predicted using a two-dimensional pose estimation network based on video data. Illustratively, video data is input into the two-dimensional pose estimation network, and a two-dimensional skeleton point sequence is output. In this implementation, the two-dimensional pose estimation network is a pre-defined neural network for performing two-dimensional pose estimation based on video data. This network can determine the two-dimensional skeleton point information for each frame of the two-dimensional image in the video data and combine the two-dimensional skeleton point information from each frame to obtain a two-dimensional skeleton point sequence.

[0107] Step 203: Based on the IMU data, a standard skeletal point sequence is obtained through kinematic calculations. The standard skeletal point sequence includes the three-dimensional coordinate data of multiple skeletal key points of the standard target object.

[0108] It should be noted that steps 202 and 203 can be executed in parallel or in a specific order. This embodiment of the present disclosure does not limit the execution order of steps 202 and 203.

[0109] Optionally, after acquiring multiple frames of IMU data, standard skeletal point information for each frame can be obtained through kinematic calculations based on the IMU data for each frame. The standard skeletal point information for each frame includes the 3D coordinate data of multiple skeletal keypoints of the standard target object for that frame. The standard skeletal point information for each frame is then combined to obtain a standard skeletal point sequence. This standard skeletal point sequence is also called a 3D standard human skeletal point sequence. The standard skeletal point sequence includes a second consecutive frame sequence corresponding to each of the multiple skeletal keypoints. The second consecutive frame sequence for each skeletal keypoint includes the 3D coordinate data of each skeletal keypoint across multiple frames. The 3D coordinate data is positional information defined in 3D space, consisting of three coordinate values: x, y, and z.

[0110] Optionally, obtaining standard skeletal point information for each frame through kinematic calculations based on each frame of IMU data may include: determining the joint angles of multiple skeletal key points of the target object in each frame based on the IMU data of each frame; and determining the standard skeletal point information for each frame based on the joint angles of multiple skeletal key points of the target object in each frame. It should be noted that the relevant details of this step can be found in the descriptions in the embodiments below, and will not be elaborated upon here.

[0111] Step 204: Determine the three-dimensional bone point sequence based on the two-dimensional bone point sequence and the standard bone point sequence. The three-dimensional bone point sequence includes the three-dimensional coordinate data of multiple skeletal key points of the predicted target object.

[0112] Optionally, the first bone feature corresponding to the two-dimensional bone point sequence and the second bone feature corresponding to the standard bone point sequence are extracted; the first bone point feature and the second bone point feature are fused to obtain the target bone point feature; and the three-dimensional bone point sequence is determined based on the target bone point feature. The bone point features (first bone point feature, second bone point feature, and target bone point feature) are used to indicate the bone structure, and the bone point features include the position, shape, size, and other features of the bone key points.

[0113] Optionally, for each type of skeletal point sequence (two-dimensional skeletal point sequence or standard skeletal point sequence), the corresponding skeletal point features can be obtained through a pre-trained feature extraction network. That is, the two-dimensional skeletal point sequence can be input into the feature extraction network to obtain the corresponding first skeletal feature, and the standard skeletal point sequence can be input into the feature extraction network to obtain the corresponding second skeletal feature.

[0114] Schematic illustration: The feature extraction network can be a Dual-Stream Temporal Transformer (DSTformer) neural network. This DSTformer neural network uses a dual-stream design to process temporal and spatial information separately, effectively capturing the spatiotemporal relationships between skeletal joints. The temporal information processing branch focuses on processing time-series data, i.e., motion information that changes over time. It uses temporal embeddings to encode temporal positional information, allowing the network to recognize and learn patterns and dynamics that change over time. The spatial information processing branch focuses on processing spatial data, i.e., the relative positions of different body parts at the same time. It encodes spatial positional information through keypoint embeddings, enabling the network to understand the spatial relationships between different body parts. After processing by their respective branches, these two features are fused together to form a comprehensive skeletal point feature representation.

[0115] Optionally, after obtaining the first and second skeletal features, feature fusion can be performed using an attention module (including a learnable weight matrix) to obtain target skeletal point features. Then, based on the target skeletal point features, a 3D skeletal point sequence is obtained through MLP regression. It should be noted that the relevant details of this step can be found in the descriptions in the embodiments below, and will not be elaborated upon here.

[0116] The three-dimensional coordinate data obtained in step 204 can be considered as a more accurate result obtained by fine-tuning the three-dimensional data obtained from the IMU data.

[0117] Please refer to Figure 3 This illustrates a flowchart of a three-dimensional pose estimation method based on a fine-tuned standard model provided in another exemplary embodiment of this disclosure. This embodiment uses this method for... Figure 1 The following example uses a computing device as an illustration. The method includes the following steps.

[0118] Step 301: Acquire target data, which includes synchronized video data, IMU data, and motion capture data.

[0119] The video data includes visual information of the target object, the IMU data includes inertial information of the target object measured by the IMU, and the motion capture data includes motion information of the target object captured by the motion capture device.

[0120] For synchronized video data, a synchronized camera can be used to record, and multiple frames of two-dimensional images can be obtained by frame extraction. Optionally, the longest complete segment of a body part can be selected from the video data as a two-dimensional image. For synchronized IMU data, the IMU data can be obtained by an inertial sensor measurement system. For synchronized motion capture data, motion capture equipment can be used to capture it.

[0121] Optionally, the "synchronization" of video data, multi-frame IMU data, and motion capture data can include frame number alignment and sampling rate alignment among the video data, multi-frame IMU data, and motion capture data. That is, there is a one-to-one correspondence between multiple frames of 2D images, multiple frames of IMU data, and multiple frames of motion capture data. It should be noted that the synchronization of these three types of data can be compared to the relevant descriptions of the synchronization of video data and multi-frame IMU data, and will not be repeated here.

[0122] Step 302: Determine the two-dimensional skeleton point information of each frame based on the two-dimensional image of each frame in the video data, and combine the two-dimensional skeleton point information of each frame to obtain a two-dimensional skeleton point sequence.

[0123] The video data includes multiple frames of two-dimensional images. The two-dimensional skeleton point information of each frame includes the two-dimensional coordinate data of multiple skeletal key points of the target object in each frame. The two-dimensional skeleton point sequence includes two-dimensional skeleton point information from multiple frames.

[0124] The target object's multiple skeletal keypoints include keypoints for multiple body parts of the person, illustratively defined as follows: Figure 4 The diagram shows multiple skeletal keypoints, including keypoints 0 to 16, for a total of 17 skeletal keypoints.

[0125] Optionally, based on each frame of the 2D image in the video data, a 2D pose estimation network predicts a sequence of 2D skeleton points. This network determines the 2D skeleton point information for each frame and combines this information to obtain the 2D skeleton point sequence. This 2D pose estimation network can be implemented through a two-stage prediction process, including but not limited to the following steps: 1. Object detection and localization: Using object detection methods, the bounding boxes of target objects appearing in each frame of the video data are identified. This step typically involves using machine learning models to identify and locate human bodies in the images. 2. Location information input: After determining the bounding boxes, the location information is used as input to the 2D pose estimation network to predict the 2D coordinate data p of multiple skeletal keypoints in each frame. t t is a positive integer, ranging from 1 to T, where T is the total number of frames in the two-dimensional images included in the video data. Then, the two-dimensional coordinate data p of each of the predicted skeletal keypoints in each frame is...t The data is combined, ensuring alignment of the first and last frames, to form a complete two-dimensional skeleton point sequence. This sequence comprises multiple consecutive first frame sequences corresponding to each skeleton keypoint. Each first consecutive frame sequence for a skeleton keypoint includes multi-frame two-dimensional coordinate data for that keypoint. (Illustratively, each first consecutive frame sequence...) in, Let be a set of real numbers, where 17 represents the total number of multiple skeletal keypoints and 3 represents the coordinate dimension.

[0126] Step 303: Based on the IMU data of each frame, the standard skeleton point information of each frame is obtained through kinematic calculation, and the standard skeleton point information of each frame is combined to obtain the standard skeleton point sequence.

[0127] Optionally, based on each frame of IMU data, the joint angles of multiple skeletal keypoints of the target object in each frame are determined; based on the joint angles of the multiple skeletal keypoints of the target object in each frame, standard skeletal point information for each frame is determined. The joint angle of each skeletal keypoint is used to indicate the angle and direction of that skeletal keypoint in three-dimensional space; for example, the joint angle can be represented as a three-dimensional pose angle. The standard skeletal point information for each frame includes the three-dimensional coordinate data of multiple standard skeletal keypoints of the target object in each frame, and the standard skeletal point sequence includes standard skeletal point information from multiple frames. The total number of IMU data is the same as the total number of frames of the two-dimensional images included in the video data.

[0128] Indicative, each frame of IMU data is T represents the total number of frames of the two-dimensional images included in the video data; 7 represents the total number of multiple skeletal keypoints, such as the pelvic keypoint, left thigh keypoint, right thigh keypoint, left calf keypoint, right calf keypoint, left ankle keypoint, and right ankle keypoint; 6 represents the dimension of inertial information, that is, inertial information can include acceleration on three spatial axes and angular velocity on three spatial axes.

[0129] Based on each frame of IMU data, the estimated joint angles, i.e., the three-dimensional pose angles, of each skeletal keypoint can be calculated using Kalman filtering. 3 represents the coordinate dimension, where A = {angle0, angle1, ..., angle2} T}, angle0 to angle T Each angle in t Satisfying angle t = [a0, a1, ..., a7], where each of a0 to a7 is a... θ is the pitch angle of a skeletal keypoint. φ is the yaw angle of a skeletal keypoint, φ is the roll angle of a skeletal keypoint, and t is a positive integer ranging from 1 to T.

[0130] Based on the joint angles of multiple skeletal keypoints of the target object in each frame, standard skeletal point information for each frame is calculated using predefined parameters such as the distance between skeletal keypoints. For example, the position of the pelvic keypoint is defined as the origin of the coordinate system, i.e., the three-dimensional coordinate data of the pelvic keypoint. The three-dimensional coordinate data of each skeletal key point then satisfy the following formula:

[0131]

[0132] Here are the 3D coordinates of the key points on the left thigh, and PelvisAngles are the joint angles of the key points on the pelvis. PT The distance between the pelvic keypoint and the thigh keypoint is preset, and RotationMatrix() is preset rotation matrix.

[0133]

[0134] Here are the 3D coordinates of key points on the left lower leg, and LeftThighAngles are the joint angles of key points on the left thigh. TS This is the preset distance between the key points on the thigh and the key points on the calf.

[0135]

[0136] Here are the 3D coordinates of key points on the left ankle, and LeftShankAngles are the joint angles of key points on the left lower leg. SF The distance between the preset lower leg key point and the ankle key point.

[0137]

[0138] Here are the 3D coordinates of the key points on the right thigh, and PelvisAngles are the joint angles of the key points on the pelvis. PT This is the preset distance between the key points of the pelvis and the key points of the thigh.

[0139]

[0140] Here are the 3D coordinates of key points on the right lower leg, and RightThighAngles are the joint angles of key points on the right thigh. TSThis is the preset distance between the key points on the thigh and the key points on the calf.

[0141]

[0142] Here are the 3D coordinates of key points on the right ankle, and RightShankAngles are the joint angles of key points on the right lower leg. SF The distance between the preset lower leg key point and the ankle key point.

[0143] RotationMatrix() is composed of the product of three matrices: pitch rotation matrix (rotation around the x-axis), roll rotation matrix (rotation around the y-axis), and yaw rotation matrix (rotation around the z-axis).

[0144] The formula for calculating the pitch rotation matrix (PitchRotationMatrix) is shown below:

[0145]

[0146] The formula for calculating the pitch rotation matrix (Roll RotationMatrix) is shown below:

[0147]

[0148] The formula for calculating the yaw rotation matrix is ​​illustrated below:

[0149]

[0150] After calculating the standard skeleton point information for each frame, the standard skeleton point information for each frame is combined to obtain a standard skeleton point sequence for a first number of skeleton keypoints. Then, a first matrix can be used to regress the standard skeleton point sequence of the first number of skeleton keypoints to a predefined target number of skeleton keypoints. The first matrix is ​​determined based on the first number and the target number; for example, the first number is 7 and the target number is 17. After obtaining the standard skeleton point sequence corresponding to 7 skeleton keypoints, a Joint Regressor matrix is ​​used to regress the standard skeleton point sequence corresponding to the 7 skeleton keypoints to a predefined standard skeleton point sequence of 17 skeleton keypoints. The first matrix is ​​a 7*17 matrix, where the element in the i-th row and j-th column represents the contribution weight of the i-th skeleton keypoint among the 7 skeleton points to the j-th skeleton keypoint among the 17 skeleton keypoints. The contribution weight is preset, and i and j are positive integers. For example, the first matrix can be a Joint Regressor matrix.

[0151] Step 304: Determine the label bone point information of each frame based on the motion capture data of each frame, and combine the label bone point information of each frame to obtain the label bone point sequence.

[0152] Based on motion capture data for each frame, a specific algorithm identifies the labeled skeleton point information for each frame. This labeled skeleton point information includes the 3D coordinates of multiple key skeletal points of the baseline target object. The labeled skeleton point sequence comprises labeled skeleton point information from multiple frames.

[0153] Optionally, the labeled skeleton point information of each frame is combined to obtain a second number of labeled skeleton point sequences. A second matrix can be used to regress this second number of labeled skeleton point sequences to a predefined target number of labeled skeleton point sequences. The second matrix is ​​determined based on the target number and the second number; for example, the second number is 39, and the target number is 17. Illustratively, a labeled skeleton point sequence of 39 skeleton keypoints is determined based on motion capture data, and this sequence is regressed to a predefined sequence of 17 labeled skeleton keypoints. Related details can be compared to the process of determining a standard skeleton point sequence based on IMU data, and will not be elaborated here.

[0154] It should be noted that steps 302, 303, and 304 can be executed in parallel or in a specific order. This embodiment does not limit the order of execution of the steps.

[0155] Step 305: Extract the first bone features corresponding to the two-dimensional bone point sequence and the second bone features corresponding to the standard bone point sequence.

[0156] Optionally, a two-dimensional skeleton point sequence can be input into a pre-trained feature extraction network to obtain the corresponding first skeleton feature, and a standard skeleton point sequence can be input into the pre-trained feature extraction network to obtain the corresponding second skeleton feature. Illustratively, the pre-trained feature extraction network is a DSTformer neural network, the first skeleton feature is a two-dimensional feature f2d, and the second skeleton feature is a standardized feature fstandard. During the training process of the feature extraction network, the parameters of the feature extraction network are not frozen, meaning that the parameters of the feature extraction network can be adjusted according to new data.

[0157] Step 306: The first and second skeletal point features are fused to obtain the target skeletal point features.

[0158] Optionally, the first and second skeleton point features are concatenated to obtain concatenated features. These concatenated features are then weighted and mapped to a two-dimensional space to obtain the target skeleton point features. In other words, a learnable weight matrix can be used to weight the concatenated features back to a two-dimensional dimension to obtain the target skeleton point features. For example, the first skeleton point feature f2d and the second skeleton point feature fstandard are concatenated to obtain the concatenated feature fcat. A learnable weight matrix W is then used to weight the concatenated feature fcat back to the same dimension as the first skeleton point feature f2d to obtain the target skeleton point feature f_fused.

[0159] Step 307: Determine the three-dimensional skeleton point sequence based on the target skeleton point features.

[0160] Optionally, the target skeletal point features are regressed into a 3D space using an MLP to obtain a 3D skeletal point sequence. This 3D skeletal point sequence is a sequence of 3D skeletal keypoints representing the target number of skeletal points, including the 3D coordinate data of multiple predicted skeletal keypoints of the target object. Illustratively, the target skeletal point features f_fused are regressed into a 3D dimension using an MLP to obtain a 3D skeletal point sequence. This sequence includes the 3D coordinate data "X_t, j" of multiple skeletal keypoints across multiple frames, where "X_t, j" represents the 3D coordinate data of the j-th skeletal keypoint in frame t, and both t and j are positive integers.

[0161] Step 308: Determine the loss value based on the 3D skeleton point sequence and the label skeleton point sequence.

[0162] The 3D skeleton point sequence and the label skeleton point sequence both contain the same amount of 3D coordinate data. A loss value is determined based on the 3D skeleton point sequence and the label skeleton point sequence. This loss value is used to indicate the difference between the 3D skeleton point sequence and the lost label skeleton point sequence.

[0163] Optionally, two losses are calculated based on the 3D skeleton point sequence and the labeled skeleton point sequence: pose loss and velocity loss. The pose loss is based on the difference between the 3D skeleton point sequence and the labeled skeleton point sequence, used to constrain the 3D human skeleton point sequence to be consistent with the labeled skeleton point sequence. The velocity loss is based on the difference between these two sequences (i.e., the rate of change or velocity), used to constrain the difference between the 3D human skeleton point sequence and the labeled skeleton point sequence to be consistent. The pose loss and velocity loss are weighted and fused to obtain the final loss value, and the network is trained under supervision to optimize the network parameters. The weight coefficients of the pose loss and velocity loss are used to adjust their relative importance in the total loss value, and these weight coefficients can be adjusted according to the specific task requirements.

[0164] Illustratively, the difference between two sequences can be calculated using Euclidean distance (i.e., L2 norm). For the pose loss L... 3D Calculate the L2 norm between the 3D human skeleton point sequence and the labeled skeleton point sequence:

[0165]

[0166] Where t is a positive integer, T is the total number of frames of the two-dimensional images included in the video data, j is a positive integer, and J is the total number of predefined skeletal keypoints, i.e., the aforementioned target number. X represents the 3D coordinates of the j-th skeletal keypoint in the t-th frame of a 3D human skeleton point sequence. t,j Let be the 3D coordinate data of the j-th bone keypoint in the t-th frame of the label bone point sequence, and ∥·∥2 be the L2 norm.

[0167] For velocity loss L o First, calculate the difference between the two sequences, then calculate the L2 norm between the difference between the two sequences:

[0168]

[0169] in, O t,j =X t,j -X t-1,j

[0170] in, O is the difference result of the j-th bone keypoint in the t-th frame of the 3D human skeleton point sequence. t,j This is the difference result of the j-th bone keypoint in the t-th frame of the labeled bone point sequence. This refers to the 3D coordinate data of the j-th bone keypoint in the t-th frame of a 3D human skeleton point sequence. X represents the 3D coordinate data of the j-th skeletal keypoint in the (t-1)-th frame of a 3D human skeleton point sequence. t,j X represents the 3D coordinates of the j-th bone keypoint in the t-th frame of the label skeleton point sequence. t-1,j This refers to the 3D coordinate data of the j-th bone keypoint in the (t-1)-th frame of the label bone point sequence.

[0171] In one possible implementation, such as Figure 5As shown, the 3D pose estimation method based on a fine-tuned standard model includes, but is not limited to, the following steps: Acquiring target data, including synchronized video data, IMU data, and motion capture data; for video data, extracting a 2D skeleton from each frame of the 2D image in the video data, and extracting the first skeleton feature; for IMU data, obtaining a standard skeleton point sequence through kinematic calculations, regressing the standard skeleton point sequence to a standard skeleton point sequence of the target number of skeletal keypoints, extracting the second skeleton feature, fusing the first and second skeleton features, and regressing the fused target skeleton point features through an MLP to obtain a 3D skeleton point sequence of the target number of skeletal keypoints; for motion capture data, determining a labeled skeleton point sequence based on the motion capture data, regressing the determined labeled skeleton point sequence to a labeled skeleton point sequence of the target number of skeletal keypoints, and then calculating the loss value based on the 3D skeleton point sequence and the labeled skeleton point sequence, and supervising the training of the network to optimize the network parameters.

[0172] In an illustrative example, taking the skeletal keypoints as head keypoints, the estimated normalized coordinates of the x, y, and z axes of the head keypoints are as follows: Figure 6 As shown, the blue line represents the estimation result using the method provided in the embodiments of this disclosure, and the yellow line represents the estimation result using the method provided in related technologies. It can be seen that the estimation result of the method provided in the embodiments of this disclosure is more stable and more continuous in coordinate representation.

[0173] In another illustrative example, taking the right ankle keypoint as the skeletal keypoint, the estimated normalized coordinates of the right ankle keypoint along the x, y, and z axes are as follows: Figure 7 As shown, the blue line represents the estimation result using the method provided in the embodiments of this disclosure, and the yellow line represents the estimation result using the method provided in related technologies. The same conclusion can be drawn: the estimation result of the method provided in the embodiments of this disclosure is more stable and more continuous in coordinate representation.

[0174] In summary, this disclosure provides a 3D pose estimation method based on fine-tuning a standard model. It calculates a standard skeletal keypoint sequence using IMU data and kinematic methods, and further fine-tunes it by combining this sequence with a 2D skeletal point sequence determined from video data. This achieves the estimation of 3D human skeleton keypoints. Specifically, by adding predefined human skeleton constraints, the spatial information that the network needs to predict is reduced, thereby improving the quantitative accuracy of the model estimation. In other words, when the input information only provides limited spatial information, prior human skeleton constraints are introduced. This transforms simple IMU data into a standard skeletal point sequence with clear physical meaning and higher relevance to the target. Without increasing the network inference burden or significantly increasing the number of network parameters, spatial information is supplemented to the model, and the accuracy of 3D pose estimation is improved by combining video data and IMU data.

[0175] The following are device embodiments of the present disclosure. For parts not described in detail in the device embodiments, please refer to the technical details disclosed in the above method embodiments.

[0176] Please refer to Figure 8 This illustration shows a schematic diagram of a three-dimensional attitude estimation device based on a fine-tuned standard model provided in an exemplary embodiment of this disclosure. The device can be implemented, in whole or in part, through software, hardware, or a combination of both, as a computing device. The device includes: an acquisition module 22, an estimation module 23, a calculation module 24, and a determination module 25.

[0177] Acquisition module 22 is used to acquire target data, which includes synchronized video data and IMU data. The video data includes visual information of the target object, and the IMU data includes inertial information of the target object obtained by IMU measurement.

[0178] Estimation module 23 is used to obtain a two-dimensional skeleton point sequence based on video data through two-dimensional pose estimation. The two-dimensional skeleton point sequence includes the two-dimensional coordinate data of multiple skeleton key points of the target object.

[0179] The calculation module 24 is used to obtain a standard skeleton point sequence based on IMU data through kinematic calculations. The standard skeleton point sequence includes the three-dimensional coordinate data of multiple skeletal key points of a standard target object.

[0180] The determination module 25 is used to determine the three-dimensional bone point sequence based on the two-dimensional bone point sequence and the standard bone point sequence. The three-dimensional bone point sequence includes the three-dimensional coordinate data of multiple skeletal key points of the predicted target object.

[0181] In one possible implementation, the determining module 25 is also used for:

[0182] Extract the first bone features corresponding to the two-dimensional bone point sequence and the second bone features corresponding to the standard bone point sequence;

[0183] The first and second skeleton point features are fused to obtain the target skeleton point features.

[0184] Based on the characteristics of the target skeleton points, determine the three-dimensional skeleton point sequence.

[0185] In another possible implementation, module 25 is also used for:

[0186] The spliced ​​features are obtained by concatenating the features of the first and second skeleton points.

[0187] The spliced ​​features are weighted and calculated and mapped to a two-dimensional space to obtain the target skeleton point features;

[0188] The target skeleton point features are regressed into three-dimensional space using MLP to obtain a three-dimensional skeleton point sequence.

[0189] In another possible implementation, the calculation module 24 is also used for:

[0190] Based on each frame of IMU data, the standard skeleton point information of each frame is obtained through kinematic calculations. The standard skeleton point information of each frame includes the three-dimensional coordinate data of multiple skeleton key points of the standard target object in each frame.

[0191] The standard skeleton point information of each frame is combined to obtain a standard skeleton point sequence.

[0192] In another possible implementation, the calculation module 24 is also used for:

[0193] Based on each frame of IMU data, determine the joint angles of multiple skeletal key points of the target object in each frame;

[0194] The standard bone point information for each frame is determined based on the joint angles of multiple skeletal key points of the target object in each frame.

[0195] In another possible implementation, estimation module 23 is also used for:

[0196] Convert video data into multi-frame two-dimensional images;

[0197] Based on each frame of two-dimensional image, two-dimensional skeletal point information of each frame is obtained through two-dimensional pose estimation. The two-dimensional skeletal point information of each frame includes the two-dimensional coordinate data of multiple skeletal key points of the target object in each frame.

[0198] The two-dimensional skeleton point information of each frame is combined to obtain a two-dimensional skeleton point sequence.

[0199] In another possible implementation, the apparatus further includes a loss determination module, for:

[0200] The loss value is determined based on the 3D skeleton point sequence and the labeled skeleton point sequence. The labeled skeleton point sequence includes the 3D coordinate data of multiple skeletal key points of the benchmark target object. The loss value is used to indicate the difference between the 3D skeleton point sequence and the lost labeled skeleton point sequence.

[0201] In another possible implementation, the target data also includes motion capture data synchronized with the video data and IMU data, and the device further includes: a tag determination module for:

[0202] The sequence of skeletal points for the labels is determined based on motion capture data, which includes motion information of the target object captured by motion capture equipment.

[0203] It should be noted that the above embodiments only illustrate the division of the above functional modules when implementing the device. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0204] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0205] This disclosure also provides a computing device, the computing device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0206] This disclosure also provides a non-volatile computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method.

[0207] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of a computing device, the processor in the computing device performs the above-described method.

[0208] Figure 9 This is a block diagram illustrating an apparatus 1900 according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device, and apparatus 1900 may be used to perform the methods described above. (Refer to...) Figure 9The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0209] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0210] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0211] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0212] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0213] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0214] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0215] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0216] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0217] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0218] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0219] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A three-dimensional pose estimation method based on a fine-tuned standard model, characterized in that, The method includes: Acquire target data, which includes synchronized video data, inertial measurement unit (IMU) data, and motion capture data. The video data includes visual information of the target object, the IMU data includes inertial information of the target object measured by the IMU, and the motion capture data includes motion information of the target object captured by a motion capture device. Based on the video data, a two-dimensional skeleton point sequence is obtained through two-dimensional pose estimation. The two-dimensional skeleton point sequence includes the two-dimensional coordinate data of multiple skeletal key points of the target object. Based on the IMU data, a standard skeletal point sequence is obtained through kinematic calculations. The standard skeletal point sequence includes the three-dimensional coordinate data of multiple skeletal key points of the target object. Based on the two-dimensional skeleton point sequence and the standard skeleton point sequence, a three-dimensional skeleton point sequence is determined, wherein the three-dimensional skeleton point sequence includes the predicted three-dimensional coordinate data of multiple skeletal key points of the target object; The step of determining the three-dimensional skeleton point sequence based on the two-dimensional skeleton point sequence and the standard skeleton point sequence includes: The first bone features corresponding to the two-dimensional bone point sequence and the second bone features corresponding to the standard bone point sequence are extracted by a dual-stream spatiotemporal transformer neural network. The first skeletal point feature and the second skeletal point feature are concatenated to obtain the concatenated feature; The spliced ​​features are weighted and calculated and mapped to a two-dimensional space to obtain the target skeleton point features; The target skeletal point features are regressed into three-dimensional space using a multilayer perceptron (MLP) to obtain the three-dimensional skeletal point sequence. The method further includes: Determine the tag skeleton point sequence based on the motion capture data; Based on the 3D skeleton point sequence and the labeled skeleton point sequence, a loss value is determined, and a supervised network is trained to optimize the network parameters. The labeled skeleton point sequence includes 3D coordinate data of multiple skeletal key points of the benchmark target object. The loss value is used to indicate the difference between the 3D skeleton point sequence and the lost labeled skeleton point sequence.

2. The method according to claim 1, characterized in that, The IMU data includes multiple frames of IMU data. The step of obtaining a standard skeleton point sequence based on the IMU data through kinematic calculations includes: Based on the IMU data of each frame, standard skeletal point information of each frame is obtained through kinematic calculations. The standard skeletal point information of each frame includes the three-dimensional coordinate data of multiple skeletal key points of the target object in each frame. The standard skeleton point information of each frame is combined to obtain the standard skeleton point sequence.

3. The method according to claim 2, characterized in that, The step of obtaining standard skeleton point information for each frame through kinematic calculations based on the IMU data of each frame includes: Based on the IMU data of each frame, determine the joint angles of multiple skeletal key points of the target object in each frame; The standard bone point information for each frame is determined based on the joint angles of multiple skeletal key points of the target object in each frame.

4. A three-dimensional attitude estimation device based on a fine-tuned standard model, characterized in that, The device includes: The acquisition module is used to acquire target data, which includes synchronized video data, inertial measurement unit (IMU) data, and motion capture data. The video data includes visual information of the target object, the IMU data includes inertial information of the target object measured by the IMU, and the motion capture data includes motion information of the target object captured by a motion capture device. The estimation module is used to obtain a two-dimensional skeleton point sequence by two-dimensional pose estimation based on the video data. The two-dimensional skeleton point sequence includes two-dimensional coordinate data of multiple skeleton key points of the target object. The calculation module is used to obtain a standard skeleton point sequence through kinematic calculation based on the IMU data. The standard skeleton point sequence includes the three-dimensional coordinate data of multiple skeletal key points of the target object. The determination module is used to determine a three-dimensional bone point sequence based on the two-dimensional bone point sequence and the standard bone point sequence, wherein the three-dimensional bone point sequence includes the predicted three-dimensional coordinate data of multiple skeletal key points of the target object; The determining module is further configured to: The first bone features corresponding to the two-dimensional bone point sequence and the second bone features corresponding to the standard bone point sequence are extracted by a dual-stream spatiotemporal transformer neural network. The first skeletal point feature and the second skeletal point feature are concatenated to obtain the concatenated feature; The spliced ​​features are weighted and calculated and mapped to a two-dimensional space to obtain the target skeleton point features; The target skeletal point features are regressed into three-dimensional space using a multilayer perceptron (MLP) to obtain the three-dimensional skeletal point sequence. The device further includes: The label determination module is used to determine the label skeleton point sequence based on the motion capture data; The loss determination module is used to determine a loss value based on the three-dimensional skeleton point sequence and the labeled skeleton point sequence, and to supervise the training of the network to optimize the network parameters. The labeled skeleton point sequence includes three-dimensional coordinate data of multiple skeletal key points of the benchmark target object. The loss value is used to indicate the difference between the three-dimensional skeleton point sequence and the lost labeled skeleton point sequence.

5. A three-dimensional attitude estimation device based on a fine-tuned standard model, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 3 when executing instructions stored in the memory.

6. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 3.

Images