Attitude detection method, training method and device of attitude detection model, equipment, storage medium and computer program product

Abstract

The invention provides a posture detection method, a posture detection model training method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring image data of a target object at a plurality of visual angles, and acquiring a two-dimensional attitude sequence of the target object at each visual angle; performing three-dimensional feature extraction processing on the two-dimensional attitude sequence of the target object at each view angle to obtain three-dimensional attitude features of the target object at each view angle; aggregating the three-dimensional attitude features of the target object at each view angle into a multi-view-angle feature; extracting common features among the three-dimensional attitude features of the target object at each view angle; performing feature fusion processing on the multi-view features and the common features to obtain fused features; and performing three-dimensional attitude prediction processing based on the fused features to obtain a target three-dimensional attitude sequence of the target object at each view angle. According to the invention, the accuracy of three-dimensional attitude detection can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a posture detection method, a training method for a posture detection model, an apparatus, a device, a storage medium, and a computer program product. Background Technology

[0002] In related technologies, single-view methods are prone to inaccurate pose prediction due to the inherent depth blur and self-occlusion problems of two-dimensional images or videos. Related technologies use multi-view information for unsupervised learning to obtain more accurate pose estimation by constraining the consistency of the 3D pose predicted from different views. However, simply adding consistency loss makes it difficult to explicitly learn the correlation between different views, resulting in poor accuracy of 3D pose detection. Summary of the Invention

[0003] This application provides a posture detection method, a posture detection model training method, an apparatus, a device, a storage medium, and a computer program product, which can improve the accuracy of three-dimensional posture detection.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides an attitude detection method, the method comprising:

[0006] Acquire image data of the target object from multiple perspectives, and acquire a two-dimensional pose sequence of the target object from each perspective;

[0007] The three-dimensional feature extraction process is performed on the two-dimensional pose sequence of the target object in each of the said views to obtain the three-dimensional pose features of the target object in each of the said views;

[0008] The three-dimensional pose features of the target object in each of the aforementioned viewpoints are aggregated into multi-view features;

[0009] Extract the common features of the target object among the three-dimensional pose features of each viewpoint;

[0010] The multi-view features and the common features are fused to obtain fused features;

[0011] Based on the fusion features, a three-dimensional pose prediction process is performed to obtain the target object's target three-dimensional pose sequence in each of the aforementioned viewpoints.

[0012] This application provides a method for training a pose detection model, the method comprising:

[0013] Obtain sample image data of the sample object from multiple perspectives, and perform the following processing using the pose detection model to be trained:

[0014] Obtain two-dimensional pose sequence samples of the sample object from each viewpoint;

[0015] The sample object is subjected to three-dimensional feature extraction processing on the two-dimensional pose sequence of each viewpoint to obtain the three-dimensional pose feature sample of the sample object in each viewpoint;

[0016] The sample object's three-dimensional pose feature samples from each of the aforementioned viewpoints are aggregated into multi-view feature samples;

[0017] Extract the common feature samples of the sample object among the three-dimensional pose feature samples of each viewpoint;

[0018] The multi-view feature samples and the common feature samples are subjected to feature fusion processing to obtain fused feature samples;

[0019] Based on the fused feature samples, a 3D pose prediction process is performed to obtain the target 3D pose sequence samples of the sample object in each of the aforementioned viewpoints;

[0020] The combined loss value is determined based on the two-dimensional attitude sequence sample, the three-dimensional attitude feature sample, and the target three-dimensional attitude sequence sample;

[0021] The parameters of the pose detection model to be trained are updated based on the combined loss value to obtain the trained pose detection model.

[0022] The above plan also includes:

[0023] The target 3D pose sequence sample includes standard 3D pose samples. Determining a first projection loss value based on the target 3D pose sequence sample and the 2D pose sequence sample includes:

[0024] The standard three-dimensional pose sample is subjected to two-dimensional spatial projection processing to obtain a projected two-dimensional pose sequence;

[0025] The two-dimensional pose sequence samples are normalized to obtain a normalized two-dimensional pose sequence.

[0026] Obtain the ratio of the projected two-dimensional attitude sequence to the regularization matrix of the projected two-dimensional attitude sequence;

[0027] Obtain the first difference between the ratio and the normalized two-dimensional pose sequence, and perform regularization on the first difference to obtain the first projection loss value.

[0028] The above plan also includes:

[0029] The target 3D pose sequence includes standard 3D pose samples. Based on the target 3D pose sequence samples and the 2D pose sequence samples, a multi-view consistency loss value is determined, including:

[0030] A first viewpoint and a second viewpoint are determined from the multiple viewpoints, and the standard three-dimensional pose sample of the second viewpoint is converted to the camera space corresponding to the first viewpoint to obtain the converted target three-dimensional pose sequence sample.

[0031] Two-dimensional spatial projection processing is performed on the standard three-dimensional pose sample of the first viewpoint and the transformed target three-dimensional pose sequence sample of the second viewpoint to obtain the projected two-dimensional pose sequence of each viewpoint.

[0032] A second projection loss value for each viewpoint is determined based on the projected two-dimensional pose sequence and the two-dimensional pose sequence sample for each viewpoint;

[0033] The sum of the second projection loss values ​​for each of the aforementioned viewpoints is taken as the multi-view consistency loss value.

[0034] The above plan also includes:

[0035] The target 3D attitude sequence sample includes camera rotation attitude samples. The camera rotation loss value is determined based on the target 3D attitude sequence sample and the 2D attitude sequence sample, including:

[0036] A first viewpoint and a second viewpoint are determined from the multiple viewpoints, and the camera rotation posture sample of the second viewpoint is converted to the camera space corresponding to the first viewpoint to obtain the converted camera posture sample.

[0037] Two-dimensional spatial projection processing is performed on the camera rotation posture samples of the first viewpoint and the transformed camera posture samples of the second viewpoint to obtain the projected two-dimensional posture sequence for each viewpoint.

[0038] A third projection loss value for each viewpoint is determined based on the projected two-dimensional pose sequence for each viewpoint and the sample of the two-dimensional pose sequence.

[0039] The sum of the third projection loss values ​​for each of the stated viewpoints is used as the camera rotation loss value.

[0040] This application provides an attitude detection device, the device comprising:

[0041] The data acquisition module is used to acquire image data of the target object from multiple perspectives and to acquire the two-dimensional pose sequence of the target object in each perspective.

[0042] The data processing module is used to perform three-dimensional feature extraction processing on the two-dimensional pose sequence of the target object in each of the viewpoints to obtain the three-dimensional pose features of the target object in each of the viewpoints.

[0043] The data processing module is also used to aggregate the three-dimensional pose features of the target object in each of the said views into multi-view features;

[0044] The data processing module is also used to extract common features of the target object among the three-dimensional pose features of each viewpoint;

[0045] The data processing module is also used to perform feature fusion processing on the multi-view features and the common features to obtain fused features;

[0046] The data processing module is also used to perform three-dimensional pose prediction processing based on the fused features to obtain the target three-dimensional pose sequence of the target object in each of the aforementioned viewpoints.

[0047] This application provides a training device for a pose detection model, the device comprising:

[0048] The data acquisition module is used to acquire sample image data of the sample object from multiple perspectives and perform the following processing through the pose detection model to be trained;

[0049] The training module is used to obtain two-dimensional pose sequence samples of the sample object at each viewpoint;

[0050] The training module is also used to perform three-dimensional feature extraction processing on the two-dimensional pose sequence of the sample object in each of the viewpoints to obtain the three-dimensional pose feature sample of the sample object in each of the viewpoints.

[0051] The training module is also used to aggregate the three-dimensional pose feature samples of the sample object in each of the viewpoints into multi-view feature samples;

[0052] The training module is also used to extract common feature samples of the sample objects among the three-dimensional pose feature samples of each viewpoint;

[0053] The training module is also used to perform feature fusion processing on the multi-view feature samples and the common feature samples to obtain fused feature samples;

[0054] The training module is also used to perform three-dimensional pose prediction processing based on the fused feature samples to obtain the target three-dimensional pose sequence samples of the sample object in each of the viewpoints;

[0055] The training module is also used to determine a combined loss value based on the two-dimensional pose sequence samples, the three-dimensional pose feature samples, and the target three-dimensional pose sequence samples;

[0056] The training module is also used to update the parameters of the pose detection model to be trained based on the combined loss value, so as to obtain the trained pose detection model.

[0057] This application provides an electronic device, the electronic device comprising:

[0058] Memory is used to store executable instructions or computer programs.

[0059] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the attitude detection method or the attitude detection model training method provided in the embodiments of this application.

[0060] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the attitude detection method or the attitude detection model training method provided in this application.

[0061] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the attitude detection method or the attitude detection model training method provided in this application.

[0062] The embodiments of this application have the following beneficial effects:

[0063] By acquiring the two-dimensional pose sequence of the target object from each viewpoint and performing three-dimensional feature extraction on the two-dimensional pose sequence, three-dimensional pose features of each viewpoint are obtained, realizing the improvement from two-dimensional pose to three-dimensional pose. By aggregating the three-dimensional pose of each viewpoint into multi-view features, the three-dimensional poses of different viewpoints are explicitly fused. By acquiring the common features of the target object in each viewpoint, the correlation between the three-dimensional poses of different viewpoints is explicitly learned. Then, by further fusing the multi-view features and common features, the target three-dimensional pose sequence of each viewpoint is obtained, achieving the beneficial effect of explicitly learning the correlation between different viewpoints and improving the accuracy of three-dimensional pose detection. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the attitude detection system architecture provided in the embodiments of this application;

[0065] Figure 2A This is a first structural diagram of the server provided in an embodiment of this application;

[0066] Figure 2B This is a schematic diagram of the second structure of the server provided in an embodiment of this application;

[0067] Figure 3 This is a schematic diagram of the structure of the attitude detection model provided in the embodiments of this application;

[0068] Figure 4A This is a first flowchart illustrating the attitude detection method provided in this application embodiment;

[0069] Figure 4B This is a second flowchart illustrating the attitude detection method provided in the embodiments of this application;

[0070] Figure 4C This is a schematic diagram of the third process of the attitude detection method provided in the embodiments of this application;

[0071] Figure 4D This is a schematic diagram of the fourth process of the attitude detection method provided in the embodiments of this application;

[0072] Figure 4E This is a schematic diagram of the fifth process of the attitude detection method provided in the embodiments of this application;

[0073] Figure 4F This is a schematic diagram of the sixth process of the attitude detection method provided in the embodiments of this application;

[0074] Figure 4G This is a schematic diagram of the seventh process of the attitude detection method provided in the embodiments of this application;

[0075] Figure 4H This is the eighth flowchart of the attitude detection method provided in the embodiments of this application;

[0076] Figure 4I This is a ninth flowchart illustrating the attitude detection method provided in the embodiments of this application;

[0077] Figure 5A This is a schematic diagram of the first process of the training method for the pose detection model provided in the embodiments of this application;

[0078] Figure 5B This is a schematic diagram of the second process of the training method for the pose detection model provided in the embodiments of this application;

[0079] Figure 5C This is a schematic diagram of the third process of the training method for the pose detection model provided in the embodiments of this application;

[0080] Figure 5D This is a schematic diagram of the fourth process of the training method for the pose detection model provided in the embodiments of this application;

[0081] Figure 5E This is a schematic diagram of the fifth step of the training method for the pose detection model provided in the embodiments of this application;

[0082] Figure 5F This is a schematic diagram of the sixth step in the training method of the pose detection model provided in the embodiments of this application;

[0083] Figure 5G This is a schematic diagram of the seventh step in the training method of the pose detection model provided in the embodiments of this application;

[0084] Figure 5H This is a schematic diagram of the eighth step in the training method of the pose detection model provided in the embodiments of this application;

[0085] Figure 6 This is a schematic diagram illustrating the application process of the pose detection method in the animation asset reconstruction scenario provided in the embodiments of this application;

[0086] Figure 7A This is a first schematic diagram of an optional structure of the attitude detection model provided in the embodiments of this application;

[0087] Figure 7B This is a second schematic diagram of an optional structure of the attitude detection model provided in the embodiments of this application;

[0088] Figure 7C This is a third schematic diagram of an optional structure of the attitude detection model provided in the embodiments of this application;

[0089] Figure 8A This is a first schematic diagram of the attitude detection effect of the attitude detection method provided in the embodiments of this application;

[0090] Figure 8B This is a second schematic diagram illustrating the attitude detection effect of the attitude detection method provided in this application embodiment;

[0091] Figure 8C This is a third schematic diagram illustrating the attitude detection effect of the attitude detection method provided in this application embodiment;

[0092] Figure 9A This is a schematic diagram illustrating the impact of different amounts of viewpoint information provided in the embodiments of this application on the performance of the pose detection model;

[0093] Figure 9B This is a schematic diagram illustrating the impact of image data with different sequence lengths as input in an embodiment of this application on the performance of the pose detection model.

[0094] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0096] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0097] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0098] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0099] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0100] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0101] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0102] 1) Image data from multiple perspectives refers to a collection of images of the same target object taken at the same time from different angles or positions. These images provide comprehensive information about the target object, which helps to better understand and analyze the target object's pose information. The distribution of multiple perspectives can be the positional distribution of various perspectives in three-dimensional space. An ideal distribution of perspective positions can cover all aspects of the target object to provide the most comprehensive information.

[0103] 2) Key points, also known as joints, are specific points in a target object. These points play an important role in defining the structure or motion of the target object. For example, in human posture estimation, key points refer to specific points on a person that can be identified, such as the head, shoulders, elbows, wrists, hips, knees, and ankles.

[0104] 3) Two-dimensional pose sequence refers to the two-dimensional pose detection result corresponding to the image data of each viewpoint (e.g., multiple frames of images corresponding to a viewpoint). It is a sequence composed of a series of two-dimensional key point coordinates corresponding to each frame of the image data. The position of each key point can be represented by a coordinate pair (x, y) on the two-dimensional plane, where x is the horizontal coordinate and y is the vertical coordinate.

[0105] 4) The target 3D pose sequence refers to the final 3D pose detection result of each viewpoint image data (including multiple frames) predicted by combining the correlation between image data from multiple viewpoints. It is a sequence composed of the 3D coordinates of key points corresponding to each frame of the image data. The target 3D pose sequence includes standard 3D pose and camera rotation pose.

[0106] 5) The intermediate 3D pose sequence is the 3D pose detection result predicted based on image data from a single viewpoint (e.g., multiple frames of images corresponding to a viewpoint). It is a sequence composed of the 3D coordinates of key points corresponding to each frame of the image data. The intermediate 3D pose sequence is the intermediate data for determining the target's 3D pose sequence, so it is called the intermediate 3D pose sequence here.

[0107] 6) Standard 3D pose refers to a standardized pose formed by locating and arranging key points in 3D space. For example, the standard 3D pose of viewpoint 1 corresponds to the position of multiple key points (e.g., a specific part of the human body) in 3D space, represented as: viewpoint 1 [(head, (x1, y1, z1)), (neck, (x2, y2, z2)) ... (ankle, (x3, y3, z3))].

[0108] 7) Camera rotation pose, also known as 6D camera rotation pose or 6-DOF camera rotation pose, refers to the camera's rotation in 3D space with six degrees of freedom. These six degrees of freedom include three translational degrees of freedom and three rotational degrees of freedom, which are used to transform the standard 3D pose corresponding to the key points into camera space.

[0109] 8) Transformer: A deep learning model based on self-attention mechanism, primarily used for sequence-to-sequence natural language processing tasks, such as machine translation and text summarization. The Transformer model mainly consists of an encoder and a decoder, responsible for processing the input sequence and generating the output sequence, respectively. The self-attention mechanism allows the Transformer to capture long-range dependencies in a sequence without using recurrent neural networks or convolutional neural network structures.

[0110] In related technologies, single-view methods are prone to inaccurate pose prediction due to the inherent depth blur and self-occlusion problems of two-dimensional images or videos. Related technologies use multi-view information for unsupervised learning to obtain more accurate pose estimation by constraining the consistency of the 3D pose predicted from different views. However, simply adding consistency loss makes it difficult to explicitly learn the correlation between different views, resulting in poor accuracy of 3D pose detection.

[0111] Inherent depth ambiguity arises from the lack of parallax information between multiple viewpoints in single-view data, resulting in insufficient clarity of depth information. Depth ambiguity refers to the camera's inability to accurately determine the depth relationships between different objects in a scene, especially when background and foreground objects have similar colors or textures. This ambiguity makes estimating the true distances of different parts of the human body difficult, thus affecting the accuracy of 3D pose detection. Self-occlusion occurs in single-view data when one part of the human body (corresponding to the target object) is occluded by another part of the body. For example, when a hand is placed in front of the body, the camera cannot directly see the hand's position. This occlusion complicates vision-based 3D pose detection because the model needs to distinguish between occluded and unoccluded parts.

[0112] This application provides a posture detection method, a posture detection model training method, an apparatus, a device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of 3D posture detection.

[0113] The following describes exemplary applications of the devices provided in the embodiments of this application. The electronic devices provided in the embodiments of this application can be implemented as various types of terminal devices such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals, or they can be implemented as servers.

[0114] See Figure 1 , Figure 1 This is a schematic diagram of the attitude detection system architecture provided in the embodiments of this application. Figure 1 The system involves server 100, terminal device 200, and network 300. Terminal device 200 is connected to server 100 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0115] In some embodiments, the present application embodiments can be implemented collaboratively by a server and a terminal device. For example, the server 100 obtains a pose detection model through the pose detection model training method provided in the present application embodiments, and the terminal device 200 sends image data of the target object from multiple perspectives to the server 100. The server 100 obtains the target three-dimensional pose sequence of the target object in each perspective through the pose detection method provided in the present application embodiments, and sends the target three-dimensional pose sequence of the target object in each perspective to the terminal device 200.

[0116] Here, server 100 can be a single server. In this case, the pose detection method and the pose detection model training method provided in this application embodiment can be implemented by the same server. Server 100 can also be a cluster of servers. In the case that server 100 is a server cluster, the pose detection method and the pose detection model training method provided in this application embodiment can be implemented by different servers. This application embodiment does not impose any limitations on this.

[0117] In other embodiments, the embodiments of this application can be implemented by a terminal device alone. The terminal device 200 sends a request to the server 100, the server 100 receives the request and sends an attitude detection model for performing the attitude detection method provided in the embodiments of this application to the terminal device 200. The terminal device 200 receives the attitude detection model sent by the server and downloads it to its local machine, and obtains the target three-dimensional attitude sequence of the target object from each viewpoint through the attitude detection model.

[0118] In some embodiments, the terminal device or server can implement the attitude detection method or attitude detection model training method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application program, module, or plugin.

[0119] Taking the server for attitude detection as an example, see [link to relevant documentation]. Figure 2A , Figure 2AThis is a first structural diagram of the server provided in an embodiment of this application. Figure 2A The server 100-1 shown includes at least one processor 110-1, memory 130-1, and at least one network interface 120-1. The various components in server 100-1 are coupled together via a bus system 140-1. It is understood that the bus system 140-1 is used to implement communication between these components. In addition to a data bus, the bus system 140-1 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2A The general designated all buses as Bus System 140-1.

[0120] The processor 110-1 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0121] The memory 130-1 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 130-1 may optionally include one or more storage devices physically located away from the processor 110-1.

[0122] The memory 130-1 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 130-1 described in this application embodiment is intended to include any suitable type of memory.

[0123] In some embodiments, memory 130-1 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0124] Operating system 131-1 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0125] The network communication module 132-1 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 120-1, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0126] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A An attitude detection device 133 stored in memory 130-1 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a data acquisition module 1331 and a data processing module 1332. These modules are logically linked and can therefore be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0127] Taking the server used for training the pose detection model as an example, see [link to relevant documentation]. Figure 2B , Figure 2B This is a schematic diagram of the second structure of the server provided in an embodiment of this application. Figure 2B The server 100-2 shown includes at least one processor 110-2, memory 130-2, and at least one network interface 120-2. The various components in server 100-2 are coupled together via a bus system 140-2. It is understood that the bus system 140-2 is used to implement communication between these components. In addition to a data bus, the bus system 140-2 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2B All buses are labeled as Bus System 140-2. For detailed explanations of Processor 110-2 and Memory 130-2, please refer to the above text; they will not be repeated here.

[0128] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2B A training device 134 for an attitude detection model stored in memory 130-2 is shown. This device can be software in the form of programs or plugins, and includes the following software modules: a data acquisition module 1341 and a training module 1342. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0129] Understandably, the above example uses the fact that server 100-1 and server 100-2 are different servers. As an alternative, the server used for pose detection and training the pose detection model can also be implemented as the same server.

[0130] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the attitude detection method or the attitude detection model training method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0131] See Figure 3 , Figure 3 This is a schematic diagram of the posture detection model provided in this application embodiment. The posture detection model may include a pre-trained two-dimensional posture extraction module, a single-view module, a cross-view module, and a regression head. The two-dimensional posture extraction module can be used to obtain the two-dimensional posture sequence of the target object in each view from image data of the target object from multiple views. The single-view module can be used to perform three-dimensional feature extraction processing on the two-dimensional posture sequence of the target object in each view to obtain the three-dimensional posture features of the target object in each view. The cross-view module can be used to aggregate the three-dimensional posture features of the target object in each view into multi-view features, extract the common features between the three-dimensional posture features of the target object in each view, and perform feature fusion processing on the multi-view features and common features to obtain fused features. The regression head can be used to perform three-dimensional posture prediction processing based on the fused features to obtain the target three-dimensional posture sequence of the target object in each view.

[0132] The posture detection method provided in this application will be described below with reference to exemplary applications and implementations of the server provided in the embodiments of this application, taking the server as the execution subject. See also Figure 4A , Figure 4A This is a first flowchart illustrating the attitude detection method provided in this application embodiment, which will be combined with... Figure 4A The steps shown are explained.

[0133] In step 101, image data of the target object from multiple perspectives are acquired, and the two-dimensional pose sequence of the target object in each perspective is acquired.

[0134] In some embodiments, image data from multiple perspectives of a target object (e.g., a game character in a game scene) refers to multiple images or videos of the same target object captured from different angles within the same time period, wherein one perspective corresponds to one image data and the image data from multiple perspectives are aligned in time.

[0135] In some embodiments, see Figure 4B , Figure 4A The step 101 shown, which involves obtaining the two-dimensional pose sequence of the target object in each viewpoint, can be achieved by performing the following steps 1011 to 1013 for multiple frames of images in each viewpoint, as explained in detail below.

[0136] In step 1011, image feature extraction processing is performed on each frame of the image from the viewpoint to obtain the image features of each frame.

[0137] In some embodiments, image feature extraction processing can be performed on each frame of the target object image using deep learning models such as Convolutional Neural Network (CNN) to obtain image features. Alternatively, image feature extraction processing can be performed on each frame of the input image using a Transformer Encoder structure to obtain image features. Here, the Transformer Encoder structure can be stacked in multiple layers for image feature extraction processing. The embodiments of this application do not limit the specific image feature extraction processing method.

[0138] In step 1012, key point detection processing is performed based on the image features of the target object in each frame image to obtain the coordinates of multiple key points of the target object in each frame image.

[0139] In some embodiments, a pre-trained two-dimensional pose extraction module (such as the Visual Transformer Baseline (ViTPose) for human pose estimation, or the Cascaded Pyramid Network (CPN) for multi-person pose estimation) can be used to perform keypoint detection processing based on the image features of the target object in each frame of the image, thereby obtaining the coordinates of multiple keypoints of the target object in each frame of the image.

[0140] For example, taking the 2D pose extraction module obtained through pre-trained ViTPose as an example, ViTPose performs keypoint detection processing on the image features obtained by the encoder (corresponding to the Transformer Encoder structure mentioned above), that is, it decodes the image features through the decoder to obtain the coordinates of multiple key points of the target object in each frame of the image. The input image can be decoded through the Transformer Decoder structure to obtain the coordinates of multiple key points of the target object in each frame of the image. Here, the Transformer Decoder structure can be stacked in multiple layers for decoding processing.

[0141] For example, training ViTPose includes: selecting a training dataset containing labeled images of keypoints (e.g., joints in the human body) for training the ViTPose model; preprocessing the images in the training dataset, such as cropping, flipping, and rotating, to increase the diversity of the training data; extracting features from the images using an encoder; predicting keypoint coordinates from the extracted features using a decoder; training the ViTPose model using the labeled images in the dataset, i.e., selecting a suitable loss function, such as mean squared error or other regression loss functions, to measure the difference between the predicted keypoint coordinates and the true keypoint coordinates; adjusting the weights of the ViTPose model using an optimizer, such as stochastic gradient descent (SGD), to minimize the loss function; evaluating the model's performance using a validation set during or after training, with evaluation metrics including average precision (AP) and mean squared error (MSE); and adjusting the model's hyperparameters, such as learning rate, batch size, and number of iterations, based on the evaluation results to optimize the performance of the ViTPose model.

[0142] In step 1013, the coordinates of multiple key points of the target object in each frame image are combined to obtain a two-dimensional pose sequence of the target object from the viewpoint.

[0143] In some embodiments, the coordinates of multiple key points in each frame of the image data corresponding to the viewpoint are combined to obtain a two-dimensional pose sequence of the target object in the viewpoint. For example, the two-dimensional pose sequence corresponding to viewpoint 1 can be represented as: two-dimensional pose sequence of viewpoint 1 = [multiple key points of the first frame image], [multiple key points of the second frame image] ... [coordinates of multiple key points of the nth frame image]. The multiple key points of the first frame image of viewpoint 1 can be represented as: [key point 1 (head, two-dimensional coordinates (*, *)), key point 2 (neck, two-dimensional coordinates (*, *)) ... key point v (foot, two-dimensional coordinates (*, *))].

[0144] In other embodiments, the two-dimensional pose sequence of the target object in the viewpoint can be normalized, and subsequent processing steps can be performed based on the normalized two-dimensional pose sequence to avoid the problem of scale ambiguity. For example, the root joints (such as the pelvic joints and chest joints in the human pose) can be decentered as the central key points, and the coordinates of the decentered key points can be divided by the normalization matrix (such as the Frobenius norm) for scaling to obtain the normalized two-dimensional pose sequence of the target object in the viewpoint.

[0145] See also Figure 4A In step 102, the three-dimensional feature extraction process is performed on the two-dimensional pose sequence of the target object from each viewpoint to obtain the three-dimensional pose features of the target object from each viewpoint.

[0146] In some embodiments, the 3D pose features of each viewpoint include temporal features of the viewpoint and an intermediate 3D pose sequence. The intermediate 3D pose sequence of each viewpoint includes the 3D pose of the target object in the image data of the corresponding viewpoint. See [link to relevant documentation]. Figure 4C , Figure 4A Step 102 shown can be achieved by performing steps 1021 to 1023 for each viewpoint's two-dimensional pose sequence, as explained in detail below.

[0147] In step 1021, the two-dimensional poses in the two-dimensional pose sequence are embedded and encoded to obtain the embedded and encoded features of each two-dimensional pose.

[0148] In some embodiments, the two-dimensional pose (corresponding to multiple keypoint coordinates) in the two-dimensional pose sequence can be divided into multiple patches to be encoded according to a preset encoding block size by an embedding layer. The multiple patches to be encoded are then combined into an encoding sequence in sequence (e.g., from left to right, from top to bottom), and the encoding sequence is embedded to obtain the embedding encoding features of each two-dimensional pose.

[0149] For example, suppose the matrix of two-dimensional poses in a two-dimensional pose sequence (such as the matrix including the coordinate information of each keypoint) is 24×24 in size. If it is divided into 4×4 patches, the result will be (24 / 4). 2 = 36 Patches, and then arrange the 36 Patches in order from top left to bottom right to obtain the sequence to be encoded. The embodiments of this application do not limit the specific method of dividing Patches.

[0150] In step 1022, the position of the two-dimensional pose in the two-dimensional pose sequence is encoded to obtain the position encoding feature of each two-dimensional pose.

[0151] In some embodiments, positional encoding can be generated by a fixed algorithm, such as a combination of sine and cosine functions. For each position (e.g., the position of a keypoint in a two-dimensional pose) pos and keypoint i, the position encoding for even-numbered i uses a sine function, while the position encoding for odd-numbered i uses a cosine function. This application does not limit the specific implementation of positional encoding.

[0152] In step 1023, feature encoding is performed based on the embedding encoding features and position encoding features of each two-dimensional pose to obtain the three-dimensional pose features of the viewpoint.

[0153] In some embodiments, the three-dimensional pose features of each viewpoint include the temporal features of the viewpoint and the intermediate three-dimensional pose sequence. The embedding encoding features and position encoding features of each two-dimensional pose are concatenated to obtain two-dimensional pose features. The two-dimensional pose features are then subjected to fourth attention encoding processing (e.g., multi-head attention (MHA)) to obtain fourth attention encoding features. Based on the fourth attention encoding features and the two-dimensional pose features, feature mapping processing is performed to obtain temporal features. The temporal features are then subjected to three-dimensional pose mapping processing to obtain the intermediate three-dimensional pose sequence.

[0154] For example, fourth attention encoding of 2D pose features can be implemented as follows: For the input 2D pose features, a linear transformation is first performed to generate three matrices: a query vector (Q), a key vector (K), and a value vector (V). The linear transformation is performed using a learnable weight matrix W. Q W K and W VNext, for each attention head, the dot product of Q and K is calculated to obtain an attention score. Then, the attention score is normalized, for example, by applying a normalization function (such as the softmax function), transforming the attention score into a probability distribution. This attention probability distribution is then used to weight V, generating a new feature representation. Finally, the outputs of all attention heads are concatenated and subjected to a linear transformation to obtain a rich representation that includes the correlation between different positions in the 2D pose features—the fourth attention-encoded feature. Through the fourth attention encoding process, parallel computation using multiple attention heads improves computational efficiency. By learning in different representation subspaces, it considers the contextual information in the 2D pose features, capturing different patterns and structures within the 2D pose features, thereby enhancing the representational power of the fourth attention-encoded feature.

[0155] For example, feature mapping based on fourth attention encoding features and two-dimensional pose features can be achieved by concatenating the fourth attention encoding features and two-dimensional pose features to obtain the features to be mapped, and then performing feedforward mapping on the features to be mapped through a feed-forward neural network (FFN) layer to obtain temporal features.

[0156] Taking a multilayer perceptron (MLP) structure as an example of a feedforward neural network layer, firstly, the features to be mapped are linearly transformed through a first linear layer. The first linear layer can be a fully connected layer with a weight matrix of W1 and a bias vector of b1. Next, the linearly transformed features are nonlinearly transformed through activation functions (such as ReLU, Sigmoid, etc.) to obtain nonlinearly transformed features. Finally, the nonlinearly transformed features are linearly transformed through a second linear layer to obtain temporal features. The second linear layer can be a fully connected layer with a weight matrix of W2 and a bias vector of b2.

[0157] For example, a regression head can be used to perform 3D pose mapping on temporal features to obtain a pose sequence. The regression head can include one or more fully connected layers, or convolutional layers or other types of network structures. The regression head receives temporal features and outputs a pose sequence. Here, the regression head is obtained by training a loss function (such as mean squared error, mean absolute error, etc.) and the network parameters of the regression head in the pose detection model are updated through the backpropagation algorithm.

[0158] See also Figure 4A In step 103, the three-dimensional pose features of the target object in each viewpoint are aggregated into multi-view features.

[0159] In some embodiments, the 3D pose features of each viewpoint include temporal features of the viewpoint, which characterize the temporal correlation of 2D poses in a 2D pose sequence of the viewpoint. The 2D pose sequence of each viewpoint includes the 2D pose of the target object in the image data of the viewpoint (e.g., each image frame of the image data). See also Figure 4D , Figure 4A Step 103 shown can be implemented through steps 1031 to 1034, which will be explained in detail below.

[0160] In step 1031, the temporal features of each viewpoint are concatenated into combined temporal features.

[0161] In some embodiments, the temporal features of each viewpoint are concatenated into a combined temporal feature. For example, the combined temporal feature can be represented as: combined temporal feature = [temporal feature of viewpoint 1, temporal feature of viewpoint 2, ..., temporal feature of viewpoint n].

[0162] In step 1032, the combined temporal features are subjected to feature encoding processing to obtain temporal encoded features.

[0163] In some embodiments, the combined temporal features can be embedded and encoded to obtain combined temporal embedded features, the combined temporal features can be position encoded to obtain position encoded features, and the combined temporal embedded features and position encoded features can be concatenated to obtain temporal encoded features. The implementation of the embedding and encoding process can be found in the description of step 1021 above, and the implementation of the position encoding process can be found in the description of step 1022 above.

[0164] In step 1033, the temporal coding features are subjected to first attention coding processing to obtain first attention coding features.

[0165] In some embodiments, the temporal coding features can be processed by an attention mechanism (e.g., multi-head attention, cross attention, etc.) to obtain the first attention coding features. Here, the first attention coding process of the temporal coding features by multi-head attention can be referred to the description of step 1023 (corresponding to the fourth attention coding process) above, which will not be repeated here.

[0166] In step 1034, feature mapping is performed based on the first attention coding features and the temporal coding features to obtain multi-view features.

[0167] In some embodiments, the first attention-encoded features and the temporal-encoded features can be concatenated to obtain the features to be mapped. The features to be mapped are then processed by a feedforward neural network layer (FFN) to obtain multi-view features. Here, the implementation method of the feedforward mapping process can be found in the description of step 1023 above, and will not be repeated here.

[0168] Through steps 1031 to 1034, the combined temporal features are used as input. Since the combined temporal features contain richer information than the pose sequence, by aggregating joint-level features from all perspectives, each joint (i.e., keypoint) can integrate information from other joints or perspectives, thereby implicitly capturing multi-perspective information. By treating each joint from different perspectives as an input unit (token) and using attention encoding, the correlation between joints under different perspectives is effectively captured, thereby generating high-dimensional features that integrate perspective information.

[0169] See also Figure 4A In step 104, common features among the three-dimensional pose features of the target object in each viewpoint are extracted.

[0170] In some embodiments, the 3D pose features of each viewpoint include an intermediate 3D pose sequence for that viewpoint, which comprises the 3D pose of the target object in each frame of the image data. See also Figure 4E , Figure 4A Step 104 shown can be implemented through steps 1041 to 1044, which will be explained in detail below.

[0171] In step 1041, the intermediate three-dimensional pose sequences of each viewpoint are spliced ​​together to form a combined pose sequence.

[0172] In some embodiments, the intermediate three-dimensional pose sequences of each viewpoint are concatenated into a combined pose sequence. For example, the combined pose sequence can be represented as: combined pose sequence = [intermediate three-dimensional pose sequence of viewpoint 1, intermediate three-dimensional pose sequence of viewpoint 2, ... intermediate three-dimensional pose sequence of viewpoint n], and the intermediate three-dimensional pose sequence of viewpoint 1 can be represented as: intermediate three-dimensional pose sequence of viewpoint 1 = [three-dimensional pose of frame 1, three-dimensional pose of frame 2, ... three-dimensional pose of frame m].

[0173] In step 1042, the combined pose sequence is subjected to feature encoding processing to obtain pose encoding features.

[0174] In some embodiments, the combined pose sequence can be embedded and encoded to obtain combined pose embedded features, and the combined pose sequence can be position encoded to obtain position encoded features. The combined pose embedded features and position encoded features are then concatenated to obtain pose encoded features. The implementation of the embedding and encoding process can be found in the description of step 1021 above, and the implementation of the position encoding process can be found in the description of step 1022 above. Here, the position encoding process is an optional operation, and the embodiments of this application do not limit the specific method of feature encoding processing.

[0175] In step 1043, a second attention encoding process is performed based on the first learnable input and pose encoding features to obtain the first encoding features.

[0176] In some embodiments, see Figure 4F , Figure 4E The step 1043 shown can be implemented by the following steps 10431 to 10433, which will be explained in detail below.

[0177] In step 10431, the first learnable input is used as the query vector, the pose encoding features are used as the key vector and the value vector, and the first attention weight is determined based on the key vector and the query vector.

[0178] In some embodiments, the first learnable input can be used as the query vector (Q), and the pose encoded features can be used as the key vector (K) and value vector (V). The attention score is obtained by calculating the dot product of Q and K. Next, the attention score is normalized, for example by applying a normalization function (such as the softma function), so that the attention score becomes a probability distribution, and the first attention weight is obtained.

[0179] In step 10432, the value vector is weighted by the first attention weight to obtain the second attention feature.

[0180] In some embodiments, V is weighted using a first attention weight to generate a new feature representation, namely a second attention feature.

[0181] In step 10433, the second attention features are subjected to the first feature mapping process to obtain the first encoded features.

[0182] In some embodiments, the second attention features can be processed by a feedforward neural network layer (FFN) to obtain the first encoded features. Here, the implementation of the first feature mapping process can be found in the description of step 1023 (corresponding to feedforward mapping process) above, and will not be repeated here.

[0183] As an example, based on the first learnable input and pose encoding features, a second attention encoding process is performed to obtain the first encoded features, which can be expressed by formula (1):

[0184] E t′ =MLP(MCA(E) t E 3D E 3D )) (1)

[0185] Among them, E t ′ represents the first encoded feature, MLP represents the first feature mapping process, MCA(E) t E 3D E 3D ) represents the second attention feature, MCA represents multi-head cross-attention, E t E represents the first learnable input. t It is the known parameters learned during the training phase of the pose detection model. In the initial stage of pose detection model training, the first learnable input has an initial value of 0, E. 3D This represents the pose encoding feature.

[0186] In some embodiments, the first encoding feature includes multiple data dimensions. After obtaining the first encoding feature, the multiple data dimensions of the first encoding feature can be rearranged to obtain an updated first encoding feature. The updated first encoding feature replaces the original first encoding feature, and then the process proceeds to step 1044 below, where a third attention encoding process is performed based on the second learnable input and the first encoding feature to obtain the second encoding feature.

[0187] For example, the pose encoding features correspond to multiple data dimensions including the number of images in a single pose detection (BatchSize, B), the number of views (V), the length of the pose sequence for each view (F), the number of keypoints (V), and the number of feature dimensions (C). Assuming the vector shape corresponding to the pose encoding features is (B*V, JF, C), a second attention encoding process is performed based on the first learnable input and the pose encoding features. The vector shape of the first encoded features can be represented as (B*V, J, C). The second attention encoding process is used to aggregate the keypoint features at the same time point under different views. The feature vector at each time point is no longer a single point, but contains all the information about this time point learned from multiple views. The focus is on the features in the time dimension (corresponding to the data dimension changing from JF to J).

[0188] For example, the multiple data dimensions of the first encoded feature are rearranged (which can be achieved using the reshape function in Python). The vector shape of the updated first encoded feature can be represented as (B, V*J, C). The data dimension rearrangement (or reshaping operation) is used to aggregate the features of each keypoint from different perspectives, that is, to aggregate the features of each keypoint at all time points, thereby reducing the time dimension and preserving the perspective invariance of the keypoint. The focus is on the features of the perspective dimension, which corresponds to the data dimension changing from J to V*J, where V*J represents all keypoints from all perspectives. Then, the features of the perspective dimension (corresponding to the updated first encoded feature) are used as the input of step 1044 to obtain the common features.

[0189] See also Figure 4E In step 1044, a third attention encoding process is performed based on the second learnable input and the first encoding feature to obtain the second encoding feature, which is then used as a common feature.

[0190] In some embodiments, see Figure 4G , Figure 4E Step 1044 shown can be implemented through steps 10441 to 10443, which will be explained in detail below.

[0191] In step 10441, a second attention weight is determined based on the second learnable input and the first encoded features.

[0192] In some embodiments, the second learnable input is used as a query vector, the first encoded features are used as a key vector and a value vector, and the second attention weight is determined based on the key vector and the query vector. For specific implementation details, please refer to the description of step 10431 above.

[0193] In step 10442, the first encoded feature is weighted by the second attention weight to obtain the third attention feature.

[0194] In some embodiments, the first encoded features are weighted using a second attention weight to generate a new feature representation, namely a third attention feature.

[0195] In step 10443, the third attention feature is subjected to a second feature mapping process to obtain the second encoding feature.

[0196] In some embodiments, the third attention feature can be processed by a feedforward neural network layer (FFN) to obtain the second encoded feature. Here, the implementation of the second feature mapping process can be found in the description of step 1023 (corresponding to feedforward mapping process) above, and will not be repeated here.

[0197] As an example, based on the second learnable input and the first encoded features, a third attention encoding process is performed to obtain the second encoded features, which can be expressed by formula (2):

[0198]

[0199] in, MLP represents the second encoded feature (i.e., the common feature), MCA represents the first feature mapping process, and E represents the second encoded feature (i.e., the common feature). v E t′ E t′ ) represents the third attention feature, MCA represents multi-head cross-attention, E v E represents the second learnable input. v It is the known parameters learned during the pose detection model training phase. In the initial stage of pose detection model training, the second learnable input has an initial value of 0, E. t ′ represents the first coding feature.

[0200] Steps 1041 to 1044 enable the learning of common features of multi-view image data. This is because the action of the target object in 3D space should remain consistent under different viewpoints, that is, there are common features between these viewpoints. Therefore, through two-stage attention encoding processing, the common features between different viewpoints are mined, and the correlation between the three-dimensional poses of different viewpoints is explicitly learned, thereby improving the accuracy of pose detection.

[0201] See also Figure 4A In step 105, feature fusion processing is performed on multi-view features and common features to obtain fused features.

[0202] In some embodiments, see Figure 4H , Figure 4A Step 105 shown can be implemented through steps 1051 to 1053, which will be explained in detail below.

[0203] In step 1051, the multi-view features are weighted using the first learnable weights to obtain the first weighted features.

[0204] In some embodiments, the multi-view features are weighted using a first learnable weight to generate a new feature representation, namely the first weighted feature.

[0205] In step 1052, the common features are weighted by the second learnable weights to obtain the second weighted features.

[0206] In some embodiments, common features are weighted using a second learnable weight to generate a new feature representation, namely the second weighted feature.

[0207] In step 1053, the sum of the first weighted feature and the second weighted feature is used as the fusion feature.

[0208] For example, feature fusion processing is performed on multi-view features and common features to obtain fused features, which can be expressed by formula (3):

[0209]

[0210] Where * indicates element-wise multiplication, and α represents the first learnable weight. This represents multi-view features, where β represents the second learnable weight. Indicates shared characteristics.

[0211] In some embodiments, the 3D pose features of each viewpoint include temporal features of the viewpoint and intermediate 3D pose sequences, see [link to relevant documentation]. Figure 4I Before performing feature fusion processing on multi-view features and common features to obtain fused features, the following steps 1054 to 1056 can be performed, which are explained in detail below.

[0212] In step 1054, the pose encoding features and the combined temporal features are concatenated to obtain the concatenated features. The pose encoding features are obtained by concatenating the intermediate three-dimensional pose sequences of each viewpoint into a combined pose sequence and performing feature encoding on the combined pose sequence. The combined temporal features are obtained by concatenating the temporal features of each viewpoint.

[0213] In some embodiments, the pose coding features and the combined temporal features are concatenated into a concatenated feature. Here, the method for obtaining the pose coding features can be found in the description of steps 1041 to 1042 above, and the method for obtaining the combined temporal features can be found in the description of step 1031 above.

[0214] In step 1055, the splicing features are weighted using pre-trained weight parameters to obtain weighted splicing features.

[0215] In some embodiments, the concatenated features are weighted using weight parameters learned during the model training phase to obtain weighted concatenated features.

[0216] In step 1056, the weighted splicing features are subjected to nonlinear mapping processing to obtain the first learnable weight and the second learnable weight.

[0217] For example, the first and second learnable weights can be obtained by formula (4):

[0218] a,β=softmax(W*Concat(Z,E)) (4)

[0219] Here, softmax corresponds to nonlinear mapping processing, W corresponds to pre-trained weights, Concat represents concatenation, Z represents combining temporal features, and E represents pose encoding features.

[0220] See also Figure 4A In step 106, three-dimensional pose prediction processing is performed based on the fused features to obtain the target three-dimensional pose sequence of the target object in each view.

[0221] In some embodiments, a regression head can be used to perform 3D pose prediction processing on the fused features to obtain the target 3D pose sequence of the target object in each view. The regression head may include one or more fully connected layers, or convolutional layers or other types of network structures. The regression head receives the fused features and outputs the target 3D pose sequence of the target object in each view. Here, the regression head may be obtained by training a loss function (such as mean squared error, mean absolute error, etc.) and the network parameters of the regression head in the pose detection model are updated by the backpropagation algorithm.

[0222] In some embodiments, the target object's target 3D pose sequence at each viewpoint includes the standard 3D pose of each keypoint and the camera rotation pose.

[0223] Taking the standard 3D pose of a key point as an example, the standard 3D pose corresponds to the position of a key point (e.g., a specific part of the human body) in 3D space, which is represented by the x, y, and z coordinates of the key point.

[0224] For example, camera rotation attitude refers to the camera's rotation in 3D space with six degrees of freedom, including three translational degrees of freedom and three rotational degrees of freedom. This is used to transform the standard 3D pose corresponding to keypoints into camera space. For instance, suppose the camera first rotates θx degrees around the X-axis, then θy degrees around the Y-axis, and finally θz degrees around the Z-axis. Simultaneously, the camera translates dx along the X-axis, dy along the Y-axis, and dz along the Z-axis. Then the rotation matrix R can be represented as R = Rz(θz)Ry(θy)Rx(θx), and the translation vector t can be represented as t = [dx, dy, dz]. T The transformation relationship can be expressed as: T C =R*T+t, where T represents the standard three-dimensional attitude, T C This represents the standard 3D pose in camera space. 6DOF camera rotation provides a continuous representation of 3D rotation in 6D space, which is more suitable for neural network learning compared to the discontinuous axis-angle representation along the rotation axis.

[0225] Through steps 101 to 106, the two-dimensional pose sequence of the target object in each viewpoint is obtained, and three-dimensional feature extraction processing is performed on the two-dimensional pose sequence to obtain the three-dimensional pose features of each viewpoint, realizing the improvement from two-dimensional pose to three-dimensional pose. By aggregating the three-dimensional pose of each viewpoint into multi-view features, the three-dimensional poses of different viewpoints are explicitly fused. By obtaining the common features of the target object in each viewpoint, the correlation between the three-dimensional poses of different viewpoints is explicitly learned. Then, by further fusing the multi-view features and common features, the target three-dimensional pose sequence of each viewpoint is obtained, achieving the beneficial effect of explicitly learning the correlation between different viewpoints and improving the accuracy of three-dimensional pose detection.

[0226] The following will describe the training method of the pose detection model provided in this application embodiment, with the server as the execution subject, using exemplary applications and implementations of the server provided in the embodiments of this application. See also Figure 5A , Figure 5A This is a schematic diagram of the first process of the training method for the pose detection model provided in this application embodiment, which will be combined with Figure 5A The steps shown are explained.

[0227] In step 201, sample image data from multiple perspectives of the sample object are acquired, and the following processing is performed using the pose detection model to be trained.

[0228] In some embodiments, sample image data from multiple perspectives of the sample object (corresponding to the target object above) are obtained, and the images in the sample image dataset from multiple perspectives are preprocessed, such as cropping, flipping, rotating, etc., to increase the diversity of the sample image data.

[0229] In step 202, two-dimensional pose sequence samples of the sample object are obtained from each viewpoint.

[0230] In some embodiments, the sample image data for each viewpoint includes multiple frames of sample images. The pose detection model includes a pre-trained two-dimensional pose extraction module. The two-dimensional pose extraction module (e.g., ViTPose, CPN, etc.) can be used to extract image features from each frame of sample images to obtain the image features of each frame of sample images (see the description of step 1011 above). Based on the image features of the sample object in each frame of sample images, keypoint detection processing is performed to obtain the coordinates of multiple keypoints of the sample object in each frame of sample images (see the description of step 1012 above). The coordinates of multiple keypoints of the sample object in each frame of sample images are combined to obtain the two-dimensional pose sequence sample of the sample object in the viewpoint (see the description of step 1013 above).

[0231] In step 203, three-dimensional feature extraction processing is performed on the two-dimensional pose sequence samples of the sample object from each viewpoint to obtain the three-dimensional pose feature samples of the sample object from each viewpoint.

[0232] In some embodiments, the pose detection model includes a single-view module, which can perform embedding encoding processing on the two-dimensional poses (corresponding to the coordinates of multiple key points of the sample object in each frame of the sample image) in the two-dimensional pose sequence samples (see the description of step 1021 above) to obtain the embedding encoding features of each two-dimensional pose; perform position encoding processing on the position of the two-dimensional pose in the two-dimensional pose sequence samples to obtain the position encoding features of each two-dimensional pose (see the description of step 1022 above); and perform feature encoding based on the embedding encoding features and position encoding features of each two-dimensional pose to obtain the three-dimensional pose feature samples of the sample object in each view (see the description of step 1023 above).

[0233] In step 204, the 3D pose feature samples of the sample object from each viewpoint are aggregated into multi-view feature samples.

[0234] In some embodiments, the 3D pose feature samples of each viewpoint include temporal features of the viewpoint, and the temporal features of each viewpoint represent the temporal correlation of the 2D poses in the 2D pose sequence samples of the viewpoint. The pose detection model includes a cross-viewpoint module, which can concatenate the temporal features of each viewpoint into a combined temporal feature sample (see the description of step 1031 above); perform feature encoding processing on the combined temporal feature sample to obtain a temporal encoded feature sample (see the description of step 1032 above); perform first attention encoding processing on the temporal encoded feature sample to obtain a first attention encoded feature sample (see the description of step 1033 above); and perform feature mapping processing based on the first attention encoded feature sample and the temporal encoded feature sample to obtain multi-viewpoint feature samples (see the description of step 1034 above).

[0235] In step 205, common feature samples are extracted among the 3D pose feature samples of the sample object from each viewpoint.

[0236] In some embodiments, the 3D pose feature samples include pose sequence samples, which include the 3D pose samples of the sample object in each frame of the sample image data. The pose detection model includes a cross-view module, which can concatenate the pose sequence samples from each viewpoint into a combined pose sequence sample (see the description of step 1041 above); perform feature encoding processing on the combined pose sequence samples to obtain pose encoded feature samples (see the description of step 1042 above); perform a second attention encoding processing based on the first learnable input and the pose encoded feature samples to obtain a first encoded feature sample (see the description of step 1043 above); perform a third attention encoding processing based on the second learnable input and the first encoded feature samples to obtain a second encoded feature sample, and use the second encoded feature sample as a common feature sample (see the description of step 1044 above).

[0237] In step 206, feature fusion processing is performed on the multi-view feature samples and the common feature samples to obtain fused feature samples.

[0238] In some embodiments, the pose detection model includes a cross-view module, which can use a first learnable weight to weight the multi-view feature samples to obtain a first weighted feature sample (see the description of step 1051 above); use a second learnable weight to weight the common feature samples to obtain a second weighted feature sample (see the description of step 1052 above); and use the sum of the first weighted feature sample and the second weighted feature sample as the fused feature sample (see the description of step 1053 above).

[0239] In step 207, three-dimensional pose prediction processing is performed based on the fused feature samples to obtain the target three-dimensional pose sequence samples of the sample object from each viewpoint.

[0240] In some embodiments, the pose detection model includes a regression head, which can perform 3D pose prediction processing on the fused feature samples to obtain the target 3D pose sequence samples of the sample object in each view. The regression head may include one or more fully connected layers, or convolutional layers or other types of network structures. The regression head receives fused features and outputs the target 3D pose sequence samples of the sample object in each view. Here, the regression head may be obtained by training a loss function (such as mean squared error, mean absolute error, etc.) and the network parameters of the regression head in the pose detection model are updated by the backpropagation algorithm. The specific composition of the target 3D pose sequence samples can be found in the description of step 106 (corresponding to the target 3D pose sequence) above.

[0241] In step 208, a combined loss value is determined based on two-dimensional attitude sequence samples, three-dimensional attitude feature samples, and target three-dimensional attitude sequence samples.

[0242] In some embodiments, see Figure 5B , Figure 5A Step 208 shown can be implemented through steps 2081 to 2084, as explained in detail below.

[0243] In step 2081, a first loss value is determined based on the target three-dimensional attitude sequence samples and two-dimensional attitude sequence samples.

[0244] In some embodiments, see Figure 5C , Figure 5B Step 2081 shown can be implemented through steps 301 to 304, which will be explained in detail below.

[0245] In step 301, a first projection loss value is determined based on the target three-dimensional attitude sequence samples and two-dimensional attitude sequence samples.

[0246] In some embodiments, see Figure 5D , Figure 5C Step 301 shown can be implemented through steps 3011 to 3014, as explained in detail below.

[0247] In step 3011, the standard three-dimensional pose sample is subjected to two-dimensional spatial projection processing to obtain a projected two-dimensional pose sequence.

[0248] In some embodiments, without 3D pose annotation, the 2D pose sequence samples extracted by the 2D pose extraction module are used as pseudo-labels, and the standard 3D pose samples are subjected to 2D spatial projection processing (e.g., weak perspective projection, orthogonal projection, etc.) to obtain the projected 2D pose sequence.

[0249] In step 3012, the two-dimensional pose sequence samples are normalized to obtain a normalized two-dimensional pose sequence.

[0250] In some embodiments, the root joints (such as the pelvic joints or chest joints in the human posture) can be used as the central key points for decentralization. The coordinates of the decentralized key points are then divided by a normalization matrix (such as the Frobenius norm) for scaling to obtain a normalized two-dimensional posture sequence. The embodiments of this application do not limit the specific implementation of the normalization process.

[0251] In step 3013, the ratio of the regularization matrix of the projected two-dimensional attitude sequence to that of the projected two-dimensional attitude sequence is obtained.

[0252] In some embodiments, the ratio of the projected two-dimensional pose sequence to the regularization matrix of the projected two-dimensional pose sequence (e.g., the Frobenius norm of the projected two-dimensional pose sequence) is obtained.

[0253] In step 3014, the first difference between the ratio and the normalized two-dimensional pose sequence is obtained, and the first difference is regularized to obtain the first projection loss value.

[0254] In some embodiments, the regularized result of the ratio and the first difference of the normalized two-dimensional pose sequence is used as the first projection loss value.

[0255] For example, the determination of the first projection loss value can be expressed by formula (5);

[0256]

[0257] Among them, W i W represents a normalized two-dimensional pose sequence (a normalized pseudo-label sequence). rep Represents the projected two-dimensional attitude sequence, ‖·‖1 represents the L1 norm, ‖·‖ F This represents the Frobenius norm.

[0258] See also Figure 5C In step 302, the multi-view consistency loss value is determined based on the target three-dimensional attitude sequence sample and the two-dimensional attitude sequence sample.

[0259] Here, the multi-view consistency loss is constructed to address the potential inaccurate prediction problem that may arise when directly constraining the 3D pose corresponding to different viewpoints. This is because the target object observed from different viewpoints may be occluded, or there may be differences in viewpoints. Directly using this observed information to constrain 3D pose detection may introduce errors. The multi-view consistency loss ensures that the information of the target object obtained from different viewpoints is consistent in 3D space, thereby improving the accuracy of 3D pose detection and reducing the impact of occlusion.

[0260] In some embodiments, see Figure 5E , Figure 5C Step 302 shown can be implemented through steps 3021 to 3024, which will be explained in detail below.

[0261] In step 3021, a first viewpoint and a second viewpoint are determined from multiple viewpoints, and the standard three-dimensional pose sample of the second viewpoint is converted to the camera space corresponding to the first viewpoint to obtain the converted target three-dimensional pose sequence sample.

[0262] In some embodiments, the standard 3D pose samples in the standard space corresponding to different viewpoints are consistent. Given viewpoint i (corresponding to the first viewpoint), the standard 3D pose samples corresponding to all other viewpoints j (corresponding to the second viewpoint) are transformed into the camera space of viewpoint i to obtain the transformed target 3D pose sequence sample.

[0263] In step 3022, the standard three-dimensional pose sample from the first perspective and the transformed target three-dimensional pose sequence sample from the second perspective are subjected to two-dimensional spatial projection processing to obtain the projected two-dimensional pose sequence for each perspective.

[0264] In some embodiments, the standard three-dimensional pose samples from the first perspective and the target three-dimensional pose sequence samples from the second perspective are subjected to two-dimensional spatial projection processing (e.g., weak perspective projection, orthogonal projection, etc.) to obtain the projected two-dimensional pose sequence for each perspective.

[0265] In step 3023, a second projection loss value for each viewpoint is determined based on the projected two-dimensional pose sequence and two-dimensional pose sequence samples for each viewpoint.

[0266] In some embodiments, a second projection loss value for each viewpoint is determined based on the projected two-dimensional pose sequence and two-dimensional pose sequence samples for each viewpoint. The method for determining the second projection loss value can be found in the explanation of formula (5) above.

[0267] In step 3024, the sum of the second projection loss values ​​for each viewpoint is used as the multi-view consistency loss value.

[0268] In some embodiments, the sum of the second projection loss values ​​corresponding to multiple viewpoints is obtained as the multi-view consistency loss value.

[0269] For example, the determination of the multi-view consistency loss can be expressed by formula (6):

[0270]

[0271] in, This represents the projected two-dimensional pose sequence obtained by reprojecting the standard three-dimensional pose sample corresponding to viewpoint j onto viewpoint i. This represents the second projection loss value. For explanations of the other parameters, please refer to formula (5) above.

[0272] See also Figure 5C In step 303, the camera rotation loss value is determined based on the target three-dimensional attitude sequence sample and the two-dimensional attitude sequence sample.

[0273] Here, camera rotation loss is constructed because in certain specific scenarios, the relative rotation between cameras with different viewpoints is fixed. Therefore, camera rotation loss can better constrain the pose detection and improve its accuracy. For example, in games, the camera follows the game character and maintains a fixed angle, which means that when cameras with different viewpoints record the same animation clip (corresponding to image data), the relative rotation between the cameras is fixed.

[0274] In some embodiments, see Figure 5F , Figure 5C Step 303 shown can be implemented through steps 3031 to 3034, which will be explained in detail below.

[0275] In step 3031, a first viewpoint and a second viewpoint are determined from multiple viewpoints, and the camera rotation posture sample of the second viewpoint is converted to the camera space corresponding to the first viewpoint to obtain the converted camera posture sample.

[0276] In some embodiments, given a viewpoint i (corresponding to the first viewpoint), the camera rotation pose samples corresponding to all other viewpoints j (corresponding to the second viewpoint) are transformed into the camera space of viewpoint i to obtain transformed camera pose samples.

[0277] In step 3032, the camera rotation posture samples from the first viewpoint and the transformed camera posture samples from the second viewpoint are subjected to two-dimensional spatial projection processing to obtain the projected two-dimensional posture sequence for each viewpoint.

[0278] In some embodiments, the camera rotation posture samples from the first viewpoint and the transformed camera posture samples from the second viewpoint are subjected to two-dimensional spatial projection processing (e.g., weak perspective projection, orthogonal projection, etc.) to obtain a projected two-dimensional posture sequence for each viewpoint.

[0279] In step 3033, a third projection loss value for each viewpoint is determined based on the projected two-dimensional pose sequence and two-dimensional pose sequence samples for each viewpoint.

[0280] In some embodiments, a third projection loss value for each viewpoint is determined based on the projected two-dimensional pose sequence and two-dimensional pose sequence samples for each viewpoint. The method for determining the third projection loss value can be found in the explanation of formula (5) above.

[0281] In step 3034, the sum of the third projection loss values ​​for each viewpoint is used as the camera rotation loss value.

[0282] In some embodiments, the sum of the third projection loss values ​​corresponding to multiple viewpoints is obtained as the camera rotation loss value.

[0283] For example, the camera rotation loss value can be determined by formula (7):

[0284]

[0285] in, This represents the projected two-dimensional pose sequence obtained by reprojecting the camera rotation pose sample corresponding to viewpoint j onto viewpoint i. This represents the third projection loss. For explanations of the other parameters, please refer to formula (5) above.

[0286] See also Figure 5CIn step 304, the projection loss value, the multi-view consistency loss value, and the camera rotation loss value are weighted and summed to obtain the first loss value.

[0287] For example, the determination of the first loss value can be expressed by formula (8):

[0288] L 2D =λ rep L rep +λ mpc L mvc +λ camera L Camera (8)

[0289] Where, λ rep , λ rep and λ camera These represent the weights corresponding to the first projection loss value, the multi-view consistency loss value, and the camera rotation loss value, respectively.

[0290] See also Figure 5B In step 2082, a second loss value is determined based on the three-dimensional pose feature samples and the target three-dimensional pose sequence samples.

[0291] Here, the second loss value, also known as the self-distillation loss, is introduced to improve the accuracy of the 3D pose output by the single-view module. Specifically, during the training phase, after several rounds of training, the cross-view module outperforms the single-view module in detecting 3D pose. At this point, the 3D pose output by the cross-view module is used to supervise the single-view module because the 3D pose output by the cross-view module provides more information than the pseudo-labels (corresponding to 2D pose sequence samples), thus achieving self-distillation. That is, the multi-view module is used to improve the pose detection accuracy of the single-view module, ensuring the usability of the single-view module when only single-view image data is used as input.

[0292] In some embodiments, see Figure 5G , Figure 5B Step 2082 shown can be achieved by performing steps 20821 to 20824 for each viewpoint, as explained in detail below.

[0293] In step 20821, the first ratio of the regularization matrix of the three-dimensional pose feature sample to the three-dimensional pose feature sample is obtained.

[0294] In some embodiments, a first ratio is obtained between the three-dimensional pose feature sample and the regularization matrix of the three-dimensional pose feature sample (e.g., the Frobenius norm of the three-dimensional pose feature sample).

[0295] In step 20822, the second ratio of the target three-dimensional attitude sequence sample to the regularization matrix of the target three-dimensional attitude sequence sample is obtained.

[0296] In some embodiments, a second ratio is obtained between the target 3D pose sequence sample and the regularization matrix of the target 3D pose sequence sample (e.g., the Frobenius norm of the target 3D pose sequence sample).

[0297] In step 20823, the second difference between the first ratio and the second ratio is obtained, and the second difference is regularized to obtain the sub-loss value.

[0298] In some embodiments, the regularized result of the second difference between the first ratio and the second ratio is used as the sub-loss value.

[0299] In step 20824, the sum of the sub-loss values ​​corresponding to each viewpoint is used as the second loss value.

[0300] In some embodiments, the sum of the sub-loss values ​​corresponding to each of the multiple perspectives is used as the second loss value.

[0301] For example, the determination of the second loss value can be expressed by formula (9):

[0302]

[0303] in, R represents the 3D pose feature sample of viewpoint i output by the single-view module. i P i R represents the target 3D pose sequence sample finally detected from viewpoint i, where R i P represents the camera rotation attitude sample. i This represents a standard three-dimensional pose sample.

[0304] See also Figure 5B In step 2083, a third loss value is determined based on the three-dimensional pose feature samples and the target three-dimensional pose sequence samples.

[0305] Here, the third loss value, also known as the temporal consistency loss, is used to constrain the two modules (single-view module and cross-view module) to predict the target 3D pose sequence (corresponding to the pose sequence included in the 3D pose features of the target object in each view output by the single-view module, and the target 3D pose sequence of the target object in each view obtained by performing 3D pose prediction processing based on the fusion features obtained by the cross-view module) to exhibit the same deformation within the same period.

[0306] In some embodiments, see Figure 5H , Figure 5BStep 2083 shown can be achieved by performing steps 20831 to 20834 for each viewpoint, as explained in detail below.

[0307] In step 20831, the first posture deformation parameters are determined based on the three-dimensional posture feature samples.

[0308] In some embodiments, the first pose deformation parameter can be determined based on the key point coordinates of the three-dimensional pose feature samples of k consecutive frames of view. For example, the average Euclidean distance between the key point coordinates of adjacent frames of k consecutive frames can be obtained as the first pose deformation parameter. The embodiments of this application do not limit the specific method of obtaining the first pose deformation parameter.

[0309] In step 20832, the second attitude deformation parameters are determined based on the target three-dimensional attitude sequence samples.

[0310] In some embodiments, a first pose deformation parameter can be determined based on the key point coordinates included in the target three-dimensional pose sequence sample of consecutive k frames from the viewpoint. For example, the average Euclidean distance between the key point coordinates of adjacent frames in consecutive k frames can be obtained as a second pose deformation parameter. The embodiments of this application do not limit the specific method for obtaining the second pose deformation parameter.

[0311] In step 20833, the third difference between the first attitude deformation parameter and the second attitude deformation parameter is obtained, and the third difference is regularized to obtain the sub-loss value.

[0312] In some embodiments, the regularization result of the third difference between the first attitude deformation parameter and the second attitude deformation parameter is used as the sub-loss value.

[0313] In step 20834, the sum of the sub-loss values ​​corresponding to each viewpoint is used as the third loss value.

[0314] In some embodiments, the sum of the sub-loss values ​​corresponding to each of the multiple perspectives is used as the third loss value.

[0315] For example, the determination of the third loss value can be expressed by formula (10):

[0316]

[0317] in, D represents the first pose deformation parameter of the 3D pose feature samples k frames apart at viewpoint i, output by the single-view module. i,k This represents the second attitude deformation parameter of the target 3D attitude sequence samples separated by k frames at viewpoint i, which is finally predicted based on the cross-view module.

[0318] See also Figure 5BIn step 2084, the first loss value, the second loss value, and the third loss value are weighted and summed to obtain the combined loss value.

[0319] For example, the determination of the combined loss value can be expressed by formula (11):

[0320] L totat =L 2d +λ d L d +λ tc L tc (11)

[0321] Where, λ d The weight λ represents the second loss value. tc This represents the weight of the third loss value.

[0322] See also Figure 5A In step 209, the parameters of the pose detection model to be trained are updated based on the combined loss value to obtain the trained pose detection model.

[0323] In some embodiments, gradient information is obtained by combining loss values, and the parameters of the pose detection model to be trained are updated according to the gradient information to obtain the trained pose detection model.

[0324] For example, the gradient information of the combined loss value for each parameter of the pose detection model can be obtained through the backpropagation algorithm. The parameters of the pose detection model can be updated using the obtained gradient information according to the gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations are reached or the pose detection model converges, thereby obtaining the trained pose detection model.

[0325] The following describes an exemplary application of the embodiments of this application in the scenario of animation asset reconstruction. Recovering 3D animation assets (e.g., 3D pose data of game characters) from published games is a critical task, for example, when animation resources are lost due to unforeseen accidents, it is necessary to recover the corresponding animation assets. Related animation asset reconstruction methods control characters to perform the required actions and utilize unsupervised 3D human pose estimation methods for capture. Supervised methods are infeasible without 3D annotations. Unsupervised pose estimation can be broadly categorized into two types: single-view pose estimation and multi-view pose estimation. Single-view methods are prone to inaccurate predictions due to the inherent depth blur and self-occlusion problems of 2D images or videos. Multi-view methods require controlling the character to perform the same animation from different perspectives, which is cumbersome for animations requiring complex game operations and impractical in real-world use. Therefore, an important strategy is to retain the ease of use of single-view estimation while improving its accuracy by utilizing information from multiple perspectives; that is, it needs to support both single-view and multi-view video input methods.

[0326] The pose detection method provided in this application can obtain the three-dimensional pose data of game characters from video data, effectively saving animators' animation asset reconstruction time. It also supports both single-view and multi-view video input methods and reconstructs high-quality animation assets from them.

[0327] See Figure 7A , Figure 7A This is a first schematic diagram of an optional structure of the pose detection model provided in this application embodiment. The two-dimensional pose extraction module is used to extract two-dimensional pose sequences from image data of each viewpoint. After obtaining the two-dimensional pose sequences of each viewpoint, the single-view module performs three-dimensional feature extraction processing on the two-dimensional pose sequences of each viewpoint to obtain the three-dimensional pose features corresponding to each viewpoint. The three-dimensional pose features include intermediate three-dimensional pose sequences and temporal features. Through the processing of the single-view module, the two-dimensional pose sequences are improved into three-dimensional pose features. In order to further improve the accuracy of pose detection, the outputs of multiple single-view modules are further aggregated and extracted through the cross-view module. Specifically, features from different viewpoints are aggregated, and common features between viewpoints are mined. Then, feature fusion is performed, and finally, the target three-dimensional pose sequence corresponding to each viewpoint is obtained based on the fused feature mapping. The target three-dimensional pose sequence includes standard three-dimensional pose and camera rotation pose.

[0328] See Figure 6 , Figure 6 This is a schematic diagram of the application process of the pose detection method in the animation asset reconstruction scenario provided in the embodiments of this application, which will be described in detail below.

[0329] In step 401, game video data is acquired and the trained pose detection model is loaded. The game video data can be single-view data or multi-view data.

[0330] In some embodiments, game video data (corresponding to image data above) is acquired, and preprocessing operations are performed on each frame of the game video data, such as cropping, scaling, and rotating. Multi-view data can be acquired by controlling the game character (corresponding to the target object above) to perform the same action from different viewpoints.

[0331] In step 402, in response to the fact that the game video data is single-view data, the single-view module in the pose detection model outputs the target 3D pose sequence corresponding to the single-view data.

[0332] In some embodiments, see Figure 7B , Figure 7B This is a second schematic diagram of an optional structure of the attitude detection model provided in the embodiments of this application. Figure 7B An optional structure of a single-view module is shown. The single-view module includes an embedding coding layer, a position coding layer, a feature coding module, and a regression head. The feature coding module can be constructed by stacking multiple layers of Transformer Encoder. The embodiments of this application do not limit the specific network structure of the single-view module.

[0333] In some embodiments, the single-view data includes multiple frames of images of a game character. When the input game video data is single-view data, image feature extraction processing is performed on each frame of the single-view data to obtain the image features of each frame (see the description of step 1011 above); keypoint detection processing is performed based on the image features of the game character in each frame to obtain the coordinates of multiple keypoints of the game character in each frame (see the description of step 1012 above); the coordinates of multiple keypoints of the game character in each frame are combined to obtain a two-dimensional pose sequence of the single-view data (see step 1013 above). (See the explanation of step 1021 above); perform embedding encoding on the two-dimensional poses (coordinates of multiple two-dimensional key points) in the two-dimensional pose sequence of the single-view data to obtain the embedding encoding features of each two-dimensional pose (see the explanation of step 1021 above); perform position encoding on the position of the two-dimensional pose in the two-dimensional pose sequence of the single-view data to obtain the position encoding features of each two-dimensional pose (see the explanation of step 1022 above); perform feature encoding based on the embedding encoding features and position encoding features of each two-dimensional pose to obtain the three-dimensional pose features of the single-view data (see the explanation of step 1023 above).

[0334] In some embodiments, the 3D pose features corresponding to the single-view data include temporal features and intermediate 3D pose sequences. The embedding encoding features and position encoding features of each 2D pose are concatenated to obtain 2D pose features, and attention encoding processing is performed on the 2D pose features (e.g., ...). Figure 7B The multi-head attention (MHA) processing of the feature encoding module is used to obtain attention-encoded features; based on the attention-encoded features and two-dimensional pose features, feature mapping processing is performed to obtain temporal features; and three-dimensional pose mapping processing is performed on the temporal features to obtain the intermediate three-dimensional pose sequence, that is, the three-dimensional pose sequence corresponding to the single-view data.

[0335] For example, feature mapping based on attention-encoded features and 2D pose features can be achieved by concatenating the attention-encoded features and 2D pose features to obtain the features to be mapped, and then performing feedforward mapping on the features to be mapped through a feedforward neural network layer (FFN) to obtain temporal features.

[0336] For example, a regression head can be used to perform 3D pose mapping on temporal features to obtain a pose sequence. The regression head can include one or more fully connected layers, or it can be a convolutional layer or other types of network structures. The regression head receives temporal features and outputs a pose sequence.

[0337] In step 403, in response to the fact that the game video data is multi-view data, the cross-view module in the pose detection model outputs the target 3D pose sequence corresponding to each view in the multi-view data.

[0338] In some embodiments, see Figure 7C , Figure 7C This is a third schematic diagram of an optional structure of the attitude detection model provided in the embodiments of this application. Figure 7C The diagram illustrates an optional structure for a cross-view module, which includes two branches: one for acquiring multi-view features and the other for acquiring common features. The input to the multi-view feature acquisition branch is multi-view temporal features (corresponding to the combined temporal features mentioned above), and the input to the common feature acquisition branch is multi-view pose sequences (corresponding to the combined pose sequences mentioned above). The features from the two branches are fused, and then the fused features are mapped by a regression head to obtain the target 3D pose sequence corresponding to each view.

[0339] In some embodiments, when the game video data is multi-view data, refer to the processing in step 402 above to obtain the three-dimensional pose features corresponding to each view. The three-dimensional pose features of the game character in each view are aggregated into multi-view features through the cross-view module. The common features between the three-dimensional pose features of the game object in each view are extracted. The multi-view features and common features are fused to obtain fused features. The fused features are then processed by a regression head to predict the three-dimensional pose of the game character in each view.

[0340] In some embodiments, the 3D pose features of each viewpoint include temporal features of the viewpoint, and the temporal features of each viewpoint represent the temporal correlation of the 2D poses in the 2D pose sequence of the viewpoint. Aggregating the 3D pose features of the game character from each viewpoint into multi-view features can be achieved as follows: concatenating the temporal features of each viewpoint into combined temporal features through a cross-viewpoint module (see the description of step 1031 above); performing feature encoding processing on the combined temporal features to obtain temporally encoded features (see the description of step 1032 above); performing first attention encoding processing on the temporal encoded features to obtain first attention encoded features (see the description of step 1033 above); and performing feature mapping processing based on the first attention encoded features and the temporally encoded features to obtain multi-view features (see the description of step 1034 above).

[0341] In some embodiments, the 3D pose features include a pose sequence, which includes the 3D pose in each frame of the game video data for each viewpoint. Extracting common features among the 3D pose features of the game object from each viewpoint can be achieved by: concatenating the pose sequences of each viewpoint into a combined pose sequence (see the description of step 1041 above); performing feature encoding processing on the combined pose sequence to obtain pose-encoded features (see the description of step 1042 above, corresponding to...). Figure 7C The pose sequence embedding encoding is performed; based on the first learnable input and pose encoding features, a second attention encoding process is performed to obtain the first encoding features (see the explanation of step 1043 above, corresponding to...). Figure 7C The first learnable input is used as Q for cross-attention encoding; based on the second learnable input and the first encoded features, a third attention encoding process is performed to obtain the second encoded features, which are then used as common features (see the explanation of step 1044 above, corresponding to...). Figure 7C (The second learnable input is used as Q for cross-attention encoding).

[0342] In some embodiments, the first coding feature includes multiple data dimensions. After obtaining the first coding feature, the multiple data dimensions of the first coding feature can be rearranged to obtain an updated first coding feature (corresponding to...). Figure 7C The reshaping operation in the process (see the above description of rearranging multiple data dimensions of the first coding feature) replaces the first coding feature before the update with the updated first coding feature, and then proceeds to the third attention coding process based on the second learnable input and the first coding feature to obtain the second coding feature.

[0343] In some embodiments, feature fusion processing of multi-view features and common features to obtain fused features can be achieved in the following way: multi-view features are weighted by a first learnable weight to obtain a first weighted feature (see the description of step 1051 above); common features are weighted by a second learnable weight to obtain a second weighted feature (see the description of step 1052 above); the sum of the first weighted feature and the second weighted feature is used as the fused feature (see the description of step 1053 above).

[0344] In some embodiments, a regression head can be used to perform 3D pose prediction processing on the fused features to obtain the target 3D pose sequence of the game character in each view. The regression head may include one or more fully connected layers, or convolutional layers or other types of network structures. The regression head receives the fused features and outputs the target 3D pose sequence of the game character in each view. Here, the regression head may be obtained by training a loss function (such as mean squared error, mean absolute error, etc.) and the network parameters of the regression head in the pose detection model are updated by the backpropagation algorithm.

[0345] In some embodiments, the target 3D pose sequence of the game character in each viewpoint includes the standard 3D pose of each keypoint and the camera rotation pose. The standard 3D pose corresponds to the position of a keypoint (e.g., a specific part of the game character) in 3D space, such as represented by the x, y, and z coordinates of the keypoint. The camera rotation pose refers to the rotation of the camera in 3D space with six degrees of freedom, including three translational degrees of freedom and three rotational degrees of freedom, used to transform the standard 3D pose corresponding to the keypoint to the camera space.

[0346] Through steps 401 to 403, support for both single-view and multi-view video input methods is achieved. This achieves the beneficial effect of improving the accuracy of pose detection by utilizing information from multiple perspectives while retaining the ease of use of single-view pose detection. Below, the performance evaluation results of the pose detection method and pose detection model training method provided in the embodiments of this application are described in detail.

[0347] In the performance evaluation of the pose detection model, it was first evaluated on two benchmark 3D human pose estimation datasets: Human3.6M and MPI-INF-3DHP. Since the task in animation asset reconstruction scenarios is to recover lost animation resources from multi-view game videos, and in the game environment, the camera follows the character's movements, unlike the fixed camera systems used in motion capture datasets such as Human3.6M and MPI-INF-3DHP, there is currently no multi-view game animation dataset available for quantitative evaluation of the pose detection method provided in this application embodiment. Therefore, to evaluate the applicability of the pose detection method and the training method of the pose detection model provided in this application embodiment in game scenarios, a new dataset, Mixamo, was created for evaluation by collecting animation assets from Mixamo.

[0348] Human3.6M is a widely used dataset for 3D human pose estimation. This dataset includes 15 everyday actions performed by 11 subjects, recorded at 50 frames per second by four calibrated high-resolution cameras. The entire dataset contains 3.6 million frames. To ensure fair evaluation, standard methods were followed during the evaluation process, using data from subjects 1, 5, 6, 7, and 8 for training and data from subjects 9 and 11 for testing. Two metrics were used for evaluation. The first is Mean Per Joint Position Error (MPJPE), which measures the average Euclidean distance between the reconstructed 3D keypoint coordinates and the ground truth. Since there is no metric data in the self-supervised setting, the reconstructed 3D pose is scaled to match the ground truth; this is often referred to as N-frame Mean Per Joint Position Error. The second metric is Procrustes-MPJPE (PMPJPE), which is the MPJPE calculated after rigidly aligning the estimated pose with the ground truth pose.

[0349] MPI-INF-3DHP is another large-scale 3D human pose estimation dataset, collected from three different scenarios: green screen, non-green screen, and outdoor environments. This dataset uses 14 synchronized cameras to record different actions performed by 8 subjects. In addition, the dataset includes a single-view test set capturing the actions of 6 subjects. Following standard methods, 5 cameras at chest height were selected for training. For evaluation, the Normalized Percentage of Correct Keypoints (NPCK) within 150mm was calculated as the evaluation metric.

[0350] Mixamo is a multi-view game animation dataset. Although the Human3.6M and MPI-INF-3DHP datasets were used to benchmark the pose detection method provided in this application embodiment against existing methods, a dedicated game animation dataset is essential for quantitatively evaluating the effectiveness of the pose detection method provided in this application embodiment in game scenes. Therefore, three animation sets were first collected from Mixamo, containing dozens of short animation clips and five characters with different skeletons. These clips were then combined into 22 coherent animations, each between 100 and 200 frames in length, and finally, these animations were redirected to each game character. A detailed virtual lab scene was also created in Blender to simulate the game environment. Specifically, five relatively fixed virtual cameras were placed around the game characters and configured to follow their movement to simulate the game scene. Furthermore, acceleration error (ACCEL) (unit: mm / s) was used. 2 This is used to evaluate the smoothness of the reconstructed human motion on this dataset. See also Figure 8A , Figure 8A This is a first schematic diagram of the pose detection effect of the pose detection method provided in this application embodiment. The first column displays rendered images from three different viewpoints (corresponding to...). Figure 8A The second column displays the skeleton formed by connecting real keypoints from multiple viewpoints (corresponding to the input video frames), and the third column displays the skeleton formed by connecting real keypoints from multiple viewpoints (corresponding to the input video frames). Figure 8A The third column shows the pose detection results of the existing technology (Canonpose) (corresponding to the actual value in the data). Figure 8A In the prior art 1), the fourth column shows the attitude detection results of the attitude detection method provided in the embodiments of this application (corresponding to...). Figure 8A (The method in the embodiments of this application).

[0351] When only multi-view video is available, CPN and ViTpose are used as 2D pose extraction modules (or 2D pose detectors). CPN is responsible for detecting 2D pose on the Human3.6M dataset, while ViTpose is responsible for detecting 2D pose on the Mixamo and MPI-INF-3DHP datasets. Since different datasets usually have different characteristics and challenges, different 2D pose detectors can be used for processing.

[0352] Referring to Table 1, which shows the comparison results of the pose detection method provided in this application embodiment with other methods using different types of supervision on the benchmark dataset Human3.6M during single-view inference, the 3D pose output by the single-view module (SM) is used as the single-view inference result.

[0353] Table 1

[0354]

[0355] As shown in Table 1, the pose detection method provided in this application embodiment outperforms other self-supervised methods in terms of NMPJPE and PMPJPE metrics. Furthermore, although Kim's proposed method uses 81 frames as the input length of the 2D pose sequence (corresponding to Kim (T=81), where T represents the input length), the pose detection method provided in this application embodiment achieves better performance with only 9 frames of input (corresponding to the method in this application embodiment (T=9)). When using 81 frames as input, the pose detection method provided in this application embodiment reduces the NMPJPE by 5.9 mm (8.6%) and the PMPJPE by 6.3 mm (12.5%) compared to Kim. This indicates that the pose detection method provided in this application embodiment achieves better performance than other video-based methods even with fewer input frames.

[0356] Furthermore, since the pose detection method provided in this application adopts a dual-module design (corresponding to a single-view module and a cross-view module), when there is multi-view test data, the 3D pose output by the cross-view module can be used for multi-view inference. See Table 2, which shows the multi-view inference results of the pose detection method provided in this application on the Human3.6M dataset.

[0357] Table 2

[0358]

[0359] As can be seen from Table 2, the attitude detection method provided in this application is superior to other comparable self-supervised methods. Compared with the method proposed by Gong, NM PJPE is reduced by 1.1 mm and PMPJPE is reduced by 7.8 mm (18.3%), further demonstrating the effectiveness of the cross-view module.

[0360] The performance of the pose detection method provided in this application embodiment was also evaluated on the MPI-INF-3DHP dataset (referring to multi-view inference results). Compared with the indoor data of Hu man 3.6M, the MPI-INF-3DHP dataset includes outdoor video, adding additional challenges. The outdoor environment of MPI-INF-3DHP introduces more variability and complexity, thus subjecting the robustness and accuracy of the pose detection method provided in this application embodiment to more rigorous testing. See Table 3, which shows the relevant results.

[0361] Table 3

[0362]

[0363] As can be seen from Table 3, the posture detection method provided in this application embodiment is superior to other self-supervised methods, indicating that even for some dynamic and unseen outdoor actions, the posture detection method provided in this application embodiment can achieve satisfactory results.

[0364] See Figure 8B , Figure 8B This is a second schematic diagram illustrating the attitude detection effect of the attitude detection method provided in this application embodiment. Figure 8B The figure shows the qualitative results of the pose detection method provided in this application on the Human3.6M dataset. See also: Figure 8C , Figure 8C This is a third schematic diagram illustrating the attitude detection effect of the attitude detection method provided in this application embodiment. Figure 8C The figure shows the qualitative results of the pose detection method provided in this application on the 3DHP dataset. Figure 8B and Figure 8C As can be seen from the pose detection results shown, the pose detection method provided in this application embodiment can obtain accurate estimation results even without 3D annotation.

[0365] To evaluate the performance of the pose detection method provided in this application embodiment in a game scene, tests were conducted on the Mixamo dataset. As shown in Table 4 below, the pose detection method provided in this application embodiment outperforms the existing method Canonpose on all metrics, demonstrating its effectiveness in accurately reconstructing 3D poses from multi-view data. This highlights the potential of the pose detection method provided in this application embodiment in real-world game animation asset restoration scenarios.

[0366] Table 4

[0367]

[0368] To demonstrate the optimal performance of the pose detection method provided in this application, 2D pose data from a dataset was used as input, and the results were compared with other existing methods. An additional benefit of this approach is that it eliminates the influence of the performance of the pre-trained 2D detector on the results. See Table 5, which shows the results tested on the Human3.6M benchmark dataset. All methods used the same 2D input data.

[0369] Table 5

[0370]

[0371]

[0372] As shown in Table 5, when using real 2D pose as input, the pose detection method provided in this application shows a significant performance improvement (NMPJPE is 7.3 mm lower than when using pseudo-2D pose). This indicates that the quality of 2D pose plays a crucial role in the 2D-to-3D conversion process, and a better 2D pose detector will help improve the performance of 3D human pose estimation. Furthermore, the pose detection method provided in this application still outperforms the method proposed by Kim. It is worth noting that all experimental results were obtained without using any additional input labels (2D pose only) or training data. These results demonstrate that even under this setting, the pose detection method provided in this application can still surpass previous methods.

[0373] See Table 6, which shows the results of the ablation experiments. In all ablation experiments, the input frame length was set to 9 frames. All metrics shown in Table 6 represent the results of single-view inference (i.e., the results output by the SM module).

[0374] Table 6

[0375]

[0376] As can be seen from Table 6, the self-distillation loss (L) was removed d After referring to the explanation of formula (9) above, both NMPJPE and PMPJPE indices decreased significantly (corresponding to the results of ID2 in Table 6). This phenomenon demonstrates the effectiveness of multi-view joint learning, indicating that features from different perspectives can provide effective explicit supervision for single-view inference results after mutual learning;

[0377] Camera rotation loss (L) camera This approach is applicable not only to game scenarios but also to multi-camera motion capture systems using fixed cameras (e.g., Huma n3.6M), because in both cases, the relative rotation of the cameras can be considered fixed. Experimental results show that although camera rotation loss increases the PMPJP PE index by 0.5 mm, it significantly improves the NMPJPE index by 7.7 mm (corresponding to the ID3 results in Table 6).

[0378] When time consistency loss is not used during training (L tc When this happens, the model's performance degrades (corresponding to the ID4 results in Table 6). This indicates that, in addition to utilizing L... d In addition to directly supervising the keypoint locations in each frame, L tc It can also be used to constrain the 3D pose sequences output by cross-view and single-view modules to have consistent pose deformation over the same time span, thereby improving the performance of 3D pose detection by making full use of temporal information.

[0379] When the feature encoding module in the single-view module is removed during training, the model's performance degrades (corresponding to the ID5 results in Table 6), with NMPJPE increasing by 4.3 mm and PMPJPE increasing by 3.5 mm. This indicates that introducing a feature encoding module in the single-view module to learn the correlation between different frames is necessary, and this module helps alleviate the self-occlusion problem commonly found in single-view models.

[0380] The dual-branch structure in the cross-view module (a branch for acquiring multi-view features and a branch for acquiring common features) can reduce errors (corresponding to the results of ID6-7 in Table 6). The branch for acquiring multi-view features allows joint keypoint-level features from different views to learn from each other, thus overcoming the limitations of single-view features. At the same time, the branch for acquiring common features can extract common feature information from the 3D poses predicted from different views;

[0381] When a joint learning strategy is not used during training (joint learning refers to joint training between single-view modules and cross-view modules) (including removing single-view modules and not using L...) d and L tc The error increased significantly (corresponding to the ID8 results in Table 6). This result highlights the importance of joint learning of multi-view features. The lack of joint learning leads to poor performance, highlighting the shortcomings of previous work in effectively utilizing multi-view features.

[0382] To investigate the impact of different inputs on the cross-view module (CM), ablation experiments were conducted by varying the inputs to the CM. Specifically, the performance of the pose detection model was evaluated when no pose sequence or temporal features were input into the CM. The results (corresponding to the results of ID9-10 in Table 6) show that omitting either the pose sequence or temporal features leads to a decrease in model performance.

[0383] See Figure 9A , Figure 9A This is a schematic diagram illustrating the impact of different amounts of viewpoint information on the performance of the pose detection model, as provided in the embodiments of this application. Figure 9A The impact of using different numbers of viewpoints during training on the performance of the pose detection model is illustrated. As the number of input 2D pose viewpoints increases, the model's ability to predict 3D pose improves, while PMPJPE continuously decreases. This means that multi-viewpoint information effectively enhances model performance by compensating for the missing depth information in single-view 3D pose estimation. Furthermore, this also demonstrates that the pose detection method provided in this application can support training using any number of viewpoints (more than one), thus making it widely applicable to learning from multi-viewpoint information.

[0384] See Figure 9B , Figure 9BThis is a schematic diagram illustrating the impact of different sequence lengths of input image data on the performance of the pose detection model, as provided in an embodiment of this application. Figure 9B The effect of input sequence length on the accuracy of the pose detection method provided in the embodiments of this application is shown: longer sequences can achieve higher accuracy.

[0385] In summary, the pose detection method and pose detection model training method provided in this application embodiment can effectively utilize the temporal relationship in the sequence frames to accurately detect the three-dimensional pose of the target object.

[0386] The following description continues to illustrate the exemplary structure of the attitude detection device 133 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2A As shown, the software module stored in the attitude detection device 133 in the memory 130-1 may include:

[0387] The data acquisition module 1331 is used to acquire image data of the target object from multiple perspectives and to acquire the two-dimensional pose sequence of the target object in each perspective.

[0388] The data processing module 1332 is used to perform three-dimensional feature extraction processing on the two-dimensional pose sequence of the target object in each of the said views, so as to obtain the three-dimensional pose features of the target object in each of the said views.

[0389] In some embodiments, the data processing module 1332 is further configured to aggregate the three-dimensional pose features of the target object in each of the said viewpoints into multi-view features.

[0390] In some embodiments, the data processing module 1332 is further configured to extract common features of the target object among the three-dimensional pose features of each of the said views.

[0391] In some embodiments, the data processing module 1332 is further configured to perform feature fusion processing on the multi-view features and the common features to obtain fused features.

[0392] In some embodiments, the data processing module 1332 is further configured to perform three-dimensional pose prediction processing based on the fused features to obtain a target three-dimensional pose sequence of the target object in each of the said viewpoints.

[0393] In some embodiments, the three-dimensional pose features of each viewpoint include the temporal features of the viewpoint, and the temporal features of each viewpoint characterize the temporal correlation of the two-dimensional poses in the two-dimensional pose sequence of the viewpoint. The data processing module 1332 is further configured to concatenate the temporal features of each viewpoint into a combined temporal feature; perform feature encoding processing on the combined temporal feature to obtain a temporal encoded feature; perform a first attention encoding processing on the temporal encoded feature to obtain a first attention encoded feature; and perform feature mapping processing based on the first attention encoded feature and the temporal encoded feature to obtain multi-view features.

[0394] In some embodiments, the three-dimensional pose features of each viewpoint include an intermediate three-dimensional pose sequence of the viewpoint. The data processing module 1332 is further configured to concatenate the intermediate three-dimensional pose sequences of each viewpoint into a combined pose sequence; perform feature encoding processing on the combined pose sequence to obtain pose encoding features; perform second attention encoding processing based on a first learnable input and the pose encoding features to obtain a first encoding feature; perform third attention encoding processing based on a second learnable input and the first encoding feature to obtain a second encoding feature, and use the second encoding feature as the common feature.

[0395] In some embodiments, the data processing module 1332 is further configured to use the first learnable input as a query vector, the pose encoding feature as a key vector and a value vector, determine a first attention weight based on the key vector and the query vector, perform weighted processing on the value vector using the first attention weight to obtain a second attention feature, and perform a first feature mapping processing on the second attention feature to obtain the first encoding feature.

[0396] In some embodiments, the first encoding feature includes multiple data dimensions. The data processing module 1332 is further configured to rearrange the multiple data dimensions of the first encoding feature to obtain an updated first encoding feature, replace the original first encoding feature with the updated first encoding feature, and proceed to the third attention encoding processing based on the second learnable input and the first encoding feature to obtain the second encoding feature.

[0397] In some embodiments, the data processing module 1332 is further configured to determine a second attention weight based on the second learnable input and the first encoded feature; to perform weighted processing on the first encoded feature using the second attention weight to obtain a third attention feature; and to perform a second feature mapping processing on the third attention feature to obtain the second encoded feature.

[0398] In some embodiments, the image data for each viewpoint includes multiple frames of images. The data processing module 1332 is further configured to perform the following processing on the multiple frames of images for each viewpoint: perform image feature extraction processing on each frame of the viewpoint to obtain image features of each frame; perform key point detection processing based on the image features of the target object in each frame to obtain multiple key point coordinates of the target object in each frame; and combine the multiple key point coordinates of the target object in each frame to obtain a two-dimensional pose sequence of the target object in the viewpoint.

[0399] In some embodiments, the data processing module 1332 is further configured to perform the following processing on the two-dimensional pose sequence for each viewpoint: perform embedding encoding processing on the two-dimensional poses in the two-dimensional pose sequence to obtain embedding encoding features of each two-dimensional pose; perform position encoding processing on the position of the two-dimensional poses in the two-dimensional pose sequence to obtain position encoding features of each two-dimensional pose; and perform feature encoding based on the embedding encoding features and the position encoding features of each two-dimensional pose to obtain the three-dimensional pose features of the viewpoint.

[0400] In some embodiments, the data processing module 1332 is further configured to perform weighted processing on the multi-view features using a first learnable weight to obtain a first weighted feature; perform weighted processing on the common features using a second learnable weight to obtain a second weighted feature; and use the sum of the first weighted feature and the second weighted feature as the fusion feature.

[0401] In some embodiments, the three-dimensional pose features of each viewpoint include the temporal features of the viewpoint and an intermediate three-dimensional pose sequence. The data processing module 1332 is further configured to concatenate the pose encoding features and the combined temporal features to obtain concatenated features. The pose encoding features are obtained by concatenating the intermediate three-dimensional pose sequences of each viewpoint into a combined pose sequence and performing feature encoding processing on the combined pose sequence. The combined temporal features are obtained by concatenating the temporal features of each viewpoint. The concatenated features are weighted using pre-trained weight parameters to obtain weighted concatenated features. The weighted concatenated features are then subjected to nonlinear mapping processing to obtain the first learnable weight and the second learnable weight.

[0402] The following description continues to illustrate the exemplary structure of the training device 134 for the pose detection model provided in this application embodiment as a software module. In some embodiments, such as... Figure 2B As shown, the software modules in the training device 134 for the pose detection model stored in memory 130-2 may include:

[0403] The data acquisition module 1341 is used to acquire sample image data of the sample object from multiple perspectives.

[0404] Training module 1342 is used to obtain two-dimensional pose sequence samples of the sample object at each viewpoint.

[0405] In some embodiments, the training module 1342 is further configured to perform three-dimensional feature extraction processing on the two-dimensional pose sequence samples of the sample object in each of the viewpoints to obtain three-dimensional pose feature samples of the sample object in each of the viewpoints.

[0406] In some embodiments, the training module 1342 is further configured to aggregate the three-dimensional pose feature samples of the sample object in each of the said views into multi-view feature samples.

[0407] In some embodiments, the training module 1342 is further configured to extract common feature samples of the sample objects among the three-dimensional pose feature samples of each of the said views.

[0408] In some embodiments, the training module 1342 is further configured to perform feature fusion processing on the multi-view feature samples and the common feature samples to obtain fused feature samples.

[0409] In some embodiments, the training module 1342 is further configured to perform three-dimensional pose prediction processing based on the fused feature samples to obtain target three-dimensional pose sequence samples of the sample object in each of the said viewpoints.

[0410] In some embodiments, the training module 1342 is further configured to determine a combined loss value based on the two-dimensional pose sequence samples, the three-dimensional pose feature samples, and the target three-dimensional pose sequence samples.

[0411] In some embodiments, the training module 1342 is further configured to update the parameters of the pose detection model to be trained based on the combined loss value, so as to obtain the trained pose detection model.

[0412] In some embodiments, the training module 1342 is further configured to determine a first loss value based on the target three-dimensional pose sequence sample and the two-dimensional pose sequence sample; determine a second loss value based on the three-dimensional pose feature sample and the target three-dimensional pose sequence sample; determine a third loss value based on the three-dimensional pose feature sample and the target three-dimensional pose sequence sample; and perform weighted summation on the first loss value, the second loss value and the third loss value to obtain a combined loss value.

[0413] In some embodiments, the training module 1342 is further configured to: determine a first projection loss value based on the target 3D pose sequence samples and the 2D pose sequence samples; determine a multi-view consistency loss value based on the target 3D pose sequence samples and the 2D pose sequence samples; determine a camera rotation loss value based on the target 3D pose sequence samples and the 2D pose sequence samples; and perform a weighted summation of the first projection loss value, the multi-view consistency loss value, and the camera rotation loss value to obtain the first loss value.

[0414] In some embodiments, the target three-dimensional pose sequence sample includes standard three-dimensional pose samples. The training module 1342 is further configured to perform two-dimensional spatial projection processing on the standard three-dimensional pose samples to obtain a projected two-dimensional pose sequence; perform normalization processing on the two-dimensional pose sequence samples to obtain a normalized two-dimensional pose sequence; obtain the ratio of the projected two-dimensional pose sequence to the regularization matrix of the projected two-dimensional pose sequence; obtain the first difference between the ratio and the normalized two-dimensional pose sequence, and perform regularization processing on the first difference to obtain the first projection loss value.

[0415] In some embodiments, the target 3D pose sequence includes standard 3D pose samples. The training module 1342 is further configured to determine a first view and a second view from the plurality of viewpoints, and convert the standard 3D pose sample of the second viewpoint to the camera space corresponding to the first viewpoint to obtain a converted target 3D pose sequence sample; perform 2D spatial projection processing on the standard 3D pose sample of the first viewpoint and the converted target 3D pose sequence sample of the second viewpoint to obtain a projected 2D pose sequence for each viewpoint; determine a second projection loss value for each viewpoint based on the projected 2D pose sequence and the 2D pose sequence sample; and use the sum of the second projection loss values ​​for each viewpoint as the multi-view consistency loss value.

[0416] In some embodiments, the target 3D pose sequence sample includes camera rotation pose samples. The training module 1342 is further configured to determine a first view and a second view from the plurality of viewpoints, and convert the camera rotation pose sample of the second viewpoint to the camera space corresponding to the first viewpoint to obtain a converted camera pose sample; perform 2D spatial projection processing on the camera rotation pose sample of the first viewpoint and the converted camera pose sample of the second viewpoint to obtain a projected 2D pose sequence for each viewpoint; determine a third projection loss value for each viewpoint based on the projected 2D pose sequence for each viewpoint and the 2D pose sequence sample; and use the sum of the third projection loss values ​​for each viewpoint as the camera rotation loss value.

[0417] In some embodiments, the training module 1342 is further configured to perform the following processing for each viewpoint: obtaining a first ratio of the regularization matrix of the three-dimensional pose feature sample to the regularization matrix of the three-dimensional pose feature sample; obtaining a second ratio of the regularization matrix of the target three-dimensional pose sequence sample to the regularization matrix of the target three-dimensional pose sequence sample; obtaining a second difference between the first ratio and the second ratio, and performing regularization processing on the second difference to obtain a sub-loss value; and using the sum of the sub-loss values ​​corresponding to each viewpoint as the second loss value.

[0418] In some embodiments, the training module 1342 is further configured to perform the following processing on each viewpoint: determine a first pose deformation parameter based on the three-dimensional pose feature sample; determine a second pose deformation parameter based on the target three-dimensional pose sequence sample; obtain a third difference between the first pose deformation parameter and the second pose deformation parameter, and perform regularization processing on the third difference to obtain a sub-loss value; and use the sum of the sub-loss values ​​corresponding to each viewpoint as the third loss value.

[0419] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the attitude detection method or attitude detection model training method described in this application.

[0420] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or program are executed by a processor, the processor will execute the attitude detection method or attitude detection model training method provided in this application. For example, ... Figure 4A The pose detection method shown or Figure 5A The training method for the pose detection model is shown.

[0421] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0422] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0423] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0424] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0425] In summary, through the embodiments of this application, the two-dimensional pose sequence of the target object at each viewpoint is obtained, and three-dimensional feature extraction processing is performed on the two-dimensional pose sequence to obtain the three-dimensional pose features of each viewpoint. This initially realizes the improvement from two-dimensional pose to three-dimensional pose. By aggregating the three-dimensional pose of each viewpoint into multi-view features, the three-dimensional poses of different viewpoints are explicitly fused. By obtaining the common features of the target object at each viewpoint, the correlation between the three-dimensional poses of different viewpoints is explicitly learned. Furthermore, by fusing the multi-view features and common features, the target three-dimensional pose sequence at each viewpoint is obtained. This achieves the beneficial effect of explicitly learning the correlation between different viewpoints and improving the accuracy of three-dimensional pose detection.

[0426] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An attitude detection method, characterized in that, The method includes: Acquire image data of the target object from multiple perspectives, and acquire a two-dimensional pose sequence of the target object from each perspective; The three-dimensional feature extraction process is performed on the two-dimensional pose sequence of the target object in each of the said views to obtain the three-dimensional pose features of the target object in each of the said views; The three-dimensional pose features of the target object in each of the aforementioned viewpoints are aggregated into multi-view features; Extract the common features of the target object among the three-dimensional pose features of each viewpoint; The multi-view features and the common features are fused to obtain fused features; Based on the fusion features, a three-dimensional pose prediction process is performed to obtain the target object's target three-dimensional pose sequence in each of the aforementioned viewpoints.

2. The method according to claim 1, characterized in that, The three-dimensional pose features of each viewpoint include the temporal features of the viewpoint, and the temporal features of each viewpoint characterize the temporal correlation of the two-dimensional poses in the two-dimensional pose sequence of the viewpoint; The step of aggregating the three-dimensional pose features of the target object in each of the aforementioned viewpoints into multi-view features includes: The temporal features of each of the aforementioned perspectives are concatenated into a combined temporal feature; The combined temporal features are subjected to feature encoding processing to obtain temporal encoded features; The temporal coding features are subjected to a first attention coding process to obtain the first attention coding features; Feature mapping is performed based on the first attention-encoded feature and the temporal-encoded feature to obtain multi-view features.

3. The method according to claim 1, characterized in that, The three-dimensional pose features of each viewpoint include an intermediate three-dimensional pose sequence of the viewpoint, and the extraction of common features of the target object among the three-dimensional pose features of each viewpoint includes: The intermediate three-dimensional pose sequences of each of the aforementioned viewpoints are concatenated into a combined pose sequence; The combined pose sequence is subjected to feature encoding processing to obtain pose encoding features; A second attention encoding process is performed based on the first learnable input and the pose encoding features to obtain the first encoding features; A third attention encoding process is performed based on the second learnable input and the first encoding feature to obtain a second encoding feature, which is then used as the common feature.

4. The method according to claim 3, characterized in that, The second attention encoding process based on the first learnable input and the pose encoding features to obtain the first encoding features includes: The first learnable input is used as a query vector, and the pose encoding features are used as a key vector and a value vector. The first attention weight is determined based on the key vector and the query vector. The value vector is weighted using the first attention weight to obtain the second attention feature; The second attention feature is subjected to a first feature mapping process to obtain the first encoded feature.

5. The method according to claim 3, characterized in that, The first encoding feature includes multiple data dimensions. After obtaining the first encoding feature, the method further includes: The multiple data dimensions of the first encoded feature are rearranged to obtain the updated first encoded feature. The updated first encoded feature replaces the original first encoded feature, and the process proceeds to the third attention encoding process based on the second learnable input and the first encoded feature to obtain the second encoded feature.

6. The method according to claim 3, characterized in that, The third attention encoding process based on the second learnable input and the first encoded features to obtain the second encoded features includes: The second attention weight is determined based on the second learnable input and the first encoded features; The first encoded feature is weighted by the second attention weight to obtain the third attention feature; The third attention feature is subjected to a second feature mapping process to obtain the second encoded feature.

7. The method according to claim 1, characterized in that, The image data for each viewpoint includes multiple frames of images; obtaining the two-dimensional pose sequence of the target object in each viewpoint includes: For each of the said viewpoints, the following processing is performed on the multi-frame images: Image feature extraction is performed on each frame of the image from the aforementioned viewpoint to obtain the image features of each frame. Based on the image features of the target object in each frame of the image, key point detection processing is performed to obtain the coordinates of multiple key points of the target object in each frame of the image; The coordinates of multiple key points of the target object in each frame of the image are combined to obtain a two-dimensional pose sequence of the target object at the viewpoint.

8. The method according to claim 1, characterized in that, The step of performing three-dimensional feature extraction processing on the two-dimensional pose sequence of the target object in each of the said views to obtain the three-dimensional pose features of the target object in each of the said views includes: For each of the two-dimensional pose sequences from the given viewpoints, the following processing is performed: Embedding encoding is performed on the two-dimensional poses in the two-dimensional pose sequence to obtain the embedding encoding features of each two-dimensional pose; The position of the two-dimensional pose in the two-dimensional pose sequence is encoded to obtain the position encoding feature of each two-dimensional pose. Feature encoding is performed based on the embedding encoding features and the position encoding features of each two-dimensional pose to obtain the three-dimensional pose features of the viewpoint.

9. The method according to claim 1, characterized in that, The feature fusion process, which integrates the multi-view features and the shared features to obtain fused features, includes: The multi-view features are weighted using a first learnable weight to obtain a first weighted feature. The common features are weighted using a second learnable weight to obtain a second weighted feature. The sum of the first weighted feature and the second weighted feature is taken as the fusion feature.

10. The method according to claim 9, characterized in that, The three-dimensional pose features of each viewpoint include the temporal features of the viewpoint and an intermediate three-dimensional pose sequence. Before performing feature fusion processing on the multi-viewpoint features and the common features to obtain fused features, the method further includes: The pose encoding features and the combined temporal features are concatenated to obtain the concatenated features. The pose encoding features are obtained by concatenating the intermediate three-dimensional pose sequences of each viewpoint into a combined pose sequence and performing feature encoding on the combined pose sequence. The combined temporal features are obtained by concatenating the temporal features of each viewpoint. The concatenated features are weighted by pre-trained weight parameters to obtain weighted concatenated features; The weighted splicing features are subjected to nonlinear mapping processing to obtain the first learnable weight and the second learnable weight.

11. A training method for a pose detection model, characterized in that, The method includes: Obtain sample image data of the sample object from multiple perspectives, and perform the following processing using the pose detection model to be trained: Obtain two-dimensional pose sequence samples of the sample object from each viewpoint; Three-dimensional feature extraction processing is performed on the two-dimensional pose sequence samples of the sample object in each of the said views to obtain the three-dimensional pose feature samples of the sample object in each of the said views; The sample object's three-dimensional pose feature samples from each of the aforementioned viewpoints are aggregated into multi-view feature samples; Extract the common feature samples of the sample object among the three-dimensional pose feature samples of each viewpoint; The multi-view feature samples and the common feature samples are subjected to feature fusion processing to obtain fused feature samples; Based on the fused feature samples, a 3D pose prediction process is performed to obtain the target 3D pose sequence samples of the sample object in each of the aforementioned viewpoints; The combined loss value is determined based on the two-dimensional attitude sequence sample, the three-dimensional attitude feature sample, and the target three-dimensional attitude sequence sample; The parameters of the pose detection model to be trained are updated based on the combined loss value to obtain the trained pose detection model.

12. The method according to claim 11, characterized in that, The determination of the combined loss value based on the two-dimensional attitude sequence samples, the three-dimensional attitude feature samples, and the target three-dimensional attitude sequence samples includes: A first loss value is determined based on the target three-dimensional attitude sequence sample and the two-dimensional attitude sequence sample; A second loss value is determined based on the three-dimensional pose feature samples and the target three-dimensional pose sequence samples; A third loss value is determined based on the three-dimensional pose feature samples and the target three-dimensional pose sequence samples; The first loss value, the second loss value, and the third loss value are weighted and summed to obtain the combined loss value.

13. The method according to claim 12, characterized in that, Determining the first loss value based on the target three-dimensional attitude sequence sample and the two-dimensional attitude sequence sample includes: A first projection loss value is determined based on the target three-dimensional attitude sequence sample and the two-dimensional attitude sequence sample; Determine the multi-view consistency loss value based on the target 3D attitude sequence sample and the 2D attitude sequence sample; The camera rotation loss value is determined based on the target 3D attitude sequence sample and the 2D attitude sequence sample; The first loss value is obtained by weighted summation of the first projection loss value, the multi-view consistency loss value, and the camera rotation loss value.

14. The method according to claim 12, characterized in that, Determining the second loss value based on the three-dimensional pose feature samples and the target three-dimensional pose sequence samples includes: Perform the following processing for each of the aforementioned viewpoints: Obtain the first ratio of the three-dimensional pose feature sample to the regularization matrix of the three-dimensional pose feature sample; Obtain the second ratio of the target three-dimensional pose sequence sample to the regularization matrix of the target three-dimensional pose sequence sample; Obtain the second difference between the first ratio and the second ratio, and perform regularization on the second difference to obtain the sub-loss value; The sum of the sub-loss values ​​corresponding to each of the aforementioned viewpoints is used as the second loss value.

15. The method according to claim 12, characterized in that, The step of determining the third loss value based on the three-dimensional pose feature samples and the target three-dimensional pose sequence samples includes: Perform the following processing for each of the aforementioned viewpoints: The first posture deformation parameters are determined based on the three-dimensional posture feature samples. Determine the second attitude deformation parameters based on the target three-dimensional attitude sequence sample; Obtain the third difference between the first attitude deformation parameter and the second attitude deformation parameter, and perform regularization on the third difference to obtain the sub-loss value; The sum of the sub-loss values ​​corresponding to each of the aforementioned viewpoints is taken as the third loss value.

16. An attitude detection device, characterized in that, The device includes: The data acquisition module is used to acquire image data of the target object from multiple perspectives and to acquire the two-dimensional pose sequence of the target object in each perspective. The data processing module is used to perform three-dimensional feature extraction processing on the two-dimensional pose sequence of the target object in each of the viewpoints to obtain the three-dimensional pose features of the target object in each of the viewpoints. The data processing module is also used to aggregate the three-dimensional pose features of the target object in each of the said views into multi-view features; The data processing module is also used to extract common features of the target object among the three-dimensional pose features of each viewpoint; The data processing module is also used to perform feature fusion processing on the multi-view features and the common features to obtain fused features; The data processing module is also used to perform three-dimensional pose prediction processing based on the fused features to obtain the target three-dimensional pose sequence of the target object in each of the aforementioned viewpoints.

17. A training device for a posture detection model, characterized in that, The device includes: The data acquisition module is used to acquire sample image data of the sample object from multiple perspectives and perform the following processing through the pose detection model to be trained; The training module is used to obtain two-dimensional pose sequence samples of the sample object at each viewpoint; The training module is also used to perform three-dimensional feature extraction processing on the two-dimensional pose sequence of the sample object in each of the viewpoints to obtain the three-dimensional pose feature sample of the sample object in each of the viewpoints. The training module is also used to aggregate the three-dimensional pose feature samples of the sample object in each of the viewpoints into multi-view feature samples; The training module is also used to extract common feature samples of the sample objects among the three-dimensional pose feature samples of each viewpoint; The training module is also used to perform feature fusion processing on the multi-view feature samples and the common feature samples to obtain fused feature samples; The training module is also used to perform three-dimensional pose prediction processing based on the fused feature samples to obtain the target three-dimensional pose sequence samples of the sample object in each of the viewpoints; The training module is also used to determine a combined loss value based on the two-dimensional pose sequence samples, the three-dimensional pose feature samples, and the target three-dimensional pose sequence samples; The training module is also used to update the parameters of the pose detection model to be trained based on the combined loss value, so as to obtain the trained pose detection model.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor is configured to execute computer-executable instructions or computer programs stored in the memory to implement the attitude detection method according to any one of claims 1 to 10, or to implement the training method for the attitude detection model according to any one of claims 11 to 15.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the attitude detection method according to any one of claims 1 to 10, or the training method for the attitude detection model according to any one of claims 11 to 15.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the attitude detection method according to any one of claims 1 to 10, or the training method for the attitude detection model according to any one of claims 11 to 15.

Images