A training method and estimation method for a single-view multi-person 3D pose estimation model

Abstract

The present application discloses a training method and estimation method for a single-view multi-person 3D pose estimation model. The training method comprises: collecting multi-person pose images; processing the multi-person pose images using a structured pose representation to obtain a set of joint points corresponding to the multi-person pose images; generating mask values ​​according to a preset masking strategy, masking the joint point set using the mask values ​​to obtain a masked pose tensor; and pre-training a backbone network for 3D pose estimation using the masked pose tensor to obtain a trained backbone network. By introducing pose mask training, the present application enables the model to learn and adapt to various possible occlusion situations, thereby improving the model's robustness and estimation accuracy in complex motion environments.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a training method and an estimation method for a single-view multi-person three-dimensional posture estimation model. Background Art

[0002] Multi-person 3D pose estimation is an important research area in computer vision. It aims to infer the positions of key joints of multiple people by analyzing image or video data. This technology has applications in many industries, such as group motion analysis, intelligent surveillance, and virtual reality.

[0003] However, human pose estimation in single-view, multi-person scenarios faces significant challenges. First, due to the lack of depth information, the model cannot accurately restore the relative positions of multiple people in three-dimensional space in a single view. In particular, when limbs overlap, the model struggles to infer their true spatial distribution from a single viewpoint. Second, multi-person interaction scenarios are prone to occlusion, which affects the accurate detection of joint positions. Furthermore, the inconsistency in the timing of multiple people's movements in dynamic scenes makes it difficult for the model to effectively separate the motion characteristics of individuals in the time series, further complicating pose estimation.

[0004] In response to the above challenges, how to provide a training method that can be used for a single-view multi-person 3D pose estimation model is a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0005] In order to solve the above technical problems, the purpose of this application is to provide a training method and estimation method for a single-view multi-person three-dimensional posture estimation model. By adopting a variety of masking strategies, such as masking a small number of joints or entire limbs, etc., to simulate common occlusion phenomena in actual scenes, it is prompted to extract and learn useful information from the data, thereby improving the robustness and reasoning ability of the model in occlusion scenarios.

[0006] To achieve the above objectives, this application provides a training method for a single-view multi-person 3D pose estimation model:

[0007] The above-mentioned application objective 1 of this application is achieved through the following technical solutions:

[0008] A training method for a single-view multi-person 3D pose estimation model, comprising:

[0009] Collect multi-person posture images;

[0010] Using structured posture representation, the multi-person posture image is processed to obtain a set of joint points corresponding to the multi-person posture image;

[0011] Generate a mask value corresponding to each joint point in the joint point set according to a preset mask strategy;

[0012] Using the mask value, masking the joint point set to obtain a mask posture tensor;

[0013] The mask pose tensor is used to pre-train a backbone network for three-dimensional pose estimation to obtain a trained backbone network.

[0014] Preferably, the joint point set is defined as , the joint point set After a fully connected layer processing, a new joint point set P is obtained. ; Wherein R is the entity space, N is the number of people in the multi-person posture picture, T is the number of time frames, K is the number of joint points contained in each person in the multi-person posture picture, D is the feature dimension, and each element in the joint point set P represents the d-th coordinate value of the j-th joint point of the i-th person in the t-th frame, i∈{1,2,...,N}, t∈{1,2,...,T}, j∈{1,2,...,K};

[0015] Define the mask value set M, , where R is the entity space, N is the number of people in the multi-person pose picture, T is the number of time frames, K is the number of joint points contained in each person in the multi-person pose picture, and each element in the mask value set Represents the mask value of the j-th joint point of the i-th person in the t-th frame, The calculation formula is:

[0016] ;

[0017] in, Represents visible, Indicates being blocked, if is visible, then The value is 1 if is obscured, then The value is 0;

[0018] The method of using the mask value to perform mask processing on the joint point set to obtain a mask posture tensor includes:

[0019] Each mask value Fusion corresponding coordinate values , get the mask pose tensor after occlusion , the calculation formula is:

[0020] .

[0021] Preferably, the backbone network includes: a single-person spatial attention module, a multi-person spatial attention module, a single-person joint temporal attention module, a multi-person joint temporal attention module and a single-person spatiotemporal attention module connected in sequence.

[0022] Preferably, the single-person spatial attention module is used to:

[0023] The mask pose tensor is used as the first input tensor X of this module. , where R is the entity space, N is the number of people in the multi-person pose image, T is the number of time frames, K is the number of joint points contained in each person in the multi-person pose image, and D is the feature dimension;

[0024] According to the first input tensor X, for each human instance i and time t, extract the corresponding first feature unit token , token , , represents the high-dimensional input feature vector of the t-th frame of the i-th person, i∈{1,2,...,N}, t∈{1,2,...,T};

[0025] Through the spatial attention mechanism, according to the first feature unit token , get each feature unit token of the first output tensor Y , token .

[0026] Preferably, the multi-person spatial attention module is used to:

[0027] The first output tensor Y in the single-person spatial attention module is used as the second input tensor of this module ;

[0028] According to the second input tensor , extracted Second feature unit token , token ,in, Represents the high-dimensional vector of input features of the t-th frame;

[0029] Through the spatial attention mechanism, according to the second feature unit token , get the second output tensor Each feature unit token , token .

[0030] Preferably, the single-person joint temporal attention module includes:

[0031] The second output tensor of the spatial attention module between multiple people As the third input tensor of this module ;

[0032] According to the third input tensor , extract the third feature unit token , token , Represents the high-dimensional input feature vector of the j-th joint of the i-th person;

[0033] Through the temporal attention mechanism, according to the third feature unit token , get the third output tensor Each feature unit token , token .

[0034] Preferably, the multi-person joint temporal attention module includes:

[0035] The third output tensor in the single-person joint temporal attention module As the fourth input tensor of this module ;

[0036] According to the fourth input tensor , extract the fourth feature unit token , token ,in, Represents the high-dimensional vector of input features of the j-th joint of all individuals;

[0037] Through the temporal attention mechanism, according to the fourth feature unit token , and get the fourth output tensor Each feature unit token , token .

[0038] Preferably, the single-person spatiotemporal attention module includes:

[0039] The fourth output tensor in the multi-person joint temporal attention module The fifth input tensor as input to this module ;

[0040] According to the fifth input tensor , extract the fifth feature unit token , token ,in, Represents the high-dimensional vector of input features of the i-th person;

[0041] Through the spatiotemporal attention mechanism, according to the fifth feature unit token , get the fifth output tensor Each feature unit token , token .

[0042] Preferably, the backbone network includes: at least 2 attention units connected in sequence, wherein each of the attention units includes the single-person spatial attention module, the multi-person spatial attention module, the single-person joint timing attention module, the multi-person joint timing attention module and the single-person spatiotemporal attention module connected in sequence.

[0043] The second objective of this application is to provide a single-view multi-person three-dimensional pose estimation method.

[0044] The second object of the present application is achieved through the following technical solutions:

[0045] A single-view multi-person 3D pose estimation method, comprising:

[0046] Collect multi-person posture images;

[0047] Using structured posture representation, the multi-person posture image is processed to obtain a set of joint points corresponding to the multi-person posture image;

[0048] According to the multi-person posture picture, a symmetric affinity matrix in the local space is constructed , and the symmetric affinity matrix in global space , , where R represents the real number space and K is the total number of joint points;

[0049] According to the symmetric affinity matrix in the local space and the symmetric affinity matrix in global space , get the overall topology matrix A;

[0050] The joint point set is passed through the fully connected layer to obtain a high-dimensional feature vector P, and the overall topological matrix A and the high-dimensional feature vector P are subjected to matrix multiplication to obtain a high-dimensional feature vector ;

[0051] The high-dimensional feature vector , input into the trained backbone network obtained by the training method of the single-view multi-person 3D pose estimation model as described above, perform 3D mapping through the multi-layer perceptron to obtain the pose estimation result.

[0052] This application provides a training method for a single-view multi-person 3D pose estimation model. By introducing pose mask training and adding masks to the input data during the training phase to simulate information loss, the model can learn and adapt to various possible occlusion situations, thereby improving the model's robustness and estimation accuracy in complex motion environments.

[0053] This application also introduces the Transformer attention and temporal attention mechanisms to efficiently capture the contextual information of the human body in time series, fully understand the complex dependencies between joints, and realize the joint learning and reasoning of the root joint position and body joint displacement of the model, thereby improving the model's learning ability in time series.

[0054] This application also introduces additional posture information on the structured posture, combines the local and global spatial topological relationships of individual joints, and fuses the eigenvectors of the joint spatial relationship in the posture feature tensor to provide more relational constraints and expressions for posture estimation, thereby enhancing the model's understanding of depth information. Finally, the trained backbone network is used to perform three-dimensional posture estimation, thereby achieving accurate estimation of the posture after occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 This is a schematic diagram of a single-view multi-person 3D pose estimation model training method in an embodiment of the present application;

[0057] Figure 2 This is a schematic diagram of posture mask training in an embodiment of the present application;

[0058] Figure 3 This is a diagram of the backbone network architecture in the embodiment of this application;

[0059] Figure 4 Schematic diagram of a single-view multi-person 3D posture model estimation method in an embodiment of the present application. DETAILED DESCRIPTION

[0060] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0061] In addition, the technical features of the various embodiments or single embodiments provided in this application can be arbitrarily combined with each other to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or structural composition pattern, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0062] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are merely illustrative. For example, the division of units and modules is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or modules can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling communication connection between the components shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0063] In addition, all functional units in the embodiments of the present application may be integrated into one processor, or each unit may be a separate device, or two or more units may be integrated into one device; each functional unit in the embodiments of the present application may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0064] Those skilled in the art will understand that all or part of the steps of the following method embodiments can be implemented by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, the steps of the following method embodiments are executed; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROMs), magnetic disks, optical disks, and other media that can store program codes.

[0065] It should be understood that the use of "system," "device," "unit," and / or "module" in this application is merely a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0066] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout the description of this application, "plurality" or "several" means two or more, unless otherwise specifically defined.

[0067] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which this application can be implemented. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in this application without affecting the efficacy and purpose that can be achieved by this application.

[0068] If a flow chart is used in this application, the flow chart is used to illustrate the operations performed by the system according to the embodiments of the application. It should be understood that the previous or subsequent operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. At the same time, other operations can be added to these processes, or one or more operations can be removed from these processes.

[0069] It should also be noted that, in this document, terms such as "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that an article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such article or device. In the absence of further limitations, elements defined by the phrase "comprises a ..." do not exclude the presence of additional identical elements in the article or device comprising the aforementioned elements.

[0070] The implementation methods of this application are written in a progressive manner.

[0071] like Figure 1 As shown, the embodiment of the present application provides a single-view multi-person 3D pose estimation model training method, comprising:

[0072] S101, collecting multi-person posture images;

[0073] Specifically, the multi-person pose images include various pose datasets, namely: MuCo-3DHP, MuPoTS-3D, Human3.6M, MPI-INF-3DHP and CMU Panoptic datasets.

[0074] S102, using structured posture representation to process the multi-person posture image to obtain a set of joint points corresponding to the multi-person posture image;

[0075] Specifically, the conventional multi-person pose estimation method estimates the human pose by inferring the body joint coordinates of all human instances in the image. Let N be the number of people in the image, K be the number of joints contained in each human body, represents the coordinates of the jth body joint point of the i-th person, Represents the set of joint points of all human instances in the image, expressed as:

[0076] ;

[0077] Therefore, the joint point coordinates in the 2D case are , then the joint point coordinates in 3D are This application uses SPR (Structured Pose Representation), which aims to unify the position information of human instances and human joints, providing a single-stage solution for multi-person pose estimation. This application introduces a root joint in SPR to represent the position of human instances.

[0078] Replace the 2D coordinates with 3D coordinates, using represents the r-root joint coordinate position of the i-th person, It is expressed as the coordinate position of the jth joint of the i-th person, Represented as the coordinate position of the jth body joint of the i-th person Relative to the introduced r root joint coordinate position The displacement of the r root joint and the jth body joint position is structured as follows:

[0079] ;

[0080] Therefore, the set of joint points of the human body posture can be expressed as:

[0081] ;

[0082] in, is the set of joint points of the SPR of all human instances in the image data, N is the number of people in the image data, and K is the number of joint points contained in each human body.

[0083] Based on the structured pose, additional pose information is introduced to model the joints. The local and global spatial topological relationships of individual joints are combined, and the eigenvectors of the joint spatial relationships are fused into the pose feature tensor to provide more relational constraints and expressions for pose estimation, thereby enhancing the model's understanding of depth information.

[0084] S103, generating a mask value corresponding to each joint point in the joint point set according to a preset mask strategy;

[0085] Specifically, according to the mask strategy set in advance, the situation where some postures are blocked is simulated, key features are extracted from the data, and mask values ​​are generated;

[0086] The masking strategies are divided into the following categories:

[0087] Full-body joint masking: Randomly select some human joints in the data for masking, such as wrists and knees. This strategy can help the model use the contextual information of full-body joints to infer the locations of occluded key points.

[0088] Full limb segment masking: Randomly select an entire limb segment, such as an arm or a leg, to mask. This strategy simulates severe occlusions and allows the model to infer the pose of the occluded limb from the remaining visible joints.

[0089] Limb joint masking: Randomly select some joints in a limb segment for masking and remove all parts except the limb. This strategy enables the model to infer the position of the invisible joints in the limb based on the contextual information of the limb.

[0090] In other embodiments of the present application, one implementation of this step is:

[0091] Choose to cover some joints or limbs of the human body randomly or according to a set ratio, then the set of multi-person 2D posture tensors is , the mask value set is M, ;

[0092] Among them, R represents the real space, N is the number of people in the image (i∈{1,2,...,N}), T is the number of time frames (t∈{1,2,...,T}), K is the number of joints contained in each human body (j∈{1,2,...,K}), D is the feature dimension, and the set After a fully connected layer, we get the set P. ;

[0093] In the set P, represents the dth coordinate value of the jth joint point of the i-th person in the t-th frame, The mask value of the j-th joint point of the i-th person in the t-th frame is expressed as:

[0094] ;

[0095] Among them, visible represents visible, occluded represents occluded, that is, if Visible, then is 1, if If blocked, is 0.

[0096] S104. Using the mask value, mask the joint point set to obtain a mask posture tensor;

[0097] Specifically, according to the mask value, the pose tensor is fused to obtain the mask pose tensor , the calculation formula is:

[0098] .

[0099] S105. Using the masked pose tensor, pre-train the backbone network for 3D pose estimation to obtain a trained backbone network. The training process is as follows: Figure 2 shown.

[0100] Through the above steps, by introducing additional posture information, the joint relationship is structured, and through posture mask training, the backbone network can learn and adapt to various possible occlusion situations, prompting it to extract and learn how to restore occluded human joints from the data, thereby improving the robustness and estimation accuracy of the model in complex motion environments.

[0101] Currently, human pose estimation in single-view, multi-person scenarios faces significant challenges. First, due to the lack of depth information, models cannot accurately restore the relative positions of multiple people in three-dimensional space in a single view. Especially when limbs overlap, it is difficult for the model to infer the true spatial distribution from a single viewpoint. Second, multi-person interaction scenarios are prone to occlusion, which affects the accurate detection of joint positions. Furthermore, the inconsistency of the timing of multiple people's movements in dynamic scenes makes it difficult for the model to effectively separate the motion characteristics of individuals in the time series, further complicating pose estimation.

[0102] The above-mentioned embodiment, through the introduction of pose mask training, strengthens the backbone network's ability to learn contextual information and structural priors, enabling the backbone network to learn and adapt to various possible occlusion situations. By employing various masking strategies, such as masking a small number of joints or entire limbs, to simulate common occlusions in real-world scenarios, the backbone network is encouraged to extract and learn useful information from the data, thereby improving the model's robustness and reasoning capabilities in occluded scenarios.

[0103] Preferably, the backbone network specifically includes, for example Figure 3As shown, the single-person spatial attention module, the multi-person spatial attention module, the single-person joint temporal attention module, the multi-person joint temporal attention module and the single-person spatiotemporal attention module are connected in sequence.

[0104] By introducing the temporal attention mechanism, the backbone network efficiently captures the contextual information of the human body in the time series, comprehensively understands the complex dependencies between joints, realizes the joint learning and reasoning of the root joint position and body joint displacement, and improves the model's learning ability in the time series.

[0105] In some embodiments, the single-person spatial attention module is configured to: take the mask pose tensor as the first input tensor X of this module, , where R is the entity space, N is the number of people in the multi-person pose image, T is the number of time frames, K is the number of joint points of each person in the multi-person pose image, and D is the feature dimension; according to the first input tensor X, for each human instance i and time t, extract the corresponding first feature unit token , token (where token represents the smallest feature unit mapped from the key point position coordinates). Represents the input feature high-dimensional vector of the t-th frame of the i-th person, i∈{1,2,...,N}, t∈{1,2,...,T}; through the spatial attention mechanism, according to the first feature unit token , get each feature unit token of the first output tensor Y , token .

[0106] In some embodiments, the multi-person spatial attention module is used to: take the first output tensor Y in the single-person spatial attention module as the second input tensor of this module ; According to the second input tensor , extracted Second feature unit token , token ; Through the spatial attention mechanism, according to the second feature unit token , get the second output tensor Each feature unit token , token .

[0107] In some embodiments, the single-person joint temporal attention module is used to: As the third input tensor of this module ; According to the third input tensor , extract the third feature unit token , token ; Through the temporal attention mechanism, according to the third feature unit token , and get the third output tensor Each feature unit token , token .

[0108] In some embodiments, the multi-person joint temporal attention module is used to: convert the third output tensor in the single-person joint temporal attention module As the fourth input tensor of this module ; According to the fourth input tensor , extract the fourth feature unit token , token ; Through the temporal attention mechanism, according to the fourth feature unit token , and get the fourth output tensor Each feature unit token , token .

[0109] In some embodiments, the single-person spatiotemporal attention module is used to: convert the fourth output tensor in the multi-person joint temporal attention module The fifth input tensor as input to this module ; According to the fifth input tensor , extract the fifth feature unit token , token ; Through the spatiotemporal attention mechanism, according to the first feature unit token , and get the fifth output tensor Each feature unit token , token .

[0110] In some embodiments, the backbone network is optionally composed of at least two attention units connected in sequence, wherein each attention unit includes a single-person spatial attention module, a multi-person spatial attention module, a single-person joint timing attention module, a multi-person joint timing attention module and a single-person spatiotemporal attention module connected in sequence.

[0111] Through the multi-layer superposition of self-attention mechanisms, higher-level and more abstract feature representations can be extracted layer by layer, and context dependencies can be iteratively refined in each layer to capture complex semantic associations from local to global.

[0112] like Figure 4As shown, the embodiment of the present application provides a single-view multi-person 3D pose estimation method, comprising:

[0113] S201, collecting multi-person posture images;

[0114] Specifically, single-view multi-person 3D pose images that require pose estimation are collected.

[0115] S202, using structured posture representation to process the multi-person posture image to obtain a set of joint points corresponding to the multi-person posture image;

[0116] S203: Construct a symmetric affinity matrix in the local space based on the multi-person posture image , and the symmetric affinity matrix in global space , , where R represents the real number space and K is the total number of joint points;

[0117] Specifically, based on the local topological relationship of the body's skeletal connections, a symmetric affinity matrix in the local space is constructed. , If two joints in the human body structure are physically connected, the corresponding element in the affinity matrix is ​​non-zero, otherwise it is zero. Based on the mutual connection between joints in any posture during the movement process, a symmetric affinity matrix in the global space is constructed. , .

[0118] S204, according to the symmetric affinity matrix in the local space and the symmetric affinity matrix in global space , get the overall topology matrix A;

[0119] Specifically, by fusing the symmetric affinity matrix in the local space and the symmetric affinity matrix in global space , and the overall topology matrix A is obtained, and the calculation formula is:

[0120] ;

[0121] S205, the joint point set is passed through the fully connected layer to obtain a high-dimensional feature vector P, and the overall topology matrix A and the high-dimensional feature vector P are multiplied to obtain a high-dimensional feature vector ;

[0122] Specifically, the joint point set After the fully connected layer, the high-dimensional feature vector P is obtained. The overall topological matrix A and the high-dimensional feature vector P are multiplied by the matrix to obtain the high-dimensional feature vector , the calculation formula is:

[0123] .

[0124] S206, the fused high-dimensional feature vector , input into the trained backbone network obtained by the training method of the single-view multi-person 3D pose estimation model as above, perform 3D mapping through the multi-layer perceptron to obtain the pose estimation result.

[0125] Specifically, the fused high-dimensional feature vector It is input as the first input tensor into the trained backbone network, and the high-dimensional feature dimension is mapped into three dimensions through a multi-layer perceptron, and finally the three-dimensional posture estimation result is obtained.

[0126] This application provides a single-view multi-person three-dimensional posture estimation method. By combining the local and global spatial topological relationships of individual joints, the eigenvectors of the joint spatial relationship are integrated into the posture feature tensor, and the input is input into a trained backbone network, so that the joint position of the occluded part can be estimated more accurately for the occluded three-dimensional posture.

[0127] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0128] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0129] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for a single-view multi-person 3D pose estimation model, characterized in that: include: Collect multi-person posture images; Using structured posture representation, the multi-person posture image is processed to obtain a set of joint points corresponding to the multi-person posture image; Generate a mask value corresponding to each joint point in the joint point set according to a preset mask strategy; Using the mask value, masking the joint point set to obtain a mask posture tensor; Pre-training a backbone network for three-dimensional pose estimation using the mask pose tensor to obtain a trained backbone network; Among them, the mask posture tensor defines the joint point set , the joint point set After a fully connected layer processing, a new joint point set P is obtained. , where R is the physical space, N is the number of people in the multi-person pose picture, T is the number of time frames, K is the number of joint points for each person in the multi-person pose picture, D is the feature dimension, and each element in the joint point set P represents the d-th coordinate value of the j-th joint point of the i-th person in the t-th frame, i∈{1,2,...,N}, t∈{1,2,...,T}, j∈{1,2,...,K}; Define the mask value set M, , R is the entity space, N is the number of people in the multi-person pose picture, T is the number of time frames, K is the number of joint points contained in each person in the multi-person pose picture, wherein each element in the mask value set Represents the mask value of the j-th joint point of the i-th person in the t-th frame, The calculation formula is: ； in, Represents visible, Indicates being blocked, if is visible, then The value is 1 if is obscured, then The value is 0; The method of using the mask value to perform mask processing on the joint point set to obtain a mask posture tensor includes: Each mask value Fusion corresponding coordinate values , get the mask pose tensor after occlusion , the calculation formula is: 。 2. The method for training a single-view multi-person 3D pose estimation model according to claim 1, characterized in that: The backbone network includes: a single-person spatial attention module, a multi-person spatial attention module, a single-person joint temporal attention module, a multi-person joint temporal attention module and a single-person spatiotemporal attention module, which are connected in sequence.

3. The training method of a single-view multi-person 3D pose estimation model according to claim 2, characterized in that: The single-person spatial attention module is used to: The mask pose tensor is used as the first input tensor X of this module. , where R is the entity space, N is the number of people in the multi-person pose image, T is the number of time frames, K is the number of joint points contained in each person in the multi-person pose image, and D is the feature dimension; According to the first input tensor X, for each human instance i and time t, extract the corresponding first feature unit token , token , Represents the high-dimensional input feature vector of the t-th frame of the i-th person, i∈{1,2,...,N}, t∈{1,2,...,T}; Through the spatial attention mechanism, according to the first feature unit token , get each feature unit token of the first output tensor Y , token .

4. The method for training a single-view multi-person 3D pose estimation model according to claim 2, wherein: The multi-person spatial attention module is used to: The first output tensor Y in the single-person spatial attention module is used as the second input tensor of this module ; According to the second input tensor , extracted Second feature unit token , token ,in, Represents the high-dimensional vector of input features of the t-th frame; Through the spatial attention mechanism, according to the second feature unit token , get the second output tensor Each feature unit token , token .

5. The method for training a single-view multi-person 3D pose estimation model according to claim 2, wherein: The single-person joint temporal attention module includes: The second output tensor of the spatial attention module between multiple people As the third input tensor of this module ; According to the third input tensor , extract the third feature unit token , token , Represents the high-dimensional input feature vector of the j-th joint of the i-th person; Through the temporal attention mechanism, according to the third feature unit token , and get the third output tensor Each feature unit token , token .

6. The method for training a single-view multi-person 3D pose estimation model according to claim 2, wherein: The multi-person joint temporal attention module includes: The third output tensor in the single-person joint temporal attention module As the fourth input tensor of this module ; According to the fourth input tensor , extract the fourth feature unit token , token ,in, Represents the high-dimensional vector of input features of the j-th joint of all individuals; Through the temporal attention mechanism, according to the fourth feature unit token , and get the fourth output tensor Each feature unit token , token .

7. The method for training a single-view multi-person 3D pose estimation model according to claim 2, wherein: The single-person spatiotemporal attention module includes: The fourth output tensor in the multi-person joint temporal attention module The fifth input tensor as input to this module ; According to the fifth input tensor , extract the fifth feature unit token , token ,in, Represents the high-dimensional vector of input features of the i-th person; Through the spatiotemporal attention mechanism, according to the fifth feature unit token , get the fifth output tensor Each feature unit token , token .

8. The method for training a single-view multi-person 3D pose estimation model according to claim 2, wherein: The backbone network includes: at least two attention units connected in sequence, wherein each of the attention units includes the single-person spatial attention module, the multi-person spatial attention module, the single-person joint timing attention module, the multi-person joint timing attention module and the single-person spatiotemporal attention module connected in sequence.

9. A single-view multi-person 3D pose estimation method, characterized in that: include: Collect multi-person posture images; Using structured posture representation, the multi-person posture image is processed to obtain a set of joint points corresponding to the multi-person posture image; According to the multi-person posture picture, a symmetric affinity matrix in the local space is constructed , and the symmetric affinity matrix in global space , , where R represents the real number space and K is the total number of joint points; According to the symmetric affinity matrix in the local space and the symmetric affinity matrix in global space , get the overall topology matrix A; The joint point set is passed through the fully connected layer to obtain a high-dimensional feature vector P, and the overall topological matrix A and the high-dimensional feature vector P are subjected to matrix multiplication to obtain a high-dimensional feature vector ; The high-dimensional feature vector , input into the trained backbone network obtained by the training method of the single-view multi-person three-dimensional pose estimation model described in any one of claims 1-8, perform three-dimensional mapping through a multi-layer perceptron, and obtain a pose estimation result.

Images