A method for reconstructing a three-dimensional model of a target object

Abstract

This application provides a method for reconstructing a three-dimensional model of a target object, comprising: obtaining image feature information corresponding to each video frame image in a video to be detected and a two-dimensional key point sequence of the target object; estimating and obtaining a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object based on the two-dimensional key point sequence of the target object; concatenating the image feature information, the two-dimensional key point sequence of the target object, and the three-dimensional key point sequence of the target object to obtain a feature sequence corresponding to the target object; and obtaining a three-dimensional model of the target object based on the feature sequence corresponding to the target object. This method uses the key point sequence as part of the model input elements, improving the accuracy of the model key point prediction and making the posture changes between the three-dimensional models of the target object corresponding to each video frame image in the video to be detected smoother and more realistic.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, training method, device, electronic device, and computer-readable storage medium for reconstructing a three-dimensional model of a target object. Background Art

[0002] With the rapid development of computer technology, the use of 3D reconstruction technology to reconstruct 3D models of objects from images or videos has been widely used in various fields such as virtual reality (VR), augmented reality (AR), mixed reality (MR), human-computer interaction, motion analysis, medical care, etc.

[0003] Existing video-based 3D model reconstruction techniques primarily utilize pre-trained neural networks to extract features from each video frame, then use the neural network to calculate and predict model parameters for each frame. Finally, a loss function is used to optimize the entire neural network. During the 3D model reconstruction process, the parameterization of the model creates a semantic gap between the model parameters and the image. This can lead to inaccurate predictions of model key points, large variations in the model's pose between each video frame, and unrealistic results. Summary of the Invention

[0004] The present application provides a method for reconstructing a three-dimensional model of a target object to solve the technical problems in existing three-dimensional model reconstruction methods, such as inaccurate prediction of model key points, large and unrealistic changes in the posture of the models corresponding to each video frame image of the video.

[0005] The present invention provides a method for reconstructing a three-dimensional model of a target object, comprising:

[0006] Obtain image feature information corresponding to each video frame image in the video to be detected and a two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected;

[0007] According to the two-dimensional key point sequence of the target object, estimating and obtaining a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object;

[0008] Connecting the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object to obtain a feature sequence corresponding to the target object in each video frame image in the video to be detected;

[0009] According to the feature sequence corresponding to the target object in each video frame image in the video to be detected, a target object three-dimensional model corresponding to the target object in each video frame image in the video to be detected is obtained.

[0010] Optionally, estimating and obtaining a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object based on the two-dimensional key point sequence of the target object includes: converting the two-dimensional key point sequence of the target object into the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object through a posture prior network.

[0011] Optionally, the pose prior network includes a first fully connected layer, a residual block, and a second fully connected layer, and converting the target object two-dimensional key point sequence into the target object three-dimensional key point sequence corresponding to the target object two-dimensional key point sequence through the pose prior network includes:

[0012] Converting the target object's two-dimensional key point sequence into a target object's two-dimensional key point feature vector based on a first fully connected layer;

[0013] Converting the two-dimensional key point feature vector of the target object into a three-dimensional key point feature vector of the target object based on the residual block;

[0014] The three-dimensional key point feature vector of the target object is converted into a three-dimensional key point sequence of the target object based on the second fully connected layer.

[0015] Optionally, the converting of the target object's two-dimensional key point sequence into the target object's three-dimensional key point sequence corresponding to the target object's two-dimensional key point sequence through the posture prior network also includes: using the target object's two-dimensional key point sequence and the target object's three-dimensional key point sequence together as the output of the posture prior network.

[0016] Optionally, the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object are connected to obtain the feature sequence corresponding to the target object in each video frame image in the video to be detected, including: dimensional summing up the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object, and combining the summation results of the same dimension into a feature sequence corresponding to the target object in each video frame image in the video to be detected.

[0017] Optionally, the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object are connected to obtain the feature sequence corresponding to the target object in each video frame image in the video to be detected, and also includes: adding the position vector corresponding to each video frame image in the video to be detected to the feature sequence corresponding to the target object in each video frame image in the video to be detected.

[0018] Optionally, obtaining a target object three-dimensional model corresponding to the target object in each video frame image in the video to be detected according to a feature sequence corresponding to the target object in each video frame image in the video to be detected includes:

[0019] Performing temporal coding processing on a feature sequence corresponding to a target object in each video frame image in the video to be detected based on an encoder, and obtaining a feature sequence corresponding to a target object in each video frame image in the video to be detected after coding;

[0020] Decoding the feature sequence corresponding to the target object in each video frame image in the encoded video to be detected based on a parameter regression network to obtain target object posture parameters corresponding to the target object in each video frame image in the video to be detected, target object shape parameters corresponding to the target object in each video frame image in the video to be detected, and target object camera parameters corresponding to the target object in each video frame image in the video to be detected;

[0021] A target object three-dimensional model corresponding to the target object in each video frame image in the video to be detected is obtained according to the target object posture parameters, the target object shape parameters, and the target object camera parameters.

[0022] The present application also provides a training method for a target object three-dimensional model reconstruction model, which is applied to the target object three-dimensional model reconstruction model. The target object three-dimensional model reconstruction model reconstructs the target object three-dimensional model of at least one video obtained from a data set using the above-mentioned target object three-dimensional model reconstruction method, and obtains target object posture parameters corresponding to the target object in each video frame image in the video, target object shape parameters corresponding to the target object in each video frame image in the video, and target object camera parameters corresponding to the target object in each video frame image in the video. The training method includes:

[0023] Obtaining model parameter loss according to the target object posture parameter and the target object shape parameter;

[0024] Obtaining a two-dimensional key point loss and a three-dimensional key point loss according to the target object posture parameter, the target object shape parameter, and the target object camera parameter;

[0025] Obtaining a generative adversarial loss based on the target object posture parameters;

[0026] Obtaining a total loss of a three-dimensional model of a target object according to the model parameter loss, the two-dimensional key point loss, the three-dimensional key point loss, and the generative adversarial loss;

[0027] The target object three-dimensional model reconstruction model is iteratively trained according to the total loss of the target object three-dimensional model.

[0028] Optionally, obtaining a model parameter loss according to the target object posture parameter and the target object shape parameter includes:

[0029] Obtaining model parameters corresponding to the target object in each video frame image in the video from the data set;

[0030] The model parameter loss is calculated by taking the model parameters corresponding to the target object in each video frame image in the video obtained from the data set as the true value.

[0031] Optionally, obtaining a two-dimensional key point loss and a three-dimensional key point loss according to the target object posture parameter, the target object shape parameter, and the target object camera parameter includes:

[0032] Obtaining a two-dimensional key point sequence corresponding to a target object in each video frame image in the video from the data set;

[0033] The two-dimensional key point loss is calculated by taking a two-dimensional key point sequence corresponding to a target object in each video frame image in the video obtained from the data set as a true value.

[0034] Optionally, the obtaining of two-dimensional key point loss and three-dimensional key point loss according to the target object posture parameter, the target object shape parameter, and the target object camera parameter further includes:

[0035] Calculating a three-dimensional key point sequence corresponding to the target object in each video frame image in the video using the target object posture parameters, the target object shape parameters, and the target object camera parameters;

[0036] The three-dimensional key point loss is calculated by taking the calculated three-dimensional key point sequence corresponding to the target object in each video frame image in the video as the true value.

[0037] Optionally, obtaining the generative adversarial loss based on the target object posture parameters includes: performing loss calculation on the target object posture parameters based on a temporal discriminant network.

[0038] The embodiment of the present application also provides a target object three-dimensional model reconstruction device, characterized by comprising: an acquisition unit, a posture prior unit, a connection unit, and a reconstruction unit;

[0039] The acquisition unit is used to acquire image feature information corresponding to each video frame image in the video to be detected and a two-dimensional key point sequence of the target object corresponding to each video frame image in the video to be detected;

[0040] The posture prior unit is used to estimate and obtain a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object based on the two-dimensional key point sequence of the target object;

[0041] The connection unit is used to connect the image feature information corresponding to each video frame image in the video to be detected, the target object two-dimensional key point sequence corresponding to the target object in each video frame image in the video to be detected, and the target object three-dimensional key point sequence corresponding to the target object two-dimensional key point sequence, to obtain a feature sequence corresponding to the target object in each video frame image in the video to be detected;

[0042] The reconstruction unit is used to obtain a three-dimensional model of the target object corresponding to the target object in each video frame image in the video to be detected based on a feature sequence corresponding to the target object in each video frame image in the video to be detected.

[0043] The embodiment of the present application also provides a training device for reconstructing a three-dimensional model of a target object, characterized by comprising: a reconstruction unit, a model parameter loss acquisition unit, a key point loss acquisition unit, a generative adversarial loss acquisition unit, a total loss acquisition unit, and a training unit;

[0044] The reconstruction unit is configured to reconstruct a three-dimensional model of a target object from at least one video obtained from a data set, and obtain target object posture parameters corresponding to the target object in each video frame image in the video, target object shape parameters corresponding to the target object in each video frame image in the video, and target object camera parameters corresponding to the target object in each video frame image in the video;

[0045] The model parameter loss acquisition unit is used to acquire the model parameter loss according to the target object posture parameter and the target object shape parameter;

[0046] The key point loss acquisition unit is used to acquire two-dimensional key point loss and three-dimensional key point loss according to the target object posture parameter, the target object shape parameter and the target object camera parameter;

[0047] The generative adversarial loss acquisition unit is used to acquire the generative adversarial loss according to the target object posture parameter;

[0048] The total loss acquisition unit is used to acquire the total loss of the target object three-dimensional model according to the model parameter loss, the two-dimensional key point loss, the three-dimensional key point loss, and the generative adversarial loss;

[0049] The training unit is used to iteratively train the target object three-dimensional model reconstruction model according to the total loss of the target object three-dimensional model.

[0050] The embodiment of the present application further provides an electronic device, characterized in that it includes: a collector, a processor and a memory;

[0051] The collector is used to collect the video to be detected;

[0052] The memory is used to store one or more computer instructions;

[0053] The processor is used to execute the one or more computer instructions to implement the above-mentioned method for reconstructing the three-dimensional model of the target object.

[0054] An embodiment of the present application further provides an electronic device, characterized in that it includes: a collector, a processor, and a memory;

[0055] The collector is configured to collect at least one video in the dataset;

[0056] The memory is used to store one or more computer instructions;

[0057] The processor is used to execute the one or more computer instructions to implement the above-mentioned training method for reconstructing a three-dimensional model of a target object.

[0058] An embodiment of the present application also provides a computer-readable storage medium on which one or more computer instructions are stored, characterized in that the instructions are executed by a processor to implement the above method.

[0059] Compared with the prior art, the method for reconstructing a three-dimensional model of a target object provided by the present application includes: obtaining image feature information corresponding to each video frame image in a video to be detected and a two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected; estimating and obtaining a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object based on the two-dimensional key point sequence of the target object; connecting the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object to obtain a feature sequence corresponding to the target object in each video frame image in the video to be detected; obtaining a three-dimensional model of the target object corresponding to the target object in each video frame image in the video to be detected based on the feature sequence corresponding to the target object in each video frame image in the video to be detected.

[0060] The method obtains image feature information corresponding to each video frame image in the video to be detected and a two-dimensional key point sequence corresponding to the target object, and converts the two-dimensional key point sequence of the target object into a three-dimensional key point sequence of the target object. The image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object, and the three-dimensional key point sequence of the target object are used as input elements to obtain a three-dimensional model of the target object corresponding to each video frame image in the video to be detected. The method for reconstructing a three-dimensional model of a target object provided by the application uses the key point sequence as part of the input elements, thereby enhancing the supervision of key points during the reconstruction process of the three-dimensional model of the target object, improving the accuracy of the model key point prediction, and making the posture changes between the three-dimensional models of the target object corresponding to each video frame image in the video to be detected more smooth and realistic. The method is particularly suitable for reconstructing a three-dimensional model of the human body. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 A diagram of an application system of a method for reconstructing a three-dimensional model of a target object provided in an embodiment of the present application;

[0063] Figure 2 A diagram illustrating an application system of another method for reconstructing a three-dimensional model of a target object provided in an embodiment of the present application;

[0064] Figure 3A flowchart of a method for reconstructing a three-dimensional model of a target object provided in one embodiment of the present application;

[0065] Figure 4 A schematic flow chart of a method for reconstructing a three-dimensional model of a target object provided in another embodiment of the present application;

[0066] Figure 5 A flowchart of converting a two-dimensional key point sequence into a three-dimensional key point sequence provided in one embodiment of the present application;

[0067] Figure 6 A schematic diagram of a process for converting a two-dimensional key point sequence into a three-dimensional key point sequence according to another embodiment of the present application;

[0068] Figure 7 A flowchart of obtaining a three-dimensional model of a target object according to a feature sequence provided in one embodiment of the present application;

[0069] Figure 8 An example diagram of a result of a three-dimensional model reconstructed by the method for reconstructing a three-dimensional model of a target object provided by an embodiment of the present application;

[0070] Figure 9 A flowchart of a method for training a target object three-dimensional model reconstruction model provided in another embodiment of the present application;

[0071] Figure 10 A schematic structural diagram of a device for reconstructing a three-dimensional model of a target object provided by another embodiment of the present application;

[0072] Figure 11 A schematic structural diagram of a device for training a target object three-dimensional model reconstruction model according to another embodiment of the present application;

[0073] Figure 12 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;

[0074] Figure 13 A schematic structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to enable those skilled in the art to better understand the technical solutions of this application, the following clearly and completely describes this application in conjunction with the drawings in the embodiments of this application. However, this application can be implemented in many other ways different from the above description. Therefore, based on the embodiments provided in this application, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of this application.

[0076] It should be noted that the terms "first", "second", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than that illustrated or described herein. In addition, the terms "including", "having" and their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0077] The present invention provides a method for reconstructing a 3D model of a target object, including reconstructing a 3D model of a person, animal, machine, or other object contained in a video or image. The present invention mainly uses the human body as the target object to explain the method provided by the present invention in detail.

[0078] The 3D human body model is a parametric model, meaning its pose and shape can be modified through parameters. The shape is typically composed of a standard human vertex model and multiple offset templates. Different body shapes can be obtained through linear combinations of the vertex model and offset templates. The pose is typically composed of joint position and rotation information, and different poses can be achieved through hinge motion and skinning information.

[0079] The Skinned Multi-Person Linear model (SMPL) is a volumetric model based on skinned vertices. It can represent various human shapes and postures, and can show the deformation of soft tissues such as skin caused by the human body in different postures. For example, it can show the bulges and depressions of muscles during limb movement, and can accurately show the morphology of muscle stretching and contraction movements.

[0080] The reconstruction of a skinned multi-body linear model depends on various parameters, including pose parameters, shape parameters, and camera parameters. Pose parameters refer to parameters that can characterize the pose of a 3D human body model, such as whether the person is standing or sitting, whether the mouth is open, smiling or laughing, and the degree of bending of the forearm relative to the upper arm (i.e., the rotation of each joint relative to its parent node). Shape parameters refer to parameters that can characterize the shape of a 3D human body model, such as height, shortness, fatness, or thinness, and facial contours, the shape and size of facial features, etc. Camera parameters refer to parameters that establish a geometric model for camera imaging. The task of reconstructing a 3D human body model is to predict pose parameters, shape parameters, and camera parameters from videos or images.

[0081] Existing methods for reconstructing a 3D human model from video generally involve three steps. First, a pre-trained neural network is used to extract feature information corresponding to each frame of the video to be tested. Second, the neural network is used to calculate the extracted feature information corresponding to each frame of the video to be tested, and the pose parameters, shape parameters, and camera parameters corresponding to the human body in each frame of the video to be tested are predicted. Third, a 3D human model corresponding to the human body in each frame of the video to be tested is reconstructed based on the predicted pose parameters, shape parameters, and camera parameters.

[0082] Since the reconstructed three-dimensional human body model is a parameterized model, and the above-mentioned existing three-dimensional human body model reconstruction method only uses the feature information corresponding to the image as an input element to reconstruct the three-dimensional human body model, there will be a semantic gap between the model parameters and the image, resulting in inaccurate prediction of the model key points, and the model posture corresponding to each video frame image in the video to be detected will vary greatly and be unrealistic.

[0083] To address the problems with the existing methods for reconstructing three-dimensional human models, the present application provides a method for reconstructing a three-dimensional model of a target object. This method uses a sequence of key points (including a two-dimensional key point sequence and a three-dimensional key point sequence) corresponding to each video frame image in the video to be detected, along with feature information, as input elements to reconstruct a three-dimensional human model. Because both two-dimensional and three-dimensional key points are introduced during the reconstruction of the three-dimensional human model, supervision of the key points is strengthened, thereby narrowing the semantic gap between the calculated model parameters and the image, enhancing the accuracy of the model key point prediction, and improving the continuity and authenticity of the model pose corresponding to each video frame image in the video to be detected.

[0084] The following is a further detailed description of the target object three-dimensional model reconstruction method, training method, device, electronic device and computer-readable storage medium described in this application in conjunction with specific embodiments and drawings.

[0085] Figure 1 This is an application system diagram of a method for reconstructing a three-dimensional model of a target object provided by an embodiment of the present application. Figure 1As shown, the application system includes: a first terminal 101 and a second terminal 102. The first terminal 101 and the second terminal 102 are communicatively connected via a network. The first terminal 101 can be various forms of video acquisition devices, such as a camera, a still camera, etc. The second terminal 102 can be a computer terminal, such as a laptop computer, a desktop computer, etc.; it can also be a touch terminal, such as a smart phone, a tablet computer, a personal digital assistant (PDA), etc. Of course, the first terminal 101 can also be a part of the second terminal 102. The first terminal 101 collects the video to be detected and sends it to the second terminal 102 via the network. The second terminal 102 reconstructs the three-dimensional model of the target object in the received video to be detected.

[0086] Figure 2 This is another application system diagram of a method for reconstructing a three-dimensional model of a target object provided by an embodiment of the present application. Figure 2 As shown, the application system includes: a terminal 201 and a server 202. The terminal 201 and the server 202 are communicatively connected via a network. The terminal 201 can be a video acquisition device in various forms, such as a camera, a still camera, etc., and can be one or more. The server 202 can be an independent server, deploying the target object three-dimensional model reconstruction method provided in this application, or it can be a server group composed of multiple servers, each of which deploys a module of the target object three-dimensional model reconstruction method provided in this application, such as a feature extraction server, a key point conversion server, a timing coding server, etc. Of course, the server 202 can also be a cloud server, and the target object three-dimensional model reconstruction method provided in this application is deployed on a cloud server. The terminal 201 collects the video to be detected and uploads it to the server 202 via the network. The server 202 reconstructs the target object three-dimensional model for the video to be detected.

[0087] The application system of the target object three-dimensional model reconstruction method can be applied to various fields such as virtual reality (VR), augmented reality (AR), mixed reality (MR), human-computer interaction, motion analysis, medical care, etc.

[0088] Figure 3 This is a flow chart of the method for reconstructing a three-dimensional model of a target object provided in this embodiment. Figure 4 This is a flow chart of a method for reconstructing a three-dimensional model of a target object provided by this embodiment, using the human body as the target object. Figure 3 and Figure 4The following description of the embodiments is used to explain the technical solution of this application and is not intended to limit actual use.

[0089] like Figure 3 The method for reconstructing a three-dimensional model of a target object provided in this embodiment includes the following steps:

[0090] Step S301 : obtaining image feature information 402 - 1 corresponding to each video frame image in the to-be-detected video 401 and a target object two-dimensional key point sequence 402 - 2 corresponding to the target object in each video frame image in the to-be-detected video.

[0091] A video can be divided into multiple frames, each corresponding to an image. Therefore, a video can be divided into multiple frames. Reconstructing a 3D model of a target object from a video actually involves reconstructing the 3D model of the target object one by one for each frame image in the video. The 3D models of the target object corresponding to each frame image in the video are then stitched together to form a 3D reconstructed video of the target object.

[0092] The image feature information acquisition method provided in this embodiment includes: extracting image feature information corresponding to each video frame image in the video to be detected through a pre-trained convolutional neural network.

[0093] Neural networks (NNs) are composed of a number of neurons and their parameters. They are systems that perform tasks by "learning" from a large number of examples and are typically not programmed with task-specific rules. For example, in image recognition, a neural network can learn the characteristics of a cat by analyzing example images labeled "cat" or "not cat," and use the learning results to identify whether other images contain cats. During neural network learning, cat characteristics are not directly input into the network. Instead, example images labeled as cats are input. Through iterative learning, the neural network automatically generates characteristic information representing a cat based on the example images.

[0094] Convolutional Neural Networks (CNNs) are a type of neural network that organizes several neurons into a convolutional layer. Data, starting from the input, propagates sequentially through several convolutional layers through the connections between neurons until it reaches the final output. Convolutional neural networks can also calculate errors based on a specified optimization objective and iteratively update the neural network parameters using backpropagation and gradient descent to optimize the network.

[0095] The convolutional neural network in this step can adopt the ResNet-50 network structure and use pre-trained parameters to extract corresponding image feature information for each video frame image in the video to be detected. The extracted image feature information can be represented as a multi-dimensional vector (for example: a vector of dimension C×H×W, where C represents the number of channels, H represents the height of the image, and W represents the width of the image).

[0096] The method for obtaining a two-dimensional key point sequence provided in this embodiment includes: using OpenPose to extract a two-dimensional key point sequence corresponding to each video frame image in a video to be detected.

[0097] OpenPose is a human posture recognition project developed based on convolutional neural networks and supervised learning and using Caffe as the framework. It can estimate the postures of one or more people in an image, including human movements, facial expressions, and finger movements, while performing real-time detection of multiple key points such as the human body, hands, and face.

[0098] This step uses OpenPose to extract a two-dimensional key point sequence corresponding to each video frame image in the video to be detected. The extracted two-dimensional key point sequence can be a two-dimensional vector including multiple key points (for example: a two-dimensional vector of J key points, with a dimension of J×2).

[0099] Step S302 : According to the target object two-dimensional key point sequence 402 - 2 , a target object three-dimensional key point sequence 402 - 3 corresponding to the target object two-dimensional key point sequence 402 - 2 is estimated and obtained.

[0100] An optional implementation method for estimating and obtaining a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object based on the two-dimensional key point sequence of the target object is: converting the two-dimensional key point sequence 402-2 of the target object into the three-dimensional key point sequence 402-3 of the target object corresponding to the two-dimensional key point sequence 402-2 of the target object through the posture prior network 402.

[0101] The posture prior network provided in this embodiment includes a first fully connected layer, a residual block, and a second fully connected layer.

[0102] The fully connected layer is a network structure in which each node is connected to all nodes in the previous layer, and is used to integrate the extracted features.

[0103] The residual block is a network structure consisting of two sequentially connected fully connected layers.

[0104] Figure 5 This is a flowchart of converting a two-dimensional key point sequence into a three-dimensional key point sequence provided by this embodiment. Figure 6This is a flow chart of converting a two-dimensional key point sequence into a three-dimensional key point sequence provided by this embodiment. Figure 5 and Figure 6 The method provided in this embodiment for converting the target object's two-dimensional key point sequence into the target object's three-dimensional key point sequence corresponding to the target object's two-dimensional key point sequence through a posture prior network is described in detail.

[0105] like Figure 5 As shown, the method provided in this embodiment converts the target object's two-dimensional key point sequence into the target object's three-dimensional key point sequence corresponding to the target object's two-dimensional key point sequence through a posture prior network, including the following steps:

[0106] Step S302 - 1 : Convert the target object two-dimensional key point sequence into a target object two-dimensional key point feature vector based on the first fully connected layer.

[0107] like Figure 6 As shown, the target object two-dimensional key point sequence corresponding to the target object in each video frame image in the video to be detected is taken as input, converted in the first fully connected layer 601, and the target object two-dimensional key point feature vector is output.

[0108] Step S302-2: Convert the two-dimensional key point feature vector of the target object into a three-dimensional key point feature vector of the target object based on the residual block.

[0109] like Figure 6 As shown, the target object's two-dimensional key point feature vector output in step S302-1 is input to residual block 602. Residual block 602 performs dimensionality processing on the input target object's two-dimensional key point feature vector, converting the two-dimensional key point feature vector into a three-dimensional key point feature vector. The residual block in this step can be one or more, and the specific number is not limited here.

[0110] Step S302-3: Convert the target object's three-dimensional key point feature vector into the target object's three-dimensional key point sequence based on the second fully connected layer.

[0111] like Figure 6 As shown, the 3D key point feature vectors corresponding to the target objects in each video frame image of the video to be detected, output in step S302-2, are input to the second fully connected layer 603. The second fully connected layer 603 transforms the input 3D key point feature vectors and outputs a 3D key point sequence corresponding to the target objects in each video frame image of the video to be detected. The 3D key point sequence can be a 3D vector including multiple key points (for example, a 3D vector of J key points with a dimension of J×3).

[0112] The conversion from a two-dimensional key point sequence to a three-dimensional key point sequence is achieved through the above three steps. Of course, the conversion method may also include other steps or contents, which will not be described here.

[0113] In addition, converting the target object's two-dimensional key point sequence into the target object's three-dimensional key point sequence corresponding to the target object's two-dimensional key point sequence through a posture prior network may also include: using the target object's two-dimensional key point sequence and the target object's three-dimensional key point sequence together as the output of the posture prior network.

[0114] In other words, the keypoint sequence output by the pose prior network includes both a 2D keypoint sequence and a 3D keypoint sequence. One implementation is to output the 2D keypoint sequence and the 3D keypoint sequence as separate sequences. Another implementation is to perform dimension-wise summation on the 2D keypoint sequence and the 3D keypoint sequence to form a dimension-merged keypoint sequence for output (for example, J×2+J×3, resulting in a dimension of J×5).

[0115] Through the above steps, the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object and the three-dimensional key point sequence of the target object are obtained, which are used as input elements for reconstructing the three-dimensional model of the target object.

[0116] Step S303, the image feature information 402-1 corresponding to each video frame image in the video to be detected, the target object two-dimensional key point sequence 402-2 corresponding to the target object in each video frame image in the video to be detected, and the target object three-dimensional key point sequence 402-3 corresponding to the target object two-dimensional key point sequence are connected to obtain the feature sequence 403 corresponding to the target object in each video frame image in the video to be detected.

[0117] Concatenation involves connecting multiple features along the channel dimension to form a complete feature sequence. This concatenation combines all features corresponding to each frame in the video being tested, preventing confusion with features corresponding to other frames. It also preserves all feature information corresponding to each frame in the video being tested, providing a complete feature sequence for subsequent calculations and learning.

[0118] An optional implementation method provided by this embodiment is: dimensional summing up the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object, and combining the summation results of the same dimension into a feature sequence corresponding to the target object in each video frame image in the video to be detected.

[0119] For example: the dimension of the feature information corresponding to each video frame image in the video to be detected is C×H×W, the dimension of the two-dimensional key point sequence is J×2, and the dimension of the three-dimensional key point sequence is J×3. Then, the dimension of the feature information, the two-dimensional key point sequence, and the three-dimensional key point sequence are added together, and the dimension of the feature sequence corresponding to each video frame image in the video to be detected is: C×H×W+J×2+J×3, where C represents the number of channels, H represents the height of the image, W represents the width of the image, and J represents the number of key points.

[0120] Since the video to be detected is composed of multiple video frame images, it is necessary to determine the relative position and absolute position of the feature sequence corresponding to each video frame image in the video to be detected in all feature sequences. An optional implementation method is to add the position vector 404 corresponding to each video frame image in the video to be detected to the feature sequence corresponding to the target object in each video frame image in the video to be detected. In this way, the feature sequences corresponding to each video frame image in the video to be detected can be arranged according to the position of each video frame image in the video to be detected, and connected into an overall feature sequence corresponding to a video. The dimension of this overall feature sequence is: (C×H×W+J×2+J×3)×T, where C represents the number of channels, H represents the height of the image, W represents the width of the image, J represents the number of key points, and T represents the number of video frames in the video to be detected.

[0121] Step S304 : obtaining a target object three-dimensional model 407 corresponding to the target object in each video frame image in the video to be detected according to the feature sequence 403 corresponding to the target object in each video frame image in the video to be detected.

[0122] Figure 7 This is a flowchart of obtaining a three-dimensional model of a target object based on a feature sequence, using a human body as the target object.

[0123] like Figure 7 As shown, the present embodiment provides a method for obtaining a human body three-dimensional model 407 corresponding to each video frame image in the video to be detected according to a feature sequence 403 corresponding to each video frame image in the video to be detected, including the following steps:

[0124] Step S304 - 1 , performing temporal coding processing on the feature sequence 403 corresponding to the target object in each video frame image in the video to be detected based on the encoder 405 , and obtaining the encoded feature sequence 405 - 1 corresponding to the target object in each video frame image in the video to be detected.

[0125] The encoder is a specific form of neural network, with the Transformer encoder being a widely used encoder. It's typically composed of multiple stacked encoder layers, each of which has two sublayer connection structures. The first sublayer connection structure includes a multi-head self-attention sublayer, a normalization layer, and a residual connection, while the second sublayer connection structure includes a feedforward fully connected sublayer, a normalization layer, and a residual connection. The multi-head self-attention sublayer in the Transformer encoder is actually a self-attention mechanism that calculates the correlation between each element in the input sequence and other elements. Therefore, the Transformer encoder has strong sequential modeling capabilities.

[0126] This embodiment uses the feature sequence corresponding to each video frame image in the video to be detected as the input sequence of the Transformer encoder. It uses the strong sequence modeling capability of the Transformer encoder to encode the feature sequence corresponding to each video frame image in the video to be detected, and ultimately outputs the encoded feature sequence corresponding to each video frame image in the video to be detected. The dimensions of the encoded feature sequence corresponding to each video frame image in the video to be detected are: C×H×W×T, where C represents the number of channels, H represents the height of the image, W represents the width of the image, and T represents the number of video frames in the video to be detected.

[0127] In step S304-2, a feature sequence 405-1 corresponding to the target object in each video frame image in the encoded video to be detected is decoded based on a parameter regression network 406 to obtain target object posture parameters 406-1 corresponding to the target object in each video frame image in the video to be detected, target object shape parameters 406-2 corresponding to the target object in each video frame image in the video to be detected, and target object camera parameters 406-3 corresponding to the target object in each video frame image in the video to be detected.

[0128] The parameter regression network is a neural network composed of two shared fully connected layers and three independent fully connected layers. It is used to decode the feature sequence corresponding to each video frame image in the encoded video to be detected and obtain the parameters corresponding to each video frame image in the video to be detected.

[0129] The posture parameters refer to parameters that characterize the current state of the human body, such as whether the person is standing or sitting, whether the mouth is open, smiling or laughing, and the degree of bending of the forearm relative to the upper arm (i.e., the rotation amount of the joint point relative to its parent node).

[0130] The shape parameters refer to parameters that characterize the inherent shape of the human body, such as height, shortness, fatness, and thinness, as well as facial contours, shapes and sizes of facial features, etc.

[0131] The camera parameters refer to the parameters for establishing the geometric model of camera imaging, which generally include external parameters and internal parameters. The external parameters may include the rotation matrix corresponding to the orientation of the human body, the translation matrix mapping the human body to the two-dimensional image coordinates, etc., and the internal parameters may include the mapping scale coefficient, etc.

[0132] In this embodiment, the feature sequence corresponding to each video frame image in the video to be detected, encoded and output by the Transformer encoder, is used as the input of the parameter regression network. After decoding by the parameter regression network, the posture parameters, shape parameters, and camera parameters corresponding to each video frame image in the video to be detected are output.

[0133] Pose parameters, shape parameters, and camera parameters can all be represented as multi-dimensional vectors. For example, pose parameters are 72-dimensional vectors, shape parameters are 10-dimensional vectors, and camera parameters are 3-dimensional vectors.

[0134] Step S304 - 3 , obtaining a target object three-dimensional model 407 corresponding to the target object in each video frame image in the video to be detected according to the target object posture parameter 406 - 1 , the target object shape parameter 406 - 2 , and the target object camera parameter 406 - 3 .

[0135] The reconstructed 3D human body model is a parameterized model. The posture parameters, shape parameters, and camera parameters corresponding to each video frame image in the video to be detected output by the parameter regression network are used as model parameters to reconstruct the 3D human body model corresponding to each video frame image in the video to be detected, such as the SMPL model.

[0136] The above embodiment uses the human body as the target object and provides an optional implementation method for reconstructing a three-dimensional model of the target object. The specific steps are as follows:

[0137] First, obtain the video to be detected.

[0138] Second, obtain the two-dimensional key point sequence corresponding to each video frame image in the video to be detected.

[0139] Third, the feature information corresponding to each video frame image in the video to be detected is extracted through the pre-trained convolutional neural network.

[0140] Fourth, the two-dimensional key point sequence corresponding to each video frame image in the video to be detected is converted into a three-dimensional key point sequence through a posture prior network.

[0141] Fifth, the two-dimensional key point sequence corresponding to each video frame image in the video to be detected obtained through the second step, the feature information corresponding to each video frame image in the video to be detected obtained through the third step, and the three-dimensional key point sequence corresponding to each video frame image in the video to be detected obtained through the fourth step are connected to obtain the feature sequence corresponding to each video frame image in the video to be detected.

[0142] Sixth, the position vector corresponding to each video frame image in the video to be detected is added to the feature sequence corresponding to each video frame image in the video to be detected obtained in the fifth step, and the feature sequence corresponding to each video frame image in the video to be detected is position-encoded.

[0143] Seventh, based on the Transformer encoder, a temporal encoding process is performed on the feature sequence corresponding to each video frame image in the video to be detected after the position encoding obtained in the sixth step.

[0144] Eighth, based on the parameter regression network, the feature sequence corresponding to each video frame image in the encoded video to be detected obtained in the seventh step is decoded to obtain the posture parameters, shape parameters, and camera parameters corresponding to each video frame image in the video to be detected.

[0145] Ninth, according to the posture parameters, shape parameters, and camera parameters corresponding to each video frame image in the to-be-detected video obtained in the eighth step, a three-dimensional human body model corresponding to each video frame image in the to-be-detected video is obtained.

[0146] Figure 8 This is an example diagram of the result of reconstructing a three-dimensional model of a human body using the target object three-dimensional model reconstruction method provided in this embodiment.

[0147] like Figure 8 As shown, line 801 shows the video frames in the video to be detected, line 802 shows the human body 3D models corresponding to the video frames in the video to be detected obtained by the existing human body 3D model reconstruction technology, and line 803 shows the human body 3D models corresponding to the video frames in the video to be detected obtained by the human body 3D model reconstruction method provided in the first embodiment of the present application. Figure 8 It can be seen that the human body three-dimensional model reconstruction method provided in the first embodiment of the present application is more accurate in predicting the key points of the human body, the continuity of the model posture corresponding to each video frame image in the video to be detected is better, and the movement is more realistic.

[0148] The target object three-dimensional model reconstruction method provided in the above embodiment can also be applied to the three-dimensional model reconstruction of objects such as animals, plants, and machines in videos or photos, and will not be discussed in detail here.

[0149] Another embodiment of the present application provides a training method for a target object three-dimensional model reconstruction model. Applied to the target object three-dimensional model reconstruction model, the target object three-dimensional model reconstruction model uses a video obtained from a data set as original training data, and reconstructs the target object three-dimensional model of the video obtained from the data set using the target object three-dimensional model reconstruction method provided in the above embodiment of the present application, thereby obtaining target object posture parameters corresponding to the target object in each video frame image in the video, target object shape parameters corresponding to the target object in each video frame image in the video, and target object camera parameters corresponding to the target object in each video frame image in the video.

[0150] Figure 9 The flowchart of the training method for the target object three-dimensional model reconstruction model provided in this embodiment. Figure 9 The training method for reconstructing a three-dimensional model of a target object provided in this embodiment is described in detail.

[0151] like Figure 9 As shown, the training method for reconstructing a three-dimensional model of a target object provided in this embodiment includes the following steps:

[0152] Step S901: Obtain model parameter loss according to the target object posture parameter and the target object shape parameter.

[0153] The model parameter loss refers to the difference between the model parameters (including posture parameters and shape parameters) obtained by the target object three-dimensional model reconstruction method provided by the first embodiment of the present application and the true value.

[0154] This embodiment provides an optional method for calculating model parameter loss, and the steps are as follows:

[0155] First, the model parameters corresponding to the target object in each video frame image in the video are obtained from the data set.

[0156] The dataset is a publicly available video collection that includes the recorded videos and the corresponding feature information, two-dimensional keypoint sequences, pose parameters, shape parameters, etc. for each video frame. InstaVariety, PoseTrack, and PennAction are all preferred datasets.

[0157] Therefore, the model parameters corresponding to each video frame image in the video, including posture parameters and shape parameters, can be directly obtained from the data set.

[0158] Second, the model parameter loss is calculated by taking the model parameters corresponding to the target object in each video frame image in the video obtained from the data set as the true value.

[0159] Taking the model parameters (including posture parameters and shape parameters) corresponding to the target object in each video frame image in the video directly obtained from the data set as the true value, the difference between the posture parameters and shape parameters corresponding to each video frame image in the video obtained by the target object three-dimensional model reconstruction method provided by the first embodiment of the present application and the true value is calculated.

[0160] The calculation formula is:

[0161] Where β represents the shape parameter, θ represents the posture parameter, μ i is the weight parameter.

[0162] Step S902 : Acquire a two-dimensional key point loss and a three-dimensional key point loss according to the target object posture parameter, the target object shape parameter, and the target object camera parameter.

[0163] The two-dimensional key point loss refers to the posture parameters, shape parameters and camera parameters corresponding to each video frame image in the video obtained according to the three-dimensional model reconstruction method of the target object provided in the first embodiment of the present application, obtaining the two-dimensional key point sequence of the model corresponding to each video frame image in the video, and calculating the difference between the two-dimensional key point sequence and the true value.

[0164] The three-dimensional key point loss refers to the calculation of the three-dimensional key point sequence of the model corresponding to each video frame image in the video according to the posture parameters, shape parameters and camera parameters corresponding to each video frame image in the video obtained by the three-dimensional model reconstruction method of the target object provided in the first embodiment of the present application; using the calculated three-dimensional key point sequence of the model corresponding to each video frame image in the video as the true value, calculating the difference between the three-dimensional key point sequence obtained by the posture prior network and the true value.

[0165] This embodiment provides an optional method for calculating two-dimensional key point loss, and the steps are as follows:

[0166] First, a two-dimensional key point sequence corresponding to the target object in each video frame image in the video is obtained from the data set.

[0167] The two-dimensional key point sequence corresponding to the target object in each video frame image in the video can be directly obtained from the data set.

[0168] Second, the two-dimensional key point loss is calculated using the two-dimensional key point sequence corresponding to the target object in each video frame image in the video obtained from the data set as the true value.

[0169] Taking the two-dimensional key point sequence corresponding to the target object in each video frame image in the video obtained directly from the data set as the true value, the difference between the two-dimensional key point sequence of the model corresponding to each video frame image in the video obtained by the target object three-dimensional model reconstruction method provided by the first embodiment of the present application and the true value is calculated.

[0170] The calculation formula is:

[0171] This embodiment provides an optional method for calculating 3D key point loss, and the steps are as follows:

[0172] First, a three-dimensional key point sequence corresponding to the target object in each video frame image in the video is calculated based on the target object's posture parameters, shape parameters, and camera parameters.

[0173] Second, the three-dimensional key point loss is calculated using the calculated three-dimensional key point sequence corresponding to the target object in each video frame image in the video as the true value.

[0174] Taking the calculated three-dimensional key point sequence corresponding to each video frame image in the video as the true value, the difference between the three-dimensional key point sequence obtained by the posture prior network provided by the first embodiment of the present application and the true value is calculated.

[0175] The calculation formula is:

[0176] Step S903: Obtain generative adversarial loss based on the target object posture parameters.

[0177] The generated adversarial loss is used to determine whether the acquired posture parameters are real.

[0178] The target object three-dimensional model reconstruction method provided in the first embodiment of the present application can be used as a generator, and the temporal discriminant network can be used as a discriminator. The loss calculation of the posture parameters obtained by the generator is performed based on the temporal discriminant network.

[0179] The generative adversarial loss of the generator can be expressed as: The adversarial loss of the discriminator can be expressed as in represents the expected value, p G represents the data distribution of the generator, p R represents the data distribution of real data (derived from the dataset), D M Represents a temporal discriminant network. For the generator, The calculation is the difference between the output and 1 when the data generated by the generator is input into the discriminator. The smaller it is, the closer the discriminator output is to 1, and the generated data is closer to the real data distribution; for the discriminator, The calculation is the difference between the discriminator's output of the real data and 1, and the difference between the output of the generated data and 0. The smaller it is, the stronger the discriminator's ability to distinguish between generated data and real data is.

[0180] Step S904: Obtain a total loss of the target object's three-dimensional model based on the model parameter loss, the two-dimensional key point loss, the three-dimensional key point loss, and the generative adversarial loss.

[0181] The total loss of the target object 3D model is the weighted sum of model parameter loss, 2D key point loss, 3D key point loss, and generative adversarial loss, and is the objective function in the entire training process of the target object 3D model reconstruction method.

[0182] The total loss of target object 3D model reconstruction can be expressed as: where λ i is the weight parameter, is the model parameter loss, is the two-dimensional key point loss, is the 3D key point loss, To generate adversarial damage, the larger the weight parameter, the greater the gradient of the corresponding loss-related parameters during the training process of the 3D model reconstruction method.

[0183] Step S905 : Iteratively training the target object 3D model reconstruction model according to the total loss of the target object 3D model.

[0184] The training method provided in this embodiment uses the chain rule to calculate the gradient of the total loss of the acquired 3D model of the target object, thereby optimizing the parameters to reduce the loss. During training, the PyTorch deep learning framework and stochastic gradient descent are used to iteratively train the 3D model reconstruction method, resulting in a high-performance neural network for 3D model reconstruction.

[0185] An embodiment of the present application provides a device for reconstructing a three-dimensional model of a target object. Figure 10 3D model reconstruction device for a target object provided in this embodiment.

[0186] like Figure 10 As shown, the target object three-dimensional model reconstruction device provided by this embodiment includes: an acquisition unit 1001, a posture prior unit 1002, a connection unit 1003, and a reconstruction unit 1004.

[0187] The acquisition unit 1001 is used to acquire image feature information corresponding to each video frame image in the video to be detected and a two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected.

[0188] The posture prior unit 1002 is used to estimate and obtain a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object based on the two-dimensional key point sequence of the target object.

[0189] Optionally, estimating and obtaining a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object based on the two-dimensional key point sequence of the target object includes: converting the two-dimensional key point sequence of the target object into the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object through a posture prior network.

[0190] Optionally, the pose prior network includes a first fully connected layer, a residual block, and a second fully connected layer, and converting the target object two-dimensional key point sequence into the target object three-dimensional key point sequence corresponding to the target object two-dimensional key point sequence through the pose prior network includes:

[0191] Converting the target object's two-dimensional key point sequence into a target object's two-dimensional key point feature vector based on a first fully connected layer;

[0192] Converting the two-dimensional key point feature vector of the target object into a three-dimensional key point feature vector of the target object based on the residual block;

[0193] The three-dimensional key point feature vector of the target object is converted into a three-dimensional key point sequence of the target object based on the second fully connected layer.

[0194] Optionally, the converting of the target object's two-dimensional key point sequence into the target object's three-dimensional key point sequence corresponding to the target object's two-dimensional key point sequence through the posture prior network also includes: using the target object's two-dimensional key point sequence and the target object's three-dimensional key point sequence together as the output of the posture prior network.

[0195] The connection unit 1003 is used to connect the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object, so as to obtain the feature sequence corresponding to the target object in each video frame image in the video to be detected.

[0196] Optionally, the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object are connected to obtain the feature sequence corresponding to the target object in each video frame image in the video to be detected, including: dimensional summing up the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object, and combining the summation results of the same dimension into a feature sequence corresponding to the target object in each video frame image in the video to be detected.

[0197] Optionally, the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object are connected to obtain the feature sequence corresponding to the target object in each video frame image in the video to be detected, and also includes: adding the position vector corresponding to each video frame image in the video to be detected to the feature sequence corresponding to the target object in each video frame image in the video to be detected.

[0198] The reconstruction unit 1004 is configured to obtain a three-dimensional model of the target object corresponding to each of the video frame images in the video to be detected based on a feature sequence corresponding to the target object in each of the video frame images in the video to be detected.

[0199] Optionally, obtaining a target object three-dimensional model corresponding to the target object in each video frame image in the video to be detected according to a feature sequence corresponding to the target object in each video frame image in the video to be detected includes:

[0200] Performing temporal coding processing on a feature sequence corresponding to a target object in each video frame image in the video to be detected based on an encoder, and obtaining a feature sequence corresponding to a target object in each video frame image in the video to be detected after coding;

[0201] Decoding the feature sequence corresponding to the target object in each video frame image in the encoded video to be detected based on a parameter regression network to obtain target object posture parameters corresponding to the target object in each video frame image in the video to be detected, target object shape parameters corresponding to the target object in each video frame image in the video to be detected, and target object camera parameters corresponding to the target object in each video frame image in the video to be detected;

[0202] A target object three-dimensional model corresponding to the target object in each video frame image in the video to be detected is obtained according to the target object posture parameters, the target object shape parameters, and the target object camera parameters.

[0203] Another embodiment of the present application provides a training device for reconstructing a three-dimensional model of a target object. Figure 11 3D model reconstruction training device for a target object provided in this embodiment.

[0204] like Figure 11 As shown, the training device for reconstructing a three-dimensional model of a target object provided in this embodiment includes: a reconstruction unit 1101, a model parameter loss acquisition unit 1102, a key point loss acquisition unit 1103, a generative adversarial loss acquisition unit 1104, a total loss acquisition unit 1105, and a training unit 1106.

[0205] The reconstruction unit 1101 is used to reconstruct a three-dimensional model of a target object for at least one video obtained in a data set, and obtain target object posture parameters corresponding to the target object in each video frame image in the video, target object shape parameters corresponding to the target object in each video frame image in the video, and target object camera parameters corresponding to the target object in each video frame image in the video.

[0206] The model parameter loss acquisition unit 1102 is configured to acquire the model parameter loss according to the target object posture parameters and the target object shape parameters.

[0207] Optionally, obtaining a model parameter loss according to the target object posture parameter and the target object shape parameter includes:

[0208] Obtaining model parameters corresponding to the target object in each video frame image in the video from the data set;

[0209] The model parameter loss is calculated by taking the model parameters corresponding to the target object in each video frame image in the video obtained from the data set as the true value.

[0210] The key point loss acquisition unit 1103 is used to acquire a two-dimensional key point loss and a three-dimensional key point loss according to the target object posture parameters, the target object shape parameters and the target object camera parameters.

[0211] Optionally, obtaining a two-dimensional key point loss and a three-dimensional key point loss according to the target object posture parameter, the target object shape parameter, and the target object camera parameter includes:

[0212] Obtaining a two-dimensional key point sequence corresponding to a target object in each video frame image in the video from the data set;

[0213] The two-dimensional key point loss is calculated by taking a two-dimensional key point sequence corresponding to a target object in each video frame image in the video obtained from the data set as a true value.

[0214] Optionally, the obtaining of two-dimensional key point loss and three-dimensional key point loss according to the target object posture parameter, the target object shape parameter, and the target object camera parameter further includes:

[0215] Calculating a three-dimensional key point sequence corresponding to the target object in each video frame image in the video using the target object posture parameters, the target object shape parameters, and the target object camera parameters;

[0216] The three-dimensional key point loss is calculated by taking the calculated three-dimensional key point sequence corresponding to the target object in each video frame image in the video as the true value.

[0217] The generative adversarial loss acquisition unit 1104 is configured to acquire the generative adversarial loss according to the target object posture parameters.

[0218] Optionally, obtaining the generative adversarial loss based on the target object posture parameters includes: performing loss calculation on the target object posture parameters based on a temporal discriminant network.

[0219] The total loss acquisition unit 1105 is used to obtain the total loss of the target object three-dimensional model based on the model parameter loss, the two-dimensional key point loss, the three-dimensional key point loss, and the generative adversarial loss.

[0220] The training unit 1106 is configured to iteratively train the target object 3D model reconstruction model according to the total loss of the target object 3D model.

[0221] An embodiment of the present application provides an electronic device. Figure 12 This is a schematic diagram of the structure of the electronic device provided in this embodiment.

[0222] like Figure 12 As shown, the electronic device provided by this embodiment includes: a collector 1201, a memory 1202 and a processor 1203.

[0223] The collector 1201 is used to collect the video to be detected.

[0224] The memory 1202 is used to store computer instructions for executing the method for reconstructing a three-dimensional model of a target object.

[0225] The processor 1203 is configured to execute computer instructions stored in the memory 1202 to perform the target object three-dimensional model reconstruction method provided in this application.

[0226] Another embodiment of the present application provides an electronic device. Figure 13 This is a schematic diagram of the structure of the electronic device provided in this embodiment.

[0227] like Figure 13 As shown, the electronic device provided by this embodiment includes: a collector 1301, a memory 1302 and a processor 1303.

[0228] The collector 1301 is configured to collect at least one video in a data set.

[0229] The memory 1302 is used to store computer instructions for executing a training method for reconstructing a three-dimensional model of a target object.

[0230] The processor 1303 is configured to execute computer instructions stored in the memory 1302 to implement the training method for reconstructing a three-dimensional model of a target object provided in this application.

[0231] Another embodiment of the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed by a processor, they are used to implement the technical solutions described in the target object three-dimensional model reconstruction method and the target object three-dimensional model reconstruction model training method provided in the present application.

[0232] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A method for reconstructing a three-dimensional model of a target object, characterized in that: include: Obtaining image feature information corresponding to each video frame image in the video to be detected and a two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected, wherein the image feature information is a multi-dimensional vector obtained by extracting the video frame image through a pre-trained convolutional neural network; Based on a pose prior network, estimating and obtaining a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object according to the two-dimensional key point sequence of the target object, the pose prior network including a first fully connected layer, at least one residual block, and a second fully connected layer; Connecting the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object to obtain a feature sequence corresponding to the target object in each video frame image in the video to be detected; According to the feature sequence corresponding to the target object in each video frame image in the video to be detected, a target object three-dimensional model corresponding to the target object in each video frame image in the video to be detected is obtained.

2. The method according to claim 1, characterized in that The method, based on the posture prior network, estimates and obtains the target object's three-dimensional key point sequence corresponding to the target object's two-dimensional key point sequence according to the target object's two-dimensional key point sequence, including: converting the target object's two-dimensional key point sequence into the target object's three-dimensional key point sequence corresponding to the target object's two-dimensional key point sequence through the posture prior network.

3. The method according to claim 2, characterized in that The converting the target object two-dimensional key point sequence into the target object three-dimensional key point sequence corresponding to the target object two-dimensional key point sequence through the posture prior network includes: Converting the target object's two-dimensional key point sequence into a target object's two-dimensional key point feature vector based on a first fully connected layer; Converting the two-dimensional key point feature vector of the target object into a three-dimensional key point feature vector of the target object based on the residual block; The three-dimensional key point feature vector of the target object is converted into a three-dimensional key point sequence of the target object based on the second fully connected layer.

4. The method according to claim 2, characterized in that The method of converting the target object's two-dimensional key point sequence into the target object's three-dimensional key point sequence corresponding to the target object's two-dimensional key point sequence through a posture prior network also includes: using the target object's two-dimensional key point sequence and the target object's three-dimensional key point sequence together as the output of the posture prior network.

5. The method according to claim 1, wherein The method comprises: performing dimensional summation on the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object, and combining the summation results of the same dimension into the feature sequence corresponding to the target object in each video frame image in the video to be detected.

6. The method according to claim 1, characterized in that The method further includes connecting the image feature information corresponding to each video frame image in the video to be detected, the two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected, and the three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object to obtain the feature sequence corresponding to the target object in each video frame image in the video to be detected, and adding the position vector corresponding to each video frame image in the video to be detected to the feature sequence corresponding to the target object in each video frame image in the video to be detected.

7. The method according to claim 1, characterized in that The step of obtaining a target object three-dimensional model corresponding to the target object in each video frame image in the video to be detected according to a feature sequence corresponding to the target object in each video frame image in the video to be detected includes: Performing temporal coding processing on a feature sequence corresponding to a target object in each video frame image in the video to be detected based on an encoder, and obtaining a feature sequence corresponding to a target object in each video frame image in the video to be detected after coding; Decoding the feature sequence corresponding to the target object in each video frame image in the encoded video to be detected based on a parameter regression network to obtain target object posture parameters corresponding to the target object in each video frame image in the video to be detected, target object shape parameters corresponding to the target object in each video frame image in the video to be detected, and target object camera parameters corresponding to the target object in each video frame image in the video to be detected; A target object three-dimensional model corresponding to the target object in each video frame image in the video to be detected is obtained according to the target object posture parameters, the target object shape parameters, and the target object camera parameters.

8. A method for training a target object three-dimensional model reconstruction model, characterized in that: The invention is applied to a target object three-dimensional model reconstruction model, wherein the target object three-dimensional model reconstruction model reconstructs the target object three-dimensional model of at least one video obtained in a data set through the target object three-dimensional model reconstruction method according to claim 1, and obtains the target object posture parameters corresponding to the target object in each video frame image in the video, the target object shape parameters corresponding to the target object in each video frame image in the video, and the target object camera parameters corresponding to the target object in each video frame image in the video. The training method includes: Obtaining model parameter loss according to the target object posture parameter and the target object shape parameter; Obtaining a two-dimensional key point loss and a three-dimensional key point loss according to the target object posture parameter, the target object shape parameter, and the target object camera parameter; Obtaining a generative adversarial loss based on the target object posture parameters; Obtaining a total loss of a three-dimensional model of a target object according to the model parameter loss, the two-dimensional key point loss, the three-dimensional key point loss, and the generative adversarial loss; The target object three-dimensional model reconstruction model is iteratively trained according to the total loss of the target object three-dimensional model.

9. The training method according to claim 8, characterized in that The obtaining of model parameter loss according to the target object posture parameter and the target object shape parameter includes: Obtaining model parameters corresponding to the target object in each video frame image in the video from the data set; The model parameter loss is calculated by taking the model parameters corresponding to the target object in each video frame image in the video obtained from the data set as the true value.

10. The training method according to claim 8, characterized in that: The obtaining of two-dimensional key point loss and three-dimensional key point loss according to the target object posture parameter, the target object shape parameter, and the target object camera parameter includes: Obtaining a two-dimensional key point sequence corresponding to a target object in each video frame image in the video from the data set; The two-dimensional key point loss is calculated by taking a two-dimensional key point sequence corresponding to a target object in each video frame image in the video obtained from the data set as a true value.

11. The training method according to claim 8, characterized in that: The acquiring of the two-dimensional key point loss and the three-dimensional key point loss according to the target object posture parameter, the target object shape parameter and the target object camera parameter further includes: Calculating a three-dimensional key point sequence corresponding to the target object in each video frame image in the video using the target object posture parameters, the target object shape parameters, and the target object camera parameters; The three-dimensional key point loss is calculated by taking the calculated three-dimensional key point sequence corresponding to the target object in each video frame image in the video as the true value.

12. The training method according to claim 8, characterized in that: The obtaining of generative adversarial loss according to the target object posture parameters includes: performing loss calculation on the target object posture parameters based on a temporal discriminant network.

13. A device for reconstructing a three-dimensional model of a target object, characterized in that: include: Acquisition unit, posture prior unit, connection unit, reconstruction unit; The acquisition unit is used to obtain image feature information corresponding to each video frame image in the video to be detected and a two-dimensional key point sequence of the target object corresponding to the target object in each video frame image in the video to be detected, wherein the image feature information is a multi-dimensional vector obtained by extracting the video frame image through a pre-trained convolutional neural network; The posture prior unit is configured to estimate and obtain a three-dimensional key point sequence of the target object corresponding to the two-dimensional key point sequence of the target object based on a posture prior network, wherein the posture prior network includes a first fully connected layer, at least one residual block, and a second fully connected layer; The connection unit is used to connect the image feature information corresponding to each video frame image in the video to be detected, the target object two-dimensional key point sequence corresponding to the target object in each video frame image in the video to be detected, and the target object three-dimensional key point sequence corresponding to the target object two-dimensional key point sequence, to obtain a feature sequence corresponding to the target object in each video frame image in the video to be detected; The reconstruction unit is used to obtain a three-dimensional model of the target object corresponding to the target object in each video frame image in the video to be detected based on a feature sequence corresponding to the target object in each video frame image in the video to be detected.

14. A training device for reconstructing a three-dimensional model of a target object, characterized in that: include: Reconstruction unit, model parameter loss acquisition unit, key point loss acquisition unit, generation adversarial loss acquisition unit, total loss acquisition unit, training unit; The reconstruction unit is configured to reconstruct a three-dimensional model of the target object on at least one video obtained from the data set by the target object three-dimensional model reconstruction method according to claim 1, and obtain target object posture parameters corresponding to the target object in each video frame image in the video, target object shape parameters corresponding to the target object in each video frame image in the video, and target object camera parameters corresponding to the target object in each video frame image in the video; The model parameter loss acquisition unit is used to acquire the model parameter loss according to the target object posture parameter and the target object shape parameter; The key point loss acquisition unit is used to acquire two-dimensional key point loss and three-dimensional key point loss according to the target object posture parameter, the target object shape parameter and the target object camera parameter; The generative adversarial loss acquisition unit is used to acquire the generative adversarial loss according to the target object posture parameter; The total loss acquisition unit is used to acquire the total loss of the target object three-dimensional model according to the model parameter loss, the two-dimensional key point loss, the three-dimensional key point loss, and the generative adversarial loss; The training unit is used to iteratively train the target object three-dimensional model reconstruction model according to the total loss of the target object three-dimensional model.

15. An electronic device, characterized in that: include: Collector, processor and memory; The collector is used to collect the video to be detected; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to implement the method according to any one of claims 1 to 7.

16. An electronic device, characterized in that: include: collectors, processors, and storage; The collector is configured to collect at least one video in the dataset; The memory is used to store one or more computer instructions; The processor is configured to execute the one or more computer instructions to implement the method according to any one of claims 8 to 12.

17. A computer-readable storage medium having one or more computer instructions stored thereon, characterized in that: The instructions are executed by a processor to implement the method according to any one of claims 1 to 12.

Images