Method, device, electronic device and storage medium for obtaining mesh topology structure

Abstract

The present application provides a method, device, electronic device, and storage medium for obtaining a mesh topology structure, which are used to improve the problem of low accuracy and real-time performance of the obtained mesh topology structure due to ambiguity in viewpoint occlusion. The method includes: obtaining multiple human body images, each of which is captured from different viewpoints of the human body region; performing key point detection and multi-view reconstruction on the multiple human body images to obtain reconstructed three-dimensional key point data; using a pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data; obtaining a fitted parameterized human body model, and inputting the three-dimensional rotation data into the parameterized human body model to obtain a mesh topology structure of the parameterized human body model.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and computer graphics, and in particular to a method, device, electronic device and storage medium for acquiring a mesh topology structure. Background Art

[0002] Motion capture, also known as dynamic capture, refers to the technology used to record and process the movements of people or other objects. In filmmaking and video game development, motion capture typically involves recording the movements of a human actor and translating them into the movements of a digital model, generating two-dimensional or three-dimensional computer animation.

[0003] Currently, motion capture techniques for capturing mesh topology typically rely on single-view color images. However, in practice, traditional fitting and iterative optimization methods can only capture the mesh topology corresponding to a parametric human model at a frame rate of 11 seconds per frame. When multiple actors are included in a single-view color image, their movements can occlude each other, and even individual actors' own movements can be blocked by the view. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for obtaining a mesh topology structure, which are used to improve the problem of low accuracy and real-time performance of the obtained mesh topology structure caused by ambiguity in perspective occlusion.

[0005] An embodiment of the present application provides a method for acquiring a mesh topology structure, including: acquiring multiple human body images, where the multiple human body images are photographed from different perspectives of the human body area; performing key point detection and multi-perspective reconstruction on the multiple human body images to obtain reconstructed three-dimensional key point data; using a pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data; acquiring a fitted parameterized human body model, and inputting the three-dimensional rotation data into the parameterized human body model to obtain a mesh topology structure of the parameterized human body model. In the above implementation process, by obtaining multiple human body images taken from different perspectives of the human body area, and performing key point detection and multi-perspective reconstruction on the multiple human body images, there is no need for human participation in labeling during the production of the parametric human body model. The automated production process improves the real-time performance of the mesh topology structure production, and effectively improves the ambiguity of the actor's movements during single-perspective photography. Furthermore, by predicting the three-dimensional rotation data corresponding to the reconstructed three-dimensional key point data, and using the fitted parametric human body model to load the three-dimensional rotation data, the mesh topology structure of the parametric human body model is obtained, thereby utilizing the multi-perspective fitting method to obtain a more accurate body posture and size of the parametric human body model, effectively improving the accuracy of the mesh topology structure of the parametric human body model.

[0006] Optionally, in an embodiment of the present application, key point detection and multi-perspective reconstruction are performed on multiple human images to obtain reconstructed three-dimensional key point data, including: performing key point detection on each human image in the multiple human images to obtain multiple groups of two-dimensional key points; and performing multi-perspective reconstruction on each group of two-dimensional key points in the multiple groups of two-dimensional key points according to parameters calibrated by the camera array to obtain reconstructed three-dimensional key point data. In the above implementation process, by performing key point detection on each human image in the multiple human images and performing multi-perspective reconstruction on each group of two-dimensional key points in the multiple groups of two-dimensional key points according to parameters calibrated by the camera array, a more accurate body posture and size of the parameterized human body model is obtained by multi-perspective fitting, effectively improving the accuracy of the mesh topology structure of the parameterized human body model.

[0007] Optionally, in an embodiment of the present application, key point detection is performed on each human body image in a plurality of human body images to obtain a plurality of groups of two-dimensional key points, including: for each human body image in the plurality of human body images, positioning target detection is performed on the human body region in the human body image to obtain a human body region image; and key point detection is performed on the human body region image to obtain a group of two-dimensional key points. In the above implementation process, by first performing positioning target detection on the human body region in the human body image and then performing key point detection on the human body region image, the problem of poor key point detection effect when the human body region in the human body image accounts for a small proportion is avoided. The human body region image in the human body image is extracted in advance and then key point detection is performed on the human body region image, thereby improving the key point detection effect of the human body image.

[0008] Optionally, in an embodiment of the present application, obtaining a fitted parameterized human body model includes: obtaining an unfitted human body model, and projecting multiple three-dimensional key points in the unfitted human body model onto a human body region image to obtain multiple two-dimensional key point data after projection; projecting the reconstructed three-dimensional key point data onto the human body region image to obtain multiple two-dimensional key point data after reconstruction, and correcting the projected multiple two-dimensional key point data based on the reconstructed multiple two-dimensional key point data to obtain multiple corrected two-dimensional key points; minimizing the error between the multiple two-dimensional key points in the unfitted human body model and the corrected multiple two-dimensional key points to obtain the parameterized human body model. In the above implementation process, by minimizing the error between the multiple two-dimensional key points in the unfitted human body model and the corrected multiple two-dimensional key points, the problem of using the unfitted human body model to process data is avoided, and a more accurate body posture and size of the parameterized human body model can be obtained, effectively improving the accuracy of the mesh topology structure of the parameterized human body model.

[0009] Optionally, in an embodiment of the present application, the deep learning network model includes: a feature extraction network and an iterative error feedback network; using a pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data, including: using the feature extraction network to perform feature extraction on the three-dimensional key point data to obtain posture features and scale features; using the iterative error feedback network to perform rotation prediction on the posture features and scale features to obtain three-dimensional rotation data.

[0010] Optionally, in an embodiment of the present application, before using a pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data, it also includes: obtaining multiple three-dimensional key point data, and obtaining three-dimensional rotation data corresponding to each three-dimensional key point data in the multiple three-dimensional key point data; using the three-dimensional key point data as training data and the three-dimensional rotation data as training labels, training the deep learning neural network to obtain a deep learning network model.

[0011] Optionally, in an embodiment of the present application, multiple three-dimensional key point data are obtained, and three-dimensional rotation data corresponding to each three-dimensional key point data in the multiple three-dimensional key point data are obtained, including: obtaining multiple human sample images, and using the key point detection algorithm of the first human body model to perform key point detection on each human body sample image in the multiple human sample images to obtain multiple groups of two-dimensional key points; performing multi-perspective reconstruction on each group of two-dimensional key points in the multiple groups of two-dimensional key points according to the parameters of the camera array calibration to obtain reconstructed three-dimensional key point data; using the second human body model to perform Rodrigues transform on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data corresponding to the three-dimensional key point data, and the first human body model and the second human body model are different.

[0012] In the implementation process of the above solution, three-dimensional key point data is generated by obtaining two-dimensional key points. The three-dimensional key point data is training data for training deep learning neural networks, and the three-dimensional rotation data obtained by performing Rodrigues transform on the three-dimensional key point data using a second human body model is training labels for training deep learning neural networks, thereby achieving the skeleton migration effect from the OpenPose model to the SMPL-X model.

[0013] Optionally, in an embodiment of the present application, a skinned multi-person linear second human body model is used to perform a Rodrigues transform on the reconstructed three-dimensional key point data, including: adding noise to one of the dimensions of the reconstructed three-dimensional key point data to obtain noisy three-dimensional key point data, or rotating the reconstructed three-dimensional key point data along one of the dimensions to obtain rotated three-dimensional key point data; and using a second human body model to perform a Rodrigues transform on the noisy three-dimensional key point data or the rotated three-dimensional key point data.

[0014] In the implementation process of the above scheme, by first obtaining the noisy or rotated three-dimensional key point data, and then using the second human body model to perform Rodrigues transform on the noisy or rotated three-dimensional key point data, more three-dimensional rotation data can be effectively added as training labels for the deep learning neural network, thereby reducing the shortage of training samples and training labels for the deep learning neural network, and improving the accuracy of the deep learning neural network in predicting three-dimensional key point data.

[0015] In the embodiment of the present application, the first human body model is an OpenPose model, and the second human body model is a skinned multi-person linear SMPL-X model.

[0016] In the implementation process of the above solution, three-dimensional key point data is generated by obtaining two-dimensional key points. The three-dimensional key point data is used as training data for training deep learning neural networks. The three-dimensional rotation data obtained by performing Rodrigues transform on the three-dimensional key point data using the SMPL-X model is used as training labels for training deep learning neural networks, thereby achieving the skeleton migration effect from the OpenPose model to the SMPL-X model.

[0017] Optionally, in an embodiment of the present application, a deep learning neural network is trained to obtain a deep learning network model, including: using the deep learning neural network to predict three-dimensional key point data to obtain predicted rotation data; respectively calculating the rotation error, key point position error, and scale factor error between the predicted rotation data and the three-dimensional rotation data; calculating the rotation error, key point position error, and scale factor error to obtain a total loss value; and training the deep learning neural network based on the total loss value to obtain a deep learning network model. In the above implementation process, the deep learning network model is trained based on the total loss value calculated based on the rotation error, key point position error, and scale factor error, thereby avoiding the problem of using only one of the errors to train the deep learning network model, and effectively improving the accuracy of the deep learning network model in predicting three-dimensional rotation data.

[0018] An embodiment of the present application also provides a mesh topology structure acquisition device, including: a human body image acquisition module, used to acquire multiple human body images, where the multiple human body images are photographed from different perspectives of the human body area; a reconstruction data acquisition module, used to perform key point detection and multi-perspective reconstruction on the multiple human body images to obtain reconstructed three-dimensional key point data; a rotation data acquisition module, used to use a pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data; a mesh topology acquisition module, used to acquire a fitted parameterized human body model, and input the three-dimensional rotation data into the parameterized human body model to obtain a mesh topology structure of the parameterized human body model.

[0019] Optionally, in an embodiment of the present application, the reconstruction data acquisition module includes: a two-dimensional key point detection module, which is used to perform key point detection on each human body image in multiple human body images to obtain multiple groups of two-dimensional key points; a two-dimensional multi-view reconstruction module, which is used to perform multi-view reconstruction on each group of two-dimensional key points in the multiple groups of two-dimensional key points according to the parameters calibrated by the camera array to obtain reconstructed three-dimensional key point data.

[0020] Optionally, in an embodiment of the present application, the two-dimensional key point detection module includes: a region image detection module, which is used to perform target positioning detection on the human body region in each human body image in multiple human body images to obtain a human body region image; and a two-dimensional key point acquisition module, which is used to perform key point detection on the human body region image to obtain a set of two-dimensional key points.

[0021] Optionally, in an embodiment of the present application, the mesh topology acquisition module includes: a three-dimensional key point projection module, used to obtain an unfitted human body model, and project multiple three-dimensional key points in the unfitted human body model to the human body area image to obtain multiple two-dimensional key point data after projection; a key point data correction module, used to project the reconstructed three-dimensional key point data to the human body area image to obtain multiple two-dimensional key point data after reconstruction, and correct the projected multiple two-dimensional key point data according to the reconstructed multiple two-dimensional key point data to obtain multiple corrected two-dimensional key points; a parameterized model acquisition module, used to minimize the error between the multiple two-dimensional key points in the fitted unfitted human body model and the corrected multiple two-dimensional key points to obtain a parameterized human body model.

[0022] Optionally, in an embodiment of the present application, the deep learning network model includes: a feature extraction network and an iterative error feedback network; the rotation data acquisition module includes: a data feature extraction module, which uses the feature extraction network to extract features of three-dimensional key point data to obtain posture features and scale features; a rotation data prediction module, which uses the iterative error feedback network to perform rotation prediction on posture features and scale features to obtain three-dimensional rotation data.

[0023] Optionally, in an embodiment of the present application, the mesh topology structure acquisition device also includes: a key point data acquisition module for acquiring multiple three-dimensional key point data; a rotation data acquisition module for acquiring three-dimensional rotation data corresponding to each three-dimensional key point data in the multiple three-dimensional key point data; a network model acquisition module for training the deep learning neural network using the three-dimensional key point data as training data and the three-dimensional rotation data as training labels to obtain a deep learning network model.

[0024] Optionally, in an embodiment of the present application, the network model acquisition module includes: a key point data prediction module, which is used to use a deep learning neural network to predict three-dimensional key point data and obtain predicted rotation data; a data error calculation module, which is used to respectively calculate the rotation error, key point position error and scale factor error between the predicted rotation data and the three-dimensional rotation data; a total loss value acquisition module, which is used to calculate the rotation error, key point position error and scale factor error to obtain a total loss value; and a neural network training module, which is used to train the deep learning neural network according to the total loss value to obtain a deep learning network model.

[0025] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method described above is performed.

[0026] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram of a flow chart of a method for obtaining a mesh topology structure provided in an embodiment of the present application is shown;

[0029] Figure 2 Schematic diagram showing shooting from different perspectives using a camera matrix provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of the process of training a deep learning model provided in an embodiment of the present application is shown;

[0031] Figure 4 A schematic diagram of key points of the human body provided by an embodiment of the present application is shown;

[0032] Figure 5 A schematic diagram of the network structure of a deep learning neural network provided by an embodiment of the present application is shown;

[0033] Figure 6 A schematic diagram of the network structure of the iterative error feedback (IEF) layer provided in an embodiment of the present application is shown;

[0034] Figure 7 A schematic diagram of the structure of a mesh topology acquisition device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application for protection, but merely represents selected embodiments of the embodiments of the present application. Based on the embodiments of the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the embodiments of the present application.

[0036] Before introducing the method for obtaining a mesh topology structure provided in the embodiment of the present application, some concepts involved in the embodiment of the present application are first introduced:

[0037] Deep learning is an algorithm in machine learning that is based on data representation learning. Deep learning is a branch of machine learning and is also an algorithm that uses artificial neural networks as its architecture to represent data.

[0038] The target detection network refers to a neural network that detects target objects in an image, that is, it detects the target objects in the image and gives the position range, classification and probability of the target objects in the image. The position range can be specifically marked in the form of a detection box, the classification refers to the specific category of the target object, and the probability refers to the probability that the target object in the detection box is of a specific category.

[0039] It should be noted that the mesh topology acquisition method provided in the embodiments of the present application can be executed by electronic devices. The electronic devices herein refer to device terminals capable of executing computer programs or the aforementioned servers. Examples of device terminals include smartphones, personal computers, tablet computers, personal digital assistants, and mobile internet devices. A server refers to a device that provides computing services over a network. Examples of servers include x86 servers and non-x86 servers. Non-x86 servers include mainframes, minicomputers, and UNIX servers.

[0040] The following introduces the application scenarios to which the mesh topology structure acquisition method is applicable. The application scenarios here include but are not limited to: after capturing human body movements to obtain human body images, the mesh topology structure acquisition method can be used to calculate the human body images to obtain the mesh topology structure of the parameterized human body model, thereby improving the accuracy of the mesh topology structure of the parameterized human body model, etc.

[0041] See Figure 1A flow chart of a method for acquiring a mesh topology structure provided by an embodiment of the present application is shown. The main idea of ​​the mesh topology structure acquisition method is to obtain multiple human body images captured from different perspectives, and perform key point detection and multi-perspective reconstruction on the multiple human body images, thereby eliminating the need for human participation in the annotation process of the parametric human body model. The automated production process improves the real-time performance of the mesh topology structure production and effectively improves the ambiguity of the actor's movements when taking pictures from a single perspective. Furthermore, by predicting the three-dimensional rotation data corresponding to the reconstructed three-dimensional key point data and loading the three-dimensional rotation data using the fitted parametric human body model, the mesh topology structure of the parametric human body model is obtained, thereby obtaining a more accurate body posture and size of the parametric human body model using a multi-perspective fitting method, effectively improving the accuracy of the mesh topology structure of the parametric human body model. The above-mentioned mesh topology structure acquisition method may include:

[0042] Step S110: Acquire a plurality of human body images, where the plurality of human body images are captured from different perspectives of the human body region.

[0043] See Figure 2 A schematic diagram of shooting from different perspectives using a camera matrix provided in an embodiment of the present application is shown; the thin straight lines in the figure are wall edges, and the thick straight lines are frames (which can be made of iron or steel), on which multiple cameras are installed. The multiple cameras here can form a camera matrix, thereby capturing all corners within the coverage area of ​​the camera matrix to the greatest extent, realizing the function of shooting from multiple different perspectives. The above-mentioned methods for obtaining human body images include: a first method of obtaining, using the camera calibration toolbox algorithm provided by OpenCV or MATLAB to calibrate the camera matrix composed of cameras, video recorders or color cameras, etc., to obtain the internal parameters and external parameters of each camera in the camera matrix, and then, the camera matrix can be used to shoot the human body area from different perspectives to obtain multiple human body images; then the terminal device sends the multiple human body images to the electronic device, and then the electronic device receives and stores the multiple human body images sent by the terminal device; a second method of obtaining, obtaining multiple pre-stored human body images, for example: obtaining multiple human body images from a file system, or obtaining multiple human body images from a database, or obtaining multiple human body images from a mobile storage device; a third method of obtaining, using a browser or other software to obtain multiple human body images on the Internet, or using other applications to access the Internet to obtain multiple human body images.

[0044] Step S120: performing key point detection and multi-view reconstruction on multiple human body images to obtain reconstructed three-dimensional key point data.

[0045] In some embodiments, if the quality of the human body image is sufficiently good (i.e., the human body image is mostly composed of the human body region, and the length and width of the human body region in the human body image account for the majority of the human body image, for example, 90%), key point detection and multi-view reconstruction can be directly performed on the human body image to obtain reconstructed 3D key point data. Since the key point detection and multi-view reconstruction process is very complex, the detailed process will be described in detail below.

[0046] In other embodiments, the quality of the human body image is usually not good enough, and it is necessary to perform target positioning detection on the human body image to obtain a human body region frame, and then crop the human body image according to the human body region frame to obtain a human body region image. Finally, key point detection and multi-perspective reconstruction are performed on the human body region image to obtain reconstructed three-dimensional key point data.

[0047] Step S130: Use a pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data.

[0048] The deep learning network model is a neural network model that takes 3D keypoint data as input and outputs 3D rotation data. This deep learning network model can predict the 3D rotation data corresponding to the 3D keypoint data. It is understood that the deep learning network model must be trained before use. Because the implementation of deep learning network model training is relatively complex, the training process is described in detail later. The specific network results of the deep learning network model are also presented in detail after the training process.

[0049] Step S140: obtaining the fitted parameterized human body model, and inputting the three-dimensional rotation data into the parameterized human body model to obtain a mesh topological structure of the parameterized human body model.

[0050] During the implementation of the above-mentioned steps S110 to S140, it was found that the traditional fitting iterative optimization method can only obtain the mesh topology structure corresponding to the parameterized human body model at a speed of 11 seconds per frame. However, the embodiment of the present application can perform rotation prediction on the reconstructed three-dimensional key point data through a pre-trained deep learning network model, and input the predicted three-dimensional rotation data into the parameterized human body model, which can achieve a speed of 1000 frames per second (Frames PerSecond, FPS) to obtain the mesh topology structure corresponding to the parameterized human body model. Therefore, compared with the traditional fitting iterative optimization method to obtain the mesh topology structure corresponding to the parameterized human body model, the real-time performance of the deep learning method in the embodiment of the present application in obtaining the mesh topology structure corresponding to the parameterized human body model has been significantly improved.

[0051] In the above implementation process, by obtaining multiple human body images taken from different perspectives of the human body area, and performing key point detection and multi-perspective reconstruction on the multiple human body images, there is no need for human participation in labeling during the production of the parametric human body model. The automated production process improves the real-time performance of the mesh topology structure production, and effectively improves the ambiguity of the actor's movements during single-perspective photography. Furthermore, by predicting the three-dimensional rotation data corresponding to the reconstructed three-dimensional key point data, and using the fitted parametric human body model to load the three-dimensional rotation data, the mesh topology structure of the parametric human body model is obtained, thereby utilizing the multi-perspective fitting method to obtain a more accurate body posture and size of the parametric human body model, effectively improving the accuracy of the mesh topology structure of the parametric human body model.

[0052] See Figure 3 The flowchart of the deep learning model training provided by the embodiment of the present application is shown; optionally, before using the deep learning network model, the deep learning network model can also be trained, or a trained deep learning model can be directly obtained from it. The process of training the deep learning model is described below, which may include:

[0053] Step S210: Acquire multiple three-dimensional key point data.

[0054] The implementation of the above step S210 may include the following:

[0055] Step S211: Acquire multiple human sample images, and use the key point detection algorithm of the first human model to perform key point detection on each of the multiple human sample images to obtain multiple groups of two-dimensional key points.

[0056] See Figure 4 The schematic diagram of the human body key points provided by the embodiment of the present application is shown; the implementation method of the above step S211 is, for example: for each human body sample image in a plurality of human body sample images, a target detection network model is used to locate the human body area in the human body sample image to detect the target, and obtain a human body area image; wherein, the target detection network models that can be used include: Feature Fusion Single Shot multi-box Detector (FSSD), YOLO network model, RCNN, fast RCNN and faster RCNN series network models. Use the target detection framework and the human body key point detection algorithm (such as the key point detection algorithm of the openpose model) to perform key point detection on the human body area image to obtain a set of two-dimensional key points, where the two-dimensional key points are the image coordinates of the human body key points.

[0057] Optionally, in the process of acquiring multiple human body images, you can create multiple human body image training data from scratch. For example, if you want actors to perform actions in the same direction, you can use Figure 2 The camera matrix in is used to take photos or record videos of the actors to obtain training data of multiple human images. It is also possible to perform image enhancement on these human images based on the human images that have been obtained, so as to obtain more human images. For example, random noise can be added to these human images to expand more training data, and the overfitting problem of the neural network model can be effectively alleviated. Similarly, these random noises can also be added to the two-dimensional key points, three-dimensional key points and / or the following three-dimensional rotation data, so as to effectively expand the training data set and improve the dependence of the deep learning neural network on direction (i.e., the overfitting problem). Therefore, by adding random noise, the time for producing the training data set can be effectively reduced, as well as the training time of the deep learning neural network.

[0058] Step S212: reconstruct each group of two-dimensional key points in the multiple groups of two-dimensional key points from multiple perspectives according to the parameters calibrated by the camera array to obtain reconstructed three-dimensional key point data.

[0059] For example, the above-mentioned step S212 may be implemented as follows: the above-mentioned parameters include: the internal parameters and external parameters of each camera in the camera matrix; using the internal parameters and external parameters obtained after the above camera calibration, and the image coordinates of the key points of the human body, the two-dimensional key points in the human body region image are three-dimensionally reconstructed to obtain the three-dimensional (3D) coordinates of the key points of the human body in the world coordinate system, where the three-dimensional coordinates can be recorded as detect_3d∈R 21×3 The three-dimensional coordinates of the human body key points in the world coordinate system are the reconstructed three-dimensional key point data.

[0060] After step S210 , step S220 is executed: obtaining three-dimensional rotation data corresponding to each three-dimensional key point data in the plurality of three-dimensional key point data.

[0061] The implementation of the above step S220 is, for example: using the second human body model (which can be a parameterized human body model) to perform Rodrigues transformation on the three-dimensional key point data to obtain the three-dimensional rotation data corresponding to each three-dimensional key point data. Here, the first human body model and the second human body model are different, and the three-dimensional rotation data includes the local rotation matrix of each joint. The local rotation matrix can be expressed as M 24×3×3 The above-mentioned parameterized human body model can specifically adopt the Skinned Multi-Person Linear Model-eXpressive (SMPL-X) model. The SMPL-X model can be expressed as body(θ,β), where θ∈R32 is the posture parameter, β∈R 10 By changing (θ, β), the function body(θ, β) can output the 3D position information of the vertices of the human body model of different postures and shapes, which is the mesh topology (mesh), denoted as V, and the position information of the 3D key points under the corresponding model can also be obtained, denoted as key_points∈R 24×3 .

[0062] In the process of implementing the above solution, it is difficult to obtain human sample images that conform to the human body kinematic structure points (such as Figure 4 Therefore, the OpenPose human key point detection algorithm is used to perform key point detection on each human sample image in multiple human sample images, and three-dimensional key point data is generated by obtaining the two-dimensional key points of the OpenPose model (which is easy to label and commonly used in academia). The three-dimensional key point data is the training data for training the deep learning neural network, and the three-dimensional rotation data obtained by performing Rodrigues transform on the three-dimensional key point data using the SMPL-X model is the training label for training the deep learning neural network, thereby achieving the skeleton migration effect from the OpenPose model to the SMPL-X model.

[0063] During the training process, the deep learning neural network also receives input skeleton directions (i.e., the orientation of 3D points, indicating the relative directions between parent and child nodes) and hierarchical structures (i.e., the parent-child node relationships between 3D points). This effectively changes the directional information mapping relationship between the OpenPose model and the SMPL-X skeleton model based on the skeleton directions and hierarchical structures. This avoids the occurrence of membrane penetration (i.e., the skeleton protruding from the membrane skin) or non-standard movements (such as a stiff waist straight line or a very stiff backflip) in the final generated mesh topology, thereby improving the accuracy of the mesh topology of the final parametric human body model.

[0064] Specifically, the above-mentioned use of the SMPL-X model to perform a Rodrigues transform on the reconstructed three-dimensional key point data can also be to first perform image data enhancement (including: adding noise or rotating along the Z axis), and then perform a Rodrigues transform on the enhanced three-dimensional key point data; a specific example of this implementation is: adding noise to one dimension of the reconstructed three-dimensional key point data to obtain noisy three-dimensional key point data, or rotating the reconstructed three-dimensional key point data along one dimension to obtain rotated three-dimensional key point data; and using the SMPL-X model to perform a Rodrigues transform on the noisy three-dimensional key point data or the rotated three-dimensional key point data.

[0065] In the implementation process of the above scheme, by first adding noise to one of the dimensions of the reconstructed three-dimensional key point data, or rotating the reconstructed three-dimensional key point data along one of the dimensions, and then using the SMPL-X model to perform Rodrigues transform on the noisy three-dimensional key point data or the rotated three-dimensional key point data, more three-dimensional rotation data can be effectively added as training labels for the deep learning neural network, thereby reducing the shortage of training samples and training labels for the deep learning neural network, and improving the accuracy of the deep learning neural network in predicting three-dimensional key point data.

[0066] After step S220, step S230 is executed: the deep learning neural network is trained using the three-dimensional key point data as training data and the three-dimensional rotation data as training labels to obtain a deep learning network model.

[0067] It can be understood that the three-dimensional key point data and the three-dimensional rotation data are one-to-one corresponding supervised data sets, which can be expressed as {X|xi=(detect_3d,M)}; where detect_3d represents the three-dimensional key point data, M represents the three-dimensional rotation data, X represents the supervised data set, and xi represents a pair of three-dimensional key point data and three-dimensional rotation data in the supervised data set.

[0068] The implementation of the above step S230 may include:

[0069] Step S231: Use a deep learning neural network to predict the three-dimensional key point data to obtain predicted rotation data.

[0070] See Figure 5 The network structure diagram of the deep learning neural network provided by the embodiment of the present application is shown; the deep learning neural network is a neural network model that can predict the three-dimensional rotation data corresponding to the three-dimensional key point data, wherein the deep learning network model includes: a feature extraction network and an iterative error feedback network, and the connection relationship between each network structure and network hierarchy is shown in the figure. The feature extraction network includes: initialization pose (init pose), initialization scale (initscale), three-dimensional key point position information J, skeleton direction and hierarchical structure D, reference key point position information J ref , reference bone direction and hierarchical structure D ref, Fully Connected Layer (FC), and Feature Layer. The iterative error feedback network includes: Iteration Error Feedback (IEF) layer, pose prediction (predpose) layer, scale prediction (scale) layer, and Forward Kinetic (FK) layer. Among them, the initial pose (init pose) is actually a mean pose, which can be obtained by averaging the pose parameters of all samples.

[0071] See Figure 6 The network structure diagram of the iterative error feedback (IEF) layer provided in the embodiment of the present application is shown; the above-mentioned iterative error feedback (IEF) layer may include: a first dropout layer, a second fully connected (FC) layer (1024), a second dropout layer, a third fully connected (FC) layer (24*6) and a fourth fully connected (FC) layer; wherein the connection relationship between each network structure is as follows Figure 5 and Figure 6 shown.

[0072] For example, the implementation of step S231 is as follows: assuming that the k-th three-dimensional key point coordinate in the three-dimensional key point data is represented by bk, the value range of k can be 0≤k≤24; the input of the deep learning neural network is I=[J,D,J ref ,D ref ]∈R 4×B×3 , where B represents the number of joints in the human body model, J is the scale-normalized three-dimensional key point position information relative to the root node, D represents the bone direction and hierarchical structure, and J ref and D ref These are the reference key point position information and reference bone direction and hierarchical structure converted from the key points of the pose template (Tpose). If only the key points are input, the deep learning network model will only have a bunch of discrete points, and the structural information of these points is missing, making it difficult for the network to train and obtain the expected results. If the position of the child node relative to the parent node is used, this hierarchical structure and direction can be expressed. For example, if the parent node of the kth joint is labeled parent k Indicates that 0≤k≤24. The human mesh topology (mesh) obtained by setting the posture parameter θ to all zero vectors is the posture template (Tpose). The key points at this time can be recorded as key_pointsTpose, and the mesh topology (mesh) and key point positions of the posture template (Tpose) are obtained.

[0073] The above deep learning neural network takes as input the initialization pose (init pose), initialization scale (initscale), three-dimensional key point position information J, skeleton direction and hierarchical structure D, reference key point position information J ref and reference bone orientation and hierarchy D ref Afterwards, the output of the deep learning neural network is the local rotation information corresponding to each joint point in the SMPL-X model and a scalar scale factor Since 3D rotation information is usually represented by quaternions, Euler angles or 3×3 rotation matrices, it should be noted that the 3D expression in the above representation is not continuous during the deep learning neural network training process, which makes it difficult for the deep learning neural network to converge. Therefore, the 3D rotation information can be converted into a continuous 6D expression. The conversion process can be expressed using formulas. For example, the formula can be used. Convert the 3×3 rotation matrix into a 6D expression and then use the formula and Convert the 6D expression to a 3×3 rotation matrix. Where N(·) represents the normalization function, N(q)=q / ||q||. Convert the 6D rotation information output by the network to Converted to the 3×3 local rotation matrix of each joint to get

[0074] Step S232: Calculate the rotation error, key point position error and scale factor error between the predicted rotation data and the three-dimensional rotation data respectively.

[0075] It's important to note that the scale factor (scale) represents the scaling relationship between the SMPL model skeleton and the detected 3D point skeleton, and it must also be considered as part of the loss. Otherwise, the network training results will be mirrored. Therefore, when constructing the loss function to calculate the total loss, the scale factor is included as part of the loss.

[0076] The implementation of the above step S232 includes: after obtaining the predicted rotation data and the three-dimensional rotation data, the formula Calculate the rotation error between the predicted rotation data and the 3D rotation data; where loss M represents the rotation error, Represents the local rotation matrix in the three-dimensional rotation data, M kij represents the local rotation matrix in the predicted rotation data. You can also use the formula Calculate the key point position error between the predicted rotation data and the 3D rotation data; where loss xyzrepresents the key point position error, Represents the estimated position of the key points of the human body posture in the three-dimensional rotation data, key_points posed,ki Represents the estimated position of the key points of the human body posture in the predicted rotation data. You can also use the formula Calculate the scale factor error between the predicted rotation data and the 3D rotation data; where loss scale represents the scale factor error, It represents the scale factor of the human body posture in the three-dimensional rotation data, and scale represents the scale factor of the human body posture in the predicted rotation data.

[0077] Step S233: Calculate the rotation error, key point position error, and scale factor error to obtain a total loss value.

[0078] The implementation of the above step S233 is, for example: using the formula total_loss=λ M loss M +λ XYZ loss xyz +λ scale loss scale The rotation error, key point position error and scale factor error are weighted and summed to obtain the total loss value; where total_loss represents the total loss value, loss M represents the rotation error, λ M Represents the weight factor of the rotation error, loss xyz represents the key point position error, λ XYZ Represents the weight factor of the key point position error, loss scale represents the scale factor error, λ scale A weighting factor representing the scale factor error.

[0079] Step S234: Train the deep learning neural network according to the total loss value to obtain a deep learning network model.

[0080] An implementation example of the above-mentioned step S234 is as follows: in the process of training the deep learning neural network according to the total loss value, the stochastic gradient descent method can be used until the total loss value (i.e., the entire loss) converges, and a trained deep learning network model can be obtained.

[0081] In the above implementation process, the deep learning network model is trained by calculating the total loss value based on the rotation error, key point position error, and scale factor error, thereby avoiding the problem of using only one of the errors to train the deep learning network model and effectively improving the accuracy of the deep learning network model in predicting three-dimensional rotation data.

[0082] Step S310: Acquire multiple human body images, where the multiple human body images are taken from different perspectives of the human body area.

[0083] Among them, the implementation principle and implementation method of step S310 are similar to those of step S110. Therefore, its implementation principle and implementation method are not explained here. If there are any unclear points, please refer to the description of step S110.

[0084] After step S310 , step S320 is executed: key point detection and multi-view reconstruction are performed on the multiple human body images to obtain reconstructed three-dimensional key point data.

[0085] The implementation of the above step S320 may include:

[0086] Step S321: performing key point detection on each of the multiple human body images to obtain multiple groups of two-dimensional key points.

[0087] Step S322: reconstruct each group of two-dimensional key points in the multiple groups of two-dimensional key points from multiple perspectives according to the parameters calibrated by the camera array to obtain reconstructed three-dimensional key point data.

[0088] Among them, the implementation principles and implementation methods of steps S321 to S322 are similar to those of steps S211 to S212. Therefore, their implementation principles and implementation methods are no longer explained here. If there are any unclear points, please refer to the description of steps S211 to S212.

[0089] After step S320, step S330 is executed: using a pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data.

[0090] The implementation of step S330 may include: It is understood that the deep learning network model includes a feature extraction network and an iterative error feedback (IEF) network; the feature extraction network can be used to extract features from the 3D key point data to obtain pose features and scale features. The iterative error feedback (IEF) network can be used to perform rotation prediction on the pose features and scale features to obtain 3D rotation data.

[0091] Among them, the implementation principle and implementation method of step S330 are similar to the implementation principle and implementation method of step S231. The only difference is that the input and output of the deep learning network model are different. Therefore, its implementation principle and implementation method will not be explained here. If there is anything unclear, please refer to the description of step S231.

[0092] After step S330 , step S340 is performed: obtaining the fitted parameterized human body model, and inputting the three-dimensional rotation data into the parameterized human body model to obtain a mesh topological structure of the parameterized human body model.

[0093] The implementation of obtaining the fitted parameterized human body model in step S340 may include:

[0094] Step S341: obtaining an unfitted human body model, and projecting a plurality of three-dimensional key points in the unfitted human body model onto a human body region image to obtain a plurality of two-dimensional key point data after projection.

[0095] The implementation method of the above-mentioned step S341, for example, can be: receiving an unfitted human body model sent by other devices, or downloading the unfitted human body model from the Internet, or creating a parameterized human body model from scratch through engineering fitting optimization, and projecting multiple three-dimensional key points in the unfitted human body model onto the human body region image to obtain multiple two-dimensional key point data after projection; wherein the above-mentioned unfitted human body model can specifically adopt an unfitted SMPL-X model.

[0096] Step S342: Project the reconstructed three-dimensional key point data onto the human body area image to obtain multiple reconstructed two-dimensional key point data, and correct the projected multiple two-dimensional key point data based on the reconstructed multiple two-dimensional key point data to obtain multiple corrected two-dimensional key points.

[0097] Step S343: Minimize the errors between the multiple two-dimensional key points in the fitted and unfitted human body model and the multiple two-dimensional key points after correction to obtain a parameterized human body model.

[0098] An implementation example of the above-mentioned step S343 is as follows: in the above-mentioned process of minimizing the fitting of the SMPL-X model, the shape parameter β can be fixed, and the posture parameter θ and the scale factor scale in the SMPL-X model, as well as the global displacement t, can be optimized to minimize the errors between the multiple two-dimensional key points in the human body model and the corrected multiple two-dimensional key points.

[0099] The implementation of inputting the 3D rotation data into the parametric human body model to obtain the mesh topology of the parametric human body model in step S340 includes: assuming that the 3D rotation data obtained in the above steps is represented by J (i.e., the position information of the 3D key points after rotation), inputting the 3D rotation data J into the parametric human body model, obtaining the mesh topology of the parametric human body model, and outputting an obj file. In practice, this process can achieve 100 fps, which is sufficient to achieve the technical effect of obtaining the mesh topology of the human body model in real time.

[0100] See Figure 7 The schematic diagram of the structure of the network topology structure acquisition device provided by the embodiment of the present application is shown. The embodiment of the present application provides a network topology structure acquisition device 300, including:

[0101] The human body image acquisition module 310 is used to acquire multiple human body images, where the multiple human body images are taken from different perspectives of the human body area.

[0102] The reconstruction data acquisition module 320 is used to perform key point detection and multi-view reconstruction on multiple human body images to obtain reconstructed three-dimensional key point data.

[0103] The rotation data acquisition module 330 is used to use a pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data.

[0104] The mesh topology obtaining module 340 is used to obtain the fitted parameterized human body model and input the three-dimensional rotation data into the parameterized human body model to obtain the mesh topology structure of the parameterized human body model.

[0105] Optionally, in an embodiment of the present application, the reconstruction data acquisition module includes:

[0106] The two-dimensional key point detection module is used to perform key point detection on each human body image in multiple human body images to obtain multiple groups of two-dimensional key points.

[0107] The two-dimensional multi-view reconstruction module is used to perform multi-view reconstruction on each group of two-dimensional key points in multiple groups of two-dimensional key points according to the parameters calibrated by the camera array to obtain the reconstructed three-dimensional key point data.

[0108] Optionally, in an embodiment of the present application, the two-dimensional key point detection module includes:

[0109] The region image detection module is used to perform target positioning detection on a human region in each human body image in a plurality of human body images to obtain a human body region image.

[0110] The two-dimensional key point acquisition module is used to detect key points in human body area images and obtain a set of two-dimensional key points.

[0111] Optionally, in an embodiment of the present application, the mesh topology obtaining module includes:

[0112] The three-dimensional key point projection module is used to obtain an unfitted human body model and project multiple three-dimensional key points in the unfitted human body model onto a human body region image to obtain multiple two-dimensional key point data after projection.

[0113] The key point data correction module is used to project the reconstructed three-dimensional key point data onto the human body area image to obtain multiple reconstructed two-dimensional key point data, and correct the projected multiple two-dimensional key point data based on the reconstructed multiple two-dimensional key point data to obtain multiple corrected two-dimensional key points.

[0114] The parameterized model acquisition module is used to minimize the errors between the multiple two-dimensional key points in the fitted and unfitted human body model and the corrected multiple two-dimensional key points to obtain a parameterized human body model.

[0115] Optionally, in an embodiment of the present application, the deep learning network model includes: a feature extraction network and an iterative error feedback network; and the rotation data acquisition module includes:

[0116] The data feature extraction module is used to extract features from three-dimensional key point data using a feature extraction network to obtain posture features and scale features.

[0117] The rotation data prediction module is used to use an iterative error feedback network to perform rotation prediction on posture features and scale features to obtain three-dimensional rotation data.

[0118] Optionally, in the embodiment of the present application, the network topology structure acquisition device further includes:

[0119] The key point data acquisition module is used to obtain multiple three-dimensional key point data.

[0120] The rotation data acquisition module is used to acquire three-dimensional rotation data corresponding to each three-dimensional key point data in a plurality of three-dimensional key point data.

[0121] The network model acquisition module is used to train the deep learning neural network using three-dimensional key point data as training data and three-dimensional rotation data as training labels to obtain a deep learning network model.

[0122] Optionally, in an embodiment of the present application, the network model obtaining module includes:

[0123] The key point data prediction module is used to predict three-dimensional key point data using a deep learning neural network to obtain predicted rotation data.

[0124] The data error calculation module is used to calculate the rotation error, key point position error and scale factor error between the predicted rotation data and the three-dimensional rotation data respectively.

[0125] The total loss value acquisition module is used to calculate the rotation error, key point position error and scale factor error to obtain the total loss value.

[0126] The neural network training module is used to train the deep learning neural network according to the total loss value to obtain a deep learning network model.

[0127] It should be understood that the device corresponds to the above-mentioned embodiment of the method for obtaining a mesh topology structure and can perform each step involved in the above-mentioned method embodiment. The specific functions of the device can be found in the description above. To avoid repetition, a detailed description is omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the operating system (OS) of the device.

[0128] An electronic device provided by an embodiment of the present application includes: a processor and a memory, the memory storing machine-readable instructions executable by the processor, and the above method being executed when the machine-readable instructions are executed by the processor.

[0129] The present application also provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to execute the above method. The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0130] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly based on the functions involved.

[0131] In addition, the functional modules of each embodiment in the embodiments of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0132] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0133] The above description is only an optional implementation method of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, and they should all be covered by the protection scope of the embodiment of the present application.

Claims

1. A method for obtaining a mesh topology structure, characterized in that: include: Acquire a plurality of human body images, wherein the plurality of human body images are captured from different perspectives of the human body region; Performing key point detection and multi-view reconstruction on the multiple human body images to obtain reconstructed three-dimensional key point data; Using a pre-trained deep learning network model, a rotation prediction is performed on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data; Acquire a fitted parameterized human body model, and input the three-dimensional rotation data into the parameterized human body model to obtain a mesh topological structure of the parameterized human body model; The deep learning network model is obtained by training a deep learning neural network using three-dimensional key point data of a first human body model as training data and three-dimensional rotation data of a second human body model as training labels, and the first human body model and the second human body model are different; The three-dimensional key point data of the first human body model is obtained by detecting two-dimensional key points from the multiple human body images using a key point detection algorithm for the first human body model, and performing multi-view reconstruction on the two-dimensional key points; The three-dimensional rotation data of the second human body model is obtained by transforming the three-dimensional key point data after multi-view reconstruction using the second human body model.

2. The method according to claim 1, characterized in that The performing key point detection and multi-view reconstruction on the plurality of human body images to obtain reconstructed three-dimensional key point data includes: Performing key point detection on each of the multiple human body images to obtain multiple groups of two-dimensional key points; Each group of two-dimensional key points in the plurality of groups of two-dimensional key points is reconstructed from multiple perspectives according to the parameters calibrated by the camera array to obtain the reconstructed three-dimensional key point data.

3. The method according to claim 2, characterized in that The step of performing key point detection on each of the plurality of human body images to obtain multiple groups of two-dimensional key points includes: For each human body image in the plurality of human body images, performing positioning target detection on a human body region in the human body image to obtain a human body region image; Key point detection is performed on the human body region image to obtain a set of two-dimensional key points.

4. The method according to claim 3, characterized in that The step of obtaining the fitted parameterized human body model includes: Acquire an unfitted human body model, and project a plurality of three-dimensional key points in the unfitted human body model onto the human body region image to obtain a plurality of two-dimensional key point data after projection; Projecting the reconstructed three-dimensional key point data onto the human body region image to obtain a plurality of reconstructed two-dimensional key point data, and correcting the projected plurality of two-dimensional key point data according to the plurality of reconstructed two-dimensional key point data to obtain a plurality of corrected two-dimensional key points; Minimize the error between the multiple two-dimensional key points in the unfitted human body model and the corrected multiple two-dimensional key points to obtain the parameterized human body model.

5. The method according to claim 1, wherein The deep learning network model includes: a feature extraction network and an iterative error feedback network; the use of the pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data includes: Using the feature extraction network to extract features from the three-dimensional key point data to obtain posture features and scale features; The iterative error feedback network is used to perform rotation prediction on the posture feature and the scale feature to obtain the three-dimensional rotation data.

6. The method according to claim 1, characterized in that The transforming of the multi-view reconstructed three-dimensional key point data using the second human body model includes: adding noise to one dimension of the reconstructed three-dimensional key point data to obtain noisy three-dimensional key point data, or rotating the reconstructed three-dimensional key point data along one dimension to obtain rotated three-dimensional key point data; A second human body model is used to perform a Rodrigues transform on the noisy three-dimensional key point data or the rotated three-dimensional key point data.

7. The method according to claim 6, characterized in that The first human body model is an OpenPose model, and the second human body model is a skinned multi-person linear SMPL-X model.

8. The method according to claim 1, characterized in that The training of the deep learning neural network includes: Using the deep learning neural network to predict the three-dimensional key point data to obtain predicted rotation data; respectively calculating a rotation error, a key point position error, and a scale factor error between the predicted rotation data and the three-dimensional rotation data; Calculating the rotation error, the key point position error, and the scale factor error to obtain a total loss value; The deep learning neural network is trained according to the total loss value to obtain the deep learning network model.

9. A device for acquiring a mesh topology structure, characterized in that: include: A human body image acquisition module, configured to acquire a plurality of human body images, wherein the plurality of human body images are obtained by photographing a human body region from different perspectives; A reconstruction data acquisition module is used to perform key point detection and multi-view reconstruction on the multiple human body images to obtain reconstructed three-dimensional key point data; A rotation data acquisition module, configured to use a pre-trained deep learning network model to perform rotation prediction on the reconstructed three-dimensional key point data to obtain three-dimensional rotation data; A mesh topology acquisition module, configured to acquire a fitted parameterized human body model and input the three-dimensional rotation data into the parameterized human body model to obtain a mesh topology structure of the parameterized human body model; The deep learning network model is obtained by training a deep learning neural network using three-dimensional key point data of a first human body model as training data and three-dimensional rotation data of a second human body model as training labels, and the first human body model and the second human body model are different; The three-dimensional key point data of the first human body model is obtained by detecting two-dimensional key points from the multiple human body images using a key point detection algorithm for the first human body model, and performing multi-view reconstruction on the two-dimensional key points; The three-dimensional rotation data of the second human body model is obtained by transforming the three-dimensional key point data after multi-view reconstruction using the second human body model.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the method according to any one of claims 1 to 8 is performed.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is executed.

Images