Three-dimensional model reconstruction method and device

Abstract

The application discloses a three-dimensional model reconstruction method and device, and belongs to the technical field of image processing. The method comprises the following steps: acquiring N frames of first images and P frames of second images of a first object in a motion process, wherein the N frames of first images comprise the P frames of second images, and N and P are positive integers; performing three-dimensional reconstruction on the first object in the N frames of first images by using a first algorithm to obtain a first reconstruction model of the first object; performing three-dimensional reconstruction on the first object in the P frames of second images by using a first template respectively to obtain a second reconstruction model of the first object; and adjusting the first object in the first reconstruction model based on the second reconstruction model to obtain a third reconstruction model of the first object.

Description

Technical Field

[0001] This application belongs to the field of image processing, and specifically relates to a three-dimensional model reconstruction method and apparatus. Background Technology

[0002] With the rapid development of 3D reconstruction technology, it has received widespread attention and in-depth research in various fields. 3D reconstruction of the human body is currently a hot research topic.

[0003] Currently, when performing 3D reconstruction of the human body, the reconstructed human body generally suffers from low accuracy, insufficient realism in details, high algorithm complexity, and long processing time, making it difficult to meet the requirements for real-time reconstruction. Summary of the Invention

[0004] The purpose of this application is to provide a three-dimensional model reconstruction method and apparatus, which solves the problems of low accuracy and long processing time in the prior art for three-dimensional reconstruction of the human body, and the inability to perform three-dimensional reconstruction of the human body in real time.

[0005] In a first aspect, embodiments of this application provide a three-dimensional model reconstruction method, the method comprising:

[0006] Acquire N frames of first images and P frames of second images of the first object during its motion, wherein the N frames of first images include the P frames of second images, and N and P are both positive integers;

[0007] The first object in the N frames of the first image is reconstructed in three dimensions using the first algorithm to obtain the first reconstructed model of the first object;

[0008] Using the first template, the first object in the second image of the P-frame is reconstructed in three dimensions to obtain a second reconstructed model of the first object;

[0009] The first object in the first reconstruction model is adjusted based on the second reconstruction model to obtain the third reconstruction model of the first object.

[0010] Secondly, embodiments of this application provide a three-dimensional model reconstruction apparatus, the apparatus comprising:

[0011] The first acquisition module is used to acquire N frames of first images and P frames of second images of the first object during its motion process, wherein the N frames of first images include the P frames of second images, and N and P are both positive integers;

[0012] The first reconstruction module is used to perform three-dimensional reconstruction of the first object in the N frames of the first image using a first algorithm to obtain a first reconstruction model of the first object.

[0013] The second reconstruction module is also used to perform three-dimensional reconstruction of the first object in the second image of the P-frame using the first template, so as to obtain a second reconstruction model of the first object.

[0014] The adjustment module is used to adjust the first object in the first reconstruction model based on the second reconstruction model to obtain a third reconstruction model of the first object.

[0015] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0016] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0017] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0018] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0019] In this embodiment, a first algorithm is used to perform 3D reconstruction on the first object in N frames of the first image during the motion process to obtain a first reconstruction model of the first object. Then, a first template is used to perform 3D reconstruction on the first object in P frames of the second image during the motion process to obtain a second reconstruction model of the first object. In this embodiment, the first reconstruction model and the second reconstruction model are obtained by performing 3D reconstruction once using the first algorithm and the first template respectively. Then, the first object in the first reconstruction model is adjusted based on the second reconstruction model to obtain a third reconstruction model of the first object, thereby improving the accuracy and timeliness of 3D reconstruction. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a three-dimensional model reconstruction method provided in some embodiments of this application;

[0021] Figure 2 This is a flowchart illustrating a three-dimensional model reconstruction method provided in some embodiments of this application;

[0022] Figure 3These are schematic diagrams illustrating the structure of a three-dimensional model reconstruction device according to some embodiments of this application;

[0023] Figure 4 These are schematic diagrams illustrating the structure of an electronic device according to some embodiments of this application;

[0024] Figure 5 These are schematic diagrams of the hardware structure of an electronic device illustrated in some embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] Before introducing the technical solutions of the embodiments of this application, the background technology of the embodiments of this application will be introduced first:

[0028] When performing 3D reconstruction of the human body, existing 3D human body templates are usually used. The acquired human body images are analyzed and processed to reconstruct the 3D human body. Alternatively, multi-view human body reconstruction methods based on computer vision and machine learning can be used to reconstruct the human body.

[0029] However, the method of using human body 3D modules to reconstruct the human body in 3D relies too heavily on human body template information, resulting in the reconstructed human body not being realistic enough. The 3D human body template is based on a human body template library trained with prior experience. Therefore, the algorithm has the advantages of robustness, fast operation speed, good tracking effect and the ability to recover the 3D human body from a single view.

[0030] The method of using multi-view human body reconstruction to reconstruct the human body in 3D is not bound by prior template information, so the accuracy of the 3D reconstruction of the human body is high, and the details and textures of the reconstructed 3D model exceed those of the human body 3D template. However, the method is relatively complex, has a long processing time, and is difficult to achieve real-time reconstruction. In addition, during the movement of the human body, there will be a large range of self-occlusion and motion blur in the image, which will lead to noise and surface defects in the reconstructed human body 3D model.

[0031] To address the aforementioned issues, this application provides a three-dimensional model reconstruction method and apparatus. The method involves performing three-dimensional reconstruction on a first object in N frames of first images during its motion using a first algorithm to obtain a first reconstructed model of the first object. Then, a first template is used to perform three-dimensional reconstruction on the first object in P frames of second images during its motion to obtain a second reconstructed model of the first object. In this application embodiment, the first and second reconstructed models are obtained by performing three-dimensional reconstruction once using the first algorithm and the first template, respectively. Then, the first object in the first reconstructed model is adjusted based on the second reconstructed model to obtain a third reconstructed model of the first object, thus improving the accuracy and timeliness of the three-dimensional reconstruction.

[0032] The technical solutions of this application embodiment can be applied to scenarios involving three-dimensional reconstruction of the human body during movement.

[0033] The three-dimensional model reconstruction method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0034] It should be noted that the examples in this application all involve a first object moving in a shooting scene, in which 100 image acquisition devices are deployed, such as 100 cameras. These 100 cameras are deployed at different positions in the shooting scene to capture images of the first object from different angles.

[0035] Figure 1 This is a flowchart illustrating a three-dimensional model reconstruction method provided in an embodiment of this application.

[0036] like Figure 1 As shown, the three-dimensional model reconstruction method provided in this application embodiment may include S110-S140.

[0037] S110. Acquire N frames of the first image and P frames of the second image of the first object during its motion.

[0038] The first object can be an object to be reconstructed in 3D, such as a person, animal, or other building. In this embodiment, a person is used as an example to illustrate the first object.

[0039] Both the first and second images can be images of the first object during its motion. Here, N frames of the first images include P frames of the second images, where N and P are both positive integers. That is, the second image is a subset of the first images.

[0040] In some embodiments of this application, images of the first object during its motion can be selected from all image acquisition devices in the shooting scene and used as the first image. Then, an image of the first object during its motion can be captured by one of the selected image acquisition devices and used as the second image.

[0041] In one example, taking a motion process of 5 minutes as an example, some cameras can be selected from 100 cameras in the shooting scene. For example, images of the first object's motion process during the 5 minutes taken by 10 cameras can be selected as the first image, and then an image of the first object's motion process during the 5 minutes taken by one of the 10 cameras can be selected as the second image.

[0042] It should be noted that the selected image acquisition devices can ensure that images of the first object can be captured from various angles, such as the front, side, and back views. One of the selected image acquisition devices is typically chosen to capture a frontal image of the first object.

[0043] S120. Using the first algorithm, perform three-dimensional reconstruction of the first object in the first N frames of the first image to obtain the first reconstruction model of the first object.

[0044] The first algorithm is an existing algorithm that processes multiple frames of images to reconstruct a first object in the images, such as a multi-view reconstruction algorithm.

[0045] The first reconstruction model can be a 3D reconstruction model of the first object obtained by performing 3D reconstruction of the first object in N frames of the first image using the first algorithm. The first reconstruction model of the first object reconstructed using the first algorithm will have some empty areas and some outliers, but the outline is clear. Here, the empty areas refer to some missing areas in the middle part of the reconstructed body of the first object, and the outliers refer to pixels outside the outline of the reconstructed first object.

[0046] In some embodiments of this application, in order to accurately obtain the first reconstructed model of the first object, S120 may specifically include:

[0047] Extract the first object from the first image of N frames to obtain the foreground image of N frames;

[0048] Extract feature points of the first object from each frame of the foreground image;

[0049] The feature points of the first object in the foreground image at the target time are fused to obtain the three-dimensional point cloud data of the first object at the target time.

[0050] The three-dimensional point cloud data at each moment during the motion of the first object are plotted to obtain a three-dimensional human reconstruction model of the first object.

[0051] Texture mapping is performed on the 3D human body reconstruction model to obtain the first reconstruction model of the first object.

[0052] The foreground image can be the image obtained by extracting the first object from the first image.

[0053] The target time can be any moment during the movement of the first object. For example, if the movement of the first object lasts for 5 minutes, such as from 8:05 to 8:10, then the target time could be 8:08.

[0054] A three-dimensional human body reconstruction model can be obtained by drawing surfaces from the three-dimensional point cloud data at various moments during the movement of the first object.

[0055] In some embodiments of this application, a first object can be extracted from each frame of the first image to obtain N frames of foreground images. The method of obtaining the foreground images will be described in detail in subsequent embodiments.

[0056] For each frame of the foreground image, when extracting feature points of the first object from that frame, it can be done based on a feature point extraction model. This model can be a deep learning-based neural network model, a support vector machine, a decision tree model, etc., and is not limited here. It should be noted that the feature points here can be feature points corresponding to the joints of the first object.

[0057] When acquiring the 3D point cloud data of the first object at the target time, the transformation matrix H between two frames of the first image taken by each pair of cameras can be calculated based on the parameters of each camera corresponding to the N frames of the first image. The specific method of calculating the transformation matrix H between two frames of the first image taken by each pair of cameras based on the parameters of each camera is existing technology and will not be elaborated here.

[0058] Then, by selecting a certain depth value and using the cost volume function constructed by the convolutional network, the probability of the first image in each frame being at that depth value can be predicted. It can be known that the first images belonging to the same depth value were taken at the same time. Then, by fusing the feature points in the first images belonging to the same depth value, the 3D point cloud data of the first object at that depth value can be obtained.

[0059] Then, the 3D point cloud data at each moment during the movement of the first object is rendered into a surface to obtain the 3D human reconstruction model of the first object. The specific surface rendering of the 3D point cloud data is existing technology and will not be elaborated here.

[0060] Then, texture mapping is performed on the 3D human body reconstruction model to obtain the first reconstruction model of the first object.

[0061] In the embodiments of this application, a first object is extracted from N frames of first images to obtain N frames of foreground images. Then, feature points of the first object are extracted from each frame of foreground images. The feature points of the first object in the foreground images at the target time are fused to obtain the three-dimensional point cloud data of the first object at the target time. The three-dimensional point cloud data at each time during the movement of the first object are drawn to obtain the three-dimensional human body reconstruction model of the first object. Then, the three-dimensional human body reconstruction model is texture mapped to obtain the accurate first reconstruction model of the first object.

[0062] In some embodiments of this application, in order to further accurately obtain the first reconstructed model of the first object, the process of performing texture mapping on the three-dimensional human body reconstruction model to obtain the first reconstructed model of the first object may specifically include:

[0063] Obtain the texel information of the first image in each frame;

[0064] The texel information of the first image in each frame is fused to obtain a texture map;

[0065] Texture mapping is performed on the 3D human body reconstruction model based on the texture map to obtain the first reconstruction model of the first object.

[0066] In some embodiments of this application, a texture map can be obtained by fusing the texel information of each frame of the first image. Then, the texture map can be applied to the three-dimensional human body reconstruction model, that is, texture mapping can be performed on the three-dimensional human body reconstruction model to obtain the first reconstruction model of the first object.

[0067] In the embodiments of this application, a texture map is obtained by fusing the texel information of each frame of the first image. Then, a texture mapping is performed on the three-dimensional human body reconstruction model based on the texture map, and the first reconstruction model of the first object can be directly obtained. Thus, a first reconstruction model with texture information can be obtained.

[0068] In some embodiments of this application, in order to further accurately obtain the first reconstructed model of the first object, the step of performing texture mapping on the three-dimensional human body reconstruction model based on the texture map to obtain the first reconstructed model of the first object may specifically include:

[0069] Based on the texture map, the texture mapping of the 3D human body reconstruction model is performed to obtain the first candidate reconstruction model of the first object;

[0070] Get the color value corresponding to each pair of adjacent texture regions;

[0071] Based on the color values ​​corresponding to each of the two adjacent texture regions, the critical region between the two adjacent texture regions is adjusted to obtain the first reconstruction model of the first object.

[0072] The first candidate reconstruction model can be a three-dimensional reconstruction model of the first object obtained by texture mapping of the three-dimensional human body reconstruction model based on the texture map. The first candidate reconstruction model can include at least two texture regions.

[0073] The critical region can be the seam area between two adjacent texture regions.

[0074] In some embodiments of this application, after texture mapping is performed on the 3D human body reconstruction model based on the texture map, the resulting reconstruction model has multiple texture regions. The color values ​​corresponding to each of the two adjacent texture regions can be obtained. Then, based on the color values ​​corresponding to each of the two adjacent texture regions, the critical region of the two adjacent texture regions is adjusted to obtain the first reconstruction model of the first object. Specifically, the color threshold of the color values ​​of the two adjacent texture regions can be adjusted using the Poisson algorithm to achieve the fusion effect. This ensures that there is no large color difference between the critical region of the two adjacent texture regions and the two adjacent texture regions.

[0075] In the embodiments of this application, a first candidate reconstruction model of the first object is obtained by performing texture mapping on the three-dimensional human body reconstruction model based on the texture map. Then, based on the color values ​​corresponding to each of the two adjacent texture regions in the first candidate reconstruction model, the critical region of the two adjacent texture regions is adjusted. This ensures that there will be no large color difference between adjacent texture regions in the first reconstruction model of the first object, thereby improving the quality of the first reconstruction model.

[0076] In some embodiments of this application, in order to improve the efficiency of acquiring N frames of foreground images, the method described above may further include the following before S110:

[0077] Obtain N frames of background images of the scene where the first object is located;

[0078] The step of extracting the first object from N frames of the first image to obtain N frames of foreground images may specifically include:

[0079] Based on the background image, the first object is extracted from the first image of N frames to obtain the foreground image of N frames.

[0080] In some embodiments of this application, before the camera captures the movement of the first object, it can first capture the scene to obtain N frames of images of the scene. These N frames are used as background images. Then, based on the background images and the relationship between background images and foreground images in the prior art, the first object can be extracted from the N frames of the first images according to the image pixels of the N frames of the first images to obtain N frames of foreground images.

[0081] In the embodiments of this application, N frames of background images of the shooting scene where the first object is located are obtained in advance, and then the first object is extracted from the N frames of first images based on the background images to obtain N frames of foreground images. In this way, the background images corresponding to the N frames of first images are known. Therefore, when extracting the first object from the N frames of first images according to the relationship between the background image and the foreground image, the calculation process can be simplified. Compared with the prior art where the background image is unknown, the solution of the embodiments of this application improves the efficiency of obtaining the foreground image.

[0082] S130. Using the first template, perform three-dimensional reconstruction on the first object in the second image of P frame to obtain the second reconstruction model of the first object.

[0083] The first template can be a pre-set template based on human body reconstruction, and the first template can be an existing human body reconstruction substrate.

[0084] The second reconstruction model can be obtained by performing three-dimensional reconstruction of the first object in the second image of P-frame using the first template, resulting in a three-dimensional reconstruction model of the first object.

[0085] It should be noted that the input to the first template is a single-frame image. Therefore, when processing the P-frame second images, a 3D reconstruction of the first object in each frame of the second image is performed. Then, based on the 3D reconstruction model of the first object corresponding to each frame of the second image, a second reconstruction model of the first object is obtained. In the second reconstruction model of the first object reconstructed based on the preset reconstruction model, the details of the first object are complete and smooth, but the details are not realistic and accurate enough.

[0086] In some embodiments of this application, in order to accurately obtain the second reconstruction model, S130 may specifically include:

[0087] Using the first template, feature points of the first object are extracted from each frame of the second image, and the feature points of the first object in each frame of the second image are fitted to obtain the pose coefficients of the first object in each frame of the second image.

[0088] Based on the pose coefficients of the first object in each frame of the second image, the first object in the second image is reconstructed in three dimensions using the first template to obtain P reconstructed images of the first object in the second images.

[0089] Using the first template, the three-dimensional reconstructed images of the first object in P second images are fused to obtain the second reconstructed model of the first object.

[0090] The pose coefficient can be a parameter used to characterize the pose of the first object in the second image.

[0091] A 3D reconstructed image can be an image obtained by performing 3D reconstruction on a first object in a second image.

[0092] In some embodiments of this application, the input to the first template is a single-frame image. Therefore, P frames of second images can be sequentially input into the first template one by one. For each frame of second image, the first template can extract feature points of the first object from the frame of second image. For example, it can be feature points corresponding to each joint of the first object. Then, it fits each feature point in the frame of second image to obtain the pose coefficient of the first object in the frame of second image. Then, it uses the pose coefficient to perform three-dimensional reconstruction of the frame of second image to obtain the three-dimensional reconstructed image of the first object in the frame of second image. Then, it fuses the three-dimensional reconstructed images corresponding to each frame of second image to obtain the second reconstruction model of the first object.

[0093] In the embodiments of this application, feature points of the first object are extracted from each frame of the second image using a first template, and the feature points of the first object in each frame of the second image are fitted to obtain the pose coefficients of the first object in each frame of the second image. Based on the pose coefficients of the first object in each frame of the second image, the first object in the second image is reconstructed in three dimensions using the first template to obtain three-dimensional reconstructed images of the first object in P second images. Then, the three-dimensional reconstructed images of the first object in the P second images are fused using the first template to accurately obtain the second reconstruction model of the first object.

[0094] In some embodiments of this application, in order to further improve the accuracy of the third reconstruction model of the first object, before performing three-dimensional reconstruction of the first object in each of the second images based on the pose coefficients of the first object in each frame of the second image using the first template to obtain three-dimensional reconstructed images of the first object in P second images, the method described above may further include:

[0095] Acquire the third image of the first object in M ​​frames during its motion;

[0096] Determine at least one motion phase in the motion process of the first object;

[0097] Based on the feature points of the first object in the third Q-frame image during the target motion phase, the shape parameters of the first object during the target motion phase are determined.

[0098] The step of performing 3D reconstruction of the first object based on the pose coefficients of the first object in each frame of the second image, using a first template, yields P 3D reconstructed images of the first object, including:

[0099] Based on the pose coefficients of the first object in each frame of the second image during the target motion phase, and the shape parameters within the target motion phase, the first object in each frame of the second image during the target motion phase is reconstructed in three dimensions to obtain the three-dimensional reconstructed image of the first object in each frame of the second image during the target motion phase.

[0100] The third image can be any image captured by all cameras during the entire motion of the first object, that is, images of the first object captured by 100 cameras in the shooting scene during the entire motion.

[0101] The target motion phase can be any one of at least one motion phase, Q is a positive integer, and N < M.

[0102] The shape parameter can be a parameter used to characterize the shape of the first object's motion. This shape parameter can be used to characterize the first object's height, weight, clothing type, etc.

[0103] In some embodiments of this application, the movement process of the first object can be divided into at least one movement stage according to the sequence of the movement process of the first object. For example, if the first object performs action A first, and then action B and action C during the 5-minute movement process, the entire movement process of the first object can be divided into at least one movement stage according to the sequence of action A, action B and action C. For example, action A can correspond to one movement stage, action B can correspond to one movement stage, and action C can also correspond to one movement stage.

[0104] Then, based on the feature points of the first object in the third Q-frame image within the target motion phase, the shape parameters of the first object within the target motion phase can be determined. Then, based on the pose coefficients corresponding to each frame image of the target motion phase and the shape parameters within the target motion phase, the first object in each frame image of the target motion phase is reconstructed in three dimensions to obtain the three-dimensional reconstructed image of the first object in each frame image of the target motion phase.

[0105] It should be noted that, since the second image of P-frame is the image captured by the camera that takes a frontal view of the first object out of 100 cameras, the third image of M-frame is the image captured by the camera that takes a view of the first object out of 100 cameras, and the third image of Q-frame during the target motion phase is the image captured by the camera that takes a view of the target motion phase out of 100 cameras, the third image of Q-frame contains the second image of P-frame during the target motion phase.

[0106] By processing the second image of each motion stage in the same way as the second image of the target motion stage, a three-dimensional reconstructed image of the first object in each frame of the P-frame second image can be obtained.

[0107] It should be noted that the shape parameters obtained during the aforementioned target motion phase are also implemented using the first algorithm. However, the images input into the first algorithm at this time are a large number of multi-frame images, namely, images taken by all cameras in the shooting scene, i.e., M-frame third images. In the first reconstruction model of the first object obtained in S120 above, the first algorithm inputs images that are fewer than several frames, i.e., images taken by some cameras in the shooting scene, i.e., P-frame second images.

[0108] In the embodiments of this application, by acquiring M frames of third images of the first object during its motion, and then determining at least one motion stage in the motion process of the first object, the shape parameters of the first object in the Q frames of the third image within the target motion stage can be determined based on the feature points of the first object in the target motion stage. Then, based on the pose coefficients of the first object in each frame of the second image within the target motion stage, and the shape parameters within the target motion stage, the first object in each frame of the second image within the target motion stage is reconstructed in three dimensions. Thus, when reconstructing the first object in each frame of the second image in three dimensions, in addition to referencing the pose parameters, the shape parameters are also referenced in each motion stage, which can further improve the reconstruction accuracy of the first object in the second image, thereby obtaining an accurate second reconstruction model.

[0109] In some embodiments of this application, in order to improve the efficiency of determining the shape parameters of the first object during the target motion phase, the determination of the shape parameters of the first object during the target motion phase based on the feature points of the first object in the Q-frame third image during the target motion phase may specifically include:

[0110] The feature points of the first object in the first target image during the target motion phase are fitted to obtain the shape parameters of the first object in the first target image.

[0111] The shape parameters of the first object in the first target image are determined as the shape parameters of the first object during the target motion phase.

[0112] The first target image can be any frame in the third image of the Q-frame.

[0113] In some embodiments of this application, since the motion shape of the first object does not change much within a motion phase, a third frame can be arbitrarily selected from the Q-frame third images within the target motion phase. Then, the feature points of the first object in the third frame are fitted to obtain the shape parameters of the first object in the third frame. The shape parameters of the first object in the third frame are then determined as the shape parameters of the first object in the Q-frame third images within the target motion phase. In other words, the shape parameters corresponding to a certain frame of the third image within the target motion phase are determined as the shape parameters corresponding to all images within the target motion phase.

[0114] In the embodiments of this application, the shape parameters of the first object in the first target image within the target motion phase are obtained by fitting the feature points of the first object in the first target image. Then, the shape parameters of the first object in the first target image are determined as the shape parameters of the first object within the target motion phase. In this way, the shape parameters corresponding to a certain frame of the third image within the target motion phase are determined as the shape parameters corresponding to all images within the target motion phase. It is not necessary to calculate the shape parameters corresponding to all images within this target motion phase, which saves computing power and improves the efficiency of shape parameter determination.

[0115] In some embodiments of this application, in order to accurately obtain a three-dimensional reconstructed image of the first object in the second image, the step of performing three-dimensional reconstruction of the first object in the second image during the target motion phase based on the pose coefficients of the first object in the second image during the target motion phase and the shape parameters within the target motion phase, to obtain a three-dimensional reconstructed image of the first object in the second image during the target motion phase, may specifically include:

[0116] Based on the pose coefficients of the first object in the second image during the target motion phase, the pose of the first object in the second image during the target motion phase is reconstructed to obtain the pose reconstruction image of the first object in the second image during the target motion phase.

[0117] Based on the shape parameters within the target motion phase, the motion shape of the pose reconstruction image is adjusted to obtain a three-dimensional reconstruction image of the first object in the second image during the target motion phase.

[0118] The pose reconstruction image can be a reconstructed image obtained by reconstructing the pose of the first object in the second image during the target motion phase, based on the pose coefficients of the first object in the second image during the target motion phase.

[0119] In some embodiments of this application, the posture of the first object in the second image during the target motion phase can be reconstructed based on the posture coefficient of the first object in the second image during the target motion phase to obtain a posture reconstruction image of the first object in the second image during the target motion phase. Then, based on the shape parameters within the target motion phase, the motion shape of the posture reconstruction image is adjusted to obtain a three-dimensional reconstruction image of the first object in the second image during the target motion phase.

[0120] In the embodiments of this application, the posture of the first object in the second image during the target motion phase is reconstructed based on the posture coefficients of the first object in the second image during the target motion phase, thereby obtaining a posture reconstruction image of the first object in the second image during the target motion phase. Then, the motion shape of the posture reconstruction image is adjusted based on the shape parameters within the target motion phase, thus accurately obtaining a three-dimensional reconstruction image of the first object in the second image during the target motion phase.

[0121] In some embodiments of this application, in order to accurately acquire N frames of first images and P frames of second images of the first object during its motion, the method described above may further include the following before S110:

[0122] Acquire multiple frames of the fourth image from K cameras captured from different perspectives;

[0123] Based on multiple frames of the fourth image, determine the first parameter of each camera and the second parameter between every two cameras;

[0124] The acquisition of the M-frame third image of the first object during its motion includes:

[0125] Acquire M frames of the third image of the first object during its motion, captured by K cameras, wherein the parameters of the K cameras are set to the first parameter and the second parameter, respectively.

[0126] Among them, K cameras can be 100 cameras in the above shooting scenario.

[0127] The preset image can be a calibration image used to calibrate K cameras, such as a checkerboard image.

[0128] The fourth image can be a preset image captured by K cameras before the acquisition of the third image in M ​​frames.

[0129] The first parameter can be the intrinsic parameter of each camera, and the second parameter can be the extrinsic parameter between any two cameras.

[0130] In some embodiments of this application, before using each camera, it is necessary to calibrate each camera. Specifically, this can be done by using each camera to capture multiple frames of a preset image, and then using the Zhang Zhengyou calibration method to calibrate based on the multiple frames of the image to obtain the intrinsic parameters of each camera and the extrinsic parameters between each pair of cameras. Then, the calibrated cameras are used to capture M frames of the third image of the first object during its motion.

[0131] In some embodiments of this application, other methods can be used to calibrate the camera. For example, the intrinsic parameters of the camera and the extrinsic parameters between each pair of cameras can be obtained by inversely solving the problem based on the imaging formula, the constraint formula of polar geometry, and the working principle of polar geometry, using the matching points in the multi-frame images of the captured scene. The specific method of inversely solving the problem based on the imaging formula, the constraint formula of polar geometry, and the working principle of polar geometry using the matching points in the multi-frame images of the captured scene is prior art and will not be elaborated here.

[0132] It should be noted that the above only shows two ways to calibrate the intrinsic and extrinsic parameters of the camera, but it is not limited to the above two calibration methods. Any method that can calibrate the intrinsic and extrinsic parameters of the camera is within the protection scope of the embodiments of this application.

[0133] In the embodiments of this application, by acquiring multiple frames of fourth images of preset images captured by K cameras at different perspectives, and then determining the first parameter of each camera and the second parameter between every two cameras based on the multiple frames of fourth images, M frames of third images of the first object during its motion can be acquired based on the calibrated K cameras, thus obtaining accurate M frames of third images, and thereby accurately obtaining the third reconstruction model of the first object.

[0134] S140. Based on the second reconstruction model, the first object in the first reconstruction model is adjusted to obtain the third reconstruction model of the first object.

[0135] The third reconstruction model can be the final three-dimensional reconstruction model of the obtained first object.

[0136] In some embodiments of this application, since the details of the first object are complete and smooth in the second reconstruction model of the first object reconstructed based on the first template, but the details are not realistic and accurate enough, the first reconstruction model of the first object reconstructed using the first algorithm will have some empty areas and some outliers, but the outline is clear. Thus, the first object in the first reconstruction model can be adjusted based on the second reconstruction model, and then the accurate third reconstruction model of the first object can be obtained.

[0137] In some embodiments of this application, in order to accurately obtain the third reconstruction model of the first object, S140 may specifically include:

[0138] The hole region of the first object in the first reconstruction model is filled based on the second reconstruction model to obtain the first reconstruction model after filling.

[0139] Align the pixels of the first and second reconstructed models after filling in the gaps.

[0140] By removing misaligned pixels from the first and second reconstruction models after filling, the first target reconstruction model and the second target reconstruction model are obtained respectively.

[0141] The first target reconstruction model and the second target reconstruction model are fused to obtain the third reconstruction model of the first object.

[0142] The first target reconstruction model can be the reconstruction model obtained by removing pixels in the first reconstruction model that are not aligned with the second reconstruction model.

[0143] The second target reconstruction model can be the reconstruction model obtained by removing pixels in the second reconstruction model that are not aligned with the first reconstruction model after filling.

[0144] In some embodiments of this application, the hole area of ​​the first object in the first reconstruction model can be filled by the second reconstruction model to obtain the filled first reconstruction model. Then, the filled first reconstruction model and the second reconstruction model can be pixel-aligned, and the misaligned pixels in the filled first reconstruction model and the second reconstruction model can be removed to obtain the first target reconstruction model and the second target reconstruction model respectively. Then, the first target reconstruction model and the second target reconstruction model can be fused to obtain the third reconstruction model of the first object.

[0145] In the embodiments of this application, the hole area of ​​the first object in the first reconstruction model is filled by the second reconstruction model to obtain the filled first reconstruction model. Then, the pixels of the filled first reconstruction model and the second reconstruction model are aligned, and the misaligned pixels in the filled first reconstruction model and the second reconstruction model are removed to obtain the first target reconstruction model and the second target reconstruction model respectively. Then, the first target reconstruction model and the second target reconstruction model are fused together to obtain the third reconstruction model of the first object accurately.

[0146] In some embodiments of this application, in order to provide a clear understanding of the technical solutions of this application, embodiments of this application also provide a three-dimensional model reconstruction method. Figure 2 This is a flowchart illustrating the 3D model reconstruction method, such as... Figure 2As shown, the 3D model reconstruction method can specifically include a real-time 3D reconstruction process and a high-precision 3D reconstruction process.

[0147] In the real-time 3D reconstruction process, a small number of multi-frame images are used, namely, N frames of the first object captured by 10 of the 100 cameras in the above embodiment, to perform 3D reconstruction of the first object based on the first algorithm, thereby obtaining the first reconstructed model of the first object. The specific implementation process can be referred to in step S120 of the above embodiment.

[0148] During the real-time 3D reconstruction process, while constructing the first reconstruction model, a single-frame image is simultaneously input into the first template. This is the second P-frame image of the first object captured by one of the 10 cameras selected from 100 cameras in the above embodiment. The first object is then reconstructed in 3D to obtain the second reconstruction model of the first object. The specific implementation process can be found in step S130 of the above embodiment.

[0149] In the real-time 3D reconstruction process, the hollow areas of the first object in the first reconstruction model are filled based on the second reconstruction model, and outliers are removed, thereby obtaining the third reconstruction model of the first object. The specific implementation process is the same as in the above embodiment, which involves filling the hollow areas of the first object in the first reconstruction model based on the second reconstruction model to obtain the filled first reconstruction model; aligning the pixels of the filled first reconstruction model and the second reconstruction model; removing misaligned pixels in the filled first reconstruction model and the second reconstruction model to obtain the first target reconstruction model and the second target reconstruction model respectively; and fusing the first target reconstruction model and the second target reconstruction model to obtain the third reconstruction model of the first object, which will not be repeated here.

[0150] In the high-precision 3D reconstruction process, a large number of multi-frame images are used, namely the M-frame third images of the first object captured by 100 cameras in the above embodiment. Based on the first algorithm, the running shape parameters of the first object in each motion stage are obtained. Then, based on the shape parameters, the process of 3D reconstruction of the first object using the first template in real-time 3D reconstruction is updated, thereby obtaining an accurate second reconstruction model.

[0151] It should be noted that in real-time 3D reconstruction, the construction of the first reconstruction model requires multiple frames of images, demanding high computational power. Therefore, this process needs to be executed on a high-performance computing platform such as a server or cloud. In contrast, the construction of the second reconstruction model utilizes a single frame of image, requiring less computational power, and can therefore be performed on a mobile device, such as a smartphone. Similarly, in high-precision 3D reconstruction, the acquisition of shape parameters also requires multiple frames of images, demanding high computational power. Therefore, high-precision 3D reconstruction also needs to be implemented on a high-performance computing platform such as a server or cloud.

[0152] The three-dimensional model reconstruction method provided in this application can be executed by a three-dimensional model reconstruction device. This application uses a three-dimensional model reconstruction device executing the three-dimensional model reconstruction method as an example to illustrate the three-dimensional model reconstruction device provided in this application.

[0153] Figure 3 This is a schematic diagram of the structure of a three-dimensional model reconstruction device according to an exemplary embodiment.

[0154] like Figure 3 As shown, the three-dimensional model reconstruction device 300 may include:

[0155] The first acquisition module 310 is used to acquire N frames of first images and P frames of second images of the first object during its motion process, wherein the N frames of first images include the P frames of second images, and N and P are both positive integers;

[0156] The first reconstruction module 320 is used to perform three-dimensional reconstruction of the first object in the N frames of the first image using a first algorithm to obtain a first reconstruction model of the first object.

[0157] The second reconstruction module 330 is also used to perform three-dimensional reconstruction of the first object in the second image of the P-frame using the first template, so as to obtain a second reconstruction model of the first object.

[0158] The adjustment module 340 is used to adjust the first object in the first reconstruction model based on the second reconstruction model to obtain a third reconstruction model of the first object.

[0159] In this embodiment, a first algorithm is used to perform 3D reconstruction on the first object in N frames of the first image during the motion process to obtain a first reconstruction model of the first object. Then, a first template is used to perform 3D reconstruction on the first object in P frames of the second image during the motion process to obtain a second reconstruction model of the first object. In this embodiment, the first reconstruction model and the second reconstruction model are obtained by performing 3D reconstruction once using the first algorithm and the first template respectively. Then, the first object in the first reconstruction model is adjusted based on the second reconstruction model to obtain a third reconstruction model of the first object, thereby improving the accuracy and timeliness of 3D reconstruction.

[0160] In some embodiments of this application, the second reconstruction module 330 may specifically include:

[0161] The first extraction unit is used to extract feature points of the first object from each frame of the second image using the first template, and to fit the feature points of the first object in each frame of the second image to obtain the pose coefficient of the first object in each frame of the second image.

[0162] The reconstruction unit is used to perform three-dimensional reconstruction of the first object in each frame of the second image based on the pose coefficient of the first object in each frame of the second image, using the first template, to obtain P three-dimensional reconstructed images of the first object in the second image;

[0163] The first fusion unit is used to fuse the three-dimensional reconstructed images of the first object in P second images using a first template to obtain a second reconstructed model of the first object.

[0164] In some embodiments of this application, the apparatus described above may further include:

[0165] The second acquisition module is used to acquire the third image of the first object in M ​​frames during the motion process;

[0166] A segmentation module is used to determine at least one motion stage in the motion process of the first object;

[0167] The first determining module is used to determine the shape parameters of the first object in the target motion phase based on the feature points of the first object in the third Q-frame image of the target motion phase, wherein the target motion phase is any one of the at least one motion phase, Q is a positive integer, and N < M.

[0168] The reconstruction unit is specifically used for:

[0169] Based on the pose coefficients of the first object in each frame of the second image during the target motion phase, and the shape parameters within the target motion phase, a three-dimensional reconstruction of the first object in each frame of the second image during the target motion phase is performed to obtain a three-dimensional reconstructed image of the first object in each frame of the second image during the target motion phase.

[0170] In some embodiments of this application, the first determining module is specifically used for:

[0171] The feature points of the first object in the first target image during the target motion phase are fitted to obtain the shape parameters of the first object in the first target image, wherein the first target image is any frame image in the third image of the Q frame;

[0172] The shape parameters of the first object in the first target image are determined as the shape parameters of the first object during the target motion phase.

[0173] In some embodiments of this application, the first reconstruction module 320 may specifically include:

[0174] The second extraction unit is used to extract the first object from the N frames of the first image to obtain N frames of foreground images;

[0175] The third extraction unit is used to extract feature points of the first object from each frame of foreground image;

[0176] The second fusion unit is used to fuse the feature points of the first object in the foreground image at the target time to obtain the three-dimensional point cloud data of the first object at the target time, wherein the target time is any time during the movement of the first object;

[0177] A surface rendering unit is used to render the three-dimensional point cloud data at each moment during the movement of the first object, so as to obtain a three-dimensional human body reconstruction model of the first object.

[0178] The texture mapping unit is used to perform texture mapping on the three-dimensional human body reconstruction model to obtain the first reconstruction model of the first object.

[0179] The 3D model reconstruction device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0180] The three-dimensional model reconstruction device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0181] The three-dimensional model reconstruction device provided in this application embodiment can achieve... Figure 1 and Figure 2 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0182] Optionally, such as Figure 4 As shown, this application embodiment also provides an electronic device 400, including a processor 401 and a memory 402. The memory 402 stores a program or instructions that can run on the processor 401. When the program or instructions are executed by the processor 401, they implement the various steps of the above-described three-dimensional model reconstruction method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0183] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0184] Figure 5 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0185] The electronic device 500 includes, but is not limited to, components such as: radio frequency unit 501, network module 502, audio output unit 503, input unit 504, sensor 505, display unit 506, user input unit 507, interface unit 508, memory 509, and processor 510.

[0186] Those skilled in the art will understand that the electronic device 500 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0187] The radio frequency unit 501 is used to acquire N frames of first images and P frames of second images of the first object during its motion process, wherein the N frames of first images include the P frames of second images, and N and P are both positive integers.

[0188] The processor 510 is configured to perform three-dimensional reconstruction of a first object in the N-frame first image using a first algorithm to obtain a first reconstruction model of the first object; perform three-dimensional reconstruction of the first object in the P-frame second image using a first template to obtain a second reconstruction model of the first object; and adjust the first object in the first reconstruction model based on the second reconstruction model to obtain a third reconstruction model of the first object.

[0189] Thus, by using the first algorithm to perform three-dimensional reconstruction on the first object in N frames of the first image during the motion process, a first reconstruction model of the first object is obtained. Then, using the first template, three-dimensional reconstruction is performed on the first object in P frames of the second image during the motion process, a second reconstruction model of the first object is obtained. In this embodiment, the first reconstruction model and the second reconstruction model are obtained by performing three-dimensional reconstruction once using the first algorithm and the first template, and then the first object in the first reconstruction model is adjusted based on the second reconstruction model to obtain a third reconstruction model of the first object, thereby improving the accuracy and timeliness of three-dimensional reconstruction.

[0190] Optionally, the processor 510 is further configured to: extract feature points of the first object from each frame of the second image using the first template; fit the feature points of the first object in each frame of the second image to obtain the pose coefficients of the first object in each frame of the second image; perform three-dimensional reconstruction of the first object in each frame of the second image based on the pose coefficients of the first object in each frame of the second image using the first template to obtain P three-dimensional reconstructed images of the first object in the second images; and fuse the P three-dimensional reconstructed images of the first object in the second images using the first template to obtain a second reconstruction model of the first object.

[0191] Thus, by using the reconstruction model, feature points of the first object are extracted from each frame of the second image, and the feature points of the first object in each frame of the second image are fitted to obtain the pose coefficients of the first object in each frame of the second image. Then, based on the pose coefficients of the first object in each frame of the second image, the first object in the second image is reconstructed in three dimensions using the first template to obtain P three-dimensional reconstructed images of the first object in the second images. By using the first template, the three-dimensional reconstructed images of the first object in the P second images are fused together to accurately obtain the second reconstruction model of the first object.

[0192] Optionally, the radio frequency unit 501 is further configured to acquire M-frame third images of the first object during its motion;

[0193] The processor 510 is further configured to determine at least one motion phase in the motion process of the first object; determine the shape parameters of the first object in the target motion phase based on feature points of the first object in a Q-frame third image within the target motion phase, wherein the target motion phase is any one of the at least one motion phase, Q is a positive integer, and N < M; and perform three-dimensional reconstruction of the first object in each frame of the second image within the target motion phase based on the pose coefficients of the first object in each frame of the second image within the target motion phase and the shape parameters within the target motion phase, to obtain a three-dimensional reconstructed image of the first object in each frame of the second image within the target motion phase.

[0194] Thus, by acquiring M frames of third images of the first object during its motion, and then determining at least one motion stage in the motion of the first object, the shape parameters of the first object in the Q frames of the third image within the target motion stage can be determined based on the feature points of the first object in the target motion stage. Then, based on the pose coefficients of the first object in each frame of the second image within the target motion stage, and the shape parameters within the target motion stage, the first object in each frame of the second image within the target motion stage is reconstructed in three dimensions. In this way, when reconstructing the first object in each frame of the second image, in addition to referencing the pose parameters, the shape parameters are also referenced in each motion stage, which can further improve the reconstruction accuracy of the first object in the second image, thereby obtaining an accurate second reconstruction model.

[0195] Optionally, the processor 510 is further configured to fit feature points of the first object in the first target image during the target motion phase to obtain shape parameters of the first object in the first target image, wherein the first target image is any frame image in the Q-frame of the third image; and to determine the shape parameters of the first object in the first target image as the shape parameters of the first object during the target motion phase.

[0196] Thus, by fitting the feature points of the first object in the first target image within the target motion phase, the shape parameters of the first object in the first target image are obtained. Then, the shape parameters of the first object in the first target image are determined as the shape parameters of the first object within the target motion phase. In this way, the shape parameters corresponding to a certain frame of the third image within the target motion phase are determined as the shape parameters corresponding to all images within the target motion phase. It is not necessary to calculate the shape parameters corresponding to all images within this target motion phase, which saves computing power and improves the efficiency of shape parameter determination.

[0197] Optionally, the processor 510 is further configured to extract the first object from the N frames of the first image to obtain N frames of foreground image; extract feature points of the first object from each frame of the foreground image; fuse the feature points of the first object in the foreground image at the target time to obtain three-dimensional point cloud data of the first object at the target time, wherein the target time is any time during the motion of the first object; perform surface rendering on the three-dimensional point cloud data at each time during the motion of the first object to obtain a three-dimensional human body reconstruction model of the first object; and perform texture mapping on the three-dimensional human body reconstruction model to obtain a first reconstruction model of the first object.

[0198] Thus, by extracting the first object from N frames of first images, N frames of foreground images are obtained. Then, feature points of the first object are extracted from each frame of foreground image. The feature points of the first object in the foreground image at the target time are fused to obtain the three-dimensional point cloud data of the first object at the target time. The three-dimensional point cloud data at each time during the movement of the first object are drawn to obtain the three-dimensional human body reconstruction model of the first object. Then, texture mapping is performed on the three-dimensional human body reconstruction model to obtain the accurate first reconstruction model of the first object.

[0199] It should be understood that, in this embodiment, the input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image data of still images or videos obtained by an image capture device (such as a color camera) in video capture mode or image capture mode. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 507 includes at least one of a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0200] The memory 509 can be used to store software programs and various data. The memory 509 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 509 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 509 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0201] Processor 510 may include one or more processing units; optionally, processor 510 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 510.

[0202] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described three-dimensional model reconstruction method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0203] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0204] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described three-dimensional model reconstruction method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0205] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0206] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the three-dimensional model reconstruction method embodiment described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0207] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0208] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0209] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for reconstructing a three-dimensional model, characterized in that, The method includes: Acquire N frames of first images and P frames of second images of the first object during its motion, wherein the N frames of first images include the P frames of second images, and N and P are both positive integers; The first object in the N frames of the first image is reconstructed in three dimensions using the first algorithm to obtain the first reconstructed model of the first object; Using the first template, the first object in the second image of the P-frame is reconstructed in three dimensions to obtain a second reconstructed model of the first object; The first object in the first reconstruction model is adjusted based on the second reconstruction model to obtain a third reconstruction model of the first object; The step of using a first algorithm to perform 3D reconstruction of a first object in the N frames of the first image to obtain a first reconstructed model of the first object includes: Extract the first object from the N frames of the first image to obtain N frames of foreground images; Extract feature points of the first object from each frame of the foreground image; The feature points of the first object in the foreground image at the target time are fused to obtain the three-dimensional point cloud data of the first object at the target time, wherein the target time is any time during the movement of the first object; The three-dimensional point cloud data at each moment during the movement of the first object are plotted to obtain a three-dimensional human body reconstruction model of the first object. Texture mapping is performed on the three-dimensional human body reconstruction model to obtain the first reconstruction model of the first object.

2. The method according to claim 1, characterized in that, The step of using a first template to perform three-dimensional reconstruction on the first object in the second image of the P-frame to obtain a second reconstruction model of the first object includes: Using the first template, feature points of the first object are extracted from each frame of the second image, and the feature points of the first object in each frame of the second image are fitted to obtain the pose coefficient of the first object in each frame of the second image. Based on the pose coefficients of the first object in each frame of the second image, the first object in the second image is reconstructed in three dimensions using the first template to obtain P three-dimensional reconstructed images of the first object in the second image. Using the first template, the three-dimensional reconstructed images of the first object in P second images are fused to obtain a second reconstructed model of the first object.

3. The method according to claim 2, characterized in that, Before performing 3D reconstruction of the first object in each of the second images based on the pose coefficients of the first object in each frame of the second image using the first template to obtain P 3D reconstructed images of the first object in the second images, the method further includes: Acquire the M-frame third image of the first object during its motion; Determine at least one motion phase in the motion process of the first object; Based on the feature points of the first object in the third Q-frame image within the target motion phase, the shape parameters of the first object within the target motion phase are determined, wherein the target motion phase is any one of the at least one motion phase, Q is a positive integer, and N < M. The step of performing 3D reconstruction on the first object in each frame of the second image based on the pose coefficients of the first object and using the first template to obtain P 3D reconstructed images of the first object in the second images includes: Based on the pose coefficients of the first object in each frame of the second image during the target motion phase, and the shape parameters within the target motion phase, a three-dimensional reconstruction of the first object in each frame of the second image during the target motion phase is performed to obtain a three-dimensional reconstructed image of the first object in each frame of the second image during the target motion phase.

4. The method according to claim 3, characterized in that, The determination of the shape parameters of the first object within the target motion phase based on feature points of the first object in the third Q-frame image within the target motion phase includes: The feature points of the first object in the first target image during the target motion phase are fitted to obtain the shape parameters of the first object in the first target image, wherein the first target image is any frame image in the third Q-frame image; The shape parameters of the first object in the first target image are determined as the shape parameters of the first object during the target motion phase.

5. A three-dimensional model reconstruction device, characterized in that, The device includes: The first acquisition module is used to acquire N frames of first images and P frames of second images of the first object during its motion process, wherein the N frames of first images include the P frames of second images, and N and P are both positive integers; The first reconstruction module is used to perform three-dimensional reconstruction of the first object in the N frames of the first image using a first algorithm to obtain a first reconstruction model of the first object. The second reconstruction module is also used to perform three-dimensional reconstruction of the first object in the second image of the P-frame using the first template, so as to obtain a second reconstruction model of the first object. The adjustment module is used to adjust the first object in the first reconstruction model based on the second reconstruction model to obtain a third reconstruction model of the first object; The first reconstruction module includes: The second extraction unit is used to extract the first object from the N frames of the first image to obtain N frames of foreground images; The third extraction unit is used to extract feature points of the first object from each frame of foreground image; The second fusion unit is used to fuse the feature points of the first object in the foreground image at the target time to obtain the three-dimensional point cloud data of the first object at the target time, wherein the target time is any time during the movement of the first object; A surface rendering unit is used to render the three-dimensional point cloud data at each moment during the movement of the first object, so as to obtain a three-dimensional human body reconstruction model of the first object. The texture mapping unit is used to perform texture mapping on the three-dimensional human body reconstruction model to obtain the first reconstruction model of the first object.

6. The apparatus according to claim 5, characterized in that, The second reconstruction module includes: The first extraction unit is used to extract feature points of the first object from each frame of the second image using the first template, and to fit the feature points of the first object in each frame of the second image to obtain the pose coefficient of the first object in each frame of the second image. The reconstruction unit is used to perform three-dimensional reconstruction of the first object in each frame of the second image based on the pose coefficient of the first object in each frame of the second image, using the first template, to obtain P three-dimensional reconstructed images of the first object in the second image; The first fusion unit is used to fuse the three-dimensional reconstructed images of the first object in P second images using a first template to obtain a second reconstructed model of the first object.

7. The apparatus according to claim 6, characterized in that, The device further includes: The second acquisition module is used to acquire the third image of the first object in M ​​frames during the motion process; A segmentation module is used to determine at least one motion stage in the motion process of the first object; The first determining module is used to determine the shape parameters of the first object in the target motion phase based on the feature points of the first object in the third Q-frame image of the target motion phase, wherein the target motion phase is any one of the at least one motion phase, Q is a positive integer, and N < M. The reconstruction unit is specifically used for: Based on the pose coefficients of the first object in each frame of the second image during the target motion phase, and the shape parameters within the target motion phase, a three-dimensional reconstruction of the first object in each frame of the second image during the target motion phase is performed to obtain a three-dimensional reconstructed image of the first object in each frame of the second image during the target motion phase.

8. The apparatus according to claim 7, characterized in that, The first determining module is specifically used for: The feature points of the first object in the first target image during the target motion phase are fitted to obtain the shape parameters of the first object in the first target image, wherein the first target image is any frame image in the third Q-frame image; The shape parameters of the first object in the first target image are determined as the shape parameters of the first object during the target motion phase.

Images