Three-dimensional reconstruction method, device and system, medium and computer equipment
Through the three-dimensional reconstruction network and supervision information optimization method, combined with two-dimensional and three-dimensional information, the problem of insufficient accuracy and reliability in traditional three-dimensional reconstruction methods is solved, and a natural and reasonable three-dimensional reconstruction result is achieved.
Patent Information
- Application Number
- CN202110506464.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-05-10
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-05-10
AI Technical Summary
Traditional three-dimensional reconstruction methods cannot take into account the accuracy and reliability of reconstruction results, and are especially deficient in generating natural and reasonable action results.
Through the three-dimensional reconstruction network, the initial parameters are obtained, and the supervision information is optimized based on the pre-acquisitioned supervision information, and multi-stage optimization is carried out in combination with two-dimensional key points and three-dimensional point cloud information to improve the accuracy and stability of the parameters .
On the basis of ensuring the reliability of the three-dimensional reconstruction results, it takes into account the accuracy of the three-dimensional reconstruction and natural and reasonable action results, and improves the stability and accuracy of the optimization process.
Smart Images

Figure CN113160418B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer vision technology, and in particular to three-dimensional reconstruction methods, devices and systems, media and computer equipment. Background Art
[0002] 3D reconstruction is one of the important problems in computer vision, and has many potential applications in the fields of augmented reality and virtual reality. By reconstructing the target object in 3D, the body posture and limb rotation of the target object can be reconstructed. However, traditional 3D reconstruction methods cannot take into account the accuracy and reliability of the reconstruction results. Summary of the invention
[0003] The present disclosure provides a three-dimensional reconstruction method, device and system, medium and computer equipment.
[0004] According to a first aspect of an embodiment of the present disclosure, a three-dimensional reconstruction method is provided, the method comprising: performing three-dimensional reconstruction of an image of a target object through a three-dimensional reconstruction network to obtain initial parameters of the target object, the initial parameters being used to establish a three-dimensional model of the target object; optimizing the initial parameters based on pre-acquired supervision information for representing characteristics of the target object to obtain optimized parameters; performing skeleton skinning processing based on the optimized parameters to establish a three-dimensional model of the target object.
[0005] In some embodiments, the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; the first supervision information includes at least one of the following: initial two-dimensional key points of the target object, semantic information of multiple pixels on the target object in the image; the second supervision information includes an initial three-dimensional point cloud on the surface of the target object. The disclosed embodiment can only use the initial two-dimensional key points or semantic information of the pixel points of the target object as supervision information to optimize the initial parameters, with high optimization efficiency and low optimization complexity; or, the initial three-dimensional point cloud on the surface of the target object and the aforementioned initial two-dimensional key points or semantic information of the pixel points can be used together as supervision information to improve the accuracy of the obtained optimization parameters.
[0006] In some embodiments, the method further includes: extracting information of initial two-dimensional key points of the target object from the image of the target object through a key point extraction network. Using the information of the initial two-dimensional key points extracted by the key point extraction network as supervision information can generate more natural and reasonable actions for the three-dimensional model.
[0007] In some embodiments, the image includes a depth image of the target object; the method further includes: extracting depth information of multiple pixels on the target object from the depth image; and reversely projecting multiple pixels on the target object in the depth image to a three-dimensional space based on the depth information to obtain an initial three-dimensional point cloud of the surface of the target object. By extracting the depth information and reversely projecting the pixels on the two-dimensional image to a three-dimensional space based on the depth information to obtain an initial three-dimensional point cloud of the surface of the target object, the initial three-dimensional point cloud can be used as supervisory information to optimize the initial parameters, further improving the accuracy of parameter optimization.
[0008] In some embodiments, the image also includes an RGB image of the target object; extracting the depth information of multiple pixels on the target object from the depth image includes: performing image segmentation on the RGB image, determining the image area where the target object is located in the RGB image based on the image segmentation result, determining the image area where the target object is located in the depth image based on the image area where the target object is located in the RGB image; and obtaining the depth information of multiple pixels in the image area where the target object is located in the depth image. By performing image segmentation on the RGB image, the position of the target object can be accurately determined, thereby accurately extracting the depth information of the target object.
[0009] In some embodiments, the method further includes: filtering out outliers from the initial three-dimensional point cloud, and using the filtered initial three-dimensional point cloud as the second supervisory information. By filtering outliers, the interference of outliers is reduced, and the accuracy of the parameter optimization process is further improved.
[0010] In some embodiments, the image of the target object is acquired by an image acquisition device, and the initial parameters include the initial global rotation parameters of the target object, the initial key point rotation parameters of each key point of the target object, the initial posture parameters of the target object, and the initial displacement parameters of the image acquisition device; the optimization of the initial parameters based on the pre-acquired supervisory information for representing the characteristics of the target object includes: optimizing the current displacement parameters and the initial global rotation parameters of the image acquisition device based on the supervisory information and the initial displacement parameters when the initial posture parameters and the initial key point rotation parameters remain unchanged, thereby obtaining optimized displacement parameters and optimized global rotation parameters; optimizing the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters, thereby obtaining optimized key point rotation parameters and optimized posture parameters. Since changing the position of the image acquisition device and changing the position of the three-dimensional key points during the optimization process can cause changes in the two-dimensional projection of the three-dimensional key points, this will cause the optimization process to be very unstable. By adopting a two-stage optimization method, the initial key point rotation parameters and initial posture parameters are first fixed to optimize the initial displacement parameters and initial global rotation parameters of the image acquisition device, and then the initial displacement parameters and initial global rotation parameters are fixed to optimize the initial key point rotation parameters and initial posture parameters, thereby improving the stability of the optimization process.
[0011] In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object; the optimization of the current displacement parameters and the initial global rotation parameters of the image acquisition device based on the supervisory information and the initial displacement parameters includes: obtaining the target two-dimensional projection key points of the target object belonging to the preset part of the target object among the two-dimensional projection key points corresponding to the three-dimensional key points of the target object; wherein the three-dimensional key points of the target object are obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters, and the two-dimensional projection key points are obtained by projecting the three-dimensional key points of the target object based on the current displacement parameters and the initial global rotation parameters; obtaining the first loss between the target two-dimensional projection key points and the initial two-dimensional key points; obtaining the second loss between the initial displacement parameters and the current displacement parameters; optimizing the current displacement parameters and the initial global rotation parameters based on the first loss and the second loss. The preset part can be a part such as a trunk. Since different actions have little effect on the key points of the trunk part, by using the key points of the trunk part to determine the first loss, the influence of different actions on the key point positions can be reduced, and the accuracy of the optimization result can be improved. Since the two-dimensional key points are supervisory information on the two-dimensional plane, and the displacement parameters of the image acquisition device are parameters on the three-dimensional plane, by obtaining the second loss, it is possible to reduce the situation where the optimization result falls into the local optimal point on the two-dimensional plane and deviates from the real point.
[0012] In some embodiments, the supervision information includes the initial two-dimensional key points of the target object; the optimization of the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters includes: obtaining a third loss between the optimized two-dimensional projection key points of the target object and the initial two-dimensional key points, the optimized two-dimensional projection key points are obtained by projecting the optimized three-dimensional key points of the target object based on the optimized displacement parameters and the optimized global rotation parameters, and the optimized three-dimensional key points are obtained based on the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters; obtaining a fourth loss, the fourth loss is used to characterize the rationality of the posture corresponding to the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters; the initial key point rotation parameters and the initial posture parameters are optimized based on the third loss and the fourth loss. This embodiment optimizes the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters, improves the stability of the optimization process, and at the same time, the rationality of the posture corresponding to the optimized parameters is guaranteed by the fourth loss.
[0013] In some embodiments, the method further comprises: after optimizing the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters, jointly optimizing the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters. Based on the above optimization, this embodiment jointly optimizes the optimized parameters, thereby further improving the accuracy of the optimization result.
[0014] In some embodiments, the supervision information includes initial two-dimensional key points of the target object and an initial three-dimensional point cloud of the surface of the target object; optimizing the current displacement parameters of the image acquisition device and the initial global rotation parameters based on the supervision information and the initial displacement parameters includes: obtaining target two-dimensional projection key points belonging to a preset part of the target object among the two-dimensional projection key points corresponding to the three-dimensional key points of the target object; wherein the three-dimensional key points of the target object are obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters, and the two-dimensional projection key points are obtained by projecting the three-dimensional key points of the target object based on the current displacement parameters and the initial global rotation parameters; obtaining a first loss between the target two-dimensional projection key points and the initial two-dimensional key points; obtaining a second loss between the initial displacement parameters and the current displacement parameters; obtaining a fifth loss between the three-dimensional point cloud to be optimized on the surface of the target object and the initial three-dimensional point cloud; the three-dimensional point cloud to be optimized is obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters; optimizing the current displacement parameters and the initial global rotation parameters based on the first loss, the second loss and the fifth loss. This embodiment adds the three-dimensional point cloud to the supervision information to optimize the initial parameters, thereby improving the accuracy of the optimization result.
[0015] In some embodiments, the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters are jointly optimized, including: obtaining the sixth loss between the optimized two-dimensional projection key points of the target object and the initial two-dimensional key points, the optimized two-dimensional projection key points are obtained by projecting the optimized three-dimensional key points of the target object based on the optimized displacement parameters and the optimized global rotation parameters, and the optimized three-dimensional key points are obtained based on the optimized global rotation parameters, the optimized key point rotation parameters and the optimized posture parameters; obtaining the seventh loss, the seventh loss is used to characterize the rationality of the posture corresponding to the optimized global rotation parameters, the optimized key point rotation parameters and the optimized posture parameters; obtaining the eighth loss between the optimized three-dimensional point cloud of the surface of the target object and the initial three-dimensional point cloud; the optimized three-dimensional point cloud is obtained based on the optimized global rotation parameters, the optimized key point rotation parameters and the optimized posture parameters; based on the sixth loss, the seventh loss and the eighth loss, the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters are jointly optimized. In this embodiment, the three-dimensional point cloud is added to the supervision information to optimize the initial parameters, thereby improving the accuracy of the optimization result.
[0016] According to a second aspect of an embodiment of the present disclosure, a three-dimensional reconstruction device is provided, comprising: a first three-dimensional reconstruction module, used to perform three-dimensional reconstruction of an image of a target object through a three-dimensional reconstruction network to obtain initial parameters of the target object, and the initial parameters are used to establish a three-dimensional model of the target object; an optimization module, used to optimize the initial parameters based on pre-acquired supervision information used to represent characteristics of the target object to obtain optimized parameters; and a second three-dimensional reconstruction module, used to perform skeleton skinning processing based on the optimized parameters to establish a three-dimensional model of the target object.
[0017] In some embodiments, the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; the first supervision information includes at least one of the following: initial two-dimensional key points of the target object, semantic information of multiple pixels on the target object in the image; the second supervision information includes an initial three-dimensional point cloud on the surface of the target object. The disclosed embodiment can only use the initial two-dimensional key points or semantic information of the pixel points of the target object as supervision information to optimize the initial parameters, with high optimization efficiency and low optimization complexity; or, the initial three-dimensional point cloud on the surface of the target object and the aforementioned initial two-dimensional key points or semantic information of the pixel points can be used together as supervision information to improve the accuracy of the obtained optimization parameters.
[0018] In some embodiments, the device further comprises: a two-dimensional key point extraction module, which is used to extract information of initial two-dimensional key points of the target object from the image of the target object through a key point extraction network. The information of the initial two-dimensional key points extracted by the key point extraction network is used as supervision information to generate more natural and reasonable actions for the three-dimensional model.
[0019] In some embodiments, the image includes a depth image of the target object; the device further includes: a depth information extraction module for extracting depth information of multiple pixel points on the target object from the depth image; and a reverse projection module for reversely projecting multiple pixel points on the target object in the depth image to a three-dimensional space based on the depth information to obtain an initial three-dimensional point cloud of the surface of the target object. By extracting the depth information and reversely projecting the pixel points on the two-dimensional image to a three-dimensional space based on the depth information, an initial three-dimensional point cloud of the surface of the target object is obtained, so that the initial three-dimensional point cloud can be used as supervision information to optimize the initial parameters, further improving the accuracy of parameter optimization.
[0020] In some embodiments, the image also includes an RGB image of the target object; the depth information extraction module includes: an image segmentation unit for performing image segmentation on the RGB image, an image region determination unit for determining the image region where the target object is located in the RGB image based on the image segmentation result, and determining the image region where the target object is located in the depth image based on the image region where the target object is located in the RGB image; a depth information acquisition unit for acquiring depth information of multiple pixels in the image region where the target object is located in the depth image. By performing image segmentation on the RGB image, the position of the target object can be accurately determined, thereby accurately extracting the depth information of the target object.
[0021] In some embodiments, the device further comprises: a filtering module, configured to filter out outliers from the initial three-dimensional point cloud, and use the filtered initial three-dimensional point cloud as the second supervisory information. By filtering outliers, the interference of outliers is reduced, and the accuracy of the parameter optimization process is further improved.
[0022] In some embodiments, the image of the target object is acquired by an image acquisition device, and the initial parameters include the initial global rotation parameters of the target object, the initial key point rotation parameters of each key point of the target object, the initial posture parameters of the target object, and the initial displacement parameters of the image acquisition device; the optimization module includes: a first optimization unit, which is used to optimize the current displacement parameters and the initial global rotation parameters of the image acquisition device based on the supervision information and the initial displacement parameters when the initial posture parameters and the initial key point rotation parameters remain unchanged, so as to obtain optimized displacement parameters and optimized global rotation parameters; a second optimization unit, which is used to optimize the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters, so as to obtain optimized key point rotation parameters and optimized posture parameters. Since changing the position of the image acquisition device and changing the position of the three-dimensional key points during the optimization process can cause the two-dimensional projection of the three-dimensional key points to change, this will cause the optimization process to be very unstable. By adopting a two-stage optimization method, the initial key point rotation parameters and initial posture parameters are first fixed to optimize the initial displacement parameters and initial global rotation parameters of the image acquisition device, and then the initial displacement parameters and initial global rotation parameters are fixed to optimize the initial key point rotation parameters and initial posture parameters, thereby improving the stability of the optimization process.
[0023] In some embodiments, the supervision information includes the initial two-dimensional key points of the target object; the first optimization unit is used to: obtain the target two-dimensional projection key points belonging to the preset part of the target object from the two-dimensional projection key points corresponding to the three-dimensional key points of the target object; wherein the three-dimensional key points of the target object are obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters, and the two-dimensional projection key points are obtained by projecting the three-dimensional key points of the target object based on the current displacement parameters and the initial global rotation parameters; obtain the first loss between the target two-dimensional projection key points and the initial two-dimensional key points; obtain the second loss between the initial displacement parameters and the current displacement parameters; optimize the current displacement parameters and the initial global rotation parameters based on the first loss and the second loss. The preset part can be a part such as the trunk. Since different actions have little effect on the key points of the trunk, by using the key points of the trunk to determine the first loss, the influence of different actions on the key point position can be reduced, and the accuracy of the optimization result can be improved. Since the two-dimensional key points are supervision information on the two-dimensional plane, and the displacement parameters of the image acquisition device are parameters on the three-dimensional plane, by obtaining the second loss, the situation where the optimization result falls into the local optimal point on the two-dimensional plane and deviates from the real point can be reduced.
[0024] In some embodiments, the supervision information includes the initial two-dimensional key points of the target object; the second optimization unit is used to: obtain a third loss between the optimized two-dimensional projection key points of the target object and the initial two-dimensional key points, the optimized two-dimensional projection key points are obtained by projecting the optimized three-dimensional key points of the target object based on the optimized displacement parameters and the optimized global rotation parameters, and the optimized three-dimensional key points are obtained based on the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters; obtain a fourth loss, the fourth loss is used to characterize the rationality of the posture corresponding to the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters; optimize the initial key point rotation parameters and the initial posture parameters based on the third loss and the fourth loss. This embodiment optimizes the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters, improves the stability of the optimization process, and at the same time, ensures the rationality of the posture corresponding to the optimized parameters through the fourth loss.
[0025] In some embodiments, the device further comprises: a joint optimization module, which is used to jointly optimize the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters after optimizing the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters. This embodiment jointly optimizes the optimized parameters based on the above optimization, thereby further improving the accuracy of the optimization result.
[0026] In some embodiments, the supervision information includes the initial two-dimensional key points of the target object and the initial three-dimensional point cloud of the surface of the target object; the first optimization unit is used to: obtain the target two-dimensional projection key points belonging to the preset part of the target object among the two-dimensional projection key points corresponding to the three-dimensional key points of the target object; wherein the three-dimensional key points of the target object are obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters, and the two-dimensional projection key points are obtained by projecting the three-dimensional key points of the target object based on the current displacement parameters and the initial global rotation parameters; obtain the first loss between the target two-dimensional projection key points and the initial two-dimensional key points; obtain the second loss between the initial displacement parameters and the current displacement parameters; obtain the fifth loss between the three-dimensional point cloud to be optimized on the surface of the target object and the initial three-dimensional point cloud; the three-dimensional point cloud to be optimized is obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters; optimize the current displacement parameters and the initial global rotation parameters based on the first loss, the second loss and the fifth loss. In this embodiment, the three-dimensional point cloud is added to the supervision information to optimize the initial parameters, thereby improving the accuracy of the optimization result.
[0027] In some embodiments, the joint optimization module includes: a first acquisition unit, used to acquire the sixth loss between the optimized two-dimensional projection key point of the target object and the initial two-dimensional key point, the optimized two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized displacement parameter and the optimized global rotation parameter, and the optimized three-dimensional key point is obtained based on the optimized global rotation parameter, the optimized key point rotation parameter and the optimized posture parameter; a second acquisition unit, used to acquire the seventh loss, the seventh loss is used to characterize the rationality of the posture corresponding to the optimized global rotation parameter, the optimized key point rotation parameter and the optimized posture parameter; a third acquisition unit, used to acquire the eighth loss between the optimized three-dimensional point cloud of the surface of the target object and the initial three-dimensional point cloud; the optimized three-dimensional point cloud is obtained based on the optimized global rotation parameter, the optimized key point rotation parameter and the optimized posture parameter; a joint optimization unit, used to jointly optimize the optimized global rotation parameter, the optimized key point rotation parameter, the optimized posture parameter and the optimized displacement parameter based on the sixth loss, the seventh loss and the eighth loss. In this embodiment, the three-dimensional point cloud is added to the supervision information to optimize the initial parameters, thereby improving the accuracy of the optimization result.
[0028] According to a third aspect of an embodiment of the present disclosure, a three-dimensional reconstruction system is provided, the system comprising: an image acquisition device, for acquiring an image of a target object; and a processing unit, which is communicatively connected to the image acquisition device, for performing three-dimensional reconstruction of the image of the target object through a three-dimensional reconstruction network to obtain initial parameters of the target object, wherein the initial parameters are used to establish a three-dimensional model of the target object; optimizing the initial parameters based on pre-acquired supervision information for representing characteristics of the target object to obtain optimized parameters; and performing skeleton skinning processing based on the optimized parameters to establish the three-dimensional model of the target object.
[0029] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in any embodiment is implemented.
[0030] According to a fifth aspect of an embodiment of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the embodiments is implemented.
[0031] The disclosed embodiment performs three-dimensional reconstruction of the image of the target object through a three-dimensional reconstruction network to obtain initial parameters, then optimizes the initial parameters based on supervision information, and establishes a three-dimensional model of the target object based on the optimized parameters obtained by the parameter optimization. The advantage of the parameter optimization method is that it can give a more accurate three-dimensional reconstruction result that conforms to the two-dimensional observation characteristics of the image, but it often gives unnatural and unreasonable action results and has low reliability. However, performing network regression through a three-dimensional reconstruction network can give a more natural and reasonable action result. Therefore, using the output results of the three-dimensional reconstruction network as initial parameters for parameter optimization can ensure the reliability of the three-dimensional reconstruction results while taking into account the accuracy of the three-dimensional reconstruction.
[0032] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.
[0034] Figure 1A and Figure 1B is a schematic diagram of a three-dimensional model of some embodiments.
[0035] Figure 2 4 is a flow chart of a three-dimensional reconstruction method according to an embodiment of the present disclosure.
[0036] Figure 3 It is an overall flow chart of an embodiment of the present disclosure.
[0037] Figure 4A and Figure 4B They are schematic diagrams of application scenarios of embodiments of the present disclosure.
[0038] Figure 5 4 is a block diagram of a 3D reconstruction apparatus according to an embodiment of the present disclosure.
[0039] Figure 6 Schematic diagram of a three-dimensional reconstruction system according to an embodiment of the present disclosure.
[0040] Figure 7 It is a schematic diagram of the structure of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0042] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.
[0043] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0044] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.
[0045] The 3D reconstruction of the target object requires the reconstruction of the target object's posture and limb rotation. Usually, a parametric model is used to express the target object's posture and limb rotation, rather than just the 3D key points. For example, the 3D reconstruction of different people reconstructs the 3D models of thinner people (such as Figure 1A ) and a 3D model of a fat person (as shown Figure 1B shown), due to Figure 1A The people and Figure 1B The people shown are in the same posture and have the same key point information. The difference in their body shapes cannot be expressed only through the key point information.
[0046] In related technologies, three-dimensional reconstruction is generally performed through two methods: parameter optimization and network regression. The parameter optimization method usually selects a set of standard parameters, and uses the gradient descent method to iteratively optimize the parameters of the three-dimensional model of the target object based on the two-dimensional visual features of the image of the target object, where the two-dimensional visual features of the image can select two-dimensional key points, etc. The advantage of the parameter optimization method is that it can give a relatively accurate parameter estimation result that conforms to the two-dimensional visual features of the image, but it often gives unnatural and unreasonable action results, and the final performance of the parameter optimization is very dependent on the initialization parameters, resulting in low reliability of the three-dimensional reconstruction method based on parameter optimization.
[0047] The network regression method usually trains an end-to-end neural network to learn the mapping from image to 3D model parameters. The advantage of the network regression method is that it can give a more natural and reasonable action result, but due to the lack of a large amount of training data, the 3D reconstruction result may not match the 2D visual features in the image. Therefore, the 3D reconstruction method based on network regression has low accuracy. The 3D reconstruction method in the related art cannot take into account both the accuracy and reliability of the 3D reconstruction result.
[0048] Based on this, the present disclosure provides a three-dimensional reconstruction method. Figure 2 As shown, the method includes:
[0049] Step 201: Performing three-dimensional reconstruction on the image of the target object through a three-dimensional reconstruction network to obtain initial parameters of the target object, wherein the initial parameters are used to establish a three-dimensional model of the target object;
[0050] Step 202: Optimizing the initial parameters based on the pre-acquired supervision information used to represent the characteristics of the target object to obtain optimized parameters;
[0051] Step 203: Perform skeleton skinning processing based on the optimization parameters to establish a three-dimensional model of the target object.
[0052] In step 201, the target object may be a three-dimensional object in various physical spaces such as a person, an animal, a robot, or one or more regions on the three-dimensional object, such as a face or a limb. For ease of description, the following description takes the case where the target object is a person and the three-dimensional reconstruction of the target object is a human body reconstruction as an example. The image of the target object may be a single image, or may include multiple images obtained by photographing the target object from multiple different perspectives. The three-dimensional human body reconstruction based on a single image is called monocular three-dimensional human body reconstruction, and the three-dimensional human body reconstruction based on multiple images from different perspectives is called multi-eye three-dimensional human body reconstruction. Each image may be a grayscale image, an RGB image, or an RGBD image. The image may be an image captured in real time by an image acquisition device (e.g., a camera or a camera) around the target object, or may be an image captured and stored in advance.
[0053] The image of the target object can be reconstructed in three dimensions by a three-dimensional reconstruction network, wherein the three-dimensional reconstruction network can be a pre-trained neural network. The three-dimensional reconstruction network can be based on the image to perform three-dimensional reconstruction, and estimate natural and reasonable initial parameters, where the initial parameters can be represented by a vector, the dimension of the vector can be, for example, 85 dimensions, and the vector contains the three parts of information of the rotation information of the moving limbs of the human body (i.e., initial posture parameters, including the initial global rotation parameters of the human body and the initial key point rotation parameters of 23 key points), initial posture parameters, and initial parameters of the camera. The human body can be represented by key points and the limb bones connecting these key points. The key points of the human body generally include one or more of the key points such as the top of the head, nose, neck, left and right eyes, left and right ears, chest, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right hips, left and right knees, left and right ankles, etc. The initial posture parameters are used to determine the position of the key points of the human body in three-dimensional space. The initial posture parameters are used to determine the body information such as the height, fatness, and thinness of the human body. The initial parameters of the camera are used to determine the absolute position of the human body in the three-dimensional space in the camera coordinate system, including the initial displacement parameters between the camera and the human body and the initial posture parameters of the camera, wherein the initial posture parameters of the camera can be replaced by the initial global rotation parameters of the human body. The human body parameters can be represented by the parameter form of the SMPL (Skinned Multi-Person Linear) model (called SMPL parameters). After obtaining the SMPL parameters, skeleton skinning processing can be performed based on the SMPL parameters, that is, a mapping function M (θ, β) is used to map the initial body parameters and the initial posture parameters to a three-dimensional model of the human body surface. The three-dimensional model includes 6890 vertices, and the vertices are connected to form triangular facets through fixed connections. A pre-trained regressor W can be used to further regress the three-dimensional key points of the human body from the vertices of the human body surface model. ,Right now:
[0054]
[0055] In step 202, the supervisory information may be a two-dimensional visual feature of the image, for example, at least one of the two-dimensional key points of the target object in the image and the semantic information of multiple pixels on the target object. The semantic information of a pixel is used to characterize the region on the target object where the pixel is located, and the region may be, for example, the region of the head, arms, torso, legs, etc. When two-dimensional key point information is used as supervisory information, a two-dimensional key point extraction network may be used to estimate the position of the key points of the human body in the image, and any two-dimensional posture estimation method may be used here, such as OpenPose. In addition to using two-dimensional visual features as supervisory information, the two-dimensional visual features and the three-dimensional point cloud on the surface of the target object may also be used together as supervisory information to further improve the accuracy of three-dimensional reconstruction.
[0056] When the image includes a depth image (for example, the image is an RGBD image), depth information of multiple pixels on the target object can be extracted from the depth image, and based on the depth information, the multiple pixels on the target object in the depth image can be projected into a three-dimensional space to obtain a three-dimensional point cloud of the surface of the target object.
[0057] The plurality of pixels may be part or all of the pixels on the target object in the image. For example, the pixels may include pixels of various regions on the target object that need to be 3D reconstructed, and the number of pixels in each region should be greater than or equal to the number required for 3D reconstruction.
[0058] Since an image generally includes both a target object and a background area. Therefore, the RGB image included in the image can be segmented to obtain the image area where the target object is located in the RGB image, and the image area where the target object is located in the depth image is determined based on the image area where the target object is located in the RGB image; and the depth information of multiple pixels in the image area where the target object is located in the depth image is obtained. By performing image segmentation, the image area where the target object that needs to be three-dimensionally reconstructed is located can be extracted from the image to avoid the influence of the background area in the image on the three-dimensional reconstruction. In some embodiments, the pixels in the depth image correspond one-to-one to the pixels in the RGB image, for example, the image is an RGBD image.
[0059] Furthermore, outliers can be filtered out from the three-dimensional point cloud, and the filtered three-dimensional point cloud can be determined as the second supervisory information. The filtering can be implemented using a point cloud filter. By filtering out outliers, a more refined three-dimensional point cloud of the target object surface can be obtained, thereby further improving the accuracy of three-dimensional reconstruction. For each target three-dimensional point in the three-dimensional point cloud, the average distance from the n three-dimensional points closest to the target three-dimensional point to the target three-dimensional point is obtained. Assuming that the average distance corresponding to each target three-dimensional point obeys a statistical distribution (for example, a Gaussian distribution), the mean and variance of the statistical distribution can be calculated, and a threshold s is set based on the mean and variance. Then, three-dimensional points with an average distance outside the threshold s range can be regarded as outliers and filtered out of the three-dimensional point cloud.
[0060] In practical applications, if the image is an RGB image, the two-dimensional observation features can be used as supervisory information to iteratively optimize the initial parameters. If the image is an RGBD image, the two-dimensional observation features and the three-dimensional point cloud of the target object surface can be used together as supervisory information to iteratively optimize the initial parameters. The optimization method can be, for example, a gradient descent method, which is not limited in the present disclosure.
[0061] In step 203, skeleton skinning processing may be performed based on the optimization parameters to obtain a three-dimensional model of the target object.
[0062] like Figure 3 As shown, it is an overall flow chart of the embodiment of the present disclosure. When the input is an RGB image, the RGB image can be reconstructed in three dimensions through a three-dimensional reconstruction network to obtain the human body parameters of the person in the image, and the key point extraction network is used to extract the key points of the person in the image to obtain the two-dimensional key points of the human body. Then, the human body parameters are used as initialization parameters, and the two-dimensional key points of the human body are used as supervision signals. The human body parameters are optimized through a parameter optimization module to obtain optimized human body parameters, and the skeleton skinning is performed based on the optimized human body parameters to obtain a human body reconstruction model.
[0063] When the input is an RGBD image, the image can be decomposed into an RGB image and a TOF (Time of Flight) depth map, and the TOF depth map includes the depth information of each pixel in the RGB image. The RGB image can be reconstructed in three dimensions through a three-dimensional reconstruction network to obtain the human body parameters of the person in the image, and the key point extraction network is used to extract the key points of the person in the image to obtain the two-dimensional key points of the human body. A point cloud reconstruction module can also be used to reconstruct the human body surface point cloud based on the depth information in the TOF depth map. Then, the human body parameters are used as initialization parameters, the two-dimensional key points of the human body and the human body surface point cloud are used together as supervision signals, and the human body parameters are optimized through the parameter optimization module to obtain the optimized human body parameters, and the skeleton skinning is performed based on the optimized human body parameters to obtain a human body reconstruction model.
[0064] Furthermore, after obtaining the human body reconstruction model, color processing may be performed on the human body reconstruction model based on the color information in the RGB image or the RGBD image, so that the human body reconstruction model matches the color information of the person in the image.
[0065] The disclosed embodiment performs three-dimensional reconstruction of the image of the target object through a three-dimensional reconstruction network to obtain initial parameters, then optimizes the initial parameters based on supervision information, and establishes a three-dimensional model of the target object based on the optimized parameters obtained by the parameter optimization. The advantage of the parameter optimization method is that it can give a more accurate three-dimensional reconstruction result that conforms to the two-dimensional observation characteristics of the image, but it often gives unnatural and unreasonable action results and has low reliability. However, performing network regression through a three-dimensional reconstruction network can give a more natural and reasonable action result. Therefore, using the output results of the three-dimensional reconstruction network as initial parameters for parameter optimization can ensure the reliability of the three-dimensional reconstruction results while taking into account the accuracy of the three-dimensional reconstruction.
[0066] In some embodiments, in the parameter optimization stage, a multi-stage optimization method can be used. The multi-stage optimization method may include a camera optimization stage and a posture optimization stage. In the camera optimization stage, the optimization targets are the global rotation parameter R and the current displacement parameter t between the image acquisition device and the target object. Wherein t and R are both three-dimensional vectors, and R is expressed in the form of axis angle. In the posture optimization stage, the optimization targets are the key point rotation parameters and body posture parameters.
[0067] Since changing the camera position and the position of the three-dimensional key points of the human body during the optimization process can both cause the two-dimensional projection of the three-dimensional key points to change, this will cause the optimization process to be very unstable. Therefore, we fix the human body posture in the camera optimization stage and the camera position in the posture optimization stage, thereby improving the stability of the optimization process. That is, under the condition that the initial posture parameters and the initial key point rotation parameters remain unchanged, based on the supervision information and the initial displacement parameters, the current displacement parameters of the image acquisition device and the initial global rotation parameters are optimized to obtain optimized displacement parameters and optimized global rotation parameters; then, the optimized displacement parameters and the optimized global rotation parameters are kept unchanged, and based on the optimized displacement parameters and the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters are optimized to obtain optimized key point rotation parameters and optimized posture parameters.
[0068] Furthermore, a target two-dimensional projection key point belonging to a preset part of the target object among the two-dimensional projection key points corresponding to the three-dimensional key points of the target object can be obtained; wherein the three-dimensional key point of the target object is obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters, and the two-dimensional projection key point is obtained by projecting the three-dimensional key point of the target object based on the current displacement parameters and the initial global rotation parameters; a first loss between the target two-dimensional projection key point and the initial two-dimensional key point is obtained; a second loss between the initial displacement parameter and the current displacement parameter is obtained; and the current displacement parameter and the initial global rotation parameter are optimized based on the first loss and the second loss.
[0069] The preset part may be a torso, and the target two-dimensional projection key points may include left and right shoulder points, left and right hip points, spine center points and other key points. Since different actions have little effect on the key points of the torso, the first loss is established by using the key points of the torso to reduce the effect of different actions on the key point positions and improve the accuracy of the optimization results. The first loss can also be called the torso key point projection loss, and the second loss can also be called the camera displacement regularization loss. The specific formula is as follows:
[0070]
[0071] L cam =||tt net ||2;
[0072] L torso and L cam Represent the first loss and the second loss respectively, x torso and Represent the target 2D projection key point and the initial 2D key point, t and tnet represents the current displacement parameter and the initial displacement parameter between the image acquisition device and the target object respectively. A first target loss L1 can be determined based on the first loss and the second loss. For example, the first target loss can be determined as the sum of the first loss and the second loss, as follows:
[0073] L1=L torso +L cam .
[0074] A third loss can be obtained between the optimized two-dimensional projection key points of the target object and the initial two-dimensional key points, wherein the optimized two-dimensional projection key points are obtained by projecting the optimized three-dimensional key points of the target object based on the optimized displacement parameters and the optimized global rotation parameters, and the optimized three-dimensional key points are obtained based on the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters; a fourth loss is obtained, wherein the fourth loss is used to characterize the rationality of the postures corresponding to the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters; and the initial key point rotation parameters and the initial posture parameters are optimized based on the third loss and the fourth loss.
[0075] The third loss can also be called the two-dimensional key point projection loss, and the fourth loss can also be called the posture rationality loss. The specific formula is as follows:
[0076] L 2d is the third loss, x and The second target loss can be determined based on the third loss and the fourth loss. For example, the second target loss can be determined as the sum of the third loss and the fourth loss, as follows:
[0077] L2=L 2d +L prior ;
[0078] Among them, L2 is the second target loss, L prior The fourth loss can be obtained by using a Gaussian mixture model (GMM) to determine whether the posture corresponding to the optimized global rotation parameters, initial key point rotation parameters and initial posture parameters is reasonable, and a larger loss is output for unreasonable postures.
[0079] After the initial key point rotation parameters and the initial posture parameters are optimized based on the optimized displacement parameters and the optimized global rotation parameters, the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters can also be jointly optimized, that is, a three-stage optimization method is adopted. In the case where the supervision information includes information of a three-dimensional point cloud of the surface of the target object, the three-stage optimization method can be adopted, including a camera optimization stage, a posture optimization stage and a point cloud optimization stage.
[0080] In the camera optimization stage, the target two-dimensional projection key points belonging to the preset part of the target object can be obtained from the two-dimensional projection key points corresponding to the three-dimensional key points of the target object; wherein the three-dimensional key points of the target object are obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters, and the two-dimensional projection key points are obtained by projecting the three-dimensional key points of the target object based on the current displacement parameters and the initial global rotation parameters; the first loss between the target two-dimensional projection key points and the initial two-dimensional key points is obtained; the second loss between the initial displacement parameters and the current displacement parameters is obtained; the fifth loss between the three-dimensional point cloud to be optimized on the surface of the target object and the initial three-dimensional point cloud is obtained; the three-dimensional point cloud to be optimized is obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters; the current displacement parameters and the initial global rotation parameters are optimized based on the first loss, the second loss and the fifth loss. The fifth loss can also be called ICP (Iterative Closest Point) point cloud registration loss, and the formula is as follows:
[0081]
[0082] Where, L icp is the fifth loss, the initial three-dimensional point cloud is regarded as point cloud P, the three-dimensional point cloud to be optimized is regarded as point cloud Q, K1 = {(p, q)} is the point pair set consisting of each point in point cloud P to the nearest point in point cloud Q, K2 = {(p, q)} is the point pair set consisting of each point in point cloud Q to the nearest point in point cloud P. The first loss and the second loss are respectively recorded as:
[0083]
[0084] L cam =||tt net ||2;
[0085] L torso and L cam Represent the first loss and the second loss respectively, x torso and Represent the target 2D projection key point and the initial 2D key point, t and t net Respectively represent the current displacement parameter and the initial displacement parameter. The first target loss L1 can be determined based on the sum of the first loss, the second loss and the fifth loss, and then the current displacement parameter and the initial global rotation parameter are optimized based on the first target loss, that is:
[0086] L1=L torso +L cam +L icp .
[0087] The optimization method of the posture optimization stage in the three-stage optimization process is the same as that of the posture optimization stage in the two-stage optimization process, which will not be repeated here.
[0088] In the point cloud optimization stage, the sixth loss between the optimized two-dimensional projection key points of the target object and the initial two-dimensional key points can be obtained. The optimized two-dimensional projection key points are obtained by projecting the optimized three-dimensional key points of the target object based on the optimized displacement parameters and the optimized global rotation parameters. The optimized three-dimensional key points are obtained based on the optimized global rotation parameters, the optimized key point rotation parameters and the optimized posture parameters; the seventh loss is obtained, and the seventh loss is used to characterize the rationality of the posture corresponding to the optimized global rotation parameters, the optimized key point rotation parameters and the optimized posture parameters; the eighth loss between the optimized three-dimensional point cloud on the surface of the target object and the initial three-dimensional point cloud is obtained; the optimized three-dimensional point cloud is obtained based on the optimized global rotation parameters, the optimized key point rotation parameters and the optimized posture parameters; based on the sixth loss, the seventh loss and the eighth loss, the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters are jointly optimized. The specific formula is as follows:
[0089]
[0090]
[0091] In the formula, For the sixth loss, To optimize the 2D projection key points, is the initial two-dimensional key point. The seventh loss can be obtained by using a Gaussian mixture model to determine whether the postures corresponding to the optimized global rotation parameters, optimized key point rotation parameters, and optimized body posture parameters are reasonable. A larger loss is output for unreasonable postures. For the eighth loss, P is the initial three-dimensional point cloud regarded as a point cloud, For said optimized three-dimensional point cloud, For each point in the point cloud P The point pair set consisting of the points with the closest distance in . Point Cloud A point pair set consisting of each point in the point cloud P and the closest point in the point cloud P. Further, the sum of the sixth loss, the seventh loss and the eighth loss can be determined as the third target loss L3, and the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters are jointly optimized based on the third target loss, as follows:
[0092] L3=L 2d +L prior +L icp .
[0093] When the image of the target object is an RGB image, parameter optimization can be performed based on the aforementioned two-stage optimization method including a camera optimization stage and a posture optimization stage; when the image of the target object is an RGBD image, parameter optimization can be performed based on the aforementioned three-stage optimization method including a camera optimization stage, a posture optimization stage and a point cloud optimization stage.
[0094] This solution has a wide range of usage scenarios and can provide a natural, reasonable and accurate human body reconstruction model in scenarios such as virtual fitting rooms, virtual anchors, and video motion migration.
[0095] like Figure 4A 4 is a schematic diagram of a virtual fitting room application scenario of an embodiment of the present disclosure. The image of the user 401 can be captured by the camera 403, and the captured image can be sent to a processor (not shown in the figure) for three-dimensional human reconstruction, so as to obtain a human body reconstruction model 404 corresponding to the user 401, and the human body reconstruction model 404 can be displayed on the display interface 402 for the user 401 to watch. At the same time, the user 401 can select the desired clothing 405, including but not limited to clothes 4051 and hats 4052, etc., and the clothing 405 can be displayed on the display interface 402 based on the human body reconstruction model 404, so that the user 401 can view the wearing effect of the clothing 405.
[0096] like Figure 4BAs shown, it is a schematic diagram of the application scenario of the virtual live broadcast room of the embodiment of the present disclosure. In the process of live broadcast, the image of the anchor user 406 can be collected by the anchor client 407, and the image of the anchor user 406 can be sent to the server 408 for three-dimensional reconstruction to obtain the human body reconstruction model of the anchor user, that is, the virtual anchor. The server 408 can return the human body reconstruction model of the anchor user to the anchor client 407 for display, as shown in the model 4071 in the figure. In addition, the anchor client 407 can also collect the voice information of the anchor user, and send the voice information to the server 408, so that the server 408 can fuse the human body reconstruction model and the voice information. The server 408 can send the fused human body reconstruction model and voice information to the audience client 409 watching the live broadcast program for display and playback, wherein the displayed human body reconstruction model is shown in the model 4091 in the figure. In the above manner, the screen of the virtual anchor performing live broadcast can be displayed on the audience client 409.
[0097] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0098] like Figure 5 As shown, the present disclosure also provides a three-dimensional reconstruction device, the device comprising:
[0099] A first 3D reconstruction module 501 is used to perform 3D reconstruction on the image of the target object through a 3D reconstruction network to obtain initial parameters of the target object, and the initial parameters are used to establish a 3D model of the target object;
[0100] An optimization module 502 is used to optimize the initial parameters based on the pre-acquired supervision information used to represent the characteristics of the target object to obtain optimized parameters;
[0101] The second 3D reconstruction module 503 is used to perform skeleton skinning processing based on the optimization parameters to establish a 3D model of the target object.
[0102] In some embodiments, the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; the first supervision information includes at least one of the following: initial two-dimensional key points of the target object, semantic information of multiple pixels on the target object in the image; the second supervision information includes an initial three-dimensional point cloud on the surface of the target object. The disclosed embodiment can only use the initial two-dimensional key points or semantic information of the pixel points of the target object as supervision information to optimize the initial parameters, with high optimization efficiency and low optimization complexity; or, the initial three-dimensional point cloud on the surface of the target object and the aforementioned initial two-dimensional key points or semantic information of the pixel points can be used together as supervision information to improve the accuracy of the obtained optimization parameters.
[0103] In some embodiments, the device further comprises: a two-dimensional key point extraction module, which is used to extract information of initial two-dimensional key points of the target object from the image of the target object through a key point extraction network. The information of the initial two-dimensional key points extracted by the key point extraction network is used as supervision information to generate more natural and reasonable actions for the three-dimensional model.
[0104] In some embodiments, the image includes a depth image of the target object; the device further includes: a depth information extraction module for extracting depth information of multiple pixel points on the target object from the depth image; and a reverse projection module for reversely projecting multiple pixel points on the target object in the depth image to a three-dimensional space based on the depth information to obtain an initial three-dimensional point cloud of the surface of the target object. By extracting the depth information and reversely projecting the pixel points on the two-dimensional image to a three-dimensional space based on the depth information, an initial three-dimensional point cloud of the surface of the target object is obtained, so that the initial three-dimensional point cloud can be used as supervision information to optimize the initial parameters, further improving the accuracy of parameter optimization.
[0105] In some embodiments, the image also includes an RGB image of the target object; the depth information extraction module includes: an image segmentation unit for performing image segmentation on the RGB image, an image region determination unit for determining the image region where the target object is located in the RGB image based on the image segmentation result, and determining the image region where the target object is located in the depth image based on the image region where the target object is located in the RGB image; a depth information acquisition unit for acquiring depth information of multiple pixels in the image region where the target object is located in the depth image. By performing image segmentation on the RGB image, the position of the target object can be accurately determined, thereby accurately extracting the depth information of the target object.
[0106] In some embodiments, the device further comprises: a filtering module, configured to filter out outliers from the initial three-dimensional point cloud, and use the filtered initial three-dimensional point cloud as the second supervisory information. By filtering outliers, the interference of outliers is reduced, and the accuracy of the parameter optimization process is further improved.
[0107] In some embodiments, the image of the target object is acquired by an image acquisition device, and the initial parameters include the initial global rotation parameters of the target object, the initial key point rotation parameters of each key point of the target object, the initial posture parameters of the target object, and the initial displacement parameters of the image acquisition device; the optimization module includes: a first optimization unit, which is used to optimize the current displacement parameters and the initial global rotation parameters of the image acquisition device based on the supervision information and the initial displacement parameters when the initial posture parameters and the initial key point rotation parameters remain unchanged, so as to obtain optimized displacement parameters and optimized global rotation parameters; a second optimization unit, which is used to optimize the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters, so as to obtain optimized key point rotation parameters and optimized posture parameters. Since changing the position of the image acquisition device and changing the position of the three-dimensional key points during the optimization process can cause the two-dimensional projection of the three-dimensional key points to change, this will cause the optimization process to be very unstable. By adopting a two-stage optimization method, the initial key point rotation parameters and initial posture parameters are first fixed to optimize the initial displacement parameters and initial global rotation parameters of the image acquisition device, and then the initial displacement parameters and initial global rotation parameters are fixed to optimize the initial key point rotation parameters and initial posture parameters, thereby improving the stability of the optimization process.
[0108] In some embodiments, the supervision information includes the initial two-dimensional key points of the target object; the first optimization unit is used to: obtain the target two-dimensional projection key points belonging to the preset part of the target object from the two-dimensional projection key points corresponding to the three-dimensional key points of the target object; wherein the three-dimensional key points of the target object are obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters, and the two-dimensional projection key points are obtained by projecting the three-dimensional key points of the target object based on the current displacement parameters and the initial global rotation parameters; obtain the first loss between the target two-dimensional projection key points and the initial two-dimensional key points; obtain the second loss between the initial displacement parameters and the current displacement parameters; optimize the current displacement parameters and the initial global rotation parameters based on the first loss and the second loss. The preset part can be a part such as the trunk. Since different actions have little effect on the key points of the trunk, by using the key points of the trunk to determine the first loss, the influence of different actions on the key point position can be reduced, and the accuracy of the optimization result can be improved. Since the two-dimensional key points are supervision information on the two-dimensional plane, and the displacement parameters of the image acquisition device are parameters on the three-dimensional plane, by obtaining the second loss, the situation where the optimization result falls into the local optimal point on the two-dimensional plane and deviates from the real point can be reduced.
[0109] In some embodiments, the supervision information includes the initial two-dimensional key points of the target object; the second optimization unit is used to: obtain a third loss between the optimized two-dimensional projection key points of the target object and the initial two-dimensional key points, the optimized two-dimensional projection key points are obtained by projecting the optimized three-dimensional key points of the target object based on the optimized displacement parameters and the optimized global rotation parameters, and the optimized three-dimensional key points are obtained based on the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters; obtain a fourth loss, the fourth loss is used to characterize the rationality of the posture corresponding to the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters; optimize the initial key point rotation parameters and the initial posture parameters based on the third loss and the fourth loss. This embodiment optimizes the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters, improves the stability of the optimization process, and at the same time, ensures the rationality of the posture corresponding to the optimized parameters through the fourth loss.
[0110] In some embodiments, the device further comprises: a joint optimization module, which is used to jointly optimize the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters after optimizing the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters. This embodiment jointly optimizes the optimized parameters based on the above optimization, thereby further improving the accuracy of the optimization result.
[0111] In some embodiments, the supervision information includes the initial two-dimensional key points of the target object and the initial three-dimensional point cloud of the surface of the target object; the first optimization unit is used to: obtain the target two-dimensional projection key points belonging to the preset part of the target object among the two-dimensional projection key points corresponding to the three-dimensional key points of the target object; wherein the three-dimensional key points of the target object are obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters, and the two-dimensional projection key points are obtained by projecting the three-dimensional key points of the target object based on the current displacement parameters and the initial global rotation parameters; obtain the first loss between the target two-dimensional projection key points and the initial two-dimensional key points; obtain the second loss between the initial displacement parameters and the current displacement parameters; obtain the fifth loss between the three-dimensional point cloud to be optimized on the surface of the target object and the initial three-dimensional point cloud; the three-dimensional point cloud to be optimized is obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters; optimize the current displacement parameters and the initial global rotation parameters based on the first loss, the second loss and the fifth loss. In this embodiment, the three-dimensional point cloud is added to the supervision information to optimize the initial parameters, thereby improving the accuracy of the optimization result.
[0112] In some embodiments, the joint optimization module includes: a first acquisition unit, used to acquire the sixth loss between the optimized two-dimensional projection key point of the target object and the initial two-dimensional key point, the optimized two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized displacement parameter and the optimized global rotation parameter, and the optimized three-dimensional key point is obtained based on the optimized global rotation parameter, the optimized key point rotation parameter and the optimized posture parameter; a second acquisition unit, used to acquire the seventh loss, the seventh loss is used to characterize the rationality of the posture corresponding to the optimized global rotation parameter, the optimized key point rotation parameter and the optimized posture parameter; a third acquisition unit, used to acquire the eighth loss between the optimized three-dimensional point cloud of the surface of the target object and the initial three-dimensional point cloud; the optimized three-dimensional point cloud is obtained based on the optimized global rotation parameter, the optimized key point rotation parameter and the optimized posture parameter; a joint optimization unit, used to jointly optimize the optimized global rotation parameter, the optimized key point rotation parameter, the optimized posture parameter and the optimized displacement parameter based on the sixth loss, the seventh loss and the eighth loss. In this embodiment, the three-dimensional point cloud is added to the supervision information to optimize the initial parameters, thereby improving the accuracy of the optimization result.
[0113] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0114] like Figure 6 As shown, the present disclosure also provides a three-dimensional reconstruction system, the system comprising:
[0115] An image acquisition device 601 is used to acquire an image of a target object; and
[0116] The processing unit 602, which is communicatively connected to the image acquisition device 601, is used to perform three-dimensional reconstruction on the image of the target object through a three-dimensional reconstruction network to obtain initial parameters of the target object, and the initial parameters are used to establish a three-dimensional model of the target object; based on pre-acquired supervision information used to represent the characteristics of the target object, the initial parameters are optimized to obtain optimized parameters; based on the optimized parameters, skeleton skinning is performed to establish a three-dimensional model of the target object.
[0117] The image acquisition device 601 in the embodiment of the present disclosure can be a device with an image acquisition function such as a camera or a webcam. The image acquired by the image acquisition device 601 can be transmitted to the processing unit 602 in real time, or stored and transmitted from the storage space to the processing unit 602 when needed. The processing unit 602 can be a single server or a server cluster composed of multiple servers. The method executed by the processing unit 602 is detailed in the embodiment of the above-mentioned three-dimensional reconstruction method, which will not be repeated here.
[0118] An embodiment of the present specification also provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.
[0119] Figure 7 A more specific hardware structure diagram of a computing device provided in an embodiment of this specification is shown, and the device may include: a processor 701, a memory 702, an input / output interface 703, a communication interface 704, and a bus 705. The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected to each other in communication within the device through the bus 705.
[0120] The processor 701 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor 701 can also include a graphics card, which can be a Nvidia titan X graphics card or a 1080Ti graphics card, etc.
[0121] The memory 702 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 702 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 702 and are called and executed by the processor 701.
[0122] The input / output interface 703 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0123] The communication interface 704 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0124] The bus 705 includes a path for transmitting information between various components of the device (eg, the processor 701 , the memory 702 , the input / output interface 703 , and the communication interface 704 ).
[0125] It should be noted that, although the above device only shows the processor 701, the memory 702, the input / output interface 703, the communication interface 704 and the bus 705, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.
[0126] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.
[0127] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0128] It can be known from the above description of the implementation mode that the technicians in this field can clearly understand that the embodiments of this specification can be implemented by means of software plus the necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of this specification can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments of this specification.
[0129] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.
[0130] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment scheme of this specification. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.
[0131] The above is only a specific implementation of the embodiments of this specification. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiments of this specification. These improvements and modifications should also be regarded as the protection scope of the embodiments of this specification.
Claims
1. A three-dimensional reconstruction method, characterized in that: The method comprises: Performing three-dimensional reconstruction on the image of the target object through a three-dimensional reconstruction network to obtain initial parameters of the target object, wherein the initial parameters are used to establish a three-dimensional model of the target object; Optimizing the initial parameters based on the pre-acquired supervision information for representing the characteristics of the target object to obtain optimized parameters; Performing skeleton skinning processing based on the optimization parameters to establish a three-dimensional model of the target object; The image of the target object is acquired by an image acquisition device, and the initial parameters include initial global rotation parameters of the target object, initial key point rotation parameters of each key point of the target object, initial posture parameters of the target object and initial displacement parameters of the image acquisition device; The optimizing the initial parameters based on the pre-acquired supervision information for representing the characteristics of the target object includes: Under the condition that the initial posture parameters and the initial key point rotation parameters remain unchanged, based on the supervision information and the initial displacement parameters, the current displacement parameters of the image acquisition device and the initial global rotation parameters are optimized to obtain optimized displacement parameters and optimized global rotation parameters; Based on the optimized displacement parameters and the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters are optimized to obtain optimized key point rotation parameters and optimized posture parameters.
2. The method according to claim 1, characterized in that The supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; The first supervisory information includes at least one of the following: an initial two-dimensional key point of the target object, semantic information of a plurality of pixels on the target object in the image; The second supervision information includes an initial three-dimensional point cloud of the surface of the target object.
3. The method according to claim 2, characterized in that The method further comprises: The information of the initial two-dimensional key points of the target object is extracted from the image of the target object through a key point extraction network.
4. The method according to claim 2, characterized in that: The image includes a depth image of the target object; the method further includes: Extracting depth information of a plurality of pixels on the target object from the depth image; Based on the depth information, a plurality of pixel points on the target object in the depth image are reversely projected into a three-dimensional space to obtain an initial three-dimensional point cloud of the surface of the target object.
5. The method according to claim 4, characterized in that The image also includes an RGB image of the target object; and extracting depth information of a plurality of pixels on the target object from the depth image includes: Performing image segmentation on the RGB image, determining an image region where the target object is located in the RGB image based on a result of the image segmentation, and determining an image region where the target object is located in the depth image based on the image region where the target object is located in the RGB image; Depth information of a plurality of pixels in an image area where the target object is located in the depth image is obtained.
6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: Outlier points are filtered out from the initial three-dimensional point cloud, and the filtered initial three-dimensional point cloud is used as the second supervisory information.
7. The method according to claim 1, characterized in that The supervision information includes initial two-dimensional key points of the target object; The optimizing the current displacement parameter of the image acquisition device and the initial global rotation parameter based on the supervision information and the initial displacement parameter comprises: Obtaining a target two-dimensional projection key point belonging to a preset part of the target object from among the two-dimensional projection key points corresponding to the three-dimensional key points of the target object; wherein the three-dimensional key point of the target object is obtained based on the initial global rotation parameter, the initial key point rotation parameter and the initial posture parameter, and the two-dimensional projection key point is obtained by projecting the three-dimensional key point of the target object based on the current displacement parameter and the initial global rotation parameter; Obtaining a first loss between the target two-dimensional projection key point and the initial two-dimensional key point; Acquire a second loss between the initial displacement parameter and the current displacement parameter; The current displacement parameters and the initial global rotation parameters are optimized based on the first loss and the second loss.
8. The method according to claim 1, characterized in that The supervision information includes the initial two-dimensional key points of the target object; the optimization of the initial key point rotation parameters and the initial posture parameters based on the optimized displacement parameters and the optimized global rotation parameters includes: Acquire a third loss between an optimized two-dimensional projection key point of the target object and the initial two-dimensional key point, wherein the optimized two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized displacement parameter and the optimized global rotation parameter, and the optimized three-dimensional key point is obtained based on the optimized global rotation parameter, the initial key point rotation parameter and the initial posture parameter; Obtaining a fourth loss, wherein the fourth loss is used to characterize the rationality of the posture corresponding to the optimized global rotation parameter, the initial key point rotation parameter, and the initial posture parameter; The initial key point rotation parameters and the initial posture parameters are optimized based on the third loss and the fourth loss.
9. The method according to claim 1, characterized in that: The method further comprises: After the initial key point rotation parameters and the initial posture parameters are optimized based on the optimized displacement parameters and the optimized global rotation parameters, the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters are jointly optimized.
10. The method according to claim 9, characterized in that The supervision information includes initial two-dimensional key points of the target object and an initial three-dimensional point cloud of the surface of the target object; The optimizing the current displacement parameter of the image acquisition device and the initial global rotation parameter based on the supervision information and the initial displacement parameter comprises: Obtaining a target two-dimensional projection key point belonging to a preset part of the target object from among the two-dimensional projection key points corresponding to the three-dimensional key points of the target object; wherein the three-dimensional key point of the target object is obtained based on the initial global rotation parameter, the initial key point rotation parameter and the initial posture parameter, and the two-dimensional projection key point is obtained by projecting the three-dimensional key point of the target object based on the current displacement parameter and the initial global rotation parameter; Obtaining a first loss between the target two-dimensional projection key point and the initial two-dimensional key point; Acquire a second loss between the initial displacement parameter and the current displacement parameter; Acquire a fifth loss between the three-dimensional point cloud to be optimized and the initial three-dimensional point cloud on the surface of the target object; the three-dimensional point cloud to be optimized is obtained based on the initial global rotation parameters, the initial key point rotation parameters and the initial posture parameters; The current displacement parameters and initial global rotation parameters are optimized based on the first loss, the second loss and the fifth loss.
11. The method according to claim 9, characterized in that The supervision information includes the initial two-dimensional key points of the target object and the initial three-dimensional point cloud of the surface of the target object; the joint optimization of the optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters includes: Acquire a sixth loss between an optimized two-dimensional projection key point of the target object and the initial two-dimensional key point, wherein the optimized two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized displacement parameter and the optimized global rotation parameter, and the optimized three-dimensional key point is obtained based on the optimized global rotation parameter, the optimized key point rotation parameter and the optimized posture parameter; Obtaining a seventh loss, wherein the seventh loss is used to characterize the rationality of the postures corresponding to the optimized global rotation parameters, the optimized key point rotation parameters, and the optimized posture parameters; Acquire an eighth loss between the optimized three-dimensional point cloud of the target object surface and the initial three-dimensional point cloud; the optimized three-dimensional point cloud is obtained based on the optimized global rotation parameters, the optimized key point rotation parameters and the optimized posture parameters; The optimized global rotation parameters, the optimized key point rotation parameters, the optimized posture parameters and the optimized displacement parameters are jointly optimized based on the sixth loss, the seventh loss and the eighth loss.
12. A three-dimensional reconstruction device, characterized in that: The device comprises: A first 3D reconstruction module, configured to perform 3D reconstruction on an image of a target object through a 3D reconstruction network to obtain initial parameters of the target object, wherein the initial parameters are used to establish a 3D model of the target object; An optimization module, used to optimize the initial parameters based on the pre-acquired supervision information used to represent the characteristics of the target object to obtain optimized parameters; A second three-dimensional reconstruction module is used to perform skeleton skinning processing based on the optimization parameters to establish a three-dimensional model of the target object; The image of the target object is acquired by an image acquisition device, and the initial parameters include initial global rotation parameters of the target object, initial key point rotation parameters of each key point of the target object, initial posture parameters of the target object and initial displacement parameters of the image acquisition device; The optimization module is specifically used for: Under the condition that the initial posture parameters and the initial key point rotation parameters remain unchanged, based on the supervision information and the initial displacement parameters, the current displacement parameters of the image acquisition device and the initial global rotation parameters are optimized to obtain optimized displacement parameters and optimized global rotation parameters; Based on the optimized displacement parameters and the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters are optimized to obtain optimized key point rotation parameters and optimized posture parameters.
13. A three-dimensional reconstruction system, characterized in that: The system comprises: An image acquisition device, used to acquire an image of a target object; and A processing unit communicatively connected to the image acquisition device is used to perform three-dimensional reconstruction on the image of the target object through a three-dimensional reconstruction network to obtain initial parameters of the target object, wherein the initial parameters are used to establish a three-dimensional model of the target object; optimize the initial parameters based on pre-acquired supervision information used to represent the characteristics of the target object to obtain optimized parameters; perform skeleton skinning processing based on the optimized parameters to establish the three-dimensional model of the target object; The initial parameters include initial global rotation parameters of the target object, initial key point rotation parameters of each key point of the target object, initial posture parameters of the target object and initial displacement parameters of the image acquisition device; The optimizing the initial parameters based on the pre-acquired supervision information for representing the characteristics of the target object includes: Under the condition that the initial posture parameters and the initial key point rotation parameters remain unchanged, based on the supervision information and the initial displacement parameters, the current displacement parameters of the image acquisition device and the initial global rotation parameters are optimized to obtain optimized displacement parameters and optimized global rotation parameters; Based on the optimized displacement parameters and the optimized global rotation parameters, the initial key point rotation parameters and the initial posture parameters are optimized to obtain optimized key point rotation parameters and optimized posture parameters.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 11 is implemented.
15. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 11 is implemented.
Citation Information
Patent Citations
Quasi-three dimensional reconstruction method for acquiring two-dimensional videos of static scenes
CN103236082A
Complex dynamic human body object three-dimensional reconstruction method and system based on multi-view video
CN107945269A
Method for establishing complete and consistent organism three-dimensional characteristic representation model
CN110288696A
Three-dimensional human body reconstruction method based on synthetic depth data
CN110298916A
Segmented SFM three-dimensional reconstruction method
CN111383333A
