Three-dimensional reconstruction method, device, equipment and storage medium

Abstract

The present application discloses a three-dimensional reconstruction method, device, equipment and storage medium, and relates to the fields of artificial intelligence technology such as computer vision and deep learning. A specific implementation of the method includes: obtaining an RGBD image sequence of the target; for the RGBD image in the RGBD image sequence, determining a first three-dimensional key point set of the target on the RGBD image; transferring the first three-dimensional key point set to the camera coordinate system to obtain a second three-dimensional key point set of the target in the camera coordinate system; based on the second three-dimensional key point set, calculating the Euler angle of the RGBD image; selecting the RGBD image based on the Euler angle; performing three-dimensional reconstruction of the target based on the selected RGBD image to obtain a three-dimensional model of the target. This implementation selects key frames from the RGBD image sequence for three-dimensional reconstruction, which can save storage and network transmission resources while ensuring the three-dimensional reconstruction effect.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, specifically to the field of artificial intelligence technologies such as computer vision and deep learning, and especially to three-dimensional reconstruction methods, devices, equipment and storage media. Background Art

[0002] Usually, the RGBD images collected by the depth camera are very dense. For example, during the face scanning process, the depth camera will collect hundreds of frames of RGBD face images. However, RGBD images are mainly stored in the form of pictures, and a huge number of RGBD images require large storage and network transmission resources. In addition, a large amount of storage and network transmission resources are required to reconstruct all frames of the RGBD image, resulting in high storage and network transmission costs. Summary of the invention

[0003] The embodiments of the present application provide a three-dimensional reconstruction method, apparatus, device and storage medium.

[0004] In a first aspect, an embodiment of the present application proposes a three-dimensional reconstruction method, including: obtaining an RGBD image sequence of a target; for an RGBD image in the RGBD image sequence, determining a first three-dimensional key point set of the target on the RGBD image; transferring the first three-dimensional key point set to a camera coordinate system to obtain a second three-dimensional key point set of the target in the camera coordinate system; based on the second three-dimensional key point set, calculating the Euler angles of the RGBD image; selecting the RGBD image based on the Euler angles; and performing three-dimensional reconstruction of the target based on the selected RGBD image to obtain a three-dimensional model of the target.

[0005] In a second aspect, an embodiment of the present application proposes a three-dimensional reconstruction device, comprising: an acquisition module, configured to acquire an RGBD image sequence of a target; a determination module, configured to determine a first three-dimensional key point set of the target on the RGBD image in the RGBD image sequence; a transfer module, configured to transfer the first three-dimensional key point set to a camera coordinate system to obtain a second three-dimensional key point set of the target in the camera coordinate system; a calculation module, configured to calculate the Euler angles of the RGBD image based on the second three-dimensional key point set; a selection module, configured to select the RGBD image based on the Euler angles; and a reconstruction module, configured to perform three-dimensional reconstruction of the target based on the selected RGBD image to obtain a three-dimensional model of the target.

[0006] In a third aspect, an embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method described in any implementation manner in the first aspect.

[0007] In a fourth aspect, an embodiment of the present application proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable a computer to execute a method as described in any implementation manner in the first aspect.

[0008] The three-dimensional reconstruction method, device, equipment and storage medium provided in the embodiment of the present application, for the RGBD image in the acquired RGBD image sequence, first determine the first three-dimensional key point set of the target on the RGBD image; then transfer the first three-dimensional key point set to the camera coordinate system to obtain the second three-dimensional key point set of the target in the camera coordinate system; finally, based on the second three-dimensional key point set, calculate the Euler angle of the RGBD image. After obtaining the Euler angle of the RGBD image in the RGBD image sequence, select the RGBD image based on the Euler angle, and perform three-dimensional reconstruction of the target based on the selected RGBD image to obtain a three-dimensional model of the target. Selecting key frames from the RGBD image sequence based on the Euler angle of the RGBD image improves the accuracy of the selected key frames. Using key frames for three-dimensional reconstruction can save storage and network transmission resources while ensuring the three-dimensional reconstruction effect. When applied to the three-dimensional reconstruction scene of the face, the storage and network transmission costs in the three-dimensional reconstruction of the face are greatly reduced.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, purposes and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present application. Among them:

[0011] Figure 1 is an exemplary system architecture in which the present application may be applied;

[0012] Figure 2 is a flow chart of an embodiment of a three-dimensional reconstruction method according to the present application;

[0013] Figure 3 is a flow chart of another embodiment of the three-dimensional reconstruction method according to the present application;

[0014] Figure 4 yes Figure 3 A decomposed flow chart of the filtering step in the three-dimensional reconstruction method shown;

[0015] Figure 5is a schematic structural diagram of an embodiment of a three-dimensional reconstruction device according to the present application;

[0016] Figure 6 It is a block diagram of an electronic device used to implement the three-dimensional reconstruction method of an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of a 3D reconstruction method or a 3D reconstruction apparatus of the present application can be applied.

[0020] like Figure 1 As shown, the system architecture 100 may include a camera 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the camera 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links, or optical fiber cables.

[0021] The camera 101 may interact with the server 103 via the network 102 to receive or send messages, etc. For example, the camera 101 may collect an RGBD image sequence of a target and send it to the server 103 .

[0022] The server 103 can provide various services. For example, the server 103 can analyze and process the RGBD image sequence of the target received from the camera 101 to generate a processing result (eg, a three-dimensional model of the target).

[0023] It should be noted that the server 103 can be hardware or software. When the server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 103 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0024] It should be noted that the three-dimensional reconstruction method provided in the embodiment of the present application is generally executed by the server 103 , and accordingly, the three-dimensional reconstruction device is generally disposed in the server 103 .

[0025] It should be understood that Figure 1 The number of cameras, networks and servers in the embodiment is only for illustration. Any number of cameras, networks and servers may be provided as required.

[0026] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a three-dimensional reconstruction method according to the present application. The three-dimensional reconstruction method comprises the following steps:

[0027] Step 201, obtaining a target RGBD image sequence.

[0028] In this embodiment, the execution subject of the 3D reconstruction method (eg Figure 1 The server 103 shown in FIG. 1 may receive data from a depth camera (eg, Figure 1 The camera 101 shown in the figure acquires the RGBD image sequence of the target. For example, the terminal device can call the API (Application Programming Interface) of the depth camera to acquire the RGBD image sequence of the target.

[0029] Typically, a depth camera can scan a target and collect an RGBD image sequence including a large number of RGBD images. Among them, an RGBD image can include an RGB image and a depth image. The pixel value of each pixel of the RGB image can be the color value of each point on the target surface. Typically, all colors that can be perceived by human vision are obtained by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. The pixel value of each pixel of the depth image can be the distance between the depth camera and each point on the target surface. Typically, the RGB image and the depth image are registered, so there is a one-to-one correspondence between the pixels of the RGB image and the depth image.

[0030] Step 202: for an RGBD image in the RGBD image sequence, determine a first three-dimensional key point set of the target on the RGBD image.

[0031] In this embodiment, for an RGBD image in an RGBD image sequence, the above-mentioned execution subject can determine a set of three-dimensional key points of the target on the RGBD image as a first set of three-dimensional key points. Among them, the key points of the target are the key semantic points of the target, and a set consisting of a large number of key points of the target is the key point set. Since the RGBD image has both the two-dimensional information of the RGB image and the depth information of the depth image, the key points of the target on the RGBD image have three-dimensional information and are three-dimensional key points.

[0032] Taking a human face as an example, the key point set of the human face may include about 150 key points of the human face, including but not limited to at least one of the following: the position, contour information and shape information of the facial features, etc.

[0033] Step 203: transfer the first three-dimensional key point set to the camera coordinate system to obtain a second three-dimensional key point set of the target in the camera coordinate system.

[0034] In this embodiment, the execution entity may transfer the first three-dimensional key point set to the camera coordinate system to obtain a three-dimensional key point set of the target in the camera coordinate system as the second three-dimensional key point set.

[0035] Typically, the execution subject may transfer the first three-dimensional key point set to the camera coordinate system by using a three-dimensional coordinate transformation method in space to obtain a second three-dimensional key point set. The three-dimensional coordinate transformation method in space may include but is not limited to rotation matrices and rotation vectors, Euler angles and quaternions, etc.

[0036] Step 204: Calculate the Euler angles of the RGBD image based on the second three-dimensional key point set.

[0037] In this embodiment, the execution subject may calculate the Euler angle of the RGBD image based on the second three-dimensional key point set. The Euler angle of the RGBD image may be a relative Euler angle or an absolute Euler angle. The relative Euler angle is the Euler angle between adjacent RGBD images in the RGBD image sequence. The absolute Euler angle may be the Euler angle of the RGBD image in the RGBD image sequence relative to the first frame RGBD image.

[0038] Step 205: Select an RGBD image based on the Euler angle.

[0039] In this embodiment, the execution subject can select RGBD images based on Euler angles. For example, when the Euler angles are relative Euler angles, for adjacent RGBD images with smaller relative Euler angles, only one RGBD image is selected; for adjacent RGBD images with larger relative Euler angles, both adjacent RGBD images are selected. For another example, when the Euler angles are absolute Euler angles, the absolute Euler angles are divided into multiple ranges, and when selecting RGBD images from the RGBD image sequence, each range is ensured to be selected.

[0040] Step 206 , performing three-dimensional reconstruction of the target based on the selected RGBD image to obtain a three-dimensional model of the target.

[0041] In this embodiment, the execution subject may perform three-dimensional reconstruction of the target based on the selected RGBD image to obtain a three-dimensional model of the target.

[0042] Usually, the selected RGBD image is the key frame of the RGBD image sequence, which is relatively sparse. At the same time, since the selected RGBD image has accurate Euler angles, the Euler angles can be used for preliminary alignment during reconstruction, which greatly reduces the number of iterations of the reconstruction process and improves the success rate of reconstruction.

[0043] The three-dimensional reconstruction method provided in the embodiment of the present application, for the RGBD image in the acquired RGBD image sequence, first determines the first three-dimensional key point set of the target on the RGBD image; then transfers the first three-dimensional key point set to the camera coordinate system to obtain the second three-dimensional key point set of the target in the camera coordinate system; finally, based on the second three-dimensional key point set, calculates the Euler angle of the RGBD image. After obtaining the Euler angle of the RGBD image in the RGBD image sequence, the RGBD image is selected based on the Euler angle, and the target is three-dimensionally reconstructed based on the selected RGBD image to obtain a three-dimensional model of the target. Selecting key frames from the RGBD image sequence based on the Euler angle of the RGBD image can retain the key frames containing the main information of the RGBD image sequence as much as possible, thereby improving the accuracy of the selected key frames. Using key frames for three-dimensional reconstruction can save storage and network transmission resources while ensuring the three-dimensional reconstruction effect. When applied to the three-dimensional reconstruction scene of the face, the storage and network transmission costs in the three-dimensional reconstruction of the face are greatly reduced.

[0044] Further references Figure 3 , which shows a process 300 of another embodiment of a three-dimensional reconstruction method according to the present application. The three-dimensional reconstruction method comprises the following steps:

[0045] Step 301, obtaining a target RGBD image sequence.

[0046] In this embodiment, the above-mentioned execution subject can obtain the RGBD image sequence of the target from the depth camera. Among them, the RGBD image may include an RGB image, a depth image and camera parameters. The camera parameters of different models of depth cameras may be different. The camera parameters are parameters related to the characteristics of the depth camera itself, including but not limited to the focal length of the camera, the optical center of the camera and other parameters.

[0047] Step 302: for an RGBD image in an RGBD image sequence, key point detection is performed on the RGB image to obtain a two-dimensional key point set of the target on the RGB image.

[0048] In this embodiment, for the RGBD image in the RGBD image sequence, the above-mentioned execution subject can perform key point detection on the RGB image to obtain a two-dimensional key point set of the target on the RGB image. Among them, the key point of the target is the key semantic point of the target, and the set composed of a large number of key points of the target is the key point set. Since the RGB image only has two-dimensional information, the key point of the target on the RGB image has two-dimensional information and is a two-dimensional key point.

[0049] Generally, the two-dimensional key point set of the target on the RGB image can be detected by the traditional detection method or by the deep learning detection method. The accuracy of the deep learning detection method is usually higher than that of the traditional detection method. Among them, the traditional detection method can be, for example, saliency detection. The deep learning detection method can be, for example, deep learning model detection. Specifically, the key point detection model is used to perform key point detection on the RGB image to obtain a two-dimensional key point set. Taking the target as a face, the personal key point detection model is used to perform face key point detection on the RGB image to obtain the two-dimensional coordinates of about 150 key points of the face on the RGB image.

[0050] Step 303: align the RGB image with the depth image to obtain a first three-dimensional key point set.

[0051] In this embodiment, the execution entity may align the RGB image with the depth image to obtain a first three-dimensional key point set.

[0052] In practice, there is a one-to-one correspondence between the pixels of the RGB image and the depth image in the RGBD image. For the two-dimensional key point of the target on the RGB image, the RGB image is aligned with the depth image to find the depth value of the two-dimensional key point in the depth image, thereby obtaining the corresponding three-dimensional key point. By performing key point detection on the RGB image and using the one-to-one correspondence between the pixels of the RGB image and the depth image, the first three-dimensional key point set of the target on the RGBD image can be quickly mapped, thereby improving the efficiency of obtaining the first three-dimensional key point set.

[0053] Step 304: Based on the camera parameters, the first three-dimensional key point set is transferred to the camera coordinate system through inverse perspective projection to obtain a second three-dimensional key point set.

[0054] In this embodiment, the execution subject can transfer the first three-dimensional key point set to the camera coordinate system through inverse perspective projection based on the camera parameters to obtain the second three-dimensional key point set. Specifically, the camera focal length and the camera optical center are first obtained from the camera parameters; then the camera focal length, the camera optical center and the depth value of the three-dimensional key points in the first three-dimensional key point set are used to perform inverse perspective projection to obtain the second three-dimensional key point set. Inverse perspective projection based on camera parameters can quickly project the second three-dimensional key point set in the camera coordinate system, thereby improving the efficiency of obtaining the second three-dimensional key point set.

[0055] Step 305: Filter the second three-dimensional key point set using a random sampling consensus algorithm.

[0056] In this embodiment, for RGBD images with larger angles (such as RGBD images of side faces with larger angles), the depth values ​​of the two-dimensional key points in the depth image will have large errors. If the two-dimensional key points with large errors are changed to the camera coordinate system, this type of key point will cause a large change in the semantics of the corresponding second three-dimensional key points, thereby affecting the calculation of the transfer relationship of the RGBD image. Therefore, in order to obtain the second three-dimensional key points with smaller errors, the second three-dimensional key point set can be filtered using RANSAC (random sample consensus), and the transfer relationship of the RGBD image is calculated for the filtered second three-dimensional key point set. The second three-dimensional key point set is filtered using a random sampling consensus algorithm, thereby filtering out the second three-dimensional key points with larger depth value errors, and only using the second three-dimensional key points with smaller depth value errors to calculate the transfer relationship, thereby improving the accuracy of the transfer relationship.

[0057] Step 306: Calculate the transfer relationship between the second three-dimensional key point sets corresponding to adjacent RGBD images in the RGBD image sequence by using a singular value decomposition method.

[0058] In this embodiment, for adjacent RGBD images in an RGBD image sequence, the execution subject may calculate the transfer relationship between the second three-dimensional key point sets corresponding to the adjacent RGBD images by a singular value decomposition method. The transfer relationship may include a rotation matrix, a translation parameter, a scaling factor, and the like. Since the second three-dimensional key point set used to calculate the transfer relationship has filtered out the second three-dimensional key points with large errors, the accuracy of the calculated transfer relationship between adjacent RGBD images is improved.

[0059] Step 307: Calculate the relative Euler angles between adjacent RGBD images according to the rotation matrix.

[0060] In this embodiment, the execution entity may calculate the relative Euler angles between adjacent RGBD images according to the rotation matrix.

[0061] It should be noted that the relative Euler angles of any adjacent RGBD images in the RGBD image sequence are calculated in the above manner.

[0062] Step 308: Calculate the absolute Euler angles of the RGBD image based on the relative Euler angles.

[0063] In this embodiment, the execution subject may calculate the absolute Euler angle of the RGBD image based on the relative Euler angle. Since the execution subject has calculated the relative Euler angle between any adjacent RGBD images in the RGBD image sequence, the relative Euler angles between adjacent RGBD images in the RGBD image sequence are sequentially added to obtain the corresponding absolute Euler angle.

[0064] The transfer relationship corresponding to adjacent RGBD images is obtained by the singular value decomposition method, and the relative Euler angles between adjacent RGBD images are calculated based on the transfer relationship, and then the absolute Euler angles are obtained. Therefore, the absolute Euler angles of the RGBD images in the RGBD image sequence can be quickly calculated, thereby improving the efficiency of obtaining the absolute Euler angles.

[0065] Step 309 , based on the absolute Euler angles, select RGBD images at equal angle intervals.

[0066] In this embodiment, the execution subject may select RGBD images at equal angle intervals based on the absolute Euler angles, so that the selected RGBD images are more uniform.

[0067] Step 310 , performing three-dimensional reconstruction of the target based on the selected RGBD image to obtain a three-dimensional model of the target.

[0068] In this embodiment, the specific operation of step 310 is already Figure 2 Step 206 is described in detail in the illustrated embodiment and will not be described again here.

[0069] from Figure 3 It can be seen that Figure 2Compared with the corresponding embodiment, the process 300 of the three-dimensional reconstruction method in this embodiment highlights the steps of obtaining the first three-dimensional key point set, the second three-dimensional key point set and the absolute Euler angle, and the step of selecting the RGBD image. In addition, a filtering step is added. Therefore, the scheme described in this embodiment performs key point detection on the RGB image, and uses the one-to-one correspondence between the pixel points of the RGB image and the depth image to quickly correspond to the first three-dimensional key point set of the target on the RGBD image, thereby improving the acquisition efficiency of the first three-dimensional key point set. Based on the camera parameters, the inverse perspective projection can be quickly projected to the second three-dimensional key point set in the camera coordinate system, thereby improving the acquisition efficiency of the second three-dimensional key point set. The transfer relationship corresponding to adjacent RGBD images is obtained by the singular value decomposition method, and the relative Euler angles between adjacent RGBD images are calculated based on the transfer relationship, and then the absolute Euler angles are obtained, so that the absolute Euler angles of the RGBD images in the RGBD image sequence can be quickly calculated, thereby improving the acquisition efficiency of the absolute Euler angles. The second 3D key point set is filtered using a random sampling consistency algorithm, thereby filtering out the second 3D key points with large depth value errors, and only using the second 3D key points with small depth value errors to calculate the transfer relationship, thereby improving the accuracy of the transfer relationship. In addition, the accurate inter-frame relationship obtained after selecting the key frame can be used for initialization in reconstruction, thereby reducing the calculation time of reconstruction and improving the reconstruction effect.

[0070] Further references Figure 4 , which shows Figure 3 The decomposition process 400 of the filtering step in the 3D reconstruction method is shown. The filtering step can be decomposed as follows:

[0071] Step 401 : Select a preset number of second three-dimensional key points from the second three-dimensional key point set to generate a second three-dimensional key point subset.

[0072] In this embodiment, the execution entity may select a preset number of second three-dimensional key points from the second three-dimensional key point set to generate a second three-dimensional key point subset, for example, randomly selecting 5 second three-dimensional key points from the second three-dimensional key point set.

[0073] Step 402: Calculate the transformation matrix of the second three-dimensional key point subset corresponding to adjacent RGBD images in the RGBD image sequence.

[0074] In this embodiment, for adjacent RGBD images in the RGBD image sequence, the execution entity may calculate a transformation matrix of the second three-dimensional key point subset corresponding to the adjacent RGBD images, and use the transformation matrix as the initial model.

[0075] Step 403, calculating the projection errors between the second three-dimensional key points in the second three-dimensional key point set and the transformation matrix, and adding the second three-dimensional key points whose projection errors are less than a preset threshold into the inlier set.

[0076] In this embodiment, the execution subject may first calculate the projection error between the second three-dimensional key point in the second three-dimensional key point set corresponding to the adjacent RGBD image and the transformation matrix; then determine whether the projection error is less than a preset threshold. If the projection error is less than the preset threshold, the second three-dimensional key point is added to the internal point set; if the projection error is not less than the preset threshold, the projection error between the next second three-dimensional key point and the transformation matrix is ​​calculated.

[0077] Step 404, determining whether the number of elements in the inlier set is greater than the number of elements in the target inlier set.

[0078] In this embodiment, the execution subject can filter the second three-dimensional key point set based on the internal point set. Specifically, when all the second three-dimensional key points corresponding to the RGBD image sequence are iterated, the execution subject can determine whether the number of elements in the internal point set is greater than the number of elements in the target internal point set. If it is greater than the number of elements in the target internal point set, execute step 405; if it is not greater than the number of elements in the target internal point set, return to execute step 401, and reselect the second three-dimensional key point subset to continue iteration. Among them, in the first iteration, the target internal point set can be empty.

[0079] Step 405: Update the inlier point set to the target inlier point set, and increase the number of iterations by 1.

[0080] In this embodiment, if the number of elements in the inlier set is greater than the number of elements in the target inlier set, the execution entity may update the inlier set to the target inlier set, and increase the number of iterations by 1.

[0081] Step 406, determining whether the number of iterations reaches a preset number of iterations.

[0082] In this embodiment, the execution subject can determine whether the number of iterations reaches the preset number of iterations. If the number of iterations reaches the preset number of iterations, step 407 is executed; if the number of iterations does not reach the preset number of iterations, step 401 is returned to reselect the second 3D key point subset for further iteration.

[0083] Step 407: Use the target internal point set as the filtered second three-dimensional key point set.

[0084] In this embodiment, if the number of iterations reaches the preset number of iterations, the execution subject may use the target internal point set as the filtered second three-dimensional key point set. At this point, the filtering of the second three-dimensional key point set is completed.

[0085] The filtering method provided in the embodiment of the present application improves the filtering accuracy by iteratively filtering the second three-dimensional key point set through the inner point set, and the filtering time can be controlled by setting the number of iterations.

[0086] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a three-dimensional reconstruction device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0087] like Figure 5 As shown, the 3D reconstruction device 500 of this embodiment may include: an acquisition module 501, a determination module 502, a transfer module 503, a calculation module 504, a selection module 505 and a reconstruction module 506. Among them, the acquisition module 501 is configured to acquire an RGBD image sequence of the target; the determination module 502 is configured to determine a first 3D key point set of the target on the RGBD image for the RGBD image in the RGBD image sequence; the transfer module 503 is configured to transfer the first 3D key point set to the camera coordinate system to obtain a second 3D key point set of the target in the camera coordinate system; the calculation module 504 is configured to calculate the Euler angle of the RGBD image based on the second 3D key point set; the selection module 505 is configured to select the RGBD image based on the Euler angle; the reconstruction module 506 is configured to perform 3D reconstruction of the target based on the selected RGBD image to obtain a 3D model of the target.

[0088] In this embodiment, the specific processing of the acquisition module 501, the determination module 502, the transfer module 503, the calculation module 504, the selection module 505 and the reconstruction module 506 and the technical effects thereof can be referred to in detail. Figure 2 The relevant descriptions of steps 201 - 206 in the corresponding embodiment are not repeated here.

[0089] In some optional implementations of this embodiment, the RGBD image includes an RGB image and a depth image; and the determination module 502 includes: a detection submodule, configured to perform key point detection on the RGB image to obtain a two-dimensional key point set of the target on the RGB image; an alignment submodule, configured to align the RGB image with the depth image to obtain a first three-dimensional key point set.

[0090] In some optional implementations of this embodiment, the detection submodule is further configured to: perform key point detection on the RGB image using a key point detection model to obtain a two-dimensional key point set.

[0091] In some optional implementations of this embodiment, the RGBD image also includes camera parameters; and the transfer module 503 includes: a projection submodule, configured to transfer the first three-dimensional key point set to the camera coordinate system through inverse perspective projection based on the camera parameters to obtain a second three-dimensional key point set.

[0092] In some optional implementations of this embodiment, the projection submodule is further configured to: obtain the camera focal length and the camera optical center from the camera parameters; perform inverse perspective projection using the camera focal length, the camera optical center and the depth values ​​of the three-dimensional key points in the first three-dimensional key point set to obtain a second three-dimensional key point set.

[0093] In some optional implementations of this embodiment, the calculation module 504 includes: a first calculation submodule, configured to calculate the transfer relationship between the second three-dimensional key point sets corresponding to adjacent RGBD images in the RGBD image sequence by a singular value decomposition method, wherein the transfer relationship includes a rotation matrix, a translation parameter, and a scaling factor; a second calculation submodule, configured to calculate the relative Euler angles between adjacent RGBD images according to the rotation matrix; and a third calculation submodule, configured to calculate the absolute Euler angles of the RGBD image based on the relative Euler angles.

[0094] In some optional implementations of this embodiment, the selection module 505 is further configured to: select RGBD images at equal angle intervals based on absolute Euler angles.

[0095] In some optional implementations of this embodiment, the three-dimensional reconstruction apparatus further includes: a filtering module configured to filter the second three-dimensional key point set using a random sampling consistency algorithm.

[0096] In some optional implementations of this embodiment, the filtering module includes: a selection submodule, configured to select a preset number of second three-dimensional key points from the second three-dimensional key point set to generate a second three-dimensional key point subset; a fourth calculation submodule, configured to calculate the transformation matrix of the second three-dimensional key point subset corresponding to adjacent RGBD images in the RGBD image sequence; a fifth calculation submodule, configured to calculate the projection error between the second three-dimensional key points in the second three-dimensional key point set and the transformation matrix, and add the second three-dimensional key points whose projection errors are less than a preset threshold to the internal point set; and a filtering submodule, configured to filter the second three-dimensional key point set based on the internal point set.

[0097] In some optional implementations of the present embodiment, the filtering submodule is further configured to: if the number of elements in the inlier set is greater than the number of elements in the target inlier set, update the inlier set to the target inlier set, and increase the number of iterations by 1; if the number of iterations does not reach the preset number of times, reselect the second three-dimensional key point subset to continue iterating; if the number of iterations reaches the preset number of iterations, use the target inlier set as the filtered second three-dimensional key point set.

[0098] In some optional implementations of this embodiment, the target is a human face, and the key points include at least one of the following: position, contour information, and shape information of facial features.

[0099] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0100] like Figure 6 , is a block diagram of an electronic device according to a three-dimensional reconstruction method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0101] like Figure 6 As shown, the electronic device includes: one or more processors 601, a memory 602, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 601 is taken as an example.

[0102] The memory 602 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the three-dimensional reconstruction method provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the three-dimensional reconstruction method provided in the present application.

[0103] The memory 602 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the three-dimensional reconstruction method in the embodiment of the present application (for example, the attached Figure 5 The processor 601 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 602, that is, implements the 3D reconstruction method in the above method embodiment.

[0104] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the electronic device of the three-dimensional reconstruction method, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely arranged relative to the processor 601, and these remote memories may be connected to the electronic device of the three-dimensional reconstruction method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0105] The electronic device of the three-dimensional reconstruction method may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0106] The input device 603 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device of the three-dimensional reconstruction method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 604 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0107] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0108] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0111] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0112] According to the technical solution of the present application, for the RGBD image in the acquired RGBD image sequence, first determine the first three-dimensional key point set of the target on the RGBD image; then transfer the first three-dimensional key point set to the camera coordinate system to obtain the second three-dimensional key point set of the target in the camera coordinate system; finally, calculate the Euler angle of the RGBD image based on the second three-dimensional key point set. After obtaining the Euler angle of the RGBD image in the RGBD image sequence, select the RGBD image based on the Euler angle, and perform three-dimensional reconstruction of the target based on the selected RGBD image to obtain a three-dimensional model of the target. Selecting key frames from the RGBD image sequence based on the Euler angle of the RGBD image improves the accuracy of the selected key frames. Using key frames for three-dimensional reconstruction can save storage and network transmission resources while ensuring the three-dimensional reconstruction effect. When applied to the three-dimensional reconstruction scene of the face, the storage and network transmission costs in the three-dimensional reconstruction of the face are greatly reduced.

[0113] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0114] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A three-dimensional reconstruction method, comprising: Get the target RGBD image sequence; For an RGBD image in the RGBD image sequence, determining a first three-dimensional key point set of the target on the RGBD image; Transferring the first three-dimensional key point set to a camera coordinate system to obtain a second three-dimensional key point set of the target in the camera coordinate system; Selecting a preset number of second three-dimensional key points from the second three-dimensional key point set to generate a second three-dimensional key point subset; Calculating a transformation matrix of a second three-dimensional key point subset corresponding to adjacent RGBD images in the RGBD image sequence; Calculating the projection errors between the second three-dimensional key points in the second three-dimensional key point set and the transformation matrix, and adding the second three-dimensional key points whose projection errors are less than a preset threshold to the inner point set; Based on the internal point set, filtering the second three-dimensional key point set; Calculating the Euler angles of the RGBD image based on the second three-dimensional key point set; Select RGBD image based on Euler angle; The target is reconstructed in three dimensions based on the selected RGBD image to obtain a three-dimensional model of the target.

2. The method according to claim 1, wherein: The RGBD image includes an RGB image and a depth image; as well as The determining a first three-dimensional key point set of the target on the RGBD image includes: Performing key point detection on the RGB image to obtain a two-dimensional key point set of the target on the RGB image; The RGB image is aligned with the depth image to obtain the first three-dimensional key point set.

3. The method according to claim 2, wherein: The step of performing key point detection on the RGB image to obtain a two-dimensional key point set of the target on the RGB image includes: The key point detection model is used to perform key point detection on the RGB image to obtain the two-dimensional key point set.

4. The method according to claim 1, wherein: The RGBD image also includes camera parameters; as well as The step of transferring the first three-dimensional key point set to a camera coordinate system to obtain a second three-dimensional key point set of the target in the camera coordinate system includes: Based on the camera parameters, the first three-dimensional key point set is transferred to a camera coordinate system through inverse perspective projection to obtain the second three-dimensional key point set.

5. The method according to claim 4, wherein: The step of transferring the first three-dimensional key point set to a camera coordinate system by inverse perspective projection based on the camera parameters to obtain the second three-dimensional key point set includes: Obtaining the camera focal length and the camera optical center from the camera parameters; The second three-dimensional key point set is obtained by performing inverse perspective projection using the camera focal length, the camera optical center, and the depth values ​​of the three-dimensional key points in the first three-dimensional key point set.

6. The method according to claim 1, wherein: The calculating the Euler angles of the RGBD image based on the second three-dimensional key point set includes: Calculating a transfer relationship between second three-dimensional key point sets corresponding to adjacent RGBD images in the RGBD image sequence by a singular value decomposition method, wherein the transfer relationship includes a rotation matrix, a translation parameter, and a scaling factor; Calculate the relative Euler angles between the adjacent RGBD images according to the rotation matrix; Based on the relative Euler angles, an absolute Euler angle of the RGBD image is calculated.

7. The method according to claim 6, wherein: The selecting of the RGBD image based on the Euler angle comprises: Based on the absolute Euler angles, RGBD images are selected at equal angle intervals.

8. The method according to claim 1, wherein: The filtering of the second three-dimensional key point set based on the interior point set includes: If the number of elements in the interior point set is greater than the number of elements in the target interior point set, the interior point set is updated to the target interior point set, and the number of iterations is increased by 1; If the number of iterations does not reach the preset number, reselect the second 3D key point subset to continue iteration; If the number of iterations reaches the preset number, the target internal point set is used as the filtered second three-dimensional key point set.

9. The method according to any one of claims 1 to 8, wherein: The target is a human face, and the key points include at least one of the following: the position, contour information and shape information of the facial features.

10. A three-dimensional reconstruction device, comprising: An acquisition module, configured to acquire an RGBD image sequence of a target; A determination module is configured to determine, for an RGBD image in the RGBD image sequence, a first three-dimensional key point set of the target on the RGBD image; A transfer module, configured to transfer the first three-dimensional key point set to a camera coordinate system to obtain a second three-dimensional key point set of the target in the camera coordinate system; A first selection module is configured to select a preset number of second three-dimensional key points from the second three-dimensional key point set to generate a second three-dimensional key point subset; A first calculation module is configured to calculate a transformation matrix of a second three-dimensional key point subset corresponding to adjacent RGBD images in the RGBD image sequence; A second calculation module is configured to calculate the projection errors between the second three-dimensional key points in the second three-dimensional key point set and the transformation matrix, and add the second three-dimensional key points whose projection errors are less than a preset threshold to the inner point set; a filtering module, configured to filter the second three-dimensional key point set based on the internal point set; A third calculation module is configured to calculate the Euler angle of the RGBD image based on the second three-dimensional key point set; A second selection module is configured to select an RGBD image based on Euler angles; The reconstruction module is configured to perform three-dimensional reconstruction of the target based on the selected RGBD image to obtain a three-dimensional model of the target.

11. The device according to claim 10, wherein: The RGBD image includes an RGB image and a depth image; as well as The determination module comprises: A detection submodule is configured to perform key point detection on the RGB image to obtain a two-dimensional key point set of the target on the RGB image; The alignment submodule is configured to align the RGB image with the depth image to obtain the first three-dimensional key point set.

12. The device according to claim 11, wherein The detection submodule is further configured to: The key point detection model is used to perform key point detection on the RGB image to obtain the two-dimensional key point set.

13. The device according to claim 10, wherein: The RGBD image also includes camera parameters; as well as The transfer module comprises: The projection submodule is configured to transfer the first three-dimensional key point set to a camera coordinate system through inverse perspective projection based on the camera parameters to obtain the second three-dimensional key point set.

14. The device according to claim 13, wherein: The projection submodule is further configured to: Obtaining the camera focal length and the camera optical center from the camera parameters; The second three-dimensional key point set is obtained by performing inverse perspective projection using the camera focal length, the camera optical center, and the depth values ​​of the three-dimensional key points in the first three-dimensional key point set.

15. The device according to claim 12, wherein: The third calculation module includes: A first calculation submodule is configured to calculate a transfer relationship between second three-dimensional key point sets corresponding to adjacent RGBD images in the RGBD image sequence by a singular value decomposition method, wherein the transfer relationship includes a rotation matrix, a translation parameter, and a scaling factor; A second calculation submodule is configured to calculate the relative Euler angle between the adjacent RGBD images according to the rotation matrix; The third calculation submodule is configured to calculate the absolute Euler angle of the RGBD image based on the relative Euler angle.

16. The device according to claim 15, wherein: The second selection module is further configured to: Based on the absolute Euler angles, RGBD images are selected at equal angle intervals.

17. The device according to claim 10, wherein: The filtering module is further configured to: If the number of elements in the interior point set is greater than the number of elements in the target interior point set, the interior point set is updated to the target interior point set, and the number of iterations is increased by 1; If the number of iterations does not reach the preset number, reselect the second 3D key point subset to continue iteration; If the number of iterations reaches the preset number, the target internal point set is used as the filtered second three-dimensional key point set.

18. The device according to any one of claims 10 to 17, wherein: The target is a human face, and the key points include at least one of the following: the position, contour information and shape information of the facial features.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method of any one of claims 1-19.

Images