Three-dimensional object generation method and device, storage medium and electronic device

Abstract

The present disclosure provides a three-dimensional object generation method, a three-dimensional object generation device, a computer-readable storage medium and an electronic device, and relates to the field of computer technology. The three-dimensional object generation method comprises: obtaining an original image, the original image contains a target object; using the original image, predicting at least two depth images of the target object, each depth image has a different direction for the target object; back-projecting at least two depth images into a three-dimensional voxel space, and using the information of the voxel points in the three-dimensional voxel space to generate a three-dimensional surface of the target object. The present disclosure can reduce the construction cost of a three-dimensional object.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a three-dimensional object generation method, a three-dimensional object generation device, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the development of games, augmented reality, virtual display, robotics, manufacturing and other fields, the demand for 3D object reconstruction is becoming more and more common.

[0003] Currently, some three-dimensional object reconstruction solutions require the use of depth cameras such as Kinect to obtain depth data, which is costly. Summary of the invention

[0004] The present disclosure provides a three-dimensional object generation method, a three-dimensional object generation device, a computer-readable storage medium and an electronic device, thereby overcoming the problem of high cost of three-dimensional object reconstruction at least to a certain extent.

[0005] According to a first aspect of the present disclosure, a three-dimensional object generation method is provided, including: acquiring an original image, the original image containing a target object; using the original image, predicting at least two depth images of the target object, each depth image having a different direction with respect to the target object; back-projecting the at least two depth images into a three-dimensional voxel space, and using information of voxel points in the three-dimensional voxel space to generate a three-dimensional surface of the target object.

[0006] According to a second aspect of the present disclosure, a three-dimensional object generation device is provided, including: an image acquisition module, used to acquire an original image, the original image containing a target object; a depth prediction module, used to use the original image to predict at least two depth images of the target object, each depth image has a different direction with respect to the target object; and a surface generation module, used to back-project the at least two depth images into a three-dimensional voxel space, and generate a three-dimensional surface of the target object using information of voxel points in the three-dimensional voxel space.

[0007] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned three-dimensional object generation method is implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising a processor; and a memory for storing one or more programs, wherein when the one or more programs are executed by the processor, the processor implements the above-mentioned three-dimensional object generation method.

[0009] In the technical solutions provided by some embodiments of the present disclosure, at least two depth images of the target object are predicted using the original image, the at least two depth images are back-projected into the three-dimensional voxel space, and the three-dimensional surface of the target object is generated using the information of the voxel points in the three-dimensional voxel space. The present disclosure only uses one original image containing the target object to obtain the three-dimensional surface of the target object, without the need for hardware equipment such as a depth camera, and has low cost.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0012] Figure 1 A schematic diagram showing an implementation scenario of the present disclosure is shown;

[0013] Figure 2 A schematic diagram of the structure of an electronic device suitable for implementing an embodiment of the present disclosure is shown;

[0014] Figure 3 A flowchart of a method for generating a three-dimensional object according to an exemplary embodiment of the present disclosure is schematically shown;

[0015] Figure 4 A schematic diagram showing a method of predicting two depth images using an original image according to an embodiment of the present disclosure is shown;

[0016] Figure 5 A schematic diagram is shown to illustrate the method for generating a three-dimensional object of the present disclosure by taking outputting a three-dimensional human body as an example;

[0017] Figure 6 A block diagram schematically shows a three-dimensional object generating device according to an exemplary embodiment of the present disclosure;

[0018] Figure 7 A block diagram schematically shows a three-dimensional object generating device according to another exemplary embodiment of the present disclosure;

[0019] Figure 8 A block diagram of a three-dimensional object generating device according to another exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0020] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0021] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0023] It should be noted that the three-dimensional object generation method of the embodiment of the present disclosure can be executed by a terminal device. In this case, the three-dimensional object generation device can be configured in the terminal device. The terminal device may include but is not limited to a smart phone, a tablet computer, a smart wearable device, a personal computer, etc., and the present disclosure does not limit the type of the terminal device.

[0024] In addition, the three-dimensional object generation method of the embodiment of the present disclosure can also be executed by a server. Specifically, the terminal device can use the captured image as the original image of the embodiment of the present disclosure, and send it to the server through the network, and the server executes the three-dimensional object generation method of the embodiment of the present disclosure according to the received image. In this case, the three-dimensional object generation device can be configured in the server. Among them, the server can be a single server, or a server cluster composed of multiple servers, etc.

[0025] The following description will be made by taking a terminal device executing the three-dimensional object generation scheme of the embodiment of the present disclosure as an example.

[0026] Figure 1 The schematic diagram of the implementation scenario of the present disclosure is shown, and the terminal device can use a single original image containing the target object to generate a three-dimensional surface of the target object. That is, in the implementation of the present disclosure, the terminal device can achieve reconstruction of the three-dimensional object based on only one image.

[0027] Specifically, the terminal device can use the original image to predict at least two depth images of the target object, back-project the depth images into a three-dimensional voxel space, and use the information of the pixel points in the three-dimensional voxel space to generate a three-dimensional surface of the target object.

[0028] Figure 2 Schematic diagram of an electronic device suitable for implementing the exemplary embodiment of the present disclosure is shown. The terminal device of the exemplary embodiment of the present disclosure may be configured as follows Figure 2 It should be noted that Figure 2 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0029] The electronic device of the present disclosure includes at least a processor and a memory, and the memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the three-dimensional object generation method of the exemplary embodiment of the present disclosure.

[0030] Specifically, Figure 2 As shown, the electronic device 200 may include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a Subscriber Identification Module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor, a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0031] It is to be understood that the structure illustrated in the embodiment of the present disclosure does not constitute a specific limitation on the electronic device 200. In other embodiments of the present disclosure, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0032] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors. In addition, a memory may be provided in the processor 210 for storing instructions and data.

[0033] The electronic device 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0034] The electronic device 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor. In some embodiments, the electronic device 200 may include 1 or N camera modules 291, where N is a positive integer greater than 1. If the electronic device 200 includes N cameras, one of the N cameras is a main camera.

[0035] The internal memory 221 can be used to store computer executable program codes, which include instructions. The internal memory 221 can include a program storage area and a data storage area. The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200.

[0036] The present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device.

[0037] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0038] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0039] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments.

[0040] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0041] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0042] Figure 3 The flowchart of the method for generating a three-dimensional object according to an exemplary embodiment of the present disclosure is schematically shown. Figure 3 , the three-dimensional object generation method may include the following steps:

[0043] S32. Obtain an original image, where the original image contains the target object.

[0044] The original image described in the present disclosure may be an RGB image, which may be captured by a camera module equipped with a terminal device. The terminal device may also obtain the original image from other devices or servers.

[0045] The original image may also be an image captured from an image. For example, first, the terminal device may capture image A with the aid of its camera module. Next, the terminal device may determine the image area containing the target object in image A through an image detection algorithm, capture the image area, and obtain the original image of the embodiment of the present disclosure.

[0046] In addition, the present disclosure does not limit the resolution of the original image. For example, the resolution of the original image may be 512×512.

[0047] The embodiments of the present disclosure do not limit the type of target object. Although some embodiments of the present disclosure are described using the human body as an example, it should be understood that in addition to the human body, the target object may also include animals, hands, any specified objects, etc.

[0048] S34. Using the original image, predict at least two depth images of the target object, where the directions of the depth images with respect to the target object are different.

[0049] In an exemplary embodiment of the present disclosure, the different directions of the depth images for the target object can be understood as determining multiple depth images from different angles of the target object. For example, the front, side, and back of the target object all belong to different angles and are different directions of the target object.

[0050] According to some embodiments of the present disclosure, a terminal device may directly predict at least two depth images of a target object using an original image.

[0051] Specifically, the prediction of depth images can be achieved with the help of machine learning models. For example, a multi-channel output depth estimation network can be used to predict multiple depth images. The present disclosure does not limit the structure of the depth estimation network, for example, it can include a DenseNet structure. The characteristic of the DenseNet structure is that each layer can reuse the features of different abstract levels of all previous layers, thereby avoiding repeated learning of similar features and improving the effectiveness of parameters. The links between different layers help the conduction of gradients during training, so that the optimization problem of deep grids can be solved.

[0052] According to some other embodiments of the present disclosure, the terminal device may predict two depth images of the target object by combining the original image and at least one semantic image converted from the original image.

[0053] First, the terminal device may convert the original image into at least one semantic image according to the semantic information contained in the target object.

[0054] It should be noted that different target object types usually contain different semantic information, and the present disclosure does not limit the specific content of the semantic information.

[0055] Taking the human body as the target object as an example, the semantic types may include head, body, upper arm, lower arm, thigh, lower leg, background, etc. That is to say, the original image may be segmented according to the semantic types such as head, body, upper arm, lower arm, thigh, lower leg, background, etc., to obtain the semantic image corresponding to each semantic type.

[0056] For example, a self-correction network trained on the Pascal-Person-Part dataset can be used to segment the original image and generate at least one semantic image, wherein the size of the obtained semantic image is the same as that of the original image.

[0057] Next, the terminal device may predict at least two depth images of the target object using the original image and at least one semantic image.

[0058] Specifically, the terminal device may perform a channel-dimensional splicing operation on the original image and each semantic image, i.e., a channel superposition operation. The spliced ​​image is input into a depth estimation network to predict at least two depth images of the target object through the depth estimation network. As mentioned above, the depth estimation network may include, for example, a DenseNet structure.

[0059] In the process of determining the depth image, adding semantic information can improve the robustness of depth estimation and improve the accuracy of the algorithm.

[0060] Taking the example that the depth image includes the front depth image of the target object and the back depth image of the target object, the training process of the depth estimation network is explained.

[0061] First, a training set may be acquired in advance, where the training set includes an original training image and a front depth image and a back depth image corresponding to the original training image.

[0062] Taking the human body as the target object, a training set can be obtained by rendering a 3D human body model with textures. This training set obtained by synthetic data has low data collection cost and good generalization to the shooting environment.

[0063] Next, the depth estimation network is trained using the training set and the loss function, which includes the front depth loss and the back depth loss.

[0064] Specifically, the loss function L can be used to train the depth estimation network, referring to Formula 1 and Formula 2:

[0065] L=L-part front +w·L back (Formula 1)

[0066]

[0067] Among them, the loss function is built based on the L1 loss function, L-part front and L back They represent the front depth loss and the back depth loss respectively; W is an adjustable hyperparameter used to reconcile the proportion of the two losses; w 1 Contains I weight parameters and represents the loss ratio of I semantic parts, which is an adjustable hyperparameter; is the true value of the depth map, is the predicted value of the depth map.

[0068] Reference below Figure 4 The process of predicting a depth image by combining a semantic image is exemplified.

[0069] First, the original image is processed by the self-correction network to obtain at least one semantic image for the target object, wherein the size of the semantic image is the same as that of the original image, both of which are 512×512.

[0070] Next, the original image and the semantic image can be spliced ​​in the channel dimension and input into the trained depth estimation network, which outputs two channels and predicts the front depth image and the back depth image of the target object at the same time. The sizes of the front depth image and the back depth image can be, for example, 256×256.

[0071] S36. Back-project at least two depth images into a three-dimensional voxel space, and use information of voxel points in the three-dimensional voxel space to generate a three-dimensional surface of the target object.

[0072] It should be noted that the back-projection operation mentioned in the embodiments of the present disclosure is the inverse process of camera imaging projection, which can be achieved through matrix operations of depth images and camera parameters.

[0073] The terminal device can back-project the at least two depth images determined in step S34 into the three-dimensional voxel space. Specifically, it can be implemented by orthogonal projection. Among them, voxel refers to the smallest unit describing the three-dimensional space. Compared with the pixel in the two-dimensional space, the size of the voxel depends on the size of the smallest rectangular bounding box and the number of preset divided voxels. For example, the size of the voxel can be a cube of 0.05m*0.05m*0.05m, and the embodiments of the present disclosure do not make special limitations on this.

[0074] Thus, the terminal device can generate the three-dimensional surface of the target object by using the information of the voxel points in the three-dimensional voxel space.

[0075] First, the terminal device can calculate the distance of the voxel point in the three-dimensional voxel space relative to the contour edge of each depth image to estimate the distance of the voxel point relative to the three-dimensional surface of the target object. It should be noted that this distance is a signed distance, for example, the voxel point is positive inside the contour edge, negative outside the contour edge, and 0 on the contour edge.

[0076] Specifically, the terminal device may determine the minimum distance of the distances between the voxel point and the contour edge of each depth image, and use the minimum distance as the estimated distance between the voxel point and the three-dimensional surface of the target object.

[0077] For example, the signed distance function of the pixel point relative to the contour edge of the depth image may be calculated, as shown in Formula 3 and Formula 4:

[0078] f(X)=min(d z (X,D f ), d z (D b , X)) (Formula 3)

[0079] d z (X, D) = z X -z π(X，D) (Formula 4)

[0080] Where f(X) represents the signed distance function, π represents the orthogonal projection, and z X represents the z-direction component of voxel point X, d z(X, D) represents the signed distance of the voxel point X to the edge of the contour of the depth image D in the z direction.

[0081] Next, the terminal device may generate a three-dimensional surface of the target object according to the distance of each pixel point relative to the three-dimensional surface of the target object.

[0082] Specifically, the terminal device may use the Marching Cube algorithm (MC algorithm for short) to generate the three-dimensional surface of the target object. The input of the Marching Cube algorithm of the disclosed embodiment may be the signed distance corresponding to each voxel point determined above, and the output may be a triangular mesh, and the three-dimensional surface of the target object is obtained based on the triangular mesh.

[0083] The main process includes comparing the distance corresponding to the voxel point with the specified threshold, determining the intersection of the voxel point and the equipotential surface, and then creating a triangular mesh.

[0084] In addition to the Marching Cube algorithm, the present disclosure may also use other three-dimensional isosurface extraction algorithms, which are not limited by the present disclosure.

[0085] Reference below Figure 5 , the three-dimensional object generation method disclosed in the present invention is explained by taking the output of a three-dimensional human body as an example.

[0086] First, the terminal device can obtain an original image containing a human body and segment the human body to obtain multiple semantic images.

[0087] Next, the terminal device splices the original image with multiple semantic images obtained after human body segmentation, and inputs them into the depth estimation network, which outputs the front depth image and the back depth image.

[0088] Then, the depth image is back-projected into the three-dimensional voxel space, and the three-dimensional surface of the human body is generated using the information of the voxel points in the three-dimensional voxel space.

[0089] In addition, after generating the three-dimensional surface of the target object, the terminal device can also predict the color of the three-dimensional surface.

[0090] Specifically, the terminal device can predict the color of the generated three-dimensional surface using the color information of the target object in the original image to determine the color of the three-dimensional surface. For example, the color texture of the target object in the original image can be reconstructed to match the three-dimensional surface to obtain the color of the three-dimensional surface. Subsequently, the three-dimensional surface can be rendered.

[0091] Taking the human body as an example, through the three-dimensional object generation method of the disclosed embodiment, on the one hand, a three-dimensional human body surface with rich details can be reconstructed using only a single RGB image, and content such as clothing wrinkles, facial details and hairstyle can be presented, without relying on hardware equipment such as depth cameras, with low cost and good presentation effect; on the other hand, the solution has strong universality and can realize real-time reasoning on the GPU, taking into account both reconstruction quality and model processing speed, and the reconstruction effect is stable and robust.

[0092] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0093] Furthermore, this example implementation also provides a three-dimensional object generating device.

[0094] Figure 6 FIG. 1 schematically shows a block diagram of a three-dimensional object generating device according to an exemplary embodiment of the present disclosure. Figure 6 According to an exemplary embodiment of the present disclosure, the three-dimensional object generating device 6 may include an image acquiring module 61 , a depth predicting module 63 , and a surface generating module 65 .

[0095] Specifically, the image acquisition module 61 can be used to acquire an original image, which contains a target object; the depth prediction module 63 can be used to use the original image to predict at least two depth images of the target object, and the directions of the depth images with respect to the target object are different; the surface generation module 65 can be used to back-project at least two depth images into a three-dimensional voxel space, and use the information of the voxel points in the three-dimensional voxel space to generate a three-dimensional surface of the target object.

[0096] According to an exemplary embodiment of the present disclosure, the process of generating a three-dimensional surface of a target object by the surface generation module 65 can be configured to perform: calculating the distance of a voxel point in a three-dimensional voxel space relative to the contour edge of each depth image to estimate the distance of the voxel point relative to the three-dimensional surface of the target object; and generating the three-dimensional surface of the target object based on the distance of each voxel point relative to the three-dimensional surface of the target object.

[0097] According to an exemplary embodiment of the present disclosure, the process of estimating the distance of a voxel point relative to the three-dimensional surface of a target object by the surface generation module 65 can be configured to perform: determining the minimum distance among the distances of the voxel point relative to the contour edge of each depth image, and taking the minimum distance as the estimated distance of the voxel point relative to the three-dimensional surface of the target object.

[0098] According to an exemplary embodiment of the present disclosure, the depth prediction module 63 may be configured to perform: converting an original image into at least one semantic image according to semantic information contained in the target object; and predicting at least two depth images of the target object using the original image and at least one semantic image.

[0099] According to an exemplary embodiment of the present disclosure, the depth prediction module 63 can also be configured to perform: performing a channel dimension stitching operation on the original image and at least one semantic image, inputting the image after the stitching operation into a depth estimation network, and predicting at least two depth images of the target object through the depth estimation network.

[0100] According to an exemplary embodiment of the present disclosure, referring to Figure 7 Compared with the three-dimensional object generating device 6 , the three-dimensional object generating device 7 may further include a network training module 71 .

[0101] Specifically, the network training module 71 can be configured to perform: pre-acquiring a training set, the training set including an original training image and a front depth image and a back depth image corresponding to the original training image; training a depth estimation network using the training set and a loss function, the loss function including a front depth loss and a back depth loss.

[0102] According to an exemplary embodiment of the present disclosure, referring to Figure 8 Compared with the three-dimensional object generating device 6, the three-dimensional object generating device 8 may further include a color determining module 81.

[0103] Specifically, the color determination module 81 may be configured to perform: predicting the color of the generated three-dimensional surface using the color information of the target object in the original image to determine the color of the three-dimensional surface.

[0104] Since the functional modules of the three-dimensional object generation device in the embodiment of the present disclosure are the same as those in the above-mentioned method implementation, they will not be described again here.

[0105] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0106] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0107] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0108] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0109] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A three-dimensional object generation method, characterized in that: include: Acquire an original image, wherein the original image contains a target object; Using the original image, predicting at least two depth images of the target object, each of the depth images having different directions with respect to the target object, wherein predicting at least two depth images of the target object comprises: converting the original image into at least one semantic image according to semantic information contained in the target object; performing a channel dimension splicing operation on the original image and the at least one semantic image, inputting the spliced ​​image into a depth estimation network, and predicting at least two depth images of the target object through the depth estimation network; The at least two depth images are back-projected into a three-dimensional voxel space, and the three-dimensional surface of the target object is generated using information of the voxel points in the three-dimensional voxel space, including: calculating the distance of the voxel points in the three-dimensional voxel space relative to the contour edge of each of the depth images to estimate the distance of the voxel points relative to the three-dimensional surface of the target object; and generating the three-dimensional surface of the target object according to the distance of each of the voxel points relative to the three-dimensional surface of the target object.

2. The three-dimensional object generation method according to claim 1, characterized in that: Estimating the distance of the voxel point relative to the three-dimensional surface of the target object includes: The minimum distance among the distances between the voxel point and the contour edge of each of the depth images is determined, and the minimum distance is used as the estimated distance between the voxel point and the three-dimensional surface of the target object.

3. The three-dimensional object generation method according to claim 1, characterized in that: The at least two depth images include a front depth image of the target object and a back depth image of the target object; The three-dimensional object generation method further comprises: Acquire a training set in advance, wherein the training set includes an original training image and a front depth image and a back depth image corresponding to the original training image; The depth estimation network is trained using the training set and a loss function, wherein the loss function includes a front depth loss and a back depth loss.

4. The three-dimensional object generation method according to claim 1, characterized in that: After generating the three-dimensional surface of the target object, the three-dimensional object generation method further includes: The color of the generated three-dimensional surface is predicted using the color information of the target object in the original image to determine the color of the three-dimensional surface.

5. A three-dimensional object generation device, characterized in that: include: An image acquisition module, used to acquire an original image, wherein the original image contains a target object; A depth prediction module, configured to predict at least two depth images of the target object using the original image, wherein the directions of the depth images with respect to the target object are different from each other; A surface generation module, configured to back-project the at least two depth images into a three-dimensional voxel space, and generate a three-dimensional surface of the target object using information of voxel points in the three-dimensional voxel space, including: calculating a distance between a voxel point in the three-dimensional voxel space and a contour edge of each of the depth images to estimate a distance between the voxel point and the three-dimensional surface of the target object; and generating the three-dimensional surface of the target object according to the distance between each of the voxel points and the three-dimensional surface of the target object; The depth prediction module is also used to convert the original image into at least one semantic image according to the semantic information contained in the target object; perform a channel dimension splicing operation on the original image and the at least one semantic image, input the spliced ​​image into a depth estimation network, and predict at least two depth images of the target object through the depth estimation network.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the three-dimensional object generation method according to any one of claims 1 to 4 is implemented.

7. An electronic device, characterized in that: include: processor; A memory for storing one or more programs, which, when executed by the processor, enables the processor to implement the three-dimensional object generation method as described in any one of claims 1 to 4.

Images