Three-dimensional photo generation method and device, computer device and storage medium

Abstract

The application relates to a three-dimensional photo generation method and device, computer equipment and a storage medium. The method comprises the following steps: determining a depth map corresponding to a two-dimensional photo according to a single two-dimensional photo taken for a three-dimensional scene; layering the depth map according to depth values in the depth map to obtain a plurality of layers of sub-depth maps, and correspondingly obtaining a plurality of layers of sub-photos corresponding to the two-dimensional photo; repairing occluded areas in each layer of the plurality of layers of sub-depth maps and the plurality of layers of sub-photos to obtain a plurality of layers of repaired sub-depth maps and a plurality of layers of repaired sub-photos; the occluded area in each layer refers to an area occluded by image content of a previous layer; and generating a three-dimensional photo containing a plurality of perspectives corresponding to the three-dimensional scene according to the plurality of layers of repaired sub-depth maps and the plurality of layers of repaired sub-photos. The method can improve the three-dimensional photo generation efficiency.

Description

Technical Field

[0001] This application relates to the fields of computer vision technology and image processing technology, and in particular to a method, apparatus, computer device and storage medium for generating three-dimensional photographs. Background Technology

[0002] Traditional 2D photography helps us capture images of the world around us, freezing the three-dimensional scenes within a photograph. However, the resulting images are static and cannot dynamically and realistically depict the three-dimensional world. 3D photography, on the other hand, is a new approach that uses parallax caused by viewpoint movement to display the three-dimensional scenes within a captured still photograph from multiple perspectives, bringing the still image to life.

[0003] Traditional methods for generating 3D images typically require capturing static 2D photographs of a 3D scene from multiple angles, and then manually combining these images to create a dynamic 3D image. This method of manually combining images from multiple angles not only requires capturing numerous static photographs but also necessitates complex compositing processes by professionals, making it extremely complicated and inefficient. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the efficiency of 3D photo generation in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for generating three-dimensional photographs. The method includes:

[0006] Determine the depth map corresponding to a single 2D photograph taken for a 3D scene;

[0007] The depth map is layered according to the depth values ​​in the depth map to obtain multiple sub-depth maps, and the corresponding multiple sub-photographs are obtained for the two-dimensional photo.

[0008] The occluded areas in each layer of the multi-layer sub-depth map and the multi-layer sub-photograph are repaired to obtain a multi-layer repaired sub-depth map and a multi-layer repaired sub-photograph; the occluded area in each layer refers to the area that is occluded by the image content of the previous layer;

[0009] Based on the multi-layer repair sub-depth map and the multi-layer repair sub-photograph, a 3D photo containing multiple perspectives corresponding to the 3D scene is generated.

[0010] Secondly, this application also provides a three-dimensional photograph generation apparatus. The apparatus includes:

[0011] The depth map determination module is used to determine the depth map corresponding to a single two-dimensional photograph taken for a three-dimensional scene.

[0012] The image layering module is used to layer the depth map according to the depth values ​​in the depth map to obtain multiple sub-depth maps, and correspondingly obtain multiple sub-photographs corresponding to the two-dimensional photo.

[0013] The image restoration module is used to restore the occluded areas in each layer of the multi-layer sub-depth map and the multi-layer sub-photograph to obtain a multi-layer restored sub-depth map and a multi-layer restored sub-photograph; the occluded area in each layer refers to the area that is occluded by the image content of the previous layer;

[0014] The 3D photo generation module is used to generate a 3D photo containing multiple perspectives corresponding to the 3D scene based on the multi-layer repair sub-depth map and the multi-layer repair sub-photo.

[0015] In one embodiment, the 3D photo generation module is further configured to perform 3D reconstruction based on the multi-layer repair sub-depth map and the multi-layer repair sub-photo to obtain a 3D point cloud under the original viewpoint corresponding to the 3D scene; the original viewpoint is the shooting viewpoint of the 2D photo; generate 3D point clouds under multiple viewpoints corresponding to the 3D scene based on the 3D point cloud under the original viewpoint; render and project the 3D point clouds under each viewpoint to obtain multiple frames of images; and generate a 3D photo containing multiple viewpoints corresponding to the 3D scene based on the multiple frames of images.

[0016] In one embodiment, the 3D photo generation module is further configured to determine the number of frames of the 3D photo based on the duration set for the 3D photo to be generated; determine multiple sets of viewpoint transformation parameters based on the range of viewpoint transformation parameters set for the 3D photo to be generated and the number of frames; and perform viewpoint transformation on the 3D point cloud under the original viewpoint according to each set of viewpoint transformation parameters to generate 3D point clouds under multiple viewpoints corresponding to the 3D scene.

[0017] In one embodiment, the 3D photo generation module is further configured to perform object segmentation processing on the 2D photo to obtain an object segmentation image and corresponding object semantic labels; if a target object semantic label exists in the object semantic labels, then semantic 3D reconstruction is performed based on the image content corresponding to the target object semantic label in the object segmentation image and the 2D photo to generate a 3D point cloud of the target object corresponding to the target object semantic label; 3D reconstruction is performed based on the multi-layer repair sub-depth map and the multi-layer repair sub-photo to obtain an initial 3D point cloud under the original viewpoint corresponding to the 3D scene; and a 3D point cloud under the original viewpoint corresponding to the 3D scene is obtained based on the initial 3D point cloud and the 3D point cloud of the target object.

[0018] In one embodiment, the image layering module is further configured to determine depth layer boundaries from the depth map; the depth layer boundaries are the locations where depth values ​​change abruptly in the depth map; based on the depth layer boundaries, the depth map is divided into multiple regions; the center of each region is used as a cluster center, and each pixel in the depth map is clustered according to the depth values ​​in the depth map to obtain multi-layer sub-depth maps; wherein each sub-depth map is determined based on each pixel in one of the clusters obtained by clustering; based on the layering of the multi-layer sub-depth maps, multi-layer sub-photographs corresponding to the two-dimensional photograph are obtained.

[0019] In one embodiment, the image layering module is further configured to perform edge detection on the object segmentation image corresponding to the two-dimensional image and the depth map respectively to obtain a first boundary image and a second boundary image; and to fuse the first boundary image and the second boundary image to obtain a depth layering boundary.

[0020] In one embodiment, the image restoration module is further configured to, for each layer of sub-depth map, restore the image content of the occluded area in the sub-depth map of the current layer based on the image content of the unoccluded area in the sub-depth map of the current layer, to obtain a multi-layer restored sub-depth map; and for each layer of sub-photograph, restore the image content of the occluded area in the sub-photograph of the current layer based on the image content of the unoccluded area in the sub-photograph of the current layer, to obtain a multi-layer restored sub-photograph.

[0021] In one embodiment, the image inpainting module is further configured to, for each layer of sub-depth map, repair the image content of the occluded area in the sub-depth map of the current layer based on the image content of the unoccluded area in the sub-depth map of the current layer, to obtain multi-layer initial repaired sub-depth maps; adjust the depth values ​​in the initial repaired sub-depth maps of each layer so that the depth value changes smoothly between the initial repaired sub-depth maps of each layer and the depth values ​​in the initial repaired sub-depth maps of each layer conform to the normal distance relationship, to obtain multi-layer repaired sub-depth maps.

[0022] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps in the three-dimensional photograph generation method described in the embodiments of this application.

[0023] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, causes the processor to perform the steps in the three-dimensional photograph generation method described in the embodiments of this application.

[0024] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the steps of the three-dimensional photograph generation method described in the embodiments of this application.

[0025] The aforementioned 3D photo generation method, apparatus, computer equipment, storage medium, and computer program product determine the corresponding depth map of a single 2D photo taken for a 3D scene. Based on the depth values ​​in the depth map, the depth map is layered to obtain multiple sub-depth maps, and correspondingly, multiple sub-photos of the 2D photo are obtained. This achieves the layering of the depth map and 2D photo according to the magnitude of the depth values, dividing them into multiple images with different spatial positions. Occlusion areas in each layer of the multi-layer sub-depth maps and multi-layer sub-photos are repaired to obtain multi-layer repaired sub-depth maps and multi-layer repaired sub-photos. This allows for the repair of areas occluded by previous layers in each layer, ensuring relatively complete image content in each layer. Based on the relatively complete multi-layer repaired sub-depth maps and multi-layer repaired sub-photos, a 3D photo containing multiple perspectives corresponding to the 3D scene is generated. This ensures that the generated 3D photo has relatively complete image content from each perspective. Therefore, only a single 2D photo needs to be taken to automatically generate a complete, high-quality 3D photo containing multiple perspectives, eliminating the need to take multiple 2D photos, perform complex compositing based on multiple 2D photos manually, or use complex shooting equipment, thus improving the efficiency of 3D photo generation. Attached Figure Description

[0026] Figure 1 This is an application environment diagram of a 3D photo generation method in one embodiment;

[0027] Figure 2 This is a flowchart illustrating a method for generating 3D photographs in one embodiment;

[0028] Figure 3(a) is a schematic diagram of a two-dimensional photograph in one embodiment;

[0029] Figure 3(b) is a schematic diagram of the object segmentation image corresponding to a two-dimensional photograph in one embodiment;

[0030] Figure 3(c) is a schematic diagram of the depth map corresponding to a two-dimensional photograph in one embodiment;

[0031] Figures 4(a), 4(b), and 4(c) are images from multiple perspectives obtained by front-to-back translational camera movement in a three-dimensional photograph in one embodiment;

[0032] Figures 5(a), 5(b), and 5(c) are images from multiple perspectives obtained by vertical camera rotation in a three-dimensional photograph in one embodiment.

[0033] Figure 6 This is a schematic diagram of the overall process of a three-dimensional photo generation method in one embodiment;

[0034] Figure 7 This is a schematic diagram of the image layering process in one embodiment;

[0035] Figure 8 This is a schematic diagram of the image restoration process in one embodiment;

[0036] Figure 9 This is a structural block diagram of a three-dimensional photo generation device in one embodiment;

[0037] Figure 10 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] In one embodiment, the 3D photo generation method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on the cloud or other network servers. Terminal 102 can acquire a single 2D photograph of a 3D scene. The 2D photograph can be taken by terminal 102 or stored on terminal 102. Terminal 102 can send the 2D photograph to server 104. Server 104 can execute the 3D photograph generation method in the embodiments of this application to obtain a 3D photograph and send the 3D photograph to terminal 102 for display. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0040] In other embodiments, the three-dimensional photo generation method of each embodiment of this application can also be executed locally by the terminal 102 without sending it to the server for execution.

[0041] In one embodiment, such as Figure 2 As shown, a method for generating three-dimensional photographs is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0042] Step 202: Determine the depth map corresponding to the single 2D photograph taken for the 3D scene.

[0043] In this context, a 3D scene refers to a 3D scene captured in a 2D photograph. A 2D photograph is a static image that freezes a 3D scene from a single perspective through photography. A depth map is an image used to represent the depth values ​​at various locations within a 2D photograph. There is a one-to-one correspondence between the pixels in a 2D photograph and the corresponding pixels in the depth map. Each pixel in the depth map has a corresponding depth value. The depth value of each pixel in the depth map corresponding to a 2D photograph represents the pixel value of the pixel corresponding to that pixel in the 2D photograph. Figure 3(a) shows a 2D photograph, and Figure 3(c) shows the depth map corresponding to that 2D photograph.

[0044] In one embodiment, the two-dimensional photograph can be any of the following: a color image or a grayscale image.

[0045] In one embodiment, the terminal can send a single 2D photo to the server, and the server can determine the depth map corresponding to the 2D photo based on the single 2D photo taken for a 3D scene.

[0046] In one embodiment, the server can perform monocular depth estimation on a single 2D photograph to obtain a depth map corresponding to the 2D photograph.

[0047] In one embodiment, the server can use a monocular depth estimation model to perform monocular depth estimation on a single 2D photograph to obtain a depth map corresponding to the 2D photograph.

[0048] Step 204: The depth map is layered according to the depth values ​​in the depth map to obtain multi-layer sub-depth maps, and the corresponding multi-layer sub-photographs are obtained from the two-dimensional photo.

[0049] In this context, layering refers to dividing an image according to the spatial position of its content. A sub-depth map is the image corresponding to each layer obtained by layering the depth map. A sub-photograph is the image corresponding to each layer obtained by layering a 2D photograph.

[0050] In one embodiment, the server can determine depth layer boundaries from the depth map, and then layer the depth map according to these boundaries to obtain multiple sub-depth maps. The depth layer boundaries are the locations where depth values ​​abruptly change in the depth map.

[0051] In another embodiment, the server can determine the depth layer boundaries from the depth map, divide the depth map into multiple regions based on the depth layer boundaries, use the center of each region as the cluster center, and cluster each pixel in the depth map based on the depth value in the depth map to obtain a multi-layer sub-depth map.

[0052] In one embodiment, the server can layer a 2D image based on the layering of the multi-layer sub-depth map to obtain multi-layer sub-images corresponding to the 2D image. Specifically, the server can map the splitting boundaries used for layering the depth map to the 2D image, and then layer the 2D image according to the splitting boundaries to obtain multi-layer sub-images.

[0053] In one embodiment, when stratification is performed directly based on depth stratification boundaries, the split boundary is the depth stratification boundary. In another embodiment, when stratification is performed through clustering, the split boundary is the boundary between the individual clusters obtained through clustering.

[0054] In one embodiment, the server can split the depth map according to the different front and back positions of the image content in space, and fill in the image content of each split part according to the size of the depth map. The filled part has no image content, resulting in a multi-layer sub-depth map with the same size.

[0055] In one embodiment, the server can map the splitting boundary used to layer the depth map to a two-dimensional photo, split the two-dimensional photo according to the splitting boundary, and fill in the image content of each split part according to the size of the depth map. The filled part has no image content, resulting in a multi-layered sub-photograph of the same size.

[0056] Step 206: Repair the occluded areas in each layer of the multi-layer sub-depth map and multi-layer sub-photograph to obtain the multi-layer repaired sub-depth map and multi-layer repaired sub-photograph; the occluded area in each layer refers to the area that is occluded by the image content of the previous layer.

[0057] In this context, "preceding layer" refers to the layer that precedes the current layer in spatial location. "Inpainting" refers to filling in occluded areas with image content. "Inpainted sub-depth map" refers to the image obtained by inpainting a sub-depth map. "Inpainted sub-photograph" refers to the image obtained by inpainting a sub-photograph. The occluded area in each layer corresponds to the area in the current layer where the image content of the preceding layer is located.

[0058] In one embodiment, for each layer, the server can repair the image content of the occluded area in the layer based on the image content of the unoccluded area in the layer.

[0059] In one embodiment, the unoccluded area in each layer can be an area containing image content in that layer, and the occluded area in each layer can be an area without image content in that layer.

[0060] Step 208: Based on the multi-layer repair sub-depth map and multi-layer repair sub-photograph, generate a 3D photo containing multiple perspectives corresponding to the 3D scene.

[0061] Here, perspective refers to the different camera angles and positions used to capture a 3D scene. A 3D photograph is a dynamic photograph that can show a 3D scene from multiple perspectives.

[0062] Figures 4(a), 4(b), and 4(c) show images from different perspectives in a 3D photograph generated from the same 2D image. Each perspective was obtained by panning the camera back and forth. Similarly, Figures 5(a), 5(b), and 5(c) show images from different perspectives in a 3D photograph generated from the same 2D image. Each perspective was obtained by rotating the camera up and down. As can be seen from Figures 4(a), 4(b), 4(c), 5(a), 5(b), and 5(c), the resulting 3D photographs contain relatively complete and high-quality perspectives.

[0063] In one embodiment, the server can perform 3D reconstruction based on multi-layer repair sub-depth maps and multi-layer repair sub-photographs to obtain 3D point clouds from the original viewpoint of the 3D scene. Then, based on the 3D point clouds from the original viewpoint, it can generate 3D point clouds from multiple viewpoints corresponding to the 3D scene. Finally, based on the 3D point clouds from each viewpoint, it can generate 3D photographs from multiple viewpoints corresponding to the 3D scene.

[0064] In one embodiment, the server can obtain multi-frame images from multiple perspectives based on multi-layer repair sub-depth maps and multi-layer repair sub-photographs, and generate a 3D photograph of the 3D scene containing multiple perspectives based on the multi-frame images.

[0065] In one embodiment, the 3D photograph can be any of a video file, an animated image file, or an interactive file. An interactive file refers to a file in which the user can interact with the image and change the perspective of the image.

[0066] The aforementioned 3D photo generation method determines the corresponding depth map of a single 2D photo taken for a 3D scene. Based on the depth values ​​in the depth map, it layers the depth map to obtain multiple sub-depth maps, and correspondingly generates multiple sub-photos of the 2D photo. This achieves the layering of the depth map and 2D photo according to their depth values, creating multiple layers of images with different spatial positions. Occlusion areas in each layer of the sub-depth maps and sub-photos are repaired, resulting in multi-layer repaired sub-depth maps and multi-layer repaired sub-photos. This repairs areas occluded by previous layers, ensuring relatively complete image content in each layer. Based on these relatively complete multi-layer repaired sub-depth maps and multi-layer repaired sub-photos, a 3D photo containing multiple perspectives corresponding to the 3D scene is generated. This ensures that the generated 3D photo has relatively complete image content from each perspective. Therefore, only a single 2D photo is needed to automatically generate a complete, high-quality 3D photo containing multiple perspectives, eliminating the need for taking multiple 2D photos, manual compositing of multiple 2D photos, and complex shooting equipment, thus improving the efficiency and quality of 3D photo generation.

[0067] In one embodiment, generating a 3D photograph containing multiple viewpoints corresponding to a 3D scene based on multi-layer repair sub-depth maps and multi-layer repair sub-photographs includes: performing 3D reconstruction based on multi-layer repair sub-depth maps and multi-layer repair sub-photographs to obtain a 3D point cloud from the original viewpoint corresponding to the 3D scene; the original viewpoint is the shooting viewpoint of the 2D photograph; generating 3D point clouds from multiple viewpoints corresponding to the 3D scene based on the 3D point clouds from the original viewpoint; rendering and projecting the 3D point clouds from each viewpoint to obtain multiple frames of images; and generating a 3D photograph containing multiple viewpoints corresponding to the 3D scene based on the multiple frames of images.

[0068] In one embodiment, the server can obtain the spatial coordinates of the 3D point cloud based on the multi-layer repair sub-depth map, and then, based on the correspondence between the pixels of the multi-layer repair sub-deep map and the multi-layer repair sub-photo, map the pixel values ​​of each pixel in the multi-layer repair sub-photo to the spatial coordinates to obtain the 3D point cloud under the original viewpoint corresponding to the 3D scene.

[0069] In one embodiment, the server can obtain the spatial coordinates of a 3D point cloud from a multi-layer repair sub-depth map according to the following formula:

[0070]

[0071] Where u and v represent the coordinates of pixels in the repaired sub-depth map. cThis represents the depth value of a pixel in the repaired sub-depth map. u0 and v0 represent the center coordinates of the repaired sub-depth map. f represents the camera's focal length. R represents the camera's rotation matrix. T represents the camera's translation matrix. x ω y ω and z ω Represents the spatial coordinates of a 3D point cloud.

[0072] In one embodiment, the rotation matrix R and translation matrix T used to obtain the 3D point cloud from the original viewpoint can be matrices in which all elements are 0.

[0073] In one embodiment, the server can filter the 3D point cloud from the original viewpoint to obtain a filtered 3D point cloud, and then use the filtered 3D point cloud to obtain 3D point clouds from multiple viewpoints. In one embodiment, the filtering process can be median filtering. In this embodiment, filtering can remove invalid flying points in the 3D point cloud caused by inconsistent boundary depth values, making the 3D point cloud more accurate and of higher quality.

[0074] In one embodiment, the server can determine multiple sets of view transformation parameters, and perform view transformation on the 3D point cloud under the original view according to each set of view transformation parameters to generate 3D point clouds under multiple view corresponding to the 3D scene. Here, view transformation is the process of transforming the 3D point cloud to different view perspectives.

[0075] In one embodiment, each set of view transformation parameters may include at least one of a rotation matrix and a translation matrix.

[0076] In one embodiment, the server can render and project the 3D point cloud for each viewpoint to obtain a 2D image, thereby obtaining multiple frames of images corresponding to each viewpoint.

[0077] In one embodiment, the server can generate video files or animated GIF files based on multiple frames of images. The video file or animated GIF file is a 3D photograph containing multiple viewpoints corresponding to a 3D scene. In one embodiment, the server can use graphics libraries such as OpenCV to encode the multiple frames of images to generate video files or animated GIF files.

[0078] In another embodiment, the server can generate an operable file based on multiple frames of images, and the user can interact with the screen of the operable file to switch the viewpoint of the screen in the operable file.

[0079] like Figure 6The diagram shows the overall flow of the 3D photo generation method in the above embodiment. First, image layering is performed, that is, the depth map and the 2D photo are layered separately to obtain multi-layer sub-depth maps and multi-layer sub-photos. Then, image inpainting is performed, that is, each layer of sub-depth map and each layer of sub-photo is repaired to obtain multi-layer repaired sub-depth maps and multi-layer repaired sub-photos. Next, 3D reconstruction is performed based on the multi-layer repaired sub-depth maps and multi-layer repaired sub-photos to obtain 3D point clouds from the original viewpoint. Then, viewpoint rendering is performed, that is, multiple viewpoint 3D point clouds are generated based on the 3D point clouds from the original viewpoint, and rendering and projection are performed based on the 3D point clouds from each viewpoint to obtain multiple frames of images. Finally, a 3D photo is generated based on the multiple frames of images.

[0080] In the above embodiments, 3D reconstruction is performed based on multi-layer repair sub-depth maps and multi-layer repair sub-photographs, and viewpoint transformation is performed to obtain relatively complete and high-quality 3D point clouds from multiple viewpoints. Then, rendering and projection are performed based on these relatively complete and high-quality 3D point clouds from multiple viewpoints to obtain multi-frame images. This improves the generation speed and quality of multi-frame images and avoids the problem of incomplete 3D point clouds and holes in 3D reconstruction directly from depth maps and 2D photographs, which leads to slow rendering speed and poor image quality. Therefore, the generation efficiency and quality of 3D photographs are improved. Because each viewpoint has a complete 3D point cloud, it can support large-angle camera movements, ensuring good consistency and stability even under large-angle camera movements, avoiding the problems of cutouts or distortions during camera movements, and further improving the quality of 3D photographs.

[0081] In one embodiment, generating 3D point clouds from multiple perspectives corresponding to a 3D scene based on the 3D point cloud from the original perspective includes: determining the number of frames of the 3D photo based on the duration set for the 3D photo to be generated; determining multiple sets of perspective transformation parameters based on the range of perspective transformation parameters and the number of frames set for the 3D photo to be generated; and performing perspective transformation on the 3D point cloud from the original perspective according to each set of perspective transformation parameters to generate 3D point clouds from multiple perspectives corresponding to the 3D scene.

[0082] In one embodiment, the number of frames is positively correlated with the duration. That is, the longer the duration, the higher the number of frames, and the shorter the duration, the lower the number of frames.

[0083] In one embodiment, the duration set for the 3D image to be generated can be a pre-set uniform duration for the 3D image. In another embodiment, the duration set for the 3D image to be generated can be specified by the user via a terminal.

[0084] In one embodiment, the range of perspective transformation parameters set for the 3D image to be generated can be pre-set uniformly for the 3D image. In another embodiment, the range of perspective transformation parameters set for the 3D image to be generated can be determined based on a user-specified perspective transformation range.

[0085] In one embodiment, the viewpoint transformation range may include at least one of a rotation angle range and a translation distance range.

[0086] In one embodiment, the server can determine the rotation angle and translation distance corresponding to each frame image based on the rotation angle range, translation distance range, and frame number, and determine multiple sets of rotation matrices and translation matrices based on the rotation angle and translation distance corresponding to each frame image.

[0087] In one embodiment, the server can perform perspective transformation on the 3D point cloud from the original viewpoint based on each set of rotation and translation matrices to generate 3D point clouds from multiple viewpoints corresponding to the 3D scene.

[0088] In the above embodiments, the number of frames of the 3D photo is determined according to the duration set for the 3D photo to be generated. Multiple sets of perspective transformation parameters are determined according to the range of change of perspective transformation parameters and the number of frames set for the 3D photo to be generated. Perspective transformation is performed on the 3D point cloud under the original perspective to generate 3D point clouds under multiple perspectives corresponding to the 3D scene. This enables the generation of complete 3D point clouds under multiple perspectives. Because each perspective can have a complete 3D point cloud, it can support large-angle camera movement, avoid the problem of hollowing or distortion during camera movement, and improve the quality of the 3D photo.

[0089] In one embodiment, object segmentation is performed on a two-dimensional photograph to obtain an object segmentation image and corresponding object semantic labels. If a target object semantic label exists in the object semantic labels, semantic 3D reconstruction is performed based on the image content corresponding to the target object semantic label in the object segmentation image and the two-dimensional photograph to generate a 3D point cloud of the target object corresponding to the target object semantic label. 3D reconstruction is performed based on multi-layer restoration sub-depth maps and multi-layer restoration sub-photographs to obtain a 3D point cloud of the original viewpoint corresponding to the 3D scene, including: 3D reconstruction is performed based on multi-layer restoration sub-depth maps and multi-layer restoration sub-photographs to obtain an initial 3D point cloud of the original viewpoint corresponding to the 3D scene; and the 3D point cloud of the target object is obtained based on the initial 3D point cloud and the 3D point cloud of the target object.

[0090] Object segmentation is the process of separating objects from a two-dimensional photograph. Object semantic labels are the semantic labels corresponding to each object in the object segmentation image obtained from the object segmentation process. Figure 3(b) shows the object segmentation image obtained by performing object segmentation on the two-dimensional photograph shown in Figure 3(a).

[0091] In one embodiment, object segmentation can be achieved using either a panoramic segmentation algorithm or an instance segmentation algorithm.

[0092] In one embodiment, the target object semantic tag can be a target object semantic tag that matches a preset semantic tag among object semantic tags.

[0093] In one embodiment, the preset semantic tags can be semantic tags from a semantic 3D reconstruction library.

[0094] In one embodiment, the preset semantic tag can be a semantic tag for common objects, such as people, cars, or trees. That is, the semantic tag for the target object is also a semantic tag for common objects, and the target object corresponding to the semantic tag for the target object is a common object, such as people, cars, or trees.

[0095] In one embodiment, the server can match each object semantic tag with each preset semantic tag to determine whether there is a target object semantic tag that matches the preset semantic tag among the object semantic tags.

[0096] In one embodiment, if the object semantic label contains a target object semantic label, the server can input the image content and two-dimensional photo corresponding to the target object semantic label in the object segmentation image into the semantic three-dimensional reconstruction model, and output the three-dimensional point cloud of the target object corresponding to the target object semantic label.

[0097] In one embodiment, the target object semantic label and the corresponding target object can be one or more. When there are multiple target object semantic labels and corresponding target objects, the server can perform semantic 3D reconstruction based on the image content and 2D photograph corresponding to each target object in the object segmentation image, generating 3D point clouds of the target objects corresponding to each target object semantic label.

[0098] In one embodiment, the server can update the initial 3D point cloud based on the 3D point cloud of the target object to obtain the 3D point cloud from the original viewpoint corresponding to the 3D scene.

[0099] In one embodiment, the server can perform 3D reconstruction based on the image content in the multi-layer repair sub-depth map and multi-layer repair sub-photograph, excluding the image content corresponding to the target object, to obtain an initial 3D point cloud under the original viewpoint corresponding to the 3D scene. Then, the 3D point cloud of the target object is filled into the missing part corresponding to the target object in the initial 3D point cloud to obtain the 3D point cloud under the original viewpoint corresponding to the 3D scene.

[0100] In another embodiment, the server can perform 3D reconstruction based on all image content in the multi-layer repair sub-depth map and multi-layer repair sub-photograph to obtain an initial 3D point cloud under the original viewpoint corresponding to the 3D scene. Then, the 3D point cloud of the target object is used to replace the point cloud corresponding to the target object in the initial 3D point cloud to obtain a 3D point cloud under the original viewpoint corresponding to the 3D scene.

[0101] In one embodiment, when there are multiple target objects and semantic labels for the target objects, the server can update the initial 3D point cloud sequentially based on the 3D point clouds of each target object to obtain the 3D point cloud from the original viewpoint corresponding to the 3D scene.

[0102] In one embodiment, the server can first normalize the 3D point cloud of the target object so that the normalized 3D point cloud of the target object is at the same scale as the initial 3D point cloud under the original viewpoint corresponding to the 3D scene. Then, based on the normalized 3D point cloud of the target object and the initial 3D point cloud, the server generates the 3D point cloud under the original viewpoint corresponding to the 3D scene.

[0103] In the above embodiments, semantic 3D reconstruction is used for common target objects, which can obtain complete and high-quality 3D point clouds of the target objects. For parts of the 2D photo other than the target objects, such as large areas like the ground or sky, 3D reconstruction is performed based on multi-layer restoration sub-depth maps and multi-layer restoration sub-photographs, which can also obtain relatively complete 3D point clouds of the ground or sky, improving the completeness and quality of the final generated 3D point cloud. Based on complete and high-quality 3D point clouds, 3D photos can be generated efficiently and with high quality.

[0104] In one embodiment, layering a depth map based on depth values ​​in the depth map to obtain multiple sub-depth maps, and correspondingly obtaining multiple sub-photographs of a two-dimensional photograph, includes: determining depth layer boundaries from the depth map; the depth layer boundaries are the locations where depth values ​​abruptly change in the depth map; dividing the depth map into multiple regions based on the depth layer boundaries; using the center of each region as a cluster center, clustering each pixel in the depth map based on the depth values ​​in the depth map to obtain multiple sub-depth maps; wherein each sub-depth map is determined based on the pixels in one of the clusters obtained from the clustering; and obtaining multiple sub-photographs of a two-dimensional photograph based on the layering of the multiple sub-depth maps.

[0105] In one embodiment, the server can perform edge detection on the depth map to obtain a second boundary image, and use the edges in the second boundary image as depth layer boundaries.

[0106] In another embodiment, the server can perform object segmentation processing on the two-dimensional photo to obtain an object segmentation image, and then perform edge detection on the object segmentation image and the depth map respectively to obtain a first boundary image and a second boundary image. The first boundary image and the second boundary image are then fused to obtain a depth layer boundary.

[0107] In one embodiment, the server can use the center of each region as the cluster center, and cluster each pixel in the depth map according to the depth value in the depth map to obtain multiple clusters, and obtain multi-layer sub-depth maps according to the pixels in each cluster.

[0108] In one embodiment, the server can pad each cluster according to the size of the depth map, with the padded parts containing no image content, to obtain multi-layer sub-depth maps of consistent size.

[0109] In the above embodiments, depth layer boundaries are determined from the depth map. Based on the depth layer boundaries, the depth map is divided into multiple regions, and the center of each region is used as the cluster center. Based on the depth values ​​in the depth map, each pixel in the depth map is clustered to obtain multi-layer sub-depth maps. Based on the layering of the multi-layer sub-depth maps, multi-layer sub-photographs corresponding to the two-dimensional photos are obtained. This allows for the accurate division of the depth map and the two-dimensional photos into multi-layer images with different spatial positions. Furthermore, it enables the repair of each layer in the multi-layer images, ultimately resulting in a complete, high-quality three-dimensional photo. This achieves the automatic generation of a complete three-dimensional photo containing multiple perspectives by only taking a single two-dimensional photo, without the need to take multiple two-dimensional photos or perform complex compositing based on multiple two-dimensional photos manually, thus improving the efficiency of three-dimensional photo generation.

[0110] In one embodiment, determining the depth layer boundary from the depth map includes: performing edge detection on the object segmentation image and the depth map corresponding to the two-dimensional image respectively to obtain a first boundary image and a second boundary image; and fusing the first boundary image and the second boundary image to obtain the depth layer boundary.

[0111] The first boundary image is obtained by performing edge detection on the object segmentation image corresponding to the two-dimensional image. The second boundary image is obtained by performing edge detection on the depth map.

[0112] In one embodiment, the server can perform edge detection using edge detection operators such as the Sobel operator (an edge detection operator) or the Canny operator (an edge detection operator).

[0113] In one embodiment, the server can perform object segmentation processing on the two-dimensional photo to obtain an object segmentation image, and then perform edge detection on the object segmentation image and the depth map corresponding to the two-dimensional image to obtain a first boundary image and a second boundary image.

[0114] In one embodiment, object segmentation can be achieved using either a panoramic segmentation algorithm or an instance segmentation algorithm.

[0115] In one embodiment, the server can fuse the first boundary image and the second boundary image to obtain a fused image, and use the edges in the fused image as depth layer boundaries.

[0116] In one embodiment, fusing the first boundary image and the second boundary image can be achieved by superimposing the pixel values ​​of corresponding pixels in the first boundary image and the second boundary image. In another embodiment, fusing the first boundary image and the second boundary image can be achieved by averaging the pixel values ​​of corresponding pixels in the first boundary image and the second boundary image.

[0117] In the above embodiments, edge detection is performed on the object segmentation image and depth map corresponding to the two-dimensional image to obtain a first boundary image and a second boundary image. By fusing the first boundary image and the second boundary image, a more accurate depth layer boundary can be obtained.

[0118] like Figure 7The diagram illustrates the process of layering depth maps and 2D photographs in the above embodiment. First, a 2D photograph is input. Depth estimation and object segmentation are performed on the 2D photograph to obtain a corresponding depth map and object segmentation image. Then, edge detection is performed on the object segmentation image and depth map to obtain a first boundary image and a second boundary image. The first boundary image and the second boundary image are then fused to obtain depth layer boundaries. Based on the depth layer boundaries, cluster centers are determined to cluster and layer the depth map, resulting in multi-layer sub-depth maps and corresponding multi-layer sub-photographs.

[0119] In one embodiment, repairing occluded areas in each layer of a multi-layer sub-depth map and a multi-layer sub-photograph to obtain a multi-layer repaired sub-depth map and a multi-layer repaired sub-photograph includes: for each layer's sub-depth map, repairing the image content of occluded areas in the sub-depth map based on the image content of unoccluded areas in the sub-depth map of that layer to obtain a multi-layer repaired sub-depth map; for each layer's sub-photograph, repairing the image content of occluded areas in the sub-photograph based on the image content of unoccluded areas in the sub-photograph of that layer to obtain a multi-layer repaired sub-photograph.

[0120] In one embodiment, the server can, for each layer's sub-depth map, fill in the image content of the occluded areas in the sub-depth map based on the image content of the unoccluded areas in the sub-depth map of that layer, thus obtaining a multi-layer repaired sub-depth map. Similarly, for each layer's sub-image, the server can fill in the image content of the occluded areas in the sub-image based on the image content of the unoccluded areas in the sub-image of that layer, thus obtaining a multi-layer repaired sub-image.

[0121] In one embodiment, the server can input the sub-depth map of each layer into the depth map restoration model. The depth map restoration model restores the image content of the occluded areas in the sub-depth map of the current layer based on the image content of the unoccluded areas in the sub-depth map of the current layer, and outputs the restored sub-depth map of each layer.

[0122] The depth map repair model is a model used to repair sub-depth maps.

[0123] In one embodiment, the depth map restoration model can be a machine learning model.

[0124] In one embodiment, the depth map inpainting model can be a deep learning model or other types of machine learning models, without limitation.

[0125] In one embodiment, the depth map restoration model can be obtained in advance through a depth map restoration model training step.

[0126] In one embodiment, the depth map inpainting model training step may include: inputting sample sub-depth maps and sample inpainting sub-depth maps into the depth map inpainting model to be trained; using the depth map inpainting model to be trained to inpaint the image content of occluded areas (i.e., areas without image content in the sample sub-depth maps) based on the image content of unoccluded areas (i.e., areas with image content in the sample sub-depth maps), to obtain predicted inpainting sub-depth maps; and iteratively adjusting the model parameters of the depth map inpainting model to be trained based on the difference between the predicted inpainting sub-depth maps and the sample inpainting sub-depth maps until the iteration stopping condition is met, to obtain the trained depth map inpainting model.

[0127] In one embodiment, the server can input the sub-photos of each layer into the photo restoration model, and the photo restoration model can restore the image content of the occluded areas in the sub-photos of the current layer based on the image content of the unoccluded areas in the sub-photos of the current layer, and output the restored sub-photos of each layer.

[0128] Among them, the photo restoration model is a model used to restore sub-photographs.

[0129] In one embodiment, the photo restoration model can be a machine learning model.

[0130] In one embodiment, the photo restoration model can be a deep learning model or other types of machine learning models, without limitation.

[0131] In one embodiment, the photo restoration model can be obtained in advance through a photo restoration model training step.

[0132] In one embodiment, the photo restoration model training step may include: inputting sample sub-photos and sample restoration sub-photos into the photo restoration model to be trained; using the photo restoration model to be trained to restore the image content of occluded areas (i.e., areas without image content in the sample sub-photos) based on the image content of unoccluded areas (i.e., areas with image content in the sample sub-photos) in the sample sub-photos, obtaining predicted restoration sub-photos; and iteratively adjusting the model parameters of the photo restoration model to be trained based on the difference between the predicted restoration sub-photos and the sample restoration sub-photos, until the iteration stopping condition is met, thus obtaining the trained photo restoration model.

[0133] In the above embodiments, for each layer's sub-depth map, the image content of the occluded areas in the sub-depth map of this layer is repaired based on the image content of the unoccluded areas in the sub-depth map of this layer, resulting in a multi-layer repaired sub-depth map. For each layer's sub-photograph, the image content of the occluded areas in the sub-photograph of this layer is repaired based on the image content of the unoccluded areas in the sub-photograph of this layer, resulting in a multi-layer repaired sub-photograph. This allows the areas occluded by previous layers in each layer to be repaired, making the image content of each layer relatively complete. This ensures that the image content of the final generated 3D photo is relatively complete from various perspectives. Thus, a complete 3D photo containing multiple perspectives can be automatically generated by taking only a single 2D photo, eliminating the need to take multiple 2D photos or perform complex compositing based on multiple 2D photos manually, thereby improving the efficiency and quality of 3D photo generation.

[0134] In one embodiment, for each layer's sub-depth map, repairing the image content of the occluded areas in the sub-depth map of that layer based on the image content of the unoccluded areas in the sub-depth map of that layer to obtain a multi-layer repaired sub-depth map includes: for each layer's sub-depth map, repairing the image content of the occluded areas in the sub-depth map of that layer based on the image content of the unoccluded areas in the sub-depth map of that layer to obtain a multi-layer initial repaired sub-depth map; adjusting the depth values ​​in the initial repaired sub-depth maps of each layer so that the depth value changes smoothly between the initial repaired sub-depth maps of each layer, and the depth values ​​in the initial repaired sub-depth maps of each layer conform to the normal distance relationship, to obtain a multi-layer repaired sub-depth map.

[0135] In one embodiment, the server can merge the initial repair sub-depth maps of each layer to obtain a multi-channel depth map, input the multi-channel depth map into a multi-layer depth map improvement model, and adjust the depth values ​​in the initial repair sub-depth maps of each layer through the multi-layer depth map improvement model so that the depth value changes smoothly between the initial repair sub-depth maps of each layer and the depth values ​​in the initial repair sub-depth maps of each layer conform to the normal distance relationship, and output the adjusted multi-layer repair sub-depth map.

[0136] The multi-layer depth map improvement model is used to adjust the depth values ​​in the initial repair sub-depth maps of each layer, so that the depth value changes smoothly between the initial repair sub-depth maps of each layer, and the depth values ​​in the initial repair sub-depth maps of each layer conform to the normal distance relationship. A multi-channel depth map contains initial repair sub-depth maps for multiple channels, with one initial repair sub-depth map in each channel.

[0137] In one embodiment, the multi-layer depth map improvement model can be a machine learning model.

[0138] In one embodiment, the multi-layer depth map improvement model can be a deep learning model or other types of machine learning models, without limitation.

[0139] In one embodiment, the multi-layer depth map improvement model can be obtained in advance through a multi-layer depth map improvement model training step.

[0140] In one embodiment, the training steps of the multi-layer depth map improvement model may include: inputting sample multi-channel depth maps and sample multi-layer repair sub-depth maps into the multi-layer depth map improvement model to be trained; adjusting the depth values ​​in the initial repair sub-depth maps of each layer through the multi-layer depth map improvement model to obtain predicted multi-layer repair sub-depth maps; and iteratively adjusting the model parameters of the multi-layer depth map improvement model to be trained based on the difference between the predicted multi-layer repair sub-depth maps and the sample multi-layer repair sub-depth maps until the iteration stopping condition is met, thereby obtaining the trained multi-layer depth map improvement model.

[0141] In the above embodiments, after repairing the sub-depth map of each layer, the depth values ​​of the initial repaired sub-depth maps of each layer are adjusted so that the depth values ​​of the initial repaired sub-depth maps of each layer change smoothly and the depth values ​​in the initial repaired sub-depth maps of each layer conform to the normal distance relationship. This enables the obtained multi-layer repaired sub-depth maps to have better hierarchical consistency, and the depth values ​​of each layer repaired sub-depth map are more realistic and accurate, thus improving the quality of the final 3D image.

[0142] like Figure 8 The diagram illustrates the process of obtaining multi-layer repaired sub-depth maps and multi-layer repaired sub-photographs through multi-layer sub-depth maps and multi-layer sub-photographs in the above embodiments. Each layer's sub-depth map is input into the depth map repair model, outputting an initial repaired sub-depth map for each layer. Then, the initial repaired sub-depth maps of each layer are merged into a multi-channel depth map, which is input into the multi-layer depth map improvement model, outputting the multi-layer repaired sub-depth map. Similarly, each layer's sub-photograph is input into the photo repair model, outputting a repaired sub-photograph for each layer.

[0143] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0144] Based on the same inventive concept, this application also provides a three-dimensional photograph generation apparatus for implementing the three-dimensional photograph generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the three-dimensional photograph generation apparatus provided below can be found in the limitations of the three-dimensional photograph generation method described above, and will not be repeated here.

[0145] In one embodiment, such as Figure 9 As shown, a three-dimensional photograph generation device 900 is provided, including: a depth map determination module 902, an image layering module 904, an image restoration module 906, and a three-dimensional photograph generation module 908, wherein:

[0146] The depth map determination module 902 is used to determine the depth map corresponding to a single 2D photograph taken for a 3D scene.

[0147] The image layering module 904 is used to layer the depth map according to the depth values ​​in the depth map to obtain multiple sub-depth maps, and correspondingly obtain multiple sub-photographs corresponding to the two-dimensional photo.

[0148] Image inpainting module 906 is used to repair the occluded areas in each layer of multi-layer sub-depth map and multi-layer sub-photograph to obtain multi-layer repaired sub-depth map and multi-layer repaired sub-photograph; the occluded area in each layer refers to the area that is occluded by the image content of the previous layer.

[0149] The 3D photo generation module 908 is used to generate a 3D photo containing multiple perspectives corresponding to a 3D scene based on the multi-layer repair sub-depth map and multi-layer repair sub-photo.

[0150] In one embodiment, the 3D photo generation module 908 is further configured to perform 3D reconstruction based on the multi-layer repair sub-depth map and the multi-layer repair sub-photo to obtain a 3D point cloud under the original viewpoint corresponding to the 3D scene; the original viewpoint is the shooting viewpoint of the 2D photo; based on the 3D point cloud under the original viewpoint, generate 3D point clouds under multiple viewpoints corresponding to the 3D scene; render and project the 3D point clouds under each viewpoint to obtain multiple frames of images; and generate a 3D photo containing multiple viewpoints corresponding to the 3D scene based on the multiple frames of images.

[0151] In one embodiment, the 3D photo generation module 908 is further configured to determine the number of frames of the 3D photo based on the duration set for the 3D photo to be generated; determine multiple sets of view transformation parameters based on the range of view transformation parameters and the number of frames set for the 3D photo to be generated; and perform view transformation on the 3D point cloud under the original view according to each set of view transformation parameters to generate 3D point clouds under multiple view corresponding to the 3D scene.

[0152] In one embodiment, the 3D photo generation module 908 is further configured to perform object segmentation processing on the 2D photo to obtain an object segmentation image and corresponding object semantic labels; if a target object semantic label exists in the object semantic label, then semantic 3D reconstruction is performed based on the image content corresponding to the target object semantic label in the object segmentation image and the 2D photo to generate a 3D point cloud of the target object corresponding to the target object semantic label; 3D reconstruction is performed based on the multi-layer repair sub-depth map and the multi-layer repair sub-photo to obtain an initial 3D point cloud under the original viewpoint corresponding to the 3D scene; and a 3D point cloud under the original viewpoint corresponding to the 3D scene is obtained based on the initial 3D point cloud and the 3D point cloud of the target object.

[0153] In one embodiment, the image layering module 904 is further configured to determine depth layer boundaries from the depth map; the depth layer boundaries are the locations where depth values ​​change abruptly in the depth map; based on the depth layer boundaries, the depth map is divided into multiple regions; the center of each region is used as the cluster center, and each pixel in the depth map is clustered according to the depth values ​​in the depth map to obtain multi-layer sub-depth maps; wherein each sub-depth map is determined based on each pixel in one of the clusters obtained by clustering; based on the layering of the multi-layer sub-depth maps, multi-layer sub-photographs corresponding to the two-dimensional photo are obtained.

[0154] In one embodiment, the image layering module 904 is further configured to perform edge detection on the object segmentation image and the depth map corresponding to the two-dimensional image respectively to obtain a first boundary image and a second boundary image; and to fuse the first boundary image and the second boundary image to obtain a depth layering boundary.

[0155] In one embodiment, the image restoration module 906 is further configured to, for each layer of sub-depth map, restore the image content of the occluded area in the sub-depth map of the current layer based on the image content of the unoccluded area in the sub-depth map of the current layer, to obtain a multi-layer restored sub-depth map; and for each layer of sub-photograph, restore the image content of the occluded area in the sub-photograph of the current layer based on the image content of the unoccluded area in the sub-photograph of the current layer, to obtain a multi-layer restored sub-photograph.

[0156] In one embodiment, the image restoration module 906 is further configured to, for each layer of sub-depth map, restore the image content of the occluded area in the sub-depth map of the current layer according to the image content of the unoccluded area in the sub-depth map of the current layer, to obtain a multi-layer initial restored sub-depth map; adjust the depth values ​​in the initial restored sub-depth maps of each layer so that the depth value changes smoothly between the initial restored sub-depth maps of each layer and the depth values ​​in the initial restored sub-depth maps of each layer conform to the normal distance relationship, to obtain a multi-layer restored sub-depth map.

[0157] The aforementioned 3D photo generation device determines the corresponding depth map of a single 2D photo taken for a 3D scene. Based on the depth values ​​in the depth map, it layers the depth map to obtain multiple sub-depth maps, and correspondingly generates multiple sub-photos of the 2D photo. This achieves the layering of the depth map and 2D photo according to their depth values, creating multiple layers of images with different spatial positions. Occlusion areas in each layer of the sub-depth maps and sub-photos are repaired, resulting in multi-layer repaired sub-depth maps and multi-layer repaired sub-photos. This repairs areas occluded by previous layers, ensuring relatively complete image content in each layer. Based on these relatively complete multi-layer repaired sub-depth maps and multi-layer repaired sub-photos, a 3D photo containing multiple perspectives corresponding to the 3D scene is generated. This ensures that the generated 3D photo has relatively complete image content from each perspective. Therefore, it can automatically generate a complete, high-quality 3D photo containing multiple perspectives from only a single 2D photo, eliminating the need to take multiple 2D photos, perform complex compositing based on multiple 2D photos, or use complex shooting equipment, thus improving the efficiency and quality of 3D photo generation.

[0158] Each module in the aforementioned 3D image generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0159] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a method for generating three-dimensional photographs.

[0160] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0161] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0162] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0163] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0164] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0165] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0166] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0167] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating three-dimensional photographs, characterized in that, The method includes: Determine the depth map corresponding to a single 2D photograph taken for a 3D scene; The depth map is layered according to the depth values ​​in the depth map to obtain multiple sub-depth maps, and the corresponding multiple sub-photographs are obtained for the two-dimensional photo. The occluded areas in each layer of the multi-layer sub-depth map and the multi-layer sub-photograph are repaired to obtain a multi-layer repaired sub-depth map and a multi-layer repaired sub-photograph; the occluded area in each layer refers to the area that is occluded by the image content of the previous layer; The spatial coordinates of the 3D point cloud are obtained based on the multi-layer restoration sub-depth map; according to the correspondence between the pixels of the multi-layer restoration sub-deep map and the multi-layer restoration sub-photograph, the pixel values ​​of each pixel in the multi-layer restoration sub-photograph are mapped to the spatial coordinates to obtain the 3D point cloud under the original viewpoint corresponding to the 3D scene; the original viewpoint is the shooting viewpoint of the 2D photo; based on the 3D point cloud under the original viewpoint, 3D point clouds under multiple viewpoints corresponding to the 3D scene are generated; the 3D point clouds under each viewpoint are rendered and projected to obtain multiple frames of images; based on the multiple frames of images, a 3D photo containing multiple viewpoints corresponding to the 3D scene is generated.

2. The method according to claim 1, characterized in that, The step of generating 3D point clouds from multiple perspectives corresponding to the 3D scene based on the 3D point cloud from the original viewpoint includes: The number of frames in the 3D photograph is determined based on the duration set for the 3D photograph to be generated; Based on the range of viewpoint transformation parameters set for the 3D photo to be generated and the number of frames, multiple sets of viewpoint transformation parameters are determined; Based on the perspective transformation parameters described in each group, the 3D point cloud under the original perspective is transformed to generate 3D point clouds under multiple perspectives corresponding to the 3D scene.

3. The method according to claim 1, characterized in that, The method further includes: The two-dimensional photograph is subjected to object segmentation processing to obtain object segmentation images and corresponding object semantic labels; If the semantic label of the object contains a semantic label of the target object, then semantic 3D reconstruction is performed based on the image content corresponding to the semantic label of the target object in the object segmentation image and the 2D photo to generate a 3D point cloud of the target object corresponding to the semantic label of the target object. The step of performing 3D reconstruction based on the multi-layer repair sub-depth map and the multi-layer repair sub-photograph to obtain the 3D point cloud under the original viewpoint corresponding to the 3D scene includes: Based on the multi-layer repair sub-depth map and the multi-layer repair sub-photograph, a three-dimensional reconstruction is performed to obtain the initial three-dimensional point cloud under the original viewpoint corresponding to the three-dimensional scene; Based on the initial 3D point cloud and the 3D point cloud of the target object, the 3D point cloud of the original viewpoint corresponding to the 3D scene is obtained.

4. The method according to claim 1, characterized in that, The step of layering the depth map according to the depth values ​​in the depth map to obtain multiple sub-depth maps, and correspondingly obtaining multiple sub-photographs corresponding to the two-dimensional photograph, includes: Depth layer boundaries are determined from the depth map; the depth layer boundaries are the locations of abrupt changes in depth values ​​in the depth map. The depth map is divided into multiple regions based on the depth layer boundaries; Using the center of each region as the cluster center, and based on the depth value in the depth map, cluster each pixel in the depth map to obtain multiple sub-depth maps; wherein, each sub-depth map is determined based on each pixel in one of the clusters obtained by clustering. Based on the layering of the multi-layer sub-depth map, the multi-layer sub-photographs corresponding to the two-dimensional photo are obtained.

5. The method according to claim 4, characterized in that, Determining the depth layer boundaries from the depth map includes: Edge detection is performed on the object segmentation image and the depth map corresponding to the two-dimensional photo to obtain a first boundary image and a second boundary image; The first boundary image and the second boundary image are fused to obtain a deep layered boundary.

6. The method according to claim 1, characterized in that, The step of repairing the occluded areas in each layer of the multi-layer sub-depth map and the multi-layer sub-photograph to obtain the multi-layer repaired sub-depth map and the multi-layer repaired sub-photograph includes: For each sub-depth map, based on the image content of the unoccluded area in the sub-depth map of this layer, the image content of the occluded area in the sub-depth map of this layer is repaired to obtain a multi-layer repaired sub-depth map; For each sub-photograph in a layer, the image content of the occluded area in the sub-photograph in that layer is repaired based on the image content of the unoccluded area in that sub-photograph, resulting in multi-layer repaired sub-photographs.

7. The method according to claim 6, characterized in that, For each layer of sub-depth map, based on the image content of the unoccluded areas in the sub-depth map of that layer, the image content of the occluded areas in the sub-depth map of that layer is repaired to obtain multi-layer repaired sub-depth maps, including: For each layer of sub-depth map, based on the image content of the unoccluded area in the sub-depth map of this layer, the image content of the occluded area in the sub-depth map of this layer is repaired to obtain multiple initial repaired sub-depth maps; The depth values ​​in the initial repair sub-depth maps of each layer are adjusted to make the depth value changes between the initial repair sub-depth maps of each layer gradual and to make the depth values ​​in the initial repair sub-depth maps of each layer conform to the normal distance relationship, thus obtaining a multi-layer repair sub-depth map.

8. A three-dimensional photograph generation device, characterized in that, The device includes: The depth map determination module is used to determine the depth map corresponding to a single two-dimensional photograph taken for a three-dimensional scene. The image layering module is used to layer the depth map according to the depth values ​​in the depth map to obtain multiple sub-depth maps, and correspondingly obtain multiple sub-photographs corresponding to the two-dimensional photo. The image restoration module is used to restore the occluded areas in each layer of the multi-layer sub-depth map and the multi-layer sub-photograph to obtain a multi-layer restored sub-depth map and a multi-layer restored sub-photograph; the occluded area in each layer refers to the area that is occluded by the image content of the previous layer; A 3D photo generation module is used to obtain the spatial coordinates of a 3D point cloud based on the multi-layer restoration sub-depth map; based on the correspondence between the pixels of the multi-layer restoration sub-deep map and the multi-layer restoration sub-photo, the pixel values ​​of each pixel in the multi-layer restoration sub-photo are mapped to the spatial coordinates to obtain the 3D point cloud under the original viewpoint corresponding to the 3D scene; the original viewpoint is the shooting viewpoint of the 2D photo; based on the 3D point cloud under the original viewpoint, multiple 3D point clouds under different viewpoints corresponding to the 3D scene are generated; the 3D point clouds under each viewpoint are rendered and projected to obtain multiple frames of images; and based on the multiple frames of images, a 3D photo containing multiple viewpoints corresponding to the 3D scene is generated.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images