Three-dimensional image generation method and device, storage medium and program product

Abstract

The embodiment of the invention provides a three-dimensional image generation method and device, a storage medium and a program product. In the embodiment of the invention, based on binocular parallax, a first monocular view with a cavity area and a second monocular view without a cavity area corresponding to a two-dimensional image are generated; according to the method, the first monocular view is taken as a global constraint, the hole area in the first monocular view is subjected to block repair, image details can be accurately restored, the image quality is remarkably improved, important image details are kept, the second monocular view without the hole area is further taken as the global constraint, and the third monocular view obtained after hole area repair is subjected to global repair, so that the image quality is improved. And the generated content is ensured to be controllable during repairing, the consistency of stereoscopic vision is maintained, and the repairing definition is higher.

Description

Technical Field

[0001] The present application relates to the field of stereoscopic vision processing technology, and in particular to a three-dimensional image generation method, device, storage medium and program product. Background Art

[0002] With the continuous development of science and technology, three-dimensional (3D) technology is being used more and more widely in all walks of life, and the demand for three-dimensional images is getting higher and higher. However, a large number of existing images are still in two-dimensional (2D) format, which limits the application scope of 3D technology.

[0003] In the prior art, there are existing methods for generating two-dimensional images to generate three-dimensional images, for example, using Generative Adversarial Networks (GANs) that originally generate two-dimensional images to generate three-dimensional images, or using Convolutional Neural Networks (CNNs) that originally generate two-dimensional images to generate three-dimensional images. However, the image quality of the three-dimensional images generated in this way is poor, for example, the image is distorted and the resolution is low. Summary of the invention

[0004] Multiple aspects of the present application provide a three-dimensional image generation method, device, storage medium and program product to improve the image quality of generated three-dimensional images.

[0005] An embodiment of the present application provides a three-dimensional image generation method, including: based on binocular parallax, generating a first monocular view with a hole area and a second monocular view without a hole area corresponding to a two-dimensional image, the two-dimensional image having a first monocular perspective, and the first monocular view and the second monocular view having a second monocular perspective; performing block-by-block repair on the hole area in the first monocular view to obtain a third monocular view, the third monocular view having the second monocular perspective; performing global repair on the third monocular view according to the second monocular view to obtain a fourth monocular view without a hole area, the fourth monocular view having the second monocular perspective; and synthesizing the two-dimensional image with the fourth monocular view to obtain a three-dimensional image corresponding to the two-dimensional image.

[0006] An embodiment of the present application also provides a three-dimensional image generation method, including: receiving a two-dimensional image uploaded by a content production end, where the two-dimensional image is captured using a two-dimensional shooting device; using the image generation method provided in the embodiment of the present application to generate a three-dimensional image corresponding to the two-dimensional image; and returning the three-dimensional image to the content production end so that the content production end can use the three-dimensional image.

[0007] An embodiment of the present application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement each step in the three-dimensional image generation method provided in the embodiment of the present application.

[0008] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor implements each step of the three-dimensional image generation method provided in the embodiment of the present application.

[0009] In the embodiment of the present application, based on binocular disparity, a first monocular view with a hole area and a second monocular view without a hole area corresponding to a two-dimensional image are generated; the hole area in the first monocular view is repaired in blocks to obtain a third monocular view; and then the third monocular view is repaired in blocks according to the second monocular view without the hole area to obtain a third monocular view without the hole area, and the two-dimensional image and the fourth monocular view are synthesized as images under two monocular perspectives to obtain a three-dimensional image corresponding to the two-dimensional image. Among them, the use of block repair for the hole area can significantly improve the image quality, reduce noise and artifacts, and maintain important image details. The second monocular view without the hole area is further used as a global constraint to perform global repair on the third monocular view after the hole area is repaired. While repairing, it can ensure that the generated content is controllable and maintain the consistency of stereoscopic vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 A schematic diagram of a three-dimensional image generation method provided by an exemplary embodiment of the present application;

[0012] Figure 2a A schematic diagram of generating a first monocular view provided by an exemplary embodiment of the present application;

[0013] Figure 2b A schematic diagram of generating a second monocular view provided by an exemplary embodiment of the present application;

[0014] Figure 3a and Figure 3b A schematic diagram of block repair and global repair provided for an exemplary embodiment of the present application;

[0015] Figure 4 A schematic structural diagram of a three-dimensional image generating device provided by an exemplary embodiment of the present application;

[0016] Figure 5 A schematic structural diagram of a three-dimensional electronic device provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0018] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0019] Spatial image generation technology plays an important role, and its development benefits from the progress in computer vision, image processing, 3D modeling and artificial intelligence. In recent years, researchers have made breakthroughs in enhancing the quality of spatial images, increasing the generation speed and optimizing the realism of images by using deep learning and neural network models, especially generative adversarial networks (GANs) and convolutional neural networks (CNNs).

[0020] With the rapid development of text-based graph technology, image generation models based on text-based graphs can generate complex image content of various styles and themes based on simple text prompts. Image generation models based on text-based graphs not only have powerful image synthesis capabilities, but also significantly improve in understanding the association between text descriptions and image content.

[0021] However, whether it is GANs, CNNs, or image generation models based on Vincent graphs, the above models are mainly used for two-dimensional image generation tasks. For three-dimensional images under different perspectives, the stereoscopic visual occlusion caused by spatial transformation cannot be processed. Moreover, due to method limitations, the image quality of the generated three-dimensional image (such as clarity or size, etc.) depends on the computing power bottleneck, hardware specifications and training data. It is impossible to generate high-quality three-dimensional images with a general method.

[0022] In addition, methods related to 3D photography have made contributions to the generation of new perspective images in recent years. 3D photography refers to the use of specific technologies to generate or capture images or photos with three-dimensional effects. This technology allows people to feel a stronger sense of three-dimensionality and space when observing photos, as if the objects are right in front of them. By introducing GANs models or SD models, features are effectively extracted from the input and used for new perspective image rendering. However, in the process of generating three-dimensional images, there are problems such as image distortion, reduced image resolution, large amount of calculation, and low conversion efficiency.

[0023] In view of the above technical problems, in an embodiment of the present application, based on binocular disparity, a first monocular view with a hole area and a second monocular view without a hole area corresponding to a two-dimensional image are generated; the hole area in the first monocular view is repaired in blocks to obtain a third monocular view, and the third monocular view is globally repaired according to the second monocular view to obtain a fourth monocular view without a hole area, and the two-dimensional image and the fourth monocular view are synthesized as images under different monocular perspectives to obtain a three-dimensional image corresponding to the two-dimensional image. Among them, the use of block repair for the hole area can accurately restore the image details, significantly improve the image quality, reduce noise and artifacts, and maintain important image details. The second monocular view without the hole area is further used as a global constraint, and the third monocular view after the hole area repair is globally repaired. While repairing, it can ensure that the generated content is controllable, maintain the consistency of stereoscopic vision, and the repair clarity is higher. For example, the image clarity of the generated three-dimensional image can include but is not limited to: 720p, 1080p, 2K or 4K.

[0024] Through the three-dimensional image generation method provided in the embodiment of the present application, it is possible to convert existing two-dimensional images or two-dimensional videos into three-dimensional images or three-dimensional videos. The image quality of the three-dimensional images or three-dimensional videos is relatively good, such as high image clarity, consistency of pixels between the foreground and background of the image, strong stereoscopic effect, and realism, thereby enhancing the immersiveness and viewing experience of the three-dimensional content, and providing high-quality three-dimensional image generation services for e-commerce shopping, film and television entertainment, advertising marketing, education and training, medicine and other fields.

[0025] A solution provided by an embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0026] Figure 1 A schematic diagram of a structure for generating a three-dimensional image provided by an exemplary embodiment of the present application. Figure 1 As shown, the method includes:

[0027] 101. Generate a first monocular view with a hole area and a second monocular view without a hole area corresponding to the two-dimensional image based on binocular disparity, wherein the two-dimensional image has a first monocular perspective, and the first monocular view and the second monocular view have a second monocular perspective;

[0028] 102. Perform block-by-block repair on the hole region in the first monocular view to obtain a third monocular view, where the third monocular view has the second monocular viewing angle;

[0029] 103. Perform global repair on the third monocular view according to the second monocular view to obtain a fourth monocular view without a hole area, where the fourth monocular view has the second monocular viewing angle;

[0030] 104. Synthesize the two-dimensional image with the fourth monocular view to obtain a three-dimensional image corresponding to the two-dimensional image.

[0031] In this embodiment, the execution subject of the 3D image generation method is not limited. For example, the 3D image generation method is implemented as an application on a terminal device, or as a system function (such as a library function) on a terminal device. In this case, the execution subject of the 3D image generation method can be the terminal device. For another example, the 3D image generation method is implemented as a functional module, application or service on a server device, and the execution subject of the 3D image generation method is the server device.

[0032] In this embodiment, a two-dimensional image can be obtained. The implementation method for obtaining a two-dimensional image is not limited. For example, the two-dimensional image can be taken by a smartphone, a camera, or downloaded from the Internet, etc. In other words, the two-dimensional image can be a two-dimensional image or a two-dimensional video taken in real time, or it can be an existing two-dimensional image or a two-dimensional video, so as to realize the automatic conversion of the existing two-dimensional images or two-dimensional videos, especially for services for B-side customers. Content creators can use the previously accumulated shooting experience and the entire workflow to create high-quality three-dimensional pictures or three-dimensional videos. In addition, there is no limitation on the content contained in the two-dimensional image, which can be determined according to the application scenario. For example, the two-dimensional image can be a portrait, a landscape image, a product image, etc.

[0033] In this embodiment, since there is a certain distance between the left and right eyes of the human eye (for example, 65 mm), this distance is called the visual distance. When the human eye looks at an object, the left eye sees more content on the left side of the object, and the right eye sees more content on the right side of the object. There is a difference in the images of the two eyes, which is binocular parallax. Based on binocular parallax, a first monocular view with a hole area and a second monocular view without a hole area corresponding to the two-dimensional image can be generated. Among them, the hole area is an area that can be seen by one eye but not by the other eye due to binocular parallax, that is, an area that is visible from one monocular perspective but invisible from another monocular perspective.

[0034] Among them, the first monocular view and the second monocular view have the same monocular perspective. The monocular perspective of the first monocular view is different from the monocular perspective of the two-dimensional image. Similarly, the monocular perspective of the second monocular view is also different from the monocular perspective of the two-dimensional image. For example, if the two-dimensional image is taken as an image under the left eye perspective (that is, the two-dimensional image has a left eye perspective), the first monocular view and the second monocular view are both images under the right eye perspective (that is, the first monocular view and the second monocular view both have a right eye perspective); if the two-dimensional image is taken as an image under the right eye perspective (that is, the two-dimensional image has a right eye perspective), the first monocular view and the second monocular view are both images under the left eye perspective (that is, the first monocular view and the second monocular view both have a left eye perspective). For the convenience of description and distinction, the monocular perspective of the two-dimensional image is recorded as the first monocular perspective, and the monocular perspective of the first monocular view and the second monocular view is recorded as the second monocular perspective, and the first monocular perspective and the second monocular perspective are different.

[0035] In this embodiment, the first monocular view can be repaired by combining block repair and global repair. Specifically, the hole area in the first monocular view can be repaired in blocks to obtain a third monocular view; the third monocular view has the second monocular perspective. Among them, the use of block repair for the hole area can accurately restore the image details, significantly improve the image quality, reduce noise and artifacts, and maintain important image details. In order to avoid the change of image content caused by the block repair of the hole area, the second monocular view without the hole area can be further used as a global constraint, and the third monocular view after the hole area repair is globally repaired to obtain a fourth monocular view without the hole area. The fourth monocular view has the second monocular view. In addition, the content of the fourth monocular view is controllable and can be consistent with the content of the two-dimensional image to a greater extent.

[0036] The two-dimensional image and the fourth monocular view have different monocular perspectives. For example, if the two-dimensional image has a left eye perspective, the fourth monocular view has a right eye perspective; if the two-dimensional image has a right eye perspective, the fourth monocular view has a left eye perspective. The two-dimensional image and the fourth monocular view are synthesized, that is, the left eye view and the right eye view are synthesized to obtain a three-dimensional image corresponding to the two-dimensional image.

[0037] The two-dimensional image and the fourth monocular view may be synthesized by side-by-side (SBS) or multi-view video coding (MV-HEVC) to obtain a three-dimensional image.

[0038] SBS is a way of presenting stereoscopic video. For the SBS format, at the sending end, the left and right viewpoint images are horizontally spliced ​​together to form a single SBS format image frame, which is convenient for the transmission and display of stereoscopic video; at the receiving end, the SBS format image frame will be decoded and separated into two single-view images on the left and right, and then presented to the user's left and right eyes respectively, thus creating a stereoscopic effect.

[0039] MV-HEVC is an efficient video coding format. MV-HEVC is an extension of High Efficiency Video Coding (HEVC). MV-HEVC is mainly used for multi-view video coding. It effectively eliminates redundant information between different perspectives by introducing inter-layer reference mode in the coding process, thereby improving coding efficiency. MV-HEVC has higher performance than traditional coding methods when processing multi-view or stereoscopic videos.

[0040] In an embodiment of the present application, based on binocular disparity, a first monocular view with a hole area and a second monocular view without a hole area corresponding to a two-dimensional image are generated; the hole area in the first monocular view is repaired in blocks to obtain a third monocular view; and then the third monocular view is globally repaired according to the second monocular view without the hole area to obtain a third monocular view without the hole area, and the two-dimensional image and the fourth monocular view are synthesized as images under two monocular perspectives to obtain a three-dimensional image corresponding to the two-dimensional image. Among them, the use of block repair for the hole area can accurately restore the image details, significantly improve the image quality, reduce noise and artifacts, and maintain important image details at the same time. The second monocular view without the hole area is further used as a global constraint to globally repair the third monocular view after the hole area repair. While repairing, it can ensure that the generated content is controllable, maintain the consistency of stereoscopic vision, and the repair clarity is higher.

[0041] In the embodiments of the present application, the implementation method of performing block repair on the hole area in the first monocular view is not limited. Among them, an implementation method of performing block repair on the hole area in the first monocular view to obtain a third monocular view includes: generating a first image feature according to the first monocular view, and performing block repair on the first image feature to obtain a third image feature; wherein the third image feature corresponds to the third monocular view, that is, the monocular view after block repair. It should be noted that the third image feature is an intermediate result in the process of generating a three-dimensional image, and the third image feature can be converted into a third monocular view through a decoder and output. Of course, the third image feature may not be converted, that is, the third monocular view is not output. In the subsequent processing process, the third image feature can be directly processed.

[0042] Similarly, in the embodiments of the present application, the implementation method of globally repairing the third monocular view according to the second monocular view is not limited. Among them, one implementation method of globally repairing the third monocular view according to the second monocular view to obtain a fourth monocular view without a hole area includes: extracting a second image feature from the second monocular view, for example, feature encoding the second monocular view to obtain a second image feature, and globally repairing the third image feature according to the second image feature to obtain a fourth image feature; generating a fourth monocular view without a hole area according to the fourth image feature, for example, feature decoding the fourth image feature to obtain a fourth monocular view. Among them, the fourth monocular view, the first monocular view and the second monocular view have the same monocular perspective (i.e., the second monocular perspective), and the fourth monocular view and the two-dimensional image have different monocular perspectives.

[0043] The second monocular view can be feature encoded by an encoder, and the fourth image feature can be feature decoded by a decoder to obtain a fourth monocular view. Feature encoding is an important step in data preprocessing, and its main purpose is to convert the original feature data in the spliced ​​image into a format that the model can understand and use. The original data may contain non-numeric data such as images, texts, dates, categories, etc., which cannot be directly input into most machine learning algorithms. Accordingly, the function of feature encoding is to convert these numerical data into numerical data for model training and prediction. Feature decoding is usually used to visualize the features learned by the model, or to recover the information of the original image from the feature map. For example, the variational autoencoder (VAE) model includes an encoder and a decoder, and the encoder compresses the spliced ​​features into a low-dimensional latent vector (e.g., the second image feature), also known as a latent variable or latent code, and performs subsequent processing of block repair and global repair based on the latent vector. The decoder decodes the latent vector (e.g., the fourth image feature) in the subsequent processing process into a reconstructed version of the original data (e.g., the fourth monocular view).

[0044] Among them, block restoration can significantly improve the quality of the image, reduce noise and artifacts, while maintaining important image details; for example, the image clarity is 4K; through global restoration, the second image feature is used as a global constraint, which can ensure the controllability of the generated content while repairing (image denoising) and maintain the consistency of stereoscopic vision.

[0045] In an optional embodiment, block repair and global repair can be achieved through a diffusion model. Among them, the diffusion model is a type of generative model based on probability likelihood. The diffusion model describes a random process in which image features gradually diffuse into noise and generate image features from noise. The diffusion model is a kind of Wensheng graph model. Applying the diffusion model to the field of three-dimensional image generation can improve the image quality of the generated three-dimensional image. The core idea of ​​the diffusion model is to capture the intrinsic structure and distribution of image features through a series of forward diffusion and reverse denoising steps. Among them, for the diffusion model, block repair can be achieved through multi-diffusion optimization. Multi-diffusion optimization is an effective image reconstruction technology that can significantly improve the quality of the image. Through reasonable diffusion and optimization strategies, it can effectively reduce noise and artifacts while maintaining important image details and improving the clarity of the generated three-dimensional image. At the same time, a global constraint term (i.e., the second image feature) can be inserted into the diffusion model to ensure that the generated content is controllable while denoising and maintain stereoscopic visual consistency.

[0046] In an optional embodiment, the diffusion model includes an encoder, an image restoration network, a denoising module and a decoder. The encoder and the decoder can be implemented by the above-mentioned VAE model, the image restoration network is used to implement block restoration, and the denoising module is used to implement global restoration. For detailed introduction of block restoration and global restoration, please refer to the subsequent embodiments.

[0047] The three-dimensional image generation method provided in the embodiment of the present application can be introduced through the following three steps. (1) Rendering the two-dimensional image to obtain a first monocular view with a hole area; (2) Generating a second monocular view without a hole area and performing foreground segmentation; (3) Realizing the conversion of the two-dimensional image to the three-dimensional image by combining local repair and global repair. The following is a detailed introduction.

[0048] (i) Based on binocular disparity, the two-dimensional image is rendered to obtain a first monocular view with a hole area.

[0049] In an optional embodiment, the two-dimensional image is used as a monocular view under the first monocular perspective, and the relative position offset information of the pixel points in the two-dimensional image under the second monocular perspective is determined based on binocular parallax; according to the position information of the pixel points in the two-dimensional image and the relative position offset information of the pixel points in the two-dimensional image under the second monocular perspective, the two-dimensional image is rendered under the second monocular perspective to obtain the first monocular view. Figure 2a In the example, the process of generating the first monocular view is exemplarily shown, wherein the relative position offset information is reflected by an optical flow map.

[0050] Among them, the first monocular perspective can be a left eye perspective or a right eye perspective, that is, the two-dimensional image can be used as a left eye view or a right eye view. Among them, the two eyes can be simulated by a virtual left eye camera and a virtual right eye camera, and the binocular parallax can be represented by the relative posture information between the left eye camera and the right eye camera, for example, by a rotation matrix and a translation matrix. For example, the left eye camera coordinate system can be transformed to the right eye camera coordinate system by a first rotation matrix and a first translation matrix, or the right eye camera coordinate system can be transformed to the left eye camera coordinate system by a second rotation matrix and a second translation matrix. Specifically, in the process of determining the relative position offset information of the pixel points in the two-dimensional image under the second monocular perspective based on the binocular parallax, the depth of the two-dimensional image is estimated by a depth estimation algorithm to obtain a depth map corresponding to the two-dimensional image, such as Figure 2a As shown; according to the set binocular perspective relative offset, including the rotation matrix (e.g., 3x3 order) and the translation matrix (e.g., 3x1 order), the binocular stereo projection formula is substituted, and the relative position offset information of each pixel in the two-dimensional image (first monocular perspective) corresponding to the second monocular perspective is obtained from the depth map.

[0051] Among them, the depth estimation algorithm is not limited. For example, the Mixed-scale Densedepth prediction (Midas) model is mainly used for monocular estimation and can predict the depth information of the scene from a two-dimensional image. It improves the accuracy and robustness of depth estimation by mixing information of multiple scales and using a densely connected convolutional neural network (CNN) architecture. For another example, the residual network (ResNet) can handle the gradient disappearance problem of deep networks and is often used in depth estimation tasks. For another example, the Fully Convolutional Networks (FCNs) can achieve end-to-end pixel-level prediction and is suitable for the generation of depth maps. For another example, the Generative Adversarial Networks (GANs) can generate realistic depth maps and improve model performance through adversarial training.

[0052] The depth map is stored in a 16-bit unsigned integer (int16) with a value range from 0 to 65535. This storage method can ensure the accuracy and integrity of the depth data and avoid data loss problems that may occur during conversion or storage.

[0053] Among them, the relative position offset information can be reflected by an optical flow map. An optical flow map is an image representation method that estimates the motion information (such as speed and direction) of each pixel by analyzing the pixel changes between images corresponding to two different monocular perspectives. An optical flow map can be a grayscale image or converted into a color image for display. In a grayscale image, motion information can be represented by changes in pixel brightness; in a color image, different motion information can be distinguished by different colors.

[0054] According to the position information of the pixels in the two-dimensional image and the relative position offset information of the pixels in the two-dimensional image in the second monocular perspective, the two-dimensional image is rendered in the second monocular perspective to obtain the first monocular view. Due to the existence of the visual occlusion area caused by the change of binocular parallax, the first monocular view includes a hole area.

[0055] Among them, in the process of rendering the two-dimensional image under the second monocular perspective, it is ensured that each step in the entire rendering process is differentiable so that it can be trained or adjusted using optimization algorithms such as gradient descent. In addition, the parallel computing capability of the graphics processing unit (GPU) can also be used to accelerate the image rendering process, for example, to accelerate computationally intensive tasks such as optical flow estimation, depth estimation, and image deformation involved in the image rendering process.

[0056] Optionally, an implementation method of rendering a two-dimensional image under a second monocular perspective based on position information of pixels in the two-dimensional image and relative position offset information under the second monocular perspective to obtain a first monocular view includes: for any pixel in the two-dimensional image, determining the initial position information of any pixel under the second monocular perspective based on the position information of any pixel in the two-dimensional image and the relative position offset information of any pixel under the second monocular perspective; correcting the initial position information of any pixel under the second perspective based on the initial position information of pixels surrounding the pixel, to obtain target position information of any pixel under the second monocular perspective; rendering under the second monocular perspective based on the pixel values ​​of each pixel in the two-dimensional image and the target position information of each pixel under the second monocular perspective to obtain the first monocular view.

[0057] Among them, according to the initial position information of the surrounding pixels of any pixel point, the initial position information of any pixel point in the second viewing angle is corrected to obtain the target position information of any pixel point in the second monocular viewing angle. The implementation method is not limited. For example, the initial position information can be corrected by splatting interpolation. For any pixel point, when the any pixel point is converted from the first monocular viewing angle to the second monocular viewing angle, the attribute information of the any pixel point (such as the initial position information or pixel value) can be "sputtered" to other pixels around the any pixel point. Among them, the weight of other pixels being "sputtered" is related to the distance between other pixels and the any pixel point, and the closer the distance, the higher the weight. Based on this, for any pixel point, the initial position information of the surrounding pixels can be weighted added to obtain a weighted result, and the weighted result is added to the initial position information of the any pixel point to obtain the target position of any pixel point.

[0058] Optionally, based on binocular vision, the two-dimensional image is rendered to obtain a first monocular view with a hole area, which can be implemented by an image rendering model. For example, the image rendering model may include but is not limited to: a depth image based rendering (Depth Image Based Rendering, DIBR) network implementation. The core of the DIBR model is to perform three-dimensional reconstruction of the two-dimensional image based on the depth image to obtain a three-dimensional scene corresponding to the two-dimensional image, and project the pixel points in the three-dimensional scene onto the imaging plane of the virtual camera to obtain the first monocular view under the second monocular perspective.

[0059] (ii) Generate a second monocular view without a hole area and perform foreground segmentation, such as Figure 2b shown.

[0060] According to the movement direction of the first monocular view relative to the two-dimensional image, at least the hole region in the first monocular view is interpolated to obtain the second monocular view.

[0061] Among them, the pixel values ​​of the hole area in the first monocular view are unknown. Considering that the motion direction of the pixels in the adjacent area will not change suddenly, it can be considered that the motion direction of the pixels in the hole area is consistent with the motion direction of the pixels in the adjacent area of ​​the hole area. Based on this, according to the motion direction of the first monocular view relative to the two-dimensional image, at least the hole area in the first monocular view can be interpolated to obtain the second monocular view. Among them, based on the interpolation processing of the motion direction, a new perspective image without holes that is consistent with the background content can be obtained, which helps to improve the subsequent image segmentation and global inpainting processing results.

[0062] Optionally, the direction of motion of the first monocular view relative to the two-dimensional image is determined based on the relative position offset information of the pixel points in the two-dimensional image under the second monocular perspective; based on the direction of motion, an interpolation operator corresponding to the first monocular view is generated, the interpolation operator includes a first operator value and a second operator value, and the position relationship of the second operator value relative to the first operator represents the direction of motion; at least the hole area in the first monocular view is interpolated according to the interpolation operator to obtain the second monocular view. Optionally, the entire first monocular view can be interpolated by the interpolation operator to obtain the second monocular view.

[0063] Among them, the implementation method of determining the direction of motion of the first monocular view relative to the two-dimensional image based on the relative position offset information of the pixel points in the two-dimensional image under the second monocular perspective is not limited. For example, considering that the motion direction of the hole area is consistent with that of the adjacent pixel points around it, the adjacent pixel points of the hole area can be selected from the two-dimensional image, and the motion direction of the first monocular view relative to the two-dimensional image can be determined based on the relative position offset information of the target pixel under the second monocular perspective. For example, the adjacent pixel points of the hole area can be determined from the optical flow map, and the motion direction of the adjacent pixel points can be used as the motion direction of the hole area.

[0064] Among them, the interpolation operator corresponding to the first monocular view can be generated based on the motion direction. Among them, the motion direction is not limited. For example, the motion direction may include but is not limited to: from left to right, from right to left, from top to bottom, from bottom to top, and from bottom left to top right, etc. It should be noted that the motion direction of the first monocular view relative to the two-dimensional image may be due to binocular vision. For example, the motion direction is from left to right, or from right to left. The motion direction of the first monocular view relative to the two-dimensional image may also be that during the process of shooting the two-dimensional image, the camera is in motion, or the object or scene in the two-dimensional image is in motion, so that the first monocular view generates a motion direction relative to the two-dimensional image. In this case, the motion direction of the first monocular view relative to the two-dimensional image may include various motion directions, which are not limited.

[0065] Among them, the interpolation operator can be a set of weights in the form of a matrix, which determines the contribution of the surrounding pixels to the central pixel. The interpolation operator includes a first operator value and a second operator value, the first value and the second value correspond to different weights, and the position relationship of the second operator value relative to the first operator value represents the direction of motion of the first monocular view relative to the two-dimensional image, and the specific position relationship of the second operator value relative to the first operator value represents the direction of motion of the hole area. Among them, the first operator value and the second operator value can be any value, for example, a value between 0 and 1. Preferably, the first operator value and the second operator value can be 0 and 1, respectively, or 1 and 0, respectively, without limitation. In general, the order of the matrix corresponding to the interpolation operator is an odd number. For example, the interpolation operator can be a 3×3-order matrix or a 5×5-order matrix. The element at the center position in the matrix corresponds to the current pixel position, indicating that the position of the current pixel point is related to the positions of the pixels around it. For example, the following takes the interpolation operator with a first operator value of 0 and a second operator value of 1, respectively, and the interpolation operator being a 5×5 matrix as an example to illustrate the interpolation operator corresponding to the first monocular view generated based on the motion direction. For example, the motion direction is from right to left, referred to as π (i.e., 180 degrees), and the interpolation operator can be expressed as Indicates that the position of the center pixel is related to the position of the pixel on the right, where the center pixel is the pixel corresponding to the center element of the matrix; for another example, the movement direction is from top to bottom, referred to as π / 2 (i.e., 90 degrees), and the interpolation operator can be expressed as Indicates that the position of the center pixel is related to the position of the upper pixel. For another example, the direction of movement is from the lower left to the upper right, referred to as 3π / 4 (i.e., 90 degrees), and the interpolation operator can be expressed as Indicates that the position of the center pixel is related to the position of the lower left pixel.

[0066] Among them, in the process of interpolating at least the hole area in the first monocular view according to the interpolation operator to obtain the second monocular view, the interpolation operator is placed on each pixel in the hole area so that the central element of the interpolation operator coincides with each pixel. For each pixel covered by the interpolation operator, the position of the pixel is multiplied by the weight of the corresponding position in the interpolation operator, and then all the products are added to obtain the sum, the sum is normalized, and the normalized result is assigned to the pixel corresponding to the central element as the position information of the pixel in the second monocular view. This process is repeated on the entire hole area until each pixel is processed to obtain the second monocular view. Optionally, the above process can be repeated on the entire first monocular view until each pixel is processed to obtain the second monocular view.

[0067] Among them, based on the direction of motion, the interpolation operator is determined, and the interpolation method of filling the direction of the hole area by the interpolation operator is referred to as the interpolation method based on the motion direction. Compared with the traditional method of interpolating through the upper, lower, left and right connected areas, the interpolation method based on the motion direction obtains a second monocular view of the hole-free area that is highly consistent with the background content, which helps to improve the processing results of subsequent image segmentation and global restoration, and helps to improve the image quality of the final three-dimensional image.

[0068] In an optional embodiment, the second monocular view may be input into an image segmentation model, the foreground of the second monocular view may be segmented in the image segmentation model, and a foreground mask image corresponding to the second monocular image may be generated. The pixel value of the foreground area in the foreground mask image may be 1 or 0, and the pixel value of the background area may be 0 or 1. Preferably, the pixel value of the foreground area in the foreground mask image is 1, and the pixel value of the background area is 0. That is, the foreground area is a white area, and the background area is a black area.

[0069] The image segmentation model is not limited. For example, the image segmentation model may include but is not limited to: Fully Convolutional Networks (FCN) or High Resolution Net (HRNet). HRNet can perform foreground segmentation on the second monocular view of different scenes or different objects to obtain a foreground mask image. In the image foreground segmentation task, HRNet can very effectively capture the details of the foreground, which is crucial for fine boundary recognition and complex foreground and background separation. HRNet can be trained on a pre-trained HRNet using a large image dataset, for example, different foreground objects such as portraits, furniture, plants or animals. Figure 2b As shown in the figure, by marking the foreground objects in the sample images and fine-tuning the pre-trained HRNet with the sample images and their annotation information, a segmentation model suitable for segmenting different foreground objects can be quickly obtained.

[0070] Among them, HRNet has the following characteristics: 1. Parallel multi-scale streams: HRNet contains multiple parallel branches, each of which processes feature maps at different resolutions. This design allows the network to learn features of different scales at the same time, thereby better preserving details in the final segmentation output. 2. Cross-scale fusion: At each stage of the network, feature maps of different resolutions are fused through cross-scale connections to ensure that the information of low-resolution branches can also be passed to high-resolution branches, which is conducive to the integration of details and semantic information. 3. Persistence of high-resolution streams: Even in the deep layers of the network, HRNet keeps at least one branch running at a higher resolution, which helps to maintain the clarity of edges and details, and is very suitable for tasks such as foreground segmentation that require precise boundaries. 4. Flexible modular design: HRNet's design is modular, and components such as attention mechanisms or deeper convolution blocks can be easily added or adjusted to meet different task requirements. 5. End-to-end training: HRNet can be trained end-to-end, directly from the original image input to the segmentation result output, without additional post-processing steps. When using HRNet for image foreground segmentation, the network is usually trained to minimize the loss function between the segmentation result and the true label, such as cross entropy loss. Once the training is completed, HRNet can predict the new input image and output the probability that each pixel belongs to the foreground, thereby obtaining the result of foreground segmentation.

[0071] (III) The conversion of 2D images to 3D images is achieved by combining local restoration and global restoration. Figure 3a and Figure 3b shown.

[0072] In an optional embodiment, a hole mask image corresponding to the first monocular view may be generated, wherein the pixel value of the hole area in the hole mask image is not limited, for example, the pixel value of the hole area may be 0 or 1. Preferably, in the hole mask image, the pixel value of the hole area is 1. For example, the hole area is white, and the non-hole area is black.

[0073] Based on the above, the hole mask image corresponding to the first monocular view and the first foreground mask image corresponding to the second monocular view can be fused to obtain a second foreground mask image with a hole area. For example, the fusion method can be an "OR operation (OR)" to merge the hole area in the hole mask image and the foreground area in the first foreground mask image. In the fusion process, if the hole area and the foreground area have a connected area, the connected area is fused, and if there is no connected area, each is independent. Further, the second foreground mask image can also be filtered, for example, Gaussian filtering or mean filtering, etc., to remove burrs and small areas in the second foreground mask image, wherein the small area can be an area less than a set area threshold. Among them, the second foreground mask image can also be called an inpaint mask image. According to the second foreground mask image and the first monocular view, a foreground image with a hole area is generated; for example, the second foreground mask image and the first monocular view are multiplied to obtain a foreground image with a hole area. In addition, other logical operations can be performed on the second foreground mask image and the first monocular view to generate a foreground image with a hole area. The foreground image, the second foreground mask image and the noise image are spliced ​​to obtain a spliced ​​image; and feature encoding is performed on the spliced ​​image to obtain first image features.

[0074] Optionally, during the feature encoding process, the spliced ​​image can be encoded into a low-dimensional space. The dimension of this low-dimensional space is usually much smaller than the dimension of the spliced ​​image. This can reduce the amount of calculation while ensuring the accuracy of the calculation and improve the efficiency of generating three-dimensional images. For example, the VAE model performs feature encoding on the spliced ​​features to obtain the first image feature in the low-dimensional space, such as Figure 3b Further, in Figure 3b In the example, the fourth image feature is decoded by the VAE model to obtain the fourth monocular view, which is only an example but not limited to this.

[0075] In an optional embodiment, the implementation method of performing block repair on the first image feature to obtain the third image feature is not limited. For example, according to the set block size and overlap rate, the first image feature is overlapped and blocked to obtain multiple initial block features, and there are overlapping features between the multiple initial block features; wherein the set block size is usually related to the input requirements or computing resources of the image repair network. Wherein, the overlap rate refers to the degree of overlap between blocks, and the overlap rate can be 50%, 20% or other ratios, depending on how much context information needs to be retained. The overlap between blocks helps to avoid boundary effects and can maintain the consistency of features between adjacent blocks. For example, a sliding window can be used to traverse the entire first image feature. The size of the sliding window is the same as the block size, and the sliding window can be moved on the first image feature according to the overlap rate, and the step size of each movement is smaller than the width or height of the block to ensure overlap. At each position, the image features in the sliding window are extracted as initial block features, thereby obtaining multiple initial block features. The multiple initial block features can be further processed, such as normalized or enhanced. The multiple initial block features are input into the image restoration network to perform image restoration respectively, so as to obtain multiple target block features; and the multiple target block features are subjected to feature fusion to obtain the third image feature.

[0076] Among them, the image restoration network is not limited. The image restoration network can repair multiple initial block features to repair or fill in the missing or damaged areas in multiple initial block images corresponding to the multiple initial block features. For example, the image restoration network can include but is not limited to: Global and Local Consistency Generative Adversarial Network (GLCIC), Context Encoder, Gated Convolution for Image Inpainting, and Unet model, etc.

[0077] Among them, the Unet network is a convolutional neural network (CNN) for image segmentation. The core idea of ​​U-Net is to introduce jump connections. This structure greatly improves the accuracy of image segmentation. UNet includes an encoder and a decoder. The encoder part extracts features through convolution and downsampling operations, while the decoder part restores image details through upsampling and jump connections. In the downsampling process, noise is continuously added to the initial block features, and the initial blocks are gradually diffused into noise. In the process of upsampling the noise, the corresponding downsampled features are spliced ​​with the upsampled features at this level in combination with the jump connection mechanism. The spliced ​​features are continuously denoised to obtain the target block features.

[0078] Among them, in the process of fusing multiple target block features to obtain the third image feature, the third image feature can be obtained by weighted superposition of each target block feature. For example, the third image feature can be obtained by weighted superposition of each target block feature by least square solution weighting. Among them, the third image feature corresponds to a third monocular view, that is, a monocular view after block restoration. It should be noted that the third image feature is an intermediate result in the process of generating a three-dimensional image, and the third image feature can be converted into a third monocular view through a decoder and output. Of course, the third image feature may not be converted, and the third monocular view may not be output. In Figure 3b The third monocular view is not shown.

[0079] In an optional embodiment, the third image feature is globally repaired according to the second image feature to obtain the fourth image feature. The global repair process can be understood as a denoising process. The denoising process can be implemented by a denoising network or by a data calculation process, which is not limited. An exemplary description is given below.

[0080] For example, the third image feature is subjected to at least one round of weighted iterative processing according to the second image feature until an iteration termination condition is met, and the result of the last round of weighted iterative processing is used as the fourth image feature; wherein the weighted iterative processing includes: fusing the second image feature with the third image feature, and performing weighted summation on the fusion result.

[0081] For another example, feature encoding is performed on the second foreground mask image to obtain a fifth image feature; based on the second image feature and the fifth image feature, the third image feature is globally repaired to obtain a fourth image feature to ensure consistency in pixel distribution in a fourth monocular view corresponding to the fourth image feature.

[0082] Further, at least one round of weighted iterative processing is performed on the third image feature according to the second image feature and the fifth image feature until the iteration termination condition is met, and the result of the last round of weighted iterative processing is used as the fourth image feature; wherein the weighted iterative processing includes: fusing the second image feature and the fifth image feature with the third image feature respectively, and weighted summing the fusion results. Wherein, for example, the second image feature, the weight C1 and the image feature in the current iteration are multiplied to obtain the first intermediate feature; the fifth image feature, the weight C2 and the image feature in the current cycle are multiplied to obtain the second intermediate feature; the first image feature and the second image feature are added to obtain the image feature of the next iteration. Wherein, in general, the weight C1 and the weight C2 are 1, for example, the weight C1 is 0.2 and the weight C2 is 0.8. As the number of iterations increases, the weight C1 becomes smaller and the weight C2 becomes larger.

[0083] The three-dimensional image generation method provided in the embodiment of the present application has the following advantages:

[0084] (1) Traditional three-dimensional image generation methods require professional three-dimensional equipment for shooting, which is expensive, costly, resource-intensive, and time-consuming, and poses great challenges to human and material resources. The three-dimensional image generation method provided in the embodiment of the present application automatically completes the tasks of two-dimensional image acquisition, spatial information extraction, and three-dimensional image generation through an algorithm, thereby improving the efficiency of three-dimensional image generation. In particular, two-dimensional images are acquired through smart terminals, cameras, or cameras, etc., without the need to acquire binocular images through professional three-dimensional equipment (such as a binocular camera).

[0085] (2) Based on binocular disparity, the two-dimensional image is rendered to obtain a first monocular view with a hole area, and the hole area is repaired based on interpolation of the motion direction to obtain a second monocular view without a hole area. Compared with the traditional rendering method, the interpolation method based on the motion direction can ensure the subsequent foreground segmentation, block repair and global repair effects, and improve the stereoscopic sense of the generated three-dimensional image.

[0086] (3) In order to better meet the binocular stereoscopic effect, the first foreground mask image of the second monocular view is obtained through multi-category segmentation and merged with the hole mask image as the second foreground mask image. This can ensure that the generated content is consistent with the background distribution and the stereoscopic effect has no ghosting.

[0087] (4) The method of repairing the hole area in blocks based on the diffusion model has stronger consistency and versatility than the traditional method. Combined with the content controllable and high-definition method (global repair), the repaired image has higher clarity and can reach 4K.

[0088] In this embodiment, the three-dimensional image generation method is implemented as a service tool, which can be a software implementation form, or a combination of software and hardware or a pure hardware implementation form. In the case of being implemented in software form, the service tool can be deployed locally on the content production end, or on the server end. This embodiment is described by taking the deployment of the service tool on the server end as an example. The content production end can be, but is not limited to: various B-end users that can provide two-dimensional images, such as merchants with a large stock of two-dimensional commodity images, merchants that can provide new two-dimensional commodity images, advertisers that can provide various two-dimensional advertising images, etc. These two-dimensional images are obtained by the content production end through a 2D shooting device. The content production end only needs to upload the two-dimensional image to the server end, and the server end uses the steps in the aforementioned embodiment to generate the corresponding three-dimensional image from the two-dimensional image, and returns the three-dimensional image to the content production end.

[0089] By adopting the method provided in this embodiment, the content production end can use the previously accumulated 2D shooting experience and the entire workflow to create high-quality three-dimensional images. There is no need to purchase expensive professional equipment or build a new workflow based on new equipment. This can greatly improve the creation efficiency of three-dimensional images and reduce creation costs.

[0090] The following takes a merchant as an example to explain the situation of this embodiment in detail. Other content production ends are the same or similar and will not be described in detail here.

[0091] In this embodiment, merchants in the e-commerce field hope to display target products through three-dimensional images, which can more clearly display the details and appearance of the target products and enhance the buyer's viewing experience. For another example, merchants in the e-commerce field hope to display target products through three-dimensional scenes to facilitate users to view the display effect of target products in three-dimensional scenes. For another example, merchants in the tourism industry hope to display the scenery of scenic spots through three-dimensional images, so that viewing users have an immersive feeling, so as to attract more tourists.

[0092] Based on the above-listed application requirements, the merchant can obtain the corresponding two-dimensional image, such as a two-dimensional image of the target product, a two-dimensional scene image, or a two-dimensional landscape image. In this embodiment, the merchant is not limited to the way in which the two-dimensional image is obtained. It can be obtained by using a 2D shooting device to shoot the target product, the actual scene or the actual landscape area, or it can be directly obtained from the image library. The captured image. After that, the merchant uploads the two-dimensional image to the server through its terminal. In this embodiment, the implementation method of the merchant uploading the two-dimensional image to the server through its terminal is not limited. For example, the server can provide an interactive interface for the merchant's terminal. The interactive interface can be a web page, or an application interface in the application corresponding to the server, or a mini program interface. There is an image upload control on the interactive interface. In response to the merchant triggering the image upload control, the terminal uploads the two-dimensional image stored locally to the server; or the terminal can also upload the two-dimensional image to the server by email, shared memory, etc. The server receives the two-dimensional image uploaded by the merchant's terminal, generates a three-dimensional image corresponding to the two-dimensional image using the steps described in the above embodiment, and provides the three-dimensional image to the merchant's terminal. The merchant terminal can receive the three-dimensional image and publish the three-dimensional image on the XR application. Alternatively, when the merchant terminal receives the three-dimensional image, it edits the visualization attribute information of the three-dimensional image, for example, adjusting the brightness and size of the three-dimensional image.

[0093] The embodiment of the present application also provides a three-dimensional video generation method, the method comprising: receiving a two-dimensional image uploaded by a content production end, the two-dimensional image being shot by a 2D shooting device; using the embodiment of the present application provided Figure 1The method shown generates a three-dimensional image corresponding to a two-dimensional image; and returns the three-dimensional image to a content production end so that the content production end can use the three-dimensional image.

[0094] Regarding the embodiments of the present application Figure 1 The detailed implementation and beneficial effects of each step in the method have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0095] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 103 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0096] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0097] Figure 4 A schematic diagram of a three-dimensional image generation device provided by an exemplary embodiment of the present application is shown in FIG. Figure 4 As shown, the device includes: a generation module 41, a block repair module 42, a global repair module 43, and a synthesis module 44.

[0098] A generating module 41 is used to generate a first monocular view with a hole area and a second monocular view without a hole area corresponding to the two-dimensional image based on binocular parallax, wherein the two-dimensional image has a first monocular viewing angle, and the first monocular view and the second monocular view have a second monocular viewing angle;

[0099] A block repair module 42, configured to perform block repair on the hole region in the first monocular view to obtain a third monocular view, wherein the third monocular view has the second monocular viewing angle;

[0100] A global restoration module 43, configured to perform global restoration on the third monocular view according to the second monocular view to obtain a fourth monocular view without a hole area, wherein the fourth monocular view has the second monocular viewing angle;

[0101] The synthesis module 44 is used to synthesize the two-dimensional image as a monocular view under another monocular perspective with the fourth monocular view to obtain a three-dimensional image corresponding to the two-dimensional image.

[0102] In an optional embodiment, when generating the first monocular view and the second monocular view, the generation module 41 is specifically used to: use the two-dimensional image as the monocular view under the first monocular perspective, and determine the relative position offset information of the pixel points in the two-dimensional image under the second monocular perspective based on the binocular parallax; render the two-dimensional image under the second monocular perspective according to the position information and relative position offset information of the pixel points in the two-dimensional image to obtain the first monocular view; and interpolate at least the hole area in the first monocular view according to the movement direction of the first monocular view relative to the two-dimensional image to obtain the second monocular view.

[0103] Optionally, when generating the first monocular view, the generation module 41 is specifically used to: for any pixel point in the two-dimensional image, determine the initial position information of any pixel point in the second monocular perspective based on the position information of any pixel point in the two-dimensional image and the relative position offset information of any pixel point in the second monocular perspective; correct the initial position information of any pixel point in the second perspective based on the initial position information of the surrounding pixels of any pixel point to obtain the target position information of any pixel point in the second monocular perspective; perform rendering processing in the second monocular perspective based on the pixel values ​​and target position information of each pixel point in the two-dimensional image to obtain the first monocular view.

[0104] Optionally, when generating the second monocular view, the generation module 41 is specifically used to: determine the movement direction of the first monocular view relative to the two-dimensional image based on the relative position offset information of the pixel points in the two-dimensional image under the second monocular perspective; based on the movement direction, generate an interpolation operator corresponding to the first monocular view, the interpolation operator includes a first operator value and a second operator value, and the position relationship of the second operator value relative to the first operator value represents the movement direction; and perform interpolation processing on at least the hole area in the first monocular view according to the interpolation operator to obtain the second monocular view.

[0105] In an optional embodiment, when the block repair module 42 performs block repair on the hole area in the first monocular view to obtain the third monocular view, it is specifically used to: generate a first image feature according to the first monocular view, and perform block repair on the first image feature to obtain a third image feature; the third image feature corresponds to the third monocular view.

[0106] In an optional embodiment, when the global repair module 43 performs global repair on the third monocular view according to the second monocular view to obtain a fourth monocular view, it is specifically used to: perform feature extraction on the second monocular view to obtain a second image feature, perform global repair on the third image feature according to the second image feature to obtain a fourth image feature; and generate a fourth monocular view without a hole area according to the fourth image feature.

[0107] In an optional embodiment, when the block repair module 42 performs block repair on the first image feature to obtain the third image feature, it is specifically used to: perform overlapping blocks on the first image feature according to a preset block size and overlap rate to obtain multiple initial block features, and the multiple initial block features have overlapping features; input the multiple initial block features into the image repair network to perform image repair separately to obtain multiple target block features; perform feature fusion on the multiple target block features to obtain the third image feature.

[0108] Optionally, when generating the first image feature according to the first monocular view, the block repair module 42 is specifically used to: fuse the hole mask image corresponding to the first monocular view and the first foreground mask image corresponding to the second monocular view to obtain a second foreground mask image with a hole area; generate a foreground image with a hole area according to the second foreground mask image and the first monocular view; splice the foreground image, the second foreground mask image and the noise image to obtain a spliced ​​image; and perform feature encoding on the spliced ​​image to obtain the first image feature.

[0109] Further optionally, when the global repair module 43 performs global repair on the third image feature according to the second image feature to obtain the fourth image feature, it is specifically used to: perform feature encoding on the second foreground mask image to obtain the fifth image feature; and perform global repair on the third image feature according to the second image feature and the fifth image feature to obtain the fourth image feature.

[0110] Further optionally, when the global repair module 43 performs global repair on the third image feature according to the second image feature and the fifth image feature to obtain the fourth image feature, it is specifically used to: perform at least one round of weighted iterative processing on the third image feature according to the second image feature and the fifth image feature until the iteration termination condition is met, and use the result of the last round of weighted iterative processing as the fourth image feature; wherein the weighted iterative processing includes: respectively fusing the second image feature and the fifth image feature with the third image feature, and performing weighted summation on the fusion results.

[0111] The embodiment of the present application also provides a three-dimensional image generation device, which includes: a receiving module, a generating module and a returning module;

[0112] A receiving module, used to receive a two-dimensional image uploaded by a content production end, where the two-dimensional image is taken by a two-dimensional shooting device;

[0113] Generate modules for using Figure 1 The method shown generates a three-dimensional image corresponding to the two-dimensional image;

[0114] The return module is used to return the three-dimensional image to the content production end so that the content production end can use the three-dimensional image.

[0115] Regarding the embodiments of the present application Figure 4 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0116] Figure 5 A schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the device includes: a memory 54 and a processor 55.

[0117] The memory 54 is used to store computer programs and can be configured to store various other data to support operations on the electronic device. Examples of such data include instructions for any application or method used to operate on the electronic device.

[0118] The processor 55 is coupled to the memory 54 and is used to execute the computer program in the memory 54, so as to: generate a first monocular view with a hole area and a second monocular view without a hole area corresponding to the two-dimensional image based on binocular parallax, the two-dimensional image has a first monocular perspective, and the first monocular view and the second monocular view have a second monocular perspective; perform block-by-block repair on the hole area in the first monocular view to obtain a third monocular view, and the third monocular view has the second monocular perspective; perform global repair on the third monocular view according to the second monocular view to obtain a fourth monocular view without a hole area, and the fourth monocular view has the second monocular perspective; synthesize the two-dimensional image with the fourth monocular view to obtain a three-dimensional image corresponding to the two-dimensional image.

[0119] In an optional embodiment, when the processor 55 generates a first monocular view with a hole area and a second monocular view without a hole area corresponding to a two-dimensional image based on binocular parallax, the processor 55 is specifically used to: use the two-dimensional image as a monocular view under a first monocular perspective, and determine the relative position offset information of pixel points in the two-dimensional image under a second monocular perspective based on binocular parallax; render the two-dimensional image under the second monocular perspective according to the position information and relative position offset information of the pixel points in the two-dimensional image to obtain the first monocular view; and interpolate at least the hole area in the first monocular view according to the movement direction of the first monocular view relative to the two-dimensional image to obtain the second monocular view.

[0120] Optionally, when the processor 55 performs rendering processing on the two-dimensional image from the second monocular perspective based on the position information of the pixel points in the two-dimensional image and the relative position offset information under the second monocular perspective to obtain the first monocular view, the processor 55 is specifically used to: determine, for any pixel point in the two-dimensional image, the initial position information of any pixel point in the second monocular perspective based on the position information of any pixel point in the two-dimensional image and the relative position offset information of any pixel point in the second monocular perspective; correct the initial position information of any pixel point in the second perspective based on the initial position information of the surrounding pixels of any pixel point to obtain the target position information of any pixel point in the second monocular perspective; perform rendering processing from the second monocular perspective based on the pixel values ​​and target position information of each pixel point in the two-dimensional image to obtain the first monocular view.

[0121] Optionally, when the processor 55 interpolates the hole area in the first monocular view according to the movement direction of the first monocular view relative to the two-dimensional image to obtain the second monocular view, it is specifically used to: determine the movement direction of the first monocular view relative to the two-dimensional image according to the relative position offset information of the pixel points in the two-dimensional image under the second monocular perspective; based on the movement direction, generate an interpolation operator corresponding to the first monocular view, the interpolation operator includes a first operator value and a second operator value, and the position relationship of the second operator value relative to the first operator value represents the movement direction; interpolate at least the hole area in the first monocular view according to the interpolation operator to obtain the second monocular view.

[0122] In an optional embodiment, when the processor 55 performs block-by-block repair on the hole area in the first monocular view to obtain a third monocular view, it is specifically used to: generate a first image feature according to the first monocular view, and perform block-by-block repair on the first image feature to obtain a third image feature; the third image feature corresponds to the third monocular view.

[0123] In an optional embodiment, when the processor 55 globally repairs the third monocular view according to the second monocular view to obtain a fourth monocular view, it is specifically used to: extract features from the second monocular view to obtain second image features, globally repair the third image features according to the second image features to obtain fourth image features; and generate a fourth monocular view without a hole area according to the fourth image features.

[0124] In an optional embodiment, when the processor 55 performs block repair on the first image feature to obtain the third image feature, it is specifically used to: perform overlapping blocks on the first image feature according to a preset block size and overlap rate to obtain multiple initial block features, and the multiple initial block features have overlapping features; input the multiple initial block features into the image repair network to perform image repair separately to obtain multiple target block features; perform feature fusion on the multiple target block features to obtain the third image feature.

[0125] Optionally, when the processor 55 generates the first image feature according to the first monocular view, it is specifically used to: fuse the hole mask image corresponding to the first monocular view and the first foreground mask image corresponding to the second monocular view to obtain a second foreground mask image with a hole area; generate a foreground image with a hole area according to the second foreground mask image and the first monocular view; splice the foreground image, the second foreground mask image and the noise image to obtain a spliced ​​image; and perform feature encoding on the spliced ​​image to obtain the first image feature.

[0126] Further optionally, when the processor 55 globally repairs the third image feature according to the second image feature to obtain the fourth image feature, it is specifically used to: feature encode the second foreground mask image to obtain the fifth image feature; and globally repair the third image feature according to the second image feature and the fifth image feature to obtain the fourth image feature.

[0127] Further optionally, when the processor 55 performs global repair on the third image feature according to the second image feature and the fifth image feature to obtain the fourth image feature, it is specifically used to: perform at least one round of weighted iterative processing on the third image feature according to the second image feature and the fifth image feature until the iteration termination condition is met, and use the result of the last round of weighted iterative processing as the fourth image feature; wherein the weighted iterative processing includes: fusing the second image feature and the fifth image feature with the third image feature respectively, and performing weighted summation on the fusion results.

[0128] Regarding the embodiments of the present application Figure 5 The detailed implementation and beneficial effects of each step in the illustrated device have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0129] Further, if Figure 5 As shown, the electronic device also includes: a communication component 56, a display 57, a power component 58, an audio component 59 and other components. Figure 5 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 5 In addition, Figure 5 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 5 If the electronic device of this embodiment is implemented as a server device such as a conventional server, a cloud server or a server array, it may not include Figure 5 Components within the dashed box.

[0130] The embodiment of the present application also provides an electronic device, the implementation structure of the electronic device is Figure 5 The implementation structure of the electronic device shown is the same or similar, and can refer to Figure 5 The electronic device provided in this embodiment is implemented as follows: Figure 5 The difference between the electronic devices in the illustrated embodiments mainly lies in that the functions implemented by the processor executing the computer program stored in the memory are different. For the electronic device provided in this embodiment, its processor executes the computer program stored in the memory, which can be used to: receive a two-dimensional image uploaded by the content production end, the two-dimensional image is taken by a two-dimensional shooting device; use the three-dimensional image generation method provided in the embodiment of the present application to generate a three-dimensional image corresponding to the two-dimensional image; return the three-dimensional image to the content production end, so that the content production end can use the three-dimensional image.

[0131] The detailed implementation and beneficial effects of the electronic device provided in the embodiments of the present application have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0132] Accordingly, the embodiment of the present application further provides a computer-readable storage medium storing a computer program, which can achieve the above-mentioned Figure 1 The steps in the method embodiment shown may be executed by an electronic device.

[0133] Accordingly, the embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, causes the processor to implement Figure 1 Steps in the method shown.

[0134] The above-mentioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0135] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0136] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0137] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0138] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage, etc.) containing computer-usable program code.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0144] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0145] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0147] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A three-dimensional image generation method, characterized in that: include: Based on binocular disparity, a first monocular view with a hole area and a second monocular view without a hole area corresponding to the two-dimensional image are generated, wherein the two-dimensional image has a first monocular perspective, and the first monocular view and the second monocular view have a second monocular perspective; Repairing the hole area in the first monocular view in blocks to obtain a third monocular view, where the third monocular view has the second monocular viewing angle; Performing global repair on the third monocular view according to the second monocular view to obtain a fourth monocular view without a hole area, wherein the fourth monocular view has the second monocular viewing angle; The two-dimensional image is synthesized with the fourth monocular view to obtain a three-dimensional image corresponding to the two-dimensional image.

2. The method according to claim 1, characterized in that Based on binocular disparity, a first monocular view with a hole area and a second monocular view without a hole area corresponding to a two-dimensional image are generated, including: Taking the two-dimensional image as a monocular view under a first monocular perspective, and determining relative position offset information of pixel points in the two-dimensional image under a second monocular perspective based on binocular parallax; According to the position information of the pixel points in the two-dimensional image and the relative position offset information, rendering processing is performed on the two-dimensional image under the second monocular perspective to obtain the first monocular view; According to the movement direction of the first monocular view relative to the two-dimensional image, at least the hole region in the first monocular view is interpolated to obtain the second monocular view.

3. The method according to claim 2, characterized in that According to the position information of the pixel points in the two-dimensional image and the relative position offset information in the second monocular perspective, rendering processing is performed on the two-dimensional image in the second monocular perspective to obtain the first monocular view, including: For any pixel point in the two-dimensional image, determine the initial position information of the any pixel point in the second monocular perspective according to the position information of the any pixel point in the two-dimensional image and the relative position offset information of the any pixel point in the second monocular perspective; Correcting the initial position information of any pixel point at the second viewing angle according to the initial position information of surrounding pixels of any pixel point to obtain the target position information of any pixel point at the second monocular viewing angle; Rendering processing is performed under the second monocular perspective according to the pixel value of each pixel point in the two-dimensional image and the target position information to obtain the first monocular view.

4. The method according to claim 2, characterized in that: The method of interpolating a hole region in the first monocular view according to a movement direction of the first monocular view relative to the two-dimensional image to obtain the second monocular view includes: determining a movement direction of the first monocular view relative to the two-dimensional image according to relative position offset information of pixels in the two-dimensional image under the second monocular view; Based on the motion direction, generating an interpolation operator corresponding to the first monocular view, the interpolation operator comprising a first operator value and a second operator value, and a positional relationship of the second operator value relative to the first operator value represents the motion direction; An interpolation process is performed on at least a hole region in the first monocular view according to the interpolation operator to obtain the second monocular view.

5. The method according to any one of claims 1 to 4, characterized in that: Repairing the hole area in the first monocular view in blocks to obtain a third monocular view includes: Generate a first image feature according to the first monocular view, and perform block-by-block restoration on the first image feature to obtain a third image feature; the third image feature corresponds to the third monocular view; Performing global repair on the third monocular view according to the second monocular view to obtain a fourth monocular view without a hole area, comprising: Performing feature extraction on the second monocular view to obtain a second image feature, and performing global restoration on the third image feature according to the second image feature to obtain a fourth image feature; A fourth monocular view without a hole area is generated according to the fourth image feature.

6. The method according to claim 5, characterized in that The first image feature is repaired in blocks to obtain a third image feature, comprising: According to a preset block size and overlap rate, overlappingly dividing the first image feature into blocks to obtain a plurality of initial block features, wherein the plurality of initial block features have overlapping features; Inputting the multiple initial block features into an image restoration network to perform image restoration respectively, to obtain multiple target block features; The plurality of target block features are subjected to feature fusion to obtain the third image feature.

7. The method according to claim 5, characterized in that Generating a first image feature according to the first monocular view includes: Fusing the hole mask image corresponding to the first monocular view and the first foreground mask image corresponding to the second monocular view to obtain a second foreground mask image with a hole area; generating a foreground image with a hole area according to the second foreground mask image and the first monocular view; splicing the foreground image, the second foreground mask image and the noise image to obtain a spliced ​​image; Feature encoding is performed on the spliced ​​image to obtain the first image feature.

8. The method according to claim 7, characterized in that Performing global restoration on the third image feature according to the second image feature to obtain a fourth image feature includes: Performing feature encoding on the second foreground mask image to obtain a fifth image feature; The third image feature is globally repaired according to the second image feature and the fifth image feature to obtain the fourth image feature.

9. The method according to claim 8, characterized in that According to the second image feature and the fifth image feature, the third image feature is globally repaired to obtain a fourth image feature, including: performing at least one round of weighted iterative processing on the third image feature according to the second image feature and the fifth image feature until an iteration termination condition is met, and taking a result of the last round of weighted iterative processing as the fourth image feature; The weighted iterative processing includes: fusing the second image feature and the fifth image feature with the third image feature respectively, and performing weighted summation on the fusion results.

10. A three-dimensional image generation method, characterized in that: include: Receiving a two-dimensional image uploaded by a content production end, wherein the two-dimensional image is taken by a two-dimensional shooting device; Using the method according to any one of claims 1 to 9, generating a three-dimensional image corresponding to the two-dimensional image; The three-dimensional image is returned to the content production end so that the content production end can use the three-dimensional image.

11. An electronic device, characterized in that: include: Memory and processor; The memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to implement the steps in the method according to any one of claims 1 to 9 and claim 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to implement the steps of the method according to any one of claims 1 to 9 and claim 10.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the processor is caused to implement the steps in the method of any one of claims 1 to 9 and claim 10.

Images