3D image generation method and device, equipment and storage medium

Abstract

The invention provides a 3D image generation method and device, equipment and a storage medium. The method comprises the steps of obtaining a left view of a target environment, and obtaining a disparity map according to the left view; wherein the left view represents a target environment seen by the left eye, and the disparity map represents a pixel difference value of the left view and the right view; obtaining a right view of the target environment according to the left view and the disparity map; wherein the right view represents a target environment seen by the right eye; and obtaining a 3D image of the target environment according to the left view, the disparity map and the right view. Through the left view and the disparity map, the obtained right view is clearer, so that the finally obtained 3D graph is clearer.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for generating a 3D image. Background Art

[0002] 2D to 3D technology is a technology that converts a flat 2D image or video into a 3D image or video with a sense of depth. This technology is widely used in many fields such as movies, games, virtual reality (VR), augmented reality (AR) and medical imaging.

[0003] In the process of generating 3D videos from 2D images in the prior art, the converted 3D videos have quality problems such as holes and blurs, which makes it difficult to meet actual business needs. Summary of the invention

[0004] The present application provides a 3D image generation method, device, equipment and storage medium to solve the quality problems of holes and blurs in converted 3D images.

[0005] In a first aspect, the present application provides a method for generating a 3D image, comprising:

[0006] Obtain a left view of the target environment, and obtain a disparity map based on the left view; wherein the left view represents the target environment seen by the left eye, and the disparity map represents the pixel difference between the left and right views;

[0007] According to the left view and the disparity map, a right view of the target environment is obtained; wherein the right view represents the target environment seen by the right eye;

[0008] According to the left view, the disparity map, and the right view, a 3D image of the target environment is obtained.

[0009] Optionally, the above method obtains a disparity map according to the left view, including:

[0010] According to the left view, based on a preset monocular depth estimation model, a depth map of the left view is obtained; wherein the preset monocular depth estimation model is a preset neural network model, which is used to determine the depth information of the pixel points, and the depth map represents the depth information of the pixel points in the left view in a preset camera coordinate system;

[0011] According to the depth map, a disparity map is obtained.

[0012] Optionally, in the above method, the disparity map is:

[0013] disparity = B × f / Depth;

[0014] Among them, disparity is the disparity map, B and f are preset parameters, and Depth is the depth map.

[0015] Optionally, the above method, according to the left view and the disparity map, obtains the right view of the target environment, including:

[0016] Performing dimensionality reduction processing on the left view, and connecting the left view after dimensionality reduction processing and the disparity map to obtain a first intermediate feature matrix; wherein the intermediate feature matrix represents the left view and the disparity map;

[0017] The preset noise characteristic matrix is ​​connected with the first intermediate characteristic matrix to obtain a second intermediate characteristic matrix;

[0018] Inputting the second intermediate feature matrix into a preset denoising model for denoising to obtain a denoised image code;

[0019] The denoised image code is decoded to obtain the right view of the target environment.

[0020] Optionally, the above method further comprises:

[0021] Obtaining a data set to be trained; wherein the data to be trained includes a left view to be trained, a disparity map to be trained, and an actual right view corresponding to the left view to be trained;

[0022] The initial model is trained according to the left view to be trained, the disparity map to be trained, and the actual right view to obtain a trained image generation model; wherein the image generation model is used to output the right view according to the left view and the disparity map.

[0023] Optionally, the above method further comprises:

[0024] Determine a right view to be trained according to the left view to be trained and the disparity map to be trained;

[0025] According to the right view to be trained and the actual right view, a new disparity map is obtained;

[0026] Input the right view to be trained and the new disparity map into the initial model to obtain a new right view;

[0027] If the new right view and the actual right view Figure 1 If the training result is the same as the actual result, the initial model is trained according to the left view to be trained, the disparity map to be trained, and the actual right view.

[0028] Optionally, the above method trains the initial model according to the left view to be trained, the disparity map to be trained, and the actual right view to obtain a trained image generation model, including:

[0029] Based on the initial model, the left view to be trained is subjected to dimensionality reduction processing, and the left view to be trained and the disparity map to be trained after the dimensionality reduction processing are connected to obtain an intermediate feature matrix to be trained; wherein the intermediate feature matrix represents the left view to be trained and the disparity map to be trained;

[0030] Adding the preset noise feature matrix to the actual right view to obtain the noisy actual right view matrix;

[0031] Connect the noisy actual right view matrix and the intermediate feature matrix to be trained to obtain the intermediate predicted right view matrix;

[0032] The intermediate predicted right view matrix is ​​input into a preset denoising model for denoising to obtain a predicted right view;

[0033] If the actual right view and the predicted right view Figure 1 If the training is complete, the image generation model is obtained.

[0034] Optionally, as in the above method, a preset noise feature matrix is ​​added to the actual right view to obtain a noisy actual right view matrix, including:

[0035] Performing dimensionality reduction processing on the actual right view to obtain the actual right view after dimensionality reduction processing;

[0036] The preset noise feature matrix is ​​added to the actual right view after the dimensionality reduction process to obtain the noisy actual right view matrix.

[0037] In a second aspect, the present application provides a 3D image generation device, comprising:

[0038] The first obtaining module is used to obtain a left view of the target environment and obtain a disparity map according to the left view; wherein the left view represents the target environment seen by the left eye, and the disparity map represents the pixel difference between the left and right views;

[0039] The second obtaining module is used to obtain a right view of the target environment according to the left view and the disparity map; wherein the right view represents the target environment seen by the right eye;

[0040] The third obtaining module is used to obtain a 3D image of the target environment according to the left view, the disparity map, and the right view.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0042] Memory stores computer-executable instructions;

[0043] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0046] The present application provides a method, device, equipment and storage medium for generating 3D images, which obtains a left view of a target environment and obtains a disparity map based on the left view; generates a disparity map through the left view, and can generate a clear right view; wherein the left view represents the target environment seen by the left eye, and the disparity map represents the pixel difference between the left and right views; obtains a right view of the target environment based on the left view and the disparity map; wherein the right view represents the target environment seen by the right eye; obtains a 3D image of the target environment based on the left view, the disparity map, and the right view. Through the left view and the disparity map, the obtained right view is clearer, and the occurrence of holes in the right view is reduced, so that the final 3D graphics are clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 A schematic diagram of a process for generating a 3D image provided in this application;

[0049] Figure 2 A schematic diagram of a flow chart of another method for generating a 3D image provided by the present application;

[0050] Figure 3 A flowchart of a method for confirming a training dataset is provided for the application;

[0051] Figure 4 A schematic diagram of the structure of a device for generating a 3D image is provided for the application;

[0052] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0053] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0054] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0055] At present, in the process of generating 3D video from 2D images, the disparity map is usually obtained through the left and right views, and then the 3D image is obtained through the disparity map and the left and right views. However, the generated 3D image often has problems such as blur and holes. This is because the internal left view image and the right view image will inevitably have invisible areas when calculating the 3D image due to the difference in viewing angles. When these areas are calculated by difference to obtain another view, blur and holes will occur.

[0056] The present application provides a method, device, equipment and storage medium for generating 3D images, which obtains a left view of a target environment and obtains a disparity map based on the left view; generates a disparity map through the left view, and can generate a clear right view; wherein the left view represents the target environment seen by the left eye, and the disparity map represents the pixel difference between the left and right views; obtains a right view of the target environment based on the left view and the disparity map; wherein the right view represents the target environment seen by the right eye; obtains a 3D image of the target environment based on the left view, the disparity map, and the right view. Through the left view and the disparity map, the obtained right view is clearer, and no hole phenomenon occurs, so that the final 3D graphics are clearer.

[0057] First, the terms involved in this application are explained:

[0058] 3D (Three-Dimensional) images: refers to images that represent objects in three-dimensional space. Unlike traditional two-dimensional images (such as photos or paintings), 3D images contain not only width and height, but also depth information. This allows 3D images to more realistically simulate objects and scenes in the real world.

[0059] Left view: In 3D movie production and display, the left view usually refers to the image seen from the audience's left eye.

[0060] Right view: In 3D movie production and display, it refers to the image seen from the right eye.

[0061] Parallax map: Also known as parallax mapping or parallax mapping, it is a technique in computer graphics used to enhance the depth and details of two-dimensional textures. It makes flat surfaces appear three-dimensional and have complex geometric structures by simulating the changes in surface details when the viewing angle changes.

[0062] Depth Map: A depth map is an image used to represent the depth information of each point in a scene. In a depth map, the value of each pixel represents the distance from the point to the camera or sensor. Depth maps are widely used in computer vision, 3D reconstruction, augmented reality (AR), virtual reality (VR), and autonomous driving.

[0063] Image encoding is the process of converting image data into a specific format for easy storage, transmission and processing.

[0064] Figure 1 A method for generating a 3D image is provided in an embodiment of the present application. Figure 1 As shown, the method can be performed by an electronic device, and this embodiment is not particularly limited here. Figure 1 As shown, the method includes:

[0065] S101 , obtaining a left view of a target environment, and obtaining a disparity map based on the left view; wherein the left view represents the target environment seen by the left eye, and the disparity map represents the pixel difference between the left and right views.

[0066] The target environment refers to the environment where the content of the left view is located. For example, the left view of the target environment can be a car picture or a motorcycle picture, etc. If the content of the left view is a car, then the corresponding right view content is also a car.

[0067] The left view is used to simulate the environment seen by the left eye of a person, and the disparity map is used to simulate the disparity of the human eye. The left view can be directly acquired through an image acquisition device such as a camera, and the disparity map is obtained by processing the pixels in the left view.

[0068] In this embodiment, a disparity map is obtained according to the left view, including:

[0069] According to the left view, based on a preset monocular depth estimation model, a depth map of the left view is obtained; wherein the preset monocular depth estimation model is a preset neural network model, which is used to determine the depth information of the pixel points, and the depth map represents the depth information of the pixel points in the left view in a preset camera coordinate system;

[0070] According to the depth map, a disparity map is obtained.

[0071] Among them, the preset monocular depth estimation model is used to obtain the depth map of the left view according to the left view. The preset monocular depth estimation model can be: marigold monocular depth estimation model, MiDAS (Muti-Factor Mixed Data Sampling, multi-factor mixed data sampling prediction) model, ZoeDepth (Zero-shot Transfer by Combining Relative and Metric Depth) depth estimation graph generation model. In this embodiment, the model structure of the preset monocular depth estimation model is not specifically limited.

[0072] The disparity map represents the pixel displacement between the left view and the right view of the same scene.

[0073] By inputting the left view into the preset monocular depth estimation model, the depth map of the left view can be obtained, and the disparity map can be obtained through the depth map. Specifically, the depth map can be substituted into the preset formula to calculate the disparity map. The depth map refers to a two-dimensional image that stores the depth values ​​of all pixels in a single view. It is a spatial unit, such as millimeters. The depth map is a two-dimensional image, the same size as the original image (i.e., the left view), and the same size as the disparity map. Each position in the depth map stores the depth value of the pixel at that position, and the depth value is the Z coordinate value in the camera coordinate system.

[0074] Specifically, the disparity map is calculated according to the formula disparity = B × f / Depth, where b and f are selected within a preset range, Depth is the depth map, and the left view is input into the depth map estimation network marigold to automatically generate the depth map Depth, and disparity represents the disparity map.

[0075] The advantage of this setting is that the depth map is generated by the monocular depth estimation model and then converted into a disparity map. Compared with the traditional stereo matching method (which needs to process the left view and the right view at the same time), the computational complexity may be lower. At the same time, by generating a depth map from the left view and converting it into a disparity map, efficient and accurate depth information acquisition can be achieved in multiple fields, bringing significant technical effects and application value.

[0076] In this embodiment, the disparity map is:

[0077] disparity = B × f / Depth;

[0078] Among them, disparity is the disparity map, B and f are preset parameters, and Depth is the depth map.

[0079] Among them, B represents the baseline length. In stereo vision, the baseline length refers to the distance between two cameras. This distance is crucial for calculating depth information and generating three-dimensional images. f represents the focal length, which refers to the distance from the center point of a lens or lens to its focus. The value range of B and the value range of f are preset. Usually, the value of B is any value between 50mm-200mm, and the value of f is any value between 4mm-50mm.

[0080] By selecting a value between 50mm-200mm as the B value and a value between 4mm-50mm as the f value, the depth map Depth is obtained, and then B, f, and Depth are substituted into the formula disparity = B×f / Depth to obtain the disparity map disparity.

[0081] The advantage of this setting is that by directly converting the depth information into disparity information, this direct relationship simplifies the conversion process from depth map to disparity map. At the same time, the random selection of B and f values ​​can solve the problem of invisible pixels in the intrinsic perspective.

[0082] S102 . Obtain a right view of the target environment according to the left view and the disparity map; wherein the right view represents the target environment seen by the right eye.

[0083] Among them, the left view and the disparity map can be input into the preset model to obtain the right view of the target environment. The preset models can be: stable diffusion model, Midjourney model, imagen model. Stable Diffusion is a generative model based on the diffusion process, which is usually used for image generation and image restoration tasks. Midjourney is an artificial intelligence model for generating images, mainly used for artistic creation and image generation. Imagen can generate high-quality images based on the input text description by combining a large language model and a diffusion model. In this embodiment, the model structure of the preset model is not specifically limited.

[0084] In this embodiment, the right view of the target environment is obtained according to the left view and the disparity map, including:

[0085] Performing dimensionality reduction processing on the left view, and connecting the left view after dimensionality reduction processing and the disparity map to obtain a first intermediate feature matrix; wherein the intermediate feature matrix represents the left view and the disparity map;

[0086] The preset noise characteristic matrix is ​​connected with the first intermediate characteristic matrix to obtain a second intermediate characteristic matrix;

[0087] Inputting the second intermediate feature matrix into a preset denoising model for denoising to obtain a denoised image code;

[0088] The denoised image code is decoded to obtain the right view of the target environment.

[0089] Among them, dimensionality reduction is a commonly used technology in data processing, which aims to reduce the number of features of a data set while retaining the main information of the data as much as possible. This technology is widely used in machine learning, data mining, image processing and other fields. Dimensionality reduction can help solve the problems caused by high-dimensional data, such as high computational complexity, large storage requirements, and model overfitting. Here, it means converting high-dimensional image data into low-dimensional data by some method, while retaining the main features and information of the original data as much as possible.

[0090] The connection process refers to directly connecting the left view and the disparity map after the dimensionality reduction process. The dimensions of both sides of the connection process need to be consistent. For example, the dimension of the left view after the dimensionality reduction process is three-dimensional, and the dimension of the disparity map is also three-dimensional. Then, after the connection process, the dimension of the first intermediate feature matrix is ​​six-dimensional.

[0091] The first intermediate matrix and the second intermediate matrix have no special meanings, and are both used to represent the products of the corresponding intermediate connection processing. Specifically, the first intermediate feature matrix is ​​used to represent the product obtained by connecting the left view and the disparity map after dimensionality reduction processing, and the second intermediate feature matrix is ​​used to represent the product obtained by connecting the preset noise feature matrix and the first intermediate feature matrix.

[0092] The preset noise feature matrix refers to artificially adding noise to the original image to simulate various interference situations that may occur in the real world. Noise refers to random or systematic interference in the image that does not belong to the original signal, usually manifested as random brightness or color changes in the image. Here, the preset noise feature matrix and the first intermediate feature matrix are connected. The preset noise feature matrix can be Gaussian noise.

[0093] Denoising refers to image denoising, which is an image processing technology that aims to restore a clear image as close to the original image as possible from an image contaminated by noise.

[0094] The denoising model is used to denoise the second intermediate feature matrix. The denoising matrix can be a U-Net model, which is a convolutional neural network (CNN) model used for image segmentation.

[0095] The left view is subjected to dimensionality reduction processing, and the left view can be reduced to a 3D image. The left view after dimensionality reduction processing can be understood as a thumbnail of the left view. Then, the left view after dimensionality reduction processing and the scaled disparity map are connected to obtain the first intermediate feature matrix. Note that the dimensions of the left view and the disparity map need to be consistent. If the left view Figure 1 Generally, the disparity map is three-dimensional, so the disparity map is also three-dimensional. Then, the first intermediate matrix and Gaussian noise are connected to obtain the second intermediate feature matrix. The second intermediate feature matrix is ​​input into the U-Net model. After T steps of denoising, the denoised image code is obtained. The denoised image code is decoded to obtain the right view of the target environment. Scaling refers to the operation of enlarging or reducing the image (i.e., the disparity map). Scaling and dimensionality reduction are to make the disparity map and the left view have the same final dimensions so that they can be connected to obtain the first intermediate feature matrix. That is, the left view and the disparity map have the same dimensions at the beginning. When the left view is reduced in dimension, the dimensions become inconsistent. Therefore, after the left view is reduced in dimension, the disparity map needs to be scaled to make the left view consistent with the disparity map. For example, the disparity map originally has the same dimension (or size) as the image, such as 128*128. After the image is reduced in dimension, it becomes 16*16. The disparity map is directly scaled to the same 16*16 by using the resize method of opencv (Open Source Computer Vision Library).

[0096] The advantages of this setting are: the left view is simplified through dimensionality reduction processing, which improves the computational efficiency. The information of the left view and the disparity map is integrated through feature matrix connection, which enhances the integrity and robustness of feature representation. The noise is effectively removed through denoising processing, which improves the image quality. The decoding process restores the denoised image encoding to a high-quality right view, ensuring the accuracy and clarity of the generated image.

[0097] S103 , obtaining a 3D image of the target environment according to the left view, the disparity map, and the right view.

[0098] Among them, if the 3D image is a 3D movie, the left view and the right view are made to be at the same level through the disparity map, thereby producing the effect of a 3D image, and then through the polarizing filter, the left view, the right view, and the disparity map, the 3D image effect can be seen when people watch.

[0099] In this embodiment, it also includes:

[0100] Obtaining a data set to be trained; wherein the data to be trained includes a left view to be trained, a disparity map to be trained, and an actual right view corresponding to the left view to be trained;

[0101] The initial model is trained according to the left view to be trained, the disparity map to be trained, and the actual right view to obtain a trained image generation model; wherein the image generation model is used to output the right view according to the left view and the disparity map.

[0102] The training data set is a data set used for model training, and can be a data set of 3D movies. If it is a data set of 3D movies, then the data set of 3D movies includes: a left view of the 3D movie, and an actual right view of the 3D movie corresponding to the left view of the 3D movie. The disparity map is also obtained based on the left view of the 3D movie.

[0103] The initial model can be any one of the stable diffusion model, midjourney model, and imagen model.

[0104] A 3D movie dataset is obtained, and a disparity map is obtained according to the left view of the 3D movie dataset. If the initial model is a stable diffusion model, the initial model is trained according to the left view of the 3D movie, the actual right view of the 3D movie corresponding to the left view of the 3D movie, and the disparity map. When the error between the right view obtained by the stable diffusion model and the actual right view is small, a trained stable diffusion model is obtained. Small error may mean that the right view obtained by the model and the actual right view of the 3D movie are highly similar. For example, small error may mean that the pixel difference between the right view obtained by the model and the actual right view of the 3D movie can be less than a preset threshold. Alternatively, small error may mean that the similarity between the right view obtained by the model and the actual right view of the 3D movie reaches a preset threshold.

[0105] The advantage of this setting is that the image generation model can be trained based on the left view, disparity map and actual right view to generate a high-quality right view, which can be applied to various computer vision and image processing tasks and a wide range of applications, such as: optimization of 2D to 3D effects of video processing software, virtual reality, augmented reality, etc.

[0106] In this embodiment, it also includes:

[0107] Determine a right view to be trained according to the left view to be trained and the disparity map to be trained;

[0108] According to the right view to be trained and the actual right view, a new disparity map is obtained;

[0109] Input the right view to be trained and the new disparity map into the initial model to obtain a new right view;

[0110] If the new right view and the actual right view Figure 1If the training result is the same as the actual result, the initial model is trained according to the left view to be trained, the disparity map to be trained, and the actual right view.

[0111] Among them, the new right view and the actual right view Figure 1 The new right view and the actual right view are highly similar. Figure 1 The pixel difference between the right view obtained by the model and the actual right view of the 3D movie can be less than a preset threshold. Figure 1 It can mean that the similarity between the two reaches a preset threshold.

[0112] If the left view to be trained is L0, the disparity map to be trained is d0, and the right view to be trained is R1, then the method for obtaining the right view R1 is:

[0113] x' = xd;

[0114] y'=y;

[0115] The coordinate value of the pixel point in the left view is (x, y), the coordinate value of the pixel point in the right view is (x', y'), and d is the disparity map to be trained.

[0116] The right view to be trained R1 and the actual right view R0 are input into the preset Raft model (RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching). Raft is a public model and can be used directly to obtain a new disparity map. The new disparity map and the right view to be trained R1 are input into the stable diffusion model to obtain a new right view R2. If the new right view R2 is consistent with the actual right view R0, it represents that the actual right view R0 can be used as a true value, so that the initial model can be trained through the left view to be trained, the disparity map to be trained, and the actual right view to obtain a trained image generation model. The trained image generation model can be a stable diffusion model.

[0117] The advantage of this setting is that this method can significantly improve the performance of the stereo matching model through supervised learning, provide higher quality disparity maps and right views, and thus achieve better results in practical applications.

[0118] In this embodiment, the initial model is trained according to the left view to be trained, the disparity map to be trained, and the actual right view to obtain a trained image generation model, including:

[0119] Based on the initial model, the left view to be trained is subjected to dimensionality reduction processing, and the left view to be trained and the disparity map to be trained after the dimensionality reduction processing are connected to obtain an intermediate feature matrix to be trained; wherein the intermediate feature matrix represents the left view to be trained and the disparity map to be trained;

[0120] Adding the preset noise feature matrix to the actual right view to obtain the noisy actual right view matrix;

[0121] Connect the noisy actual right view matrix and the intermediate feature matrix to be trained to obtain the intermediate predicted right view matrix;

[0122] The intermediate predicted right view matrix is ​​input into a preset denoising model for denoising to obtain a predicted right view;

[0123] If the actual right view and the predicted right view Figure 1 If the training is complete, the image generation model is obtained.

[0124] The left view to be trained is a known image, which is used to input the model to obtain the predicted right view.

[0125] The preset noise may be Gaussian noise.

[0126] The denoising model may be a U-Net model, which is a Convolutional Neural Network (CNN) model used for image segmentation.

[0127] If the actual right view and the predicted right view Figure 1 If the actual right view and the predicted right view are consistent, the representation model training is completed. Figure 1 If the preset model is consistent with the training image, it is sufficient to retrain the preset model based on the left view to be trained, the disparity map to be trained, and the actual right view.

[0128] Connecting means connecting two corresponding things.

[0129] The intermediate feature matrix to be trained and the intermediate predicted right view matrix have no special meaning. Here, they refer to the corresponding connection products. Specifically, the left view to be trained and the disparity map to be trained after dimensionality reduction are connected, and the product obtained after the connection is the intermediate feature matrix to be trained. The product obtained by connecting the noisy actual right view matrix and the intermediate feature matrix to be trained is the intermediate predicted right view matrix.

[0130] The initial model and the trained image generation model are corresponding and identical models. For example, if the initial model is a stable diffusion model, then the trained image generation model is also a stable diffusion model, and the initial model is an unconverged model, that is, the predicted right view obtained by the initial model may be different from the actual right view. Figure 1 The right view predicted by the trained image generation model and the actual right view Figure 1 Must be consistent.

[0131] When the initial model is a stable diffusion model, the steps of processing the image in sequence are: dimensionality reduction, denoising, and decryption. Therefore, it is based on the initial model.

[0132] The noisy actual right view matrix refers to the right view after adding noise. After the left view to be trained is reduced in dimension (at this time, the left view to be trained becomes a three-dimensional left view to be trained), the reduced-dimensional left view to be trained is connected in parallel with the scaled disparity map to be trained (at this time, a six-dimensional intermediate feature matrix to be trained is obtained), the actual right view is reduced in dimension (at this time, the right view to be trained becomes a three-dimensional right view to be trained), and then random Gaussian noise is added to the actual right view to obtain the noisy actual right view matrix, the noisy actual right view matrix and the intermediate feature matrix to be trained are connected to obtain the intermediate predicted right view matrix, and then the intermediate predicted right view matrix is ​​input into the U-Net model to train and generate some parameters of the U-Net model. At the same time, the intermediate predicted right view matrix is ​​denoised to obtain the predicted right view. If the actual right view and the predicted right view are Figure 1 If the training is complete, the image generation model is obtained.

[0133] The benefit of this setting is that it can significantly improve the performance of the image generation model, including image quality, robustness, training efficiency, and prediction accuracy.

[0134] In this embodiment, a preset noise feature matrix is ​​added to the actual right view to obtain a noisy actual right view matrix, including:

[0135] Performing dimensionality reduction processing on the actual right view to obtain the actual right view after dimensionality reduction processing;

[0136] The preset noise feature matrix is ​​added to the actual right view after the dimensionality reduction process to obtain the noisy actual right view matrix.

[0137] The preset noise characteristic matrix may be Gaussian noise.

[0138] The actual right view is subjected to dimensionality reduction processing to obtain a three-dimensional actual right view (i.e., the actual right view after dimensionality reduction processing), and a preset noise feature matrix is ​​added to the actual right view after dimensionality reduction processing to obtain a noisy actual right view matrix. Specifically, the dimensionality reduction processing can be performed by the EncoderNetwork of the vae (Variational Autoencoder) of the generated network.

[0139] The advantage of this setting is that by reducing the dimensionality of the actual right view and adding a preset noise feature matrix, the performance of the image generation and processing system can be significantly improved, including computational efficiency, model robustness, and generalization ability.

[0140] The embodiments of the present application provide a 3D image generation method, device, equipment and storage medium. The left view and the disparity map are input into a preset model to train a trained image generation model. The trained image generation model can generate a high-quality, clear and hole-free right view. At the same time, the problem of coordinate deviation is solved by randomly selecting preset parameters, so that the final 3D image is clearer.

[0141] Figure 2 A method for generating a 3D image is provided in an embodiment of the present application. Figure 2 As shown, the execution subject of the method can be a server, and this embodiment does not make any special restrictions here. Figure 2 As shown, the method includes:

[0142] S201 , confirming a training data set, wherein the training data set includes a predicted right view R1 , a real right view R0 , and a disparity map d1 .

[0143] in, Figure 3 A method for confirming a training data set is provided in an embodiment of the present application. Figure 3 As shown, the execution subject of the method can be a server, and this embodiment does not make any special restrictions here. Figure 3 As shown, the method includes:

[0144] Get a real 2D left view L0 and a real right view R0 corresponding to L0.

[0145] The left view image L0 is processed using the depth map estimation model to generate the corresponding depth map Depth, and then the corresponding disparity map d0 is calculated according to the formula disparity = Bf / Depth. Among them, B and f can be randomly selected, the value range of B is: any value between 4mm-50mm, the value range of f is: any value between 4mm-50mm, disparity is the disparity map, and Depth is the depth map.

[0146] According to the disparity map d0 and the left view L0, the corresponding right view R1 is calculated. According to the disparity map d0 and the left view L0, the corresponding right view R1 is calculated by: inputting the right view R1 and the real right view R0 into the Raft model (RAFT-Stereo: Multilevel Recurrent Field Transforms for StereoMatching) to calculate the new disparity map d1.

[0147] The disparity map d1 and the right view R1 are taken as input and input into the stable diffusion model, which outputs a new right view R2. The generated right view R2 is consistent with the real right view R0 in coordinates, so R0 can be used directly as the ground truth. Therefore, the stable diffusion model can be trained with the right view R0. In machine learning and data science, ground truth refers to real-world data that is directly observed, measured, or collected. These data are considered to be accurate and can be used as a benchmark for judging and verifying the accuracy of other data or models. In the field of machine learning, ground truth usually refers to the labels or classification results in the data set used for training.

[0148] S202: Train a stable diffusion model according to the training data set, and obtain a trained stable diffusion model.

[0149] According to the training data set, the stable diffusion model is trained and the method of obtaining the trained stable diffusion model is as follows:

[0150] The predicted right view R1 is processed by the encoder network of the generative network vae (Variational Autoencoder) for dimensionality reduction, and then connected in parallel with the scaled disparity map d1 to obtain the parallel product Z (x), the right view d1 is also processed by vae's Encoder Network for dimensionality reduction, and then random Gaussian noise is added to obtain the noisy right view d1, and then the parallel product Z(x) and the disparity map d1 are connected in parallel to obtain the parallel result, and then the parallel result is input into the U-Net of the training generation network, and the parameters of some U-Net are trained. After T steps of denoising operation, the denoised image encoding is obtained, and the denoised image encoding is decoded to obtain the predicted right view R2, and the loss of the right view R2 and the real right view R0 is calculated (for example, the pixel difference between R2 and R0). If the loss is within the preset range at this time, the trained stable diffusion model is obtained. If the loss is not within the preset range at this time, the stable diffusion model is retrained according to the training data set.

[0151] S203, calculating a disparity map according to the left view, and inputting the disparity map and the left view into the trained model to obtain a right view.

[0152] The disparity map is calculated based on the left view, and the disparity map and the left view are input into the trained model to obtain the right view:

[0153] According to the left view, the corresponding depth map can be obtained through the depth map estimation model, and then the corresponding disparity map can be calculated according to the formula disparity = Bf / Depth. Then, the VAE of the generative network is used to reduce the dimension of the left view, and the reduced-dimensional left view and the disparity map are connected to obtain an intermediate connection product. Then, the left view-disparity map is connected with random Gaussian noise again to obtain a predicted connection product. The predicted connection product is input into the U-Net model, and after T-step denoising operation, the denoised image code is obtained. Then, the decoder Network of the VAE of the generative network is partially decoded to obtain the final high-quality right view without holes.

[0154] Another 3D image generation method, device, equipment and storage medium provided in the embodiments of the present application outputs a high-quality right view without holes by inputting a left view image and a disparity map corresponding to the left view into a stable diffusion model. At the same time, because B and f can be selected arbitrarily, the coordinate position of the generated right view R1 can be controlled to be at any position between L0 and R0, that is, the coordinate distance between the input and output during training will not be much different from the coordinate distance obtained in the actual application. This method not only improves the quality of the right view, but also optimizes the performance of the generated network by comparing it with the actual right view.

[0155] Figure 4The schematic diagram of the structure of a 3D image generation device provided in an embodiment of the present application. The device of this embodiment can be in the form of software and / or hardware. Figure 4 As shown, a 3D image generation device 40 provided in an embodiment of the present application includes: an acquisition module 401, a determination module 402, and a processing module 403, wherein:

[0156] The first obtaining module 401 is used to obtain a left view of the target environment and obtain a disparity map according to the left view; wherein the left view represents the target environment seen by the left eye, and the disparity map represents the pixel difference between the left and right views;

[0157] The second obtaining module 402 is used to obtain a right view of the target environment according to the left view and the disparity map; wherein the right view represents the target environment seen by the right eye;

[0158] The third obtaining module 403 is used to obtain a 3D image of the target environment according to the left view, the disparity map, and the right view.

[0159] In one example, the first obtaining module 401 includes:

[0160] A first obtaining unit is used to obtain a depth map of the left view based on the left view and a preset monocular depth estimation model; wherein the preset monocular depth estimation model is a preset neural network model, which is used to determine the depth information of the pixel points, and the depth map represents the depth information of the pixel points in the left view in a preset camera coordinate system;

[0161] The second obtaining unit is used to obtain a disparity map according to the depth map.

[0162] In a possible implementation, the second obtaining unit is specifically configured to:

[0163] disparity = Bf / Depth;

[0164] Among them, disparity is the disparity map, B and f are preset parameters, and Depth is the depth map.

[0165] In one example, the second obtaining module 402 includes:

[0166] The third obtaining unit is used to perform dimensionality reduction processing on the left view, and connect the left view after dimensionality reduction processing and the disparity map to obtain a first intermediate feature matrix; wherein the intermediate feature matrix represents the left view and the disparity map;

[0167] A fourth obtaining unit is used to connect the preset noise characteristic matrix with the first intermediate characteristic matrix to obtain a second intermediate characteristic matrix;

[0168] A fifth obtaining unit is used to input the second intermediate feature matrix into a preset denoising model for denoising to obtain a denoised image code;

[0169] The sixth obtaining unit is used to decode the denoised image code to obtain a right view of the target environment.

[0170] In one example, the third obtaining module 403 includes:

[0171] An acquisition unit, used for acquiring a data set to be trained; wherein the data to be trained includes a left view to be trained, a disparity map to be trained, and an actual right view corresponding to the left view to be trained;

[0172] The seventh obtaining unit is used to train the initial model according to the left view to be trained, the disparity map to be trained, and the actual right view to obtain a trained image generation model; wherein the image generation model is used to output the right view according to the left view and the disparity map.

[0173] In a possible implementation, the seventh obtaining unit is specifically used to:

[0174] Determine a right view to be trained according to the left view to be trained and the disparity map to be trained;

[0175] According to the right view to be trained and the actual right view, a new disparity map is obtained;

[0176] Input the right view to be trained and the new disparity map into the initial model to obtain a new right view;

[0177] If the new right view and the actual right view Figure 1 If the training result is the same as the actual result, the initial model is trained according to the left view to be trained, the disparity map to be trained, and the actual right view.

[0178] In a possible implementation, the seventh obtaining unit is specifically used to:

[0179] Based on the initial model, the left view to be trained is subjected to dimensionality reduction processing, and the left view to be trained and the disparity map to be trained after the dimensionality reduction processing are connected to obtain an intermediate feature matrix to be trained; wherein the intermediate feature matrix represents the left view to be trained and the disparity map to be trained;

[0180] Adding the preset noise feature matrix to the actual right view to obtain the noisy actual right view matrix;

[0181] Connect the noisy actual right view matrix and the intermediate feature matrix to be trained to obtain the intermediate predicted right view matrix;

[0182] The intermediate predicted right view matrix is ​​input into a preset denoising model for denoising to obtain a predicted right view;

[0183] If the actual right view and the predicted right view Figure 1 If the training is complete, the image generation model is obtained.

[0184] In a possible implementation, the seventh obtaining unit is specifically used to:

[0185] Performing dimensionality reduction processing on the actual right view to obtain the actual right view after dimensionality reduction processing;

[0186] The preset noise feature matrix is ​​added to the actual right view after the dimensionality reduction process to obtain the noisy actual right view matrix.

[0187] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 50 includes:

[0188] The electronic device 50 may include a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a communication component 503 and other components. The processor 501 , the memory 502 and the communication component 503 are connected via a bus 504 .

[0189] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that the at least one processor 501 executes the above-mentioned method for generating a 3D image.

[0190] In the above Figure 5 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0191] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0192] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0193] In some embodiments, a computer program product is further provided, comprising a computer program or instructions, which implement the steps in any of the above-mentioned methods for generating a 3D image when executed by a processor.

[0194] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0195] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0196] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0197] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0198] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0199] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0200] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0202] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for generating a 3D image, characterized in that: The method comprises: Acquire a left view of the target environment, and obtain a disparity map based on the left view; wherein the left view represents the target environment seen by the left eye, and the disparity map represents the pixel difference between the left and right views; Obtaining a right view of the target environment according to the left view and the disparity map; wherein the right view represents the target environment seen by the right eye; A 3D image of the target environment is obtained according to the left view, the disparity map, and the right view.

2. The method according to claim 1, characterized in that: The step of obtaining a disparity map according to the left view includes: According to the left view, based on a preset monocular depth estimation model, a depth map of the left view is obtained; wherein the preset monocular depth estimation model is a preset neural network model, which is used to determine the depth information of the pixel points, and the depth map represents the depth information of the pixel points in the left view in a preset camera coordinate system; The disparity map is obtained according to the depth map.

3. The method according to claim 2, characterized in that The disparity map is: disparity = B × f / Depth; Among them, the disparity is the disparity map, B and f are preset parameters, and the Depth is the depth map.

4. The method according to claim 1, characterized in that: The step of obtaining a right view of the target environment according to the left view and the disparity map includes: Performing dimensionality reduction processing on the left view, and connecting the left view after dimensionality reduction processing and the disparity map to obtain a first intermediate feature matrix; wherein the intermediate feature matrix represents the left view and the disparity map; Connecting a preset noise characteristic matrix with the first intermediate characteristic matrix to obtain a second intermediate characteristic matrix; Inputting the second intermediate feature matrix into a preset denoising model for denoising to obtain a denoised image code; The denoised image code is decoded to obtain a right view of the target environment.

5. The method according to claim 1, characterized in that Also includes: Acquire a data set to be trained; wherein the data to be trained includes a left view to be trained, a disparity map to be trained, and an actual right view corresponding to the left view to be trained; The initial model is trained according to the left view to be trained, the disparity map to be trained, and the actual right view to obtain a trained image generation model; wherein the image generation model is used to output the right view according to the left view and the disparity map.

6. The method according to claim 5, characterized in that Also includes: Determining a right view to be trained according to the left view to be trained and the disparity map to be trained; Obtaining a new disparity map according to the right view to be trained and the actual right view; Inputting the right view to be trained and the new disparity map into the initial model to obtain a new right view; If the new right view is consistent with the actual right view, the initial model is trained according to the left view to be trained, the disparity map to be trained, and the actual right view.

7. The method according to claim 6, characterized in that The training of the initial model according to the left view to be trained, the disparity map to be trained, and the actual right view to obtain a trained image generation model includes: Based on the initial model, the left view to be trained is subjected to dimensionality reduction processing, and the left view to be trained and the disparity map to be trained after the dimensionality reduction processing are connected to obtain an intermediate feature matrix to be trained; wherein the intermediate feature matrix represents the left view to be trained and the disparity map to be trained; Adding a preset noise feature matrix to the actual right view to obtain a noisy actual right view matrix; Connecting the noisy actual right view matrix and the intermediate feature matrix to be trained to obtain an intermediate predicted right view matrix; Inputting the intermediate predicted right view matrix into a preset denoising model for denoising to obtain a predicted right view; If the actual right view is consistent with the predicted right view, a trained image generation model is obtained.

8. The method according to claim 7, characterized in that The step of adding a preset noise feature matrix to the actual right view to obtain the noisy actual right view matrix includes: Performing dimensionality reduction processing on the actual right view to obtain an actual right view after dimensionality reduction processing; A preset noise feature matrix is ​​added to the actual right view after the dimensionality reduction process to obtain the noisy actual right view matrix.

9. A 3D image generation device, characterized in that: include: A first obtaining module is used to obtain a left view of the target environment and obtain a disparity map according to the left view; wherein the left view represents the target environment seen by the left eye, and the disparity map represents the pixel difference between the left and right views; A second obtaining module is used to obtain a right view of the target environment according to the left view and the disparity map; wherein the right view represents the target environment seen by the right eye; The third obtaining module is used to obtain a 3D image of the target environment according to the left view, the disparity map, and the right view.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images