Stereo image generation method and device, equipment, storage medium and product

Abstract

The invention discloses a three-dimensional image generation method and device, equipment, a storage medium and a product, relates to the technical field of computers, and discloses a three-dimensional image generation method which comprises the following steps: acquiring an initial left eye image and parallax information of the initial left eye image; mapping the initial left eye image according to the parallax information to obtain a to-be-filled right eye image; based on a preset hole filling model, hole filling is carried out on the to-be-filled right eye image to obtain a target right eye image, and the hole filling model is obtained by training a preset to-be-trained model based on an initial image sample; and combining the initial left eye image and the target right eye image to obtain a target three-dimensional image. According to the method and the device, high-accuracy hole filling is realized through the hole filling model, more accurate right eye image generation is realized, a more accurate stereo image is further obtained, and finally the accuracy of generating the stereo image is remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment, storage medium and product for generating a stereoscopic image. Background Art

[0002] With the rapid development of digital media and computer graphics, stereoscopic images play an increasingly important role in film production, video games, virtual reality, augmented reality, and medical imaging. The importance of stereoscopic image generation technology is becoming increasingly prominent with the rapid development of various fields.

[0003] The existing stereo image generation technology is to calculate the depth image through the left eye image; calculate the right eye disparity based on the depth image; then, based on the calculated disparity, map each pixel in the left eye image to the corresponding position of the right eye image to be filled to obtain the right eye image, and combine the left eye image and the right eye image to obtain a stereo image.

[0004] However, in the mapping process, the pixels of the right eye image model cannot be matched, resulting in a problem of empty pixels. The current approach is to directly fill the empty pixels with the average color value around the empty pixels. The existing technology directly uses the average color value around the empty pixels to fill, which will generate an inaccurate right eye image, thereby greatly reducing the accuracy of the generated stereoscopic image.

[0005] Therefore, how to improve the accuracy of generating stereo images is a problem that needs to be solved urgently.

[0006] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0007] The main purpose of the present application is to provide a stereoscopic image generation method, device, equipment, storage medium and product, aiming to solve the technical problem of low accuracy in generating stereoscopic images.

[0008] To achieve the above object, the present application proposes a method for generating a stereoscopic image, the method comprising:

[0009] Acquire an initial left-eye image and disparity information of the initial left-eye image;

[0010] According to the disparity information, the initial left-eye image is mapped to obtain a right-eye image to be filled;

[0011] Based on a preset hole filling model, the hole filling is performed on the right eye image to be filled to obtain a target right eye image, wherein the hole filling model is obtained by training the preset model to be trained;

[0012] The initial left-eye image and the target right-eye image are combined to obtain a target stereoscopic image.

[0013] In one embodiment, before the step of performing hole filling on the image to be filled based on a preset hole filling model to obtain a target image, the step includes:

[0014] Get the initial image sample;

[0015] Randomly dividing the initial image samples into blocks to obtain secondary image samples;

[0016] Randomly marking pixel points in the secondary image sample to obtain a pseudo hole image sample;

[0017] Acquire a hole structure data sample of the pseudo hole image sample;

[0018] The preset first model to be trained is trained by using the pseudo hole image samples and the hole structure data samples to obtain a hole filling model.

[0019] In one embodiment, the step of training a preset model to be trained by using the pseudo hole image samples and the hole structure data samples to obtain a hole filling model includes:

[0020] Inputting the pseudo hole image sample and the hole structure data sample into a preset generator model to obtain a target image sample;

[0021] Inputting the pseudo hole image sample and the target image sample into a preset judgement model to obtain a first judgment result;

[0022] If the first judgment result does not meet the preset condition, the generator loss of the generator model is calculated according to a preset generator loss function to obtain an updated generator model;

[0023] According to a preset discriminator loss function, the discriminator loss of the discriminator model is calculated to obtain an updated discriminator model;

[0024] Returning to the step of inputting the pseudo hole image sample and the hole structure data sample into a preset generator model to obtain a target image sample;

[0025] If the first judgment result meets the preset conditions, the generator model and the judger model are combined to obtain a hole filling model.

[0026] In one embodiment, the step of obtaining the initial left-eye image and the disparity information of the initial left-eye image includes:

[0027] Acquire an initial left eye image;

[0028] Converting the initial left-eye image into an initial grayscale image;

[0029] Based on a preset depth estimation model, performing depth estimation on the initial grayscale image to obtain an initial depth map, wherein the depth estimation model is obtained by training a preset second to-be-trained model using grayscale image samples and depth map labels of the grayscale image samples;

[0030] Acquire texture information and edge information of the initial depth map;

[0031] Based on the texture information and the edge information, disparity information of the initial depth map is calculated.

[0032] In one embodiment, the step of calculating the disparity information of the initial depth map based on the texture information and the edge information includes:

[0033] Calculating a texture richness map based on the texture information;

[0034] Acquire depth information of the initial depth map;

[0035] A depth weight is calculated by using a preset first weight formula according to the depth information, the texture richness map and a preset first basic weight;

[0036] According to the edge information, the texture richness map and the preset second basic weight, an edge weight is calculated by a preset second weight formula;

[0037] According to the depth weight, the edge weight, the depth information and the edge information, a dynamic adjustment weight is calculated by a preset dynamic weight formula;

[0038] The disparity information of the initial depth map is calculated according to the dynamic adjustment weight and the preset maximum disparity by using a preset disparity calculation formula.

[0039] In one embodiment, before the step of calculating the disparity information of the initial depth map based on the texture information and the edge information, the step includes:

[0040] Get grayscale image training samples;

[0041] Determining a depth map label for the grayscale image training sample;

[0042] Based on the grayscale image training samples and the depth map labels of the grayscale image training samples, the preset second model to be trained is iteratively trained to obtain a depth estimation model that meets the accuracy condition.

[0043] In addition, to achieve the above-mentioned purpose, the present application also proposes a stereoscopic image generating device, the stereoscopic image generating device comprising:

[0044] An acquisition module, used for acquiring an initial left-eye image and disparity information of the initial left-eye image;

[0045] A mapping module, used for mapping the initial left-eye image according to the disparity information to obtain a right-eye image to be filled;

[0046] A filling module, used for performing hole filling on the right eye image to be filled based on a preset hole filling model to obtain a target right eye image, wherein the hole filling model is obtained by training a preset model to be trained;

[0047] The combining module is used to combine the initial left-eye image and the target right-eye image to obtain a target stereoscopic image.

[0048] In addition, to achieve the above-mentioned purpose, the present application also proposes a stereoscopic image generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the stereoscopic image generation method as described above.

[0049] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the stereoscopic image generation method described above are implemented.

[0050] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the stereoscopic image generation method described above are implemented.

[0051] One or more technical solutions proposed in this application have at least the following technical effects:

[0052] Compared with the related art, which directly uses the color average value around the hole pixel for filling, which will generate an inaccurate right eye image, thereby greatly reducing the accuracy of the generated stereo image, the present application obtains the initial left eye image and the disparity information of the initial left eye image; maps the initial left eye image according to the disparity information to obtain the right eye image to be filled; based on the preset hole filling model, the right eye image to be filled is hole filled to obtain the target right eye image, wherein the hole filling model is obtained by training the preset model to be trained based on the initial image sample; and combines the initial left eye image with the target right eye image to obtain the target stereo image. It can be understood that the present application uses the initial image sample to perform hole filling training on the model to be trained to obtain the hole filling model. When the right eye image to be filled corresponding to the left eye image is calculated by the disparity information, the hole filling model is used to achieve high-accuracy hole filling. Therefore, based on the preset hole filling model, the hole filling of the image to be filled can achieve more accurate right eye image generation, thereby obtaining a more accurate stereo image, and finally significantly improving the accuracy of generating stereo images. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 A schematic diagram of a flow chart of a first embodiment of a method for generating a stereoscopic image according to the present application;

[0056] Figure 2 A schematic diagram of a model provided for the first embodiment of the method for generating a stereoscopic image of the present application;

[0057] Figure 3 This is a schematic diagram of the module structure of a stereoscopic image generating device according to an embodiment of the present application;

[0058] Figure 4 Schematic diagram of the device structure of the hardware operating environment involved in the stereoscopic image generation method in the embodiment of the present application.

[0059] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0061] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0062] The main solution of the embodiment of the present application is: obtaining an initial left-eye image and disparity information of the initial left-eye image;

[0063] According to the disparity information, the initial left-eye image is mapped to obtain a right-eye image to be filled;

[0064] Based on a preset hole filling model, the hole filling is performed on the right eye image to be filled to obtain a target right eye image, wherein the hole filling model is obtained by training a preset model to be trained based on an initial image sample;

[0065] The initial left-eye image and the target right-eye image are combined to obtain a target stereoscopic image.

[0066] The existing stereo image generation technology is to calculate the depth image through the left eye image; calculate the right eye disparity based on the depth image; and then map each pixel in the left eye image to the corresponding position of the right eye image to be filled based on the calculated disparity to obtain the right eye image, and combine the left eye image and the right eye image to obtain a stereo image. In the mapping process, there will be a problem that the pixels of the right eye image model cannot match, resulting in a hole pixel. The current practice is to directly fill the hole pixel with the color average value around the hole pixel. The existing technology directly uses the color average value around the hole pixel for filling, which will generate an inaccurate right eye image, which will greatly reduce the accuracy of the generated stereo image.

[0067] The present application provides a solution, which uses initial image samples to perform hole filling training on a training model to obtain a hole filling model. When the right eye image to be filled corresponding to the left eye image is calculated through disparity information, the hole filling model is used to achieve high-accuracy hole filling. Therefore, based on the preset hole filling model, hole filling of the image to be filled can achieve more accurate right eye image generation, thereby obtaining a more accurate stereoscopic image, and ultimately significantly improving the accuracy of generating stereoscopic images.

[0068] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. In this embodiment, the present application takes the stereoscopic image generation device as the execution subject, and for the convenience of description, it is specifically described as "device" below.

[0069] Based on this, the present application embodiment provides a method for generating a stereoscopic image. Figure 1 , Figure 1 This is a flowchart of the first embodiment of the stereoscopic image generation method of the present application.

[0070] In this embodiment, the stereoscopic image generation method includes steps S10 to S40:

[0071] Step S10, obtaining an initial left-eye image and disparity information of the initial left-eye image;

[0072] It should be noted that the initial left-eye image refers to image information captured or calculated by a visual system from the left eye perspective, including but not limited to left-eye image information captured using a monocular vision acquisition terminal, and the disparity information is information containing the relative position differences of objects in the image in different perspectives.

[0073] In a feasible implementation, step S10 may include:

[0074] Acquire an initial left eye image;

[0075] Converting the initial left-eye image into an initial grayscale image;

[0076] Based on a preset depth estimation model, performing depth estimation on the initial grayscale image to obtain an initial depth map, wherein the depth estimation model is obtained by training a preset second to-be-trained model using grayscale image samples and depth map labels of the grayscale image samples;

[0077] Acquire texture information and edge information of the initial depth map;

[0078] Based on the texture information and the edge information, disparity information of the initial depth map is calculated.

[0079] It should be noted that the initial grayscale image is an image that only contains brightness information but does not contain color information. The depth estimation model is an algorithm model trained using machine learning technology, which can predict the depth value of each point in the scene based on the input grayscale image. The initial depth map is an image containing depth information. The grayscale image samples are grayscale image training samples for testing. The depth map labels are depth maps corresponding to the grayscale image training samples, which are used to train the depth estimation model. The second model to be trained is a depth estimation model that can perform depth estimation on grayscale images.

[0080] In addition, it should be noted that the texture information is information that describes repetitive patterns or structures in an image, the edge information is information that describes the boundaries of objects in an image or areas of significant changes, and the parallax information is information that includes the position differences of objects between perspectives when the same object is observed from different perspectives.

[0081] It can be understood that by performing depth estimation on the initial grayscale image corresponding to the initial left eye image through the trained depth estimation model, a high-precision initial depth map can be obtained, and texture analysis and edge analysis are performed on the initial depth map to obtain the corresponding texture information and edge information, and then the disparity information of the initial depth map is obtained by calculation.

[0082] In a specific implementation, before the step of calculating the disparity information of the initial depth map based on the texture information and the edge information, the step includes:

[0083] Get grayscale image training samples;

[0084] Determining a depth map label for the grayscale image training sample;

[0085] Based on the grayscale image training samples and the depth map labels of the grayscale image training samples, the preset second model to be trained is iteratively trained to obtain a depth estimation model that meets the accuracy condition.

[0086] It can be understood that the depth estimation model is an algorithm model trained using machine learning technology, which can predict the depth value of each point in the scene based on the input grayscale image. The initial depth map is an image containing depth information, and the grayscale image samples are grayscale image training samples for testing. The depth map label is the depth map corresponding to the grayscale image training sample, which is used to train the depth estimation model, and the second model to be trained is a depth estimation model that can perform depth estimation on grayscale images.

[0087] In a specific implementation, a pre-trained monocular depth estimation model is used to perform depth estimation on the grayscale image corresponding to the left eye initial image to obtain a left eye depth map.

[0088] In a specific implementation, the step of calculating the disparity information of the initial depth map based on the texture information and the edge information includes:

[0089] Calculating a texture richness map based on the texture information;

[0090] Acquire depth information of the initial depth map;

[0091] A depth weight is calculated by using a preset first weight formula according to the depth information, the texture richness map and a preset first basic weight;

[0092] According to the edge information, the texture richness map and the preset second basic weight, an edge weight is calculated by a preset second weight formula;

[0093] According to the depth weight, the edge weight, the depth information and the edge information, a dynamic adjustment weight is calculated by a preset dynamic weight formula;

[0094] The disparity information of the initial depth map is calculated according to the dynamic adjustment weight and the preset maximum disparity by using a preset disparity calculation formula.

[0095] It should be noted that the texture information is the detailed features of the surface of an object in the image, such as lines, patterns, etc. The edge information is the information of the outline or boundary of an object in the image, which is usually manifested as a significant change in pixel values. The initial depth map is a map composed of the depth values ​​of each point in the image preliminarily estimated by a certain algorithm. The texture richness map is a representation of the richness of the texture in the image calculated based on the texture information. The depth information is the depth value of each point in the image. The first basic weight is a preset starting weight value for calculating the depth weight.

[0096] In addition, it should be noted that the first weight formula is a formula for calculating the depth weight by combining depth information, texture richness mapping and the first basic weight, the depth weight is a weight value reflecting the importance of depth information in disparity calculation, the second basic weight is a preset starting weight value for calculating edge weight, the second weight formula is a formula for calculating edge weight by combining edge information, texture richness mapping and the second basic weight, the edge weight is a weight value reflecting the importance of edge information in disparity calculation, the dynamic weight formula is a formula for calculating the dynamically adjusted weight based on the depth weight, edge weight, depth information and edge information, the dynamically adjusted weight is a weight value dynamically adjusted according to the texture richness mapping during the disparity calculation process, the maximum disparity is a preset hyperparameter used for adaptive adjustment in different devices, the disparity calculation formula is a formula for calculating the disparity information of the initial depth map based on the dynamic adjustment weight and the maximum disparity, the disparity information is the position difference of each point in the image relative to the corresponding points in other perspectives, and is an important parameter in stereoscopic vision.

[0097] In a specific implementation, a dynamic weight adjustment mechanism based on texture and edge information in a pixel neighborhood is used to optimize disparity calculation and obtain disparity information as follows:

[0098] First, the Canny operator is applied to the input left eye depth map to detect edges. The output of the Canny operator is a binary image, where white pixels represent edges. The Canny operator is a multi-level edge detection algorithm, which is mainly used for edge detection tasks in image processing.

[0099] The output of the Canny operator is normalized to the range [0,1] and combined with the depth information, where the normalization is achieved by dividing the pixel values ​​in the edge image by the maximum possible value.

[0100] Then, the texture energy is evaluated by calculating the variance of the gradient magnitude in the region, as follows:

[0101]

[0102] Among them, |R| is the total number of pixels in the regional photo R, |G(x)| is the gradient amplitude of pixel x, and the gradient amplitude G is calculated by the Sobel operator, which is a discrete differential operator mainly used for edge detection.

[0103] Then, for the entire image, the texture energy of each pixel is calculated to obtain a texture richness map T(x), as follows:

[0104] T(x)=f(E(R x ))

[0105] Among them, R x is a local region centered on x, and f is a normalization function to ensure that the value of T(x) is in the range [0,1].

[0106] Then use the texture richness map T(x) to dynamically adjust the weights of depth information and edge information. The formula is as follows:

[0107] D combined (x)=α(x)·D(X)+β(x)·GN(x)

[0108] Among them, D(x) is the depth information in the original depth map, GN(x) is the normalized Canny operator output, representing edge information, α(x) and β(x) are dynamic weights, which are adjusted according to the texture richness T(x).

[0109] The weight function is defined as:

[0110] α(x)=w1+(1-w1)·(1-T(x))

[0111] β(x)=w2+(1-w2)·T(x)

[0112] Among them, w1 and w2 are the basic weights of the texture-poor areas and can be adjusted according to the specific application.

[0113] In order to ensure that the sum of α(x) and β(x) is 1, normalization is performed as follows:

[0114]

[0115] Finally, the fused values ​​are used to calculate the disparity h, using the following formula:

[0116] h=-D combined MaxDisp

[0117] Among them, MaxDisp is the maximum disparity.

[0118] In this embodiment, an initial left-eye image is obtained; the initial left-eye image is converted into an initial grayscale image; based on a preset depth estimation model, the initial grayscale image is depth estimated to obtain an initial depth map, wherein the depth estimation model is obtained by training a preset second model to be trained using grayscale image samples and depth map labels of the grayscale image samples; texture information and edge information of the initial depth map are obtained; based on the texture information and the edge information, the disparity information of the initial depth map is calculated. This embodiment uses a pre-trained depth estimation model to generate a depth image, and then a dynamic weight adjustment mechanism based on texture and edge information in a pixel neighborhood is used to optimize the disparity calculation process, obtain disparity information, and solve the technical problem of inaccurate disparity.

[0119] The above is only a feasible implementation of step S10 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S10.

[0120] Step S20, mapping the initial left-eye image according to the disparity information to obtain a right-eye image to be filled;

[0121] It should be noted that the mapping is the process of converting pixels in one image to another image according to a certain rule or algorithm in image processing. The right eye image to be filled refers to the image to be generated based on the left eye image and disparity information. Due to the mismatch of pixels during mapping, there are empty pixels waiting to be filled.

[0122] It is understandable that the left eye image is first mapped according to the parallax information calculated from the left eye perspective to generate a right eye image. In this process, the problem of pixel mismatch in the right eye image may occur, so the obtained right eye image is the right eye image to be filled.

[0123] In the specific implementation, each pixel in the left eye image is mapped to the corresponding position in the right eye image according to the calculated disparity. The calculation method is as follows:

[0124] For each pixel (x, y) in the left eye image, calculate its mapped position (x′, y) in the right eye image:

[0125] x′=x+h

[0126] Where x′ is the horizontal coordinate in the right eye image, and y remains unchanged because the parallax only affects the horizontal position;

[0127] If the new position is within the image range (ie, non-negative and less than the image width), the pixel value in the left eye image is directly copied to the corresponding position in the right eye image; if the new position exceeds the image range, hole filling is required.

[0128] The above is only a feasible implementation of step S20 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S20.

[0129] Step S30, performing hole filling on the to-be-filled right eye image based on a preset hole filling model to obtain a target right eye image, wherein the hole filling model is obtained by training a preset to-be-trained model based on the initial image sample;

[0130] It should be noted that the hole filling model is an algorithm model used to fill the hole pixels in the image to achieve image integrity. The right eye image to be filled is the right eye perspective image that needs to be processed to fill the hole. The target right eye image is the right eye perspective image with the hole pixels filled after being processed by the hole filling model. The initial image sample is the original image data set used to train the hole filling model. These image samples contain a variety of scenes and objects, which help the model learn how to effectively fill different types of holes. The preset model to be trained refers to the hole filling model architecture that has been designed before starting the training process.

[0131] In a feasible implementation manner, before step S30, the following may be included:

[0132] Get the initial image sample;

[0133] Randomly dividing the initial image samples into blocks to obtain secondary image samples;

[0134] Randomly marking pixel points in the secondary image sample to obtain a pseudo hole image sample;

[0135] Acquire a hole structure data sample of the pseudo hole image sample;

[0136] The preset first model to be trained is trained by using the pseudo hole image samples and the hole structure data samples to obtain a hole filling model.

[0137] It should be noted that the initial image sample is the original image data set used to train the model, which usually contains images of various scenes and objects to ensure that the model can learn diverse features and patterns. The random blocking is a data enhancement technology that creates new training samples by randomly cutting images into small blocks, which can increase the diversity of samples and improve the generalization ability of the model. The secondary image sample is an image block obtained after random blocking, which is a subset of the original image and is used for further training. The random marking is to randomly select and specially mark the pixels in the image during image processing to simulate the hole area in the image.

[0138] In addition, it should be noted that the pseudo-hole image samples are image samples with holes that are artificially created by randomly marking the pixels in the secondary image samples, wherein these holes do not exist in reality but are simulated and are used to train the model on how to process and fill the holes. The hole structure data samples are structural information about the hole area extracted from the pseudo-hole image samples, including but not limited to the location, size, shape, etc. of the holes. The preset first model to be trained is an initial model architecture that has been designed before training, which is used to detect and fill the holes in the image. The hole filling model is a model obtained after the training is completed, which can identify the hole pixels in the image and effectively fill them to generate visually reasonable image content.

[0139] In the specific implementation, a GAN-MAVC network (Generative Adversarial Network with Multi-scale Attention and Vector Completion) is built to establish a hole filling model. The specific steps are as follows:

[0140] First, random cutting is performed in the initial image sample according to the preset length, width and number; the image block obtained after the random cutting process is a subset of the initial image sample, and the subset of the initial image sample is used as training data.

[0141] Then, random pixels in the cut image blocks are randomly marked to construct pseudo hole image data;

[0142] It is understandable that the image samples with holes created by randomly marking the pixels in the cut image blocks do not exist in reality but are simulated for training the model on how to process and fill the holes.

[0143] Then, build the GAN-MAVC network, refer to Figure 2 The input data is divided into two parts. The first part is the pseudo-hole image data; the second part is the hole surrounding structure data, which improves the model's understanding of the relative position and function of the hole area in the image. The hole surrounding structure data is calculated based on the mean of the non-empty pixels around the marked pixels.

[0144] Among them, the generator is built mainly considering that the purpose of the model is to perform image restoration, so a structure of cross-connected hole convolution and traditional convolution is adopted, and finally flattened into a hidden variable Z; then the structural data around the hole is spliced ​​into the hidden variable Z; by incorporating the self-attention mechanism structure, the model can judge how much structural information and semantic information to use when restoring the image, and then restore the target image to the original image through the upsampling structure. The overall structure of the discriminator is the same as the generator downsampling structure, which is a cross-connected structure of traditional convolution and hole convolution. After flattening, it is passed to two linear layers to determine whether it is a true or false image.

[0145] Among them, the loss function design when building the GAN-MAVC network is divided into the generator loss function and the discriminator loss function:

[0146] The generator loss function is divided into adversarial loss, reconstruction loss and structural information loss;

[0147] It should be noted that the adversarial loss refers to the generator generating images that are realistic enough to deceive the discriminator, which is calculated using binary cross entropy loss; the reconstruction loss refers to the generator's ability to reconstruct the original image, which is calculated using pixel-level loss (mean absolute error); the structural information loss is to ensure that the generated image contains structural information around the hole, which is calculated using feature-level loss (perceptual loss).

[0148] The discriminator loss function is divided into real image loss and generated image loss.

[0149] It should be noted that the real image loss refers to the discriminator classifying the real image as real, using binary cross entropy loss; the generated image loss refers to the discriminator classifying the image generated by the generator as false, also using binary cross entropy loss.

[0150] Among them, the training process of the GAN-MAVC network is as follows:

[0151] Initialize the network parameters of the generator and discriminator;

[0152] Then choose the learning rate and optimizer for the training cycle, which is divided into two parts:

[0153] The first part is the generator training. First, a batch of inputs are generated from random noise or a predefined distribution. Then, the generator network is used to generate images. Then, the discriminator is used to evaluate the authenticity of the generated images. Then, the adversarial loss, reconstruction loss, and structural information loss of the generator are calculated. Finally, the optimizer is used to update the parameters of the generator.

[0154] The second part is the discriminator training. First, a batch of real images are selected from the training set, then a batch of fake images are generated from the generator, and then the loss of the discriminator on the real images and fake images is calculated. Finally, the optimizer is used to update the parameters of the discriminator.

[0155] Repeat the above process, alternating the training of the generator and the discriminator.

[0156] It can be understood that the goal of training the discriminator is to enable it to distinguish between real data and fake data generated by the generator as much as possible, while the goal of training the generator is to enable it to deceive the discriminator so that it cannot distinguish between real data and fake data.

[0157] After each training cycle, the performance of the generator and discriminator is evaluated; the test set is used to evaluate the quality of the generated images.

[0158] The model parameters are saved periodically so that the training can be resumed when needed. Finally, the trained model is used to fill the holes in the right eye image to be filled.

[0159] It can be understood that this embodiment can achieve more accurate right eye image generation by filling holes in the image to be filled through a preset hole filling model, thereby obtaining a more accurate stereoscopic image, thereby solving the technical problem of low accuracy in generating stereoscopic images.

[0160] The above is only a feasible implementation of step S30 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S30.

[0161] Step S40 , combining the initial left-eye image and the target right-eye image to obtain a target stereoscopic image.

[0162] It should be noted that the target right-eye image refers to a processed or generated image that should be directly captured by the right eye, and the target stereoscopic image is a result image obtained by combining the initial left-eye image and the target right-eye image.

[0163] It can be understood that the target stereoscopic image is created by combining the initial left eye image and the processed target right eye image. This process involves aligning and fusing two images from different perspectives to simulate the binocular vision effect when the human eye views the real world. In this way, a stereoscopic image with a sense of depth can be obtained, which enhances the viewer's visual experience.

[0164] The above is only a feasible implementation of step S40 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S40.

[0165] It is understandable that after all the above models are serialized, they are deployed in the server as microservices. Users upload a single left-eye image, and the right-eye image is rendered in real time according to the maximum parallax set by different device terminals to form a stereoscopic visual image. When connected to AR glasses, users upload a single image or video (essentially multiple images), and a stereoscopic visual image or 3D video is rendered in real time.

[0166] This embodiment provides a stereoscopic image generation method, which uses initial image samples to perform hole filling training on a training model to obtain a hole filling model. When the right eye image to be filled corresponding to the left eye image is calculated through disparity information, the hole filling model is used to achieve high-accuracy hole filling. Therefore, based on the preset hole filling model, more accurate right eye image generation can be achieved by filling the holes of the image to be filled, thereby obtaining a more precise stereoscopic image, and ultimately significantly improving the accuracy of generating stereoscopic images.

[0167] In a feasible implementation, a left-eye character image is captured by using a monocular vision capture terminal, the input 2D image is converted into a grayscale image, and a pre-trained depth estimation model developed based on the MonoFormer model is used to estimate the depth map of the left-eye image to generate a left-eye depth map;

[0168] Then a method that takes into account the texture and edge information in the pixel neighborhood (a dynamic weight adjustment mechanism based on image content) is proposed to optimize the disparity calculation. Specifically:

[0169] Apply the Canny operator to the input 2D image to detect edges, output a binary image and normalize it; calculate the gradient amplitude through the Sobel operator, and evaluate the texture energy by calculating the variance of the gradient amplitude in the area; for the entire image, calculate the texture energy of each pixel to obtain a texture richness map; use the texture richness map to dynamically adjust the weights of the depth information and edge information, and normalize the weights of the depth information and edge information; calculate the dynamic adjustment weight based on the depth information, edge information and corresponding weights, and calculate the disparity information based on the maximum disparity and the dynamic adjustment weight;

[0170] The mapping position of the pixel in the left eye image is calculated based on the coordinates and disparity information of the pixel in the right eye image. If the mapping position is within the image range, the pixel value in the left eye image is directly copied to the corresponding position in the right eye image. If the new position exceeds the image range, hole filling is required.

[0171] Then, a hole filling model is built based on the GAN-MAVC network. The generator and discriminator are built using a structure of cross-connection of hole convolution and traditional convolution. The model is trained based on pseudo-hole image data and hole surrounding structure data.

[0172] Finally, the model is deployed in the server as a microservice, and the trained model is used to fill the holes in the right eye image to form a stereoscopic visual picture.

[0173] It can be seen from the above embodiments that the present application uses initial image samples to perform hole filling training on the training model to obtain a hole filling model. When the right eye image to be filled corresponding to the left eye image is calculated through disparity information, the hole filling model is used to achieve high-accuracy hole filling. Therefore, based on the preset hole filling model, hole filling of the image to be filled can achieve more accurate right eye image generation, and then obtain a more accurate stereoscopic image, and finally achieve the technical effect of significantly improving the accuracy of generating stereoscopic images.

[0174] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the stereoscopic image generation method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0175] This application also provides a stereoscopic image generation device, please refer to Figure 3 , the stereoscopic image generating device comprises:

[0176] An acquisition module 10, configured to acquire an initial left-eye image and disparity information of the initial left-eye image;

[0177] A mapping module 20, configured to map the initial left-eye image according to the disparity information to obtain a right-eye image to be filled;

[0178] A filling module 30 is used to fill holes in the right eye image to be filled based on a preset hole filling model to obtain a target right eye image, wherein the hole filling model is obtained by training a preset model to be trained;

[0179] The combining module 40 is used to combine the initial left-eye image and the target right-eye image to obtain a target stereoscopic image.

[0180] In a possible implementation manner of the present application, the stereoscopic image generating device further includes:

[0181] A first acquisition module, used for acquiring an initial image sample;

[0182] A random block module, used for randomly dividing the initial image samples into blocks to obtain secondary image samples;

[0183] A random marking module, used for randomly marking pixel points in the secondary image sample to obtain a pseudo hole image sample;

[0184] A second acquisition module, used for acquiring a hole structure data sample of the pseudo hole image sample;

[0185] The first training module is used to train a preset first model to be trained through the pseudo hole image samples and the hole structure data samples to obtain a hole filling model.

[0186] In a possible implementation manner of the present application, the first training module includes:

[0187] A generator module, used for inputting the pseudo hole image sample and the hole structure data sample into a preset generator model to obtain a target image sample;

[0188] A judgement module, used for inputting the pseudo hole image sample and the target image sample into a preset judgement model to obtain a first judgment result;

[0189] A first updating module, configured to calculate the generator loss of the generator model according to a preset generator loss function to obtain an updated generator model if the first judgment result does not meet a preset condition;

[0190] A second updating module, used to calculate the discriminator loss of the discriminator model according to a preset discriminator loss function to obtain an updated discriminator model;

[0191] A loop module, used for returning to the step of inputting the pseudo hole image sample and the hole structure data sample into a preset generator model to obtain a target image sample;

[0192] The first combination module is used to combine the generator model and the judger model to obtain a hole filling model if the first judgment result meets a preset condition.

[0193] In a possible implementation manner of the present application, the acquisition module 10 includes:

[0194] A third acquisition module, used for acquiring an initial left-eye image;

[0195] A first conversion module, used for converting the initial left-eye image into an initial grayscale image;

[0196] A depth estimation module, configured to perform depth estimation on the initial grayscale image based on a preset depth estimation model to obtain an initial depth map, wherein the depth estimation model is obtained by training a preset second to-be-trained model using grayscale image samples and depth map labels of the grayscale image samples;

[0197] A fourth acquisition module, used to acquire texture information and edge information of the initial depth map;

[0198] The first calculation module is used to calculate the disparity information of the initial depth map based on the texture information and the edge information.

[0199] In a possible implementation manner of the present application, the first calculation module includes:

[0200] A second calculation module, used for calculating a texture richness map according to the texture information;

[0201] A fifth acquisition module, used to acquire depth information of the initial depth map;

[0202] A third calculation module, configured to calculate a depth weight by using a preset first weight formula according to the depth information, the texture richness map and a preset first basic weight;

[0203] A fourth calculation module, configured to calculate an edge weight by using a preset second weight formula according to the edge information, the texture richness map and a preset second basic weight;

[0204] a fifth calculation module, configured to calculate a dynamic adjustment weight by using a preset dynamic weight formula according to the depth weight, the edge weight, the depth information and the edge information;

[0205] The sixth calculation module is used to calculate the disparity information of the initial depth map according to the dynamic adjustment weight and the preset maximum disparity by using a preset disparity calculation formula.

[0206] In a possible implementation of the present application, the depth estimation module includes:

[0207] A sixth acquisition module, used for acquiring grayscale image training samples;

[0208] A first determination module, used to determine the depth map label of the grayscale image training sample;

[0209] The second training module is used to iteratively train a preset second model to be trained based on the grayscale image training samples and the depth map labels of the grayscale image training samples to obtain a depth estimation model that meets the accuracy requirements.

[0210] The stereoscopic image generation device provided by the present application adopts the stereoscopic image generation method in the above embodiment, which can solve the technical problem of low accuracy in generating stereoscopic images. Compared with the prior art, the beneficial effects of the stereoscopic image generation device provided by the present application are the same as those of the stereoscopic image generation method provided by the above embodiment, and other technical features in the stereoscopic image generation device are the same as those disclosed in the above embodiment method, which will not be described in detail here.

[0211] The present application provides a stereoscopic image generating device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the stereoscopic image generating method in the above-mentioned embodiment 1.

[0212] Reference below Figure 4 , which shows a schematic diagram of the structure of a stereoscopic image generating device suitable for implementing the embodiment of the present application. The stereoscopic image generating device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, tablet computers, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The stereoscopic image generating device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0213] like Figure 4As shown, the stereoscopic image generating device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the stereoscopic image generating device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the stereoscopic image generation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a stereoscopic image generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0214] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0215] The stereoscopic image generation device provided by the present application adopts the stereoscopic image generation method in the above embodiment, which can solve the technical problem of low accuracy in generating stereoscopic images. Compared with the prior art, the beneficial effects of the stereoscopic image generation device provided by the present application are the same as the beneficial effects of the stereoscopic image generation method provided by the above embodiment, and other technical features in the stereoscopic image generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0216] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0217] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0218] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the stereoscopic image generation method in the above-mentioned embodiment.

[0219] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0220] The computer-readable storage medium may be included in the stereoscopic image generating device; or may exist independently without being assembled into the stereoscopic image generating device.

[0221] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by a stereoscopic image generating device, the stereoscopic image generating device: obtains an initial left-eye image and disparity information of the initial left-eye image; maps the initial left-eye image according to the disparity information to obtain a right-eye image to be filled; based on a preset hole filling model, performs hole filling on the right-eye image to be filled to obtain a target right-eye image, wherein the hole filling model is obtained by training a preset model to be trained based on the initial image sample; and combines the initial left-eye image and the target right-eye image to obtain a target stereoscopic image.

[0222] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0223] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0224] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0225] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned stereoscopic image generation method, and can solve the technical problem of low accuracy in generating stereoscopic images. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the stereoscopic image generation method provided by the above-mentioned embodiment, and will not be repeated here.

[0226] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned stereoscopic image generation method when executed by a processor.

[0227] The computer program product provided by the present application can solve the technical problem of low accuracy in generating stereoscopic images. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the stereoscopic image generation method provided by the above embodiment, which will not be described in detail here.

[0228] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating a stereoscopic image, characterized in that: The method includes: Acquire an initial left-eye image and disparity information of the initial left-eye image; According to the disparity information, the initial left-eye image is mapped to obtain a right-eye image to be filled; Based on a preset hole filling model, the hole filling is performed on the right eye image to be filled to obtain a target right eye image, wherein the hole filling model is obtained by training a preset model to be trained based on an initial image sample; The initial left-eye image and the target right-eye image are combined to obtain a target stereoscopic image.

2. The method according to claim 1, characterized in that Based on a preset hole filling model, the step of performing hole filling on the right eye image to be filled to obtain a target right eye image includes: Get an initial image sample; Randomly dividing the initial image samples into blocks to obtain secondary image samples; Randomly marking pixel points in the secondary image sample to obtain a pseudo hole image sample; Acquire a hole structure data sample of the pseudo hole image sample; The preset first model to be trained is trained by using the pseudo hole image samples and the hole structure data samples to obtain a hole filling model.

3. The method according to claim 2, characterized in that The step of training a preset model to be trained by using the pseudo hole image samples and the hole structure data samples to obtain a hole filling model comprises: Inputting the pseudo hole image sample and the hole structure data sample into a preset generator model to obtain a target image sample; Inputting the pseudo hole image sample and the target image sample into a preset judgement model to obtain a first judgment result; If the first judgment result does not meet the preset condition, the generator loss of the generator model is calculated according to a preset generator loss function to obtain an updated generator model; According to a preset discriminator loss function, the discriminator loss of the discriminator model is calculated to obtain an updated discriminator model; Returning to the step of inputting the pseudo hole image sample and the hole structure data sample into a preset generator model to obtain a target image sample; If the first judgment result meets the preset conditions, the generator model and the judger model are combined to obtain a hole filling model.

4. The method according to claim 1, characterized in that The step of acquiring the initial left-eye image and the disparity information of the initial left-eye image comprises: Acquire an initial left eye image; Converting the initial left-eye image into an initial grayscale image; Based on a preset depth estimation model, performing depth estimation on the initial grayscale image to obtain an initial depth map, wherein the depth estimation model is obtained by training a preset second to-be-trained model using grayscale image samples and depth map labels of the grayscale image samples; Acquire texture information and edge information of the initial depth map; Based on the texture information and the edge information, disparity information of the initial depth map is calculated.

5. The method according to claim 4, characterized in that The step of calculating the disparity information of the initial depth map based on the texture information and the edge information comprises: Calculating a texture richness map based on the texture information; Acquire depth information of the initial depth map; A depth weight is calculated by using a preset first weight formula according to the depth information, the texture richness map and a preset first basic weight; According to the edge information, the texture richness map and the preset second basic weight, an edge weight is calculated by a preset second weight formula; According to the depth weight, the edge weight, the depth information and the edge information, a dynamic adjustment weight is calculated by a preset dynamic weight formula; The disparity information of the initial depth map is calculated according to the dynamic adjustment weight and the preset maximum disparity by using a preset disparity calculation formula.

6. The method according to claim 4, characterized in that The step of calculating the disparity information of the initial depth map based on the texture information and the edge information includes: Get grayscale image training samples; Determining a depth map label for the grayscale image training sample; Based on the grayscale image training samples and the depth map labels of the grayscale image training samples, the preset second model to be trained is iteratively trained to obtain a depth estimation model that meets a preset accuracy standard.

7. A stereoscopic image generating device, characterized in that: The device comprises: An acquisition module, used for acquiring an initial left-eye image and disparity information of the initial left-eye image; A mapping module, used for mapping the initial left-eye image according to the disparity information to obtain a right-eye image to be filled; A filling module, used for performing hole filling on the right eye image to be filled based on a preset hole filling model to obtain a target right eye image, wherein the hole filling model is obtained by training a preset model to be trained; The combining module is used to combine the initial left-eye image and the target right-eye image to obtain a target stereoscopic image.

8. A stereoscopic image generating device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the stereoscopic image generation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the stereoscopic image generation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the stereoscopic image generation method according to any one of claims 1 to 6 are implemented.

Images