Light field image synthesis method and device, electronic equipment and storage medium

Abstract

The application provides a light field image synthesis method and device, electronic equipment and storage medium. The method comprises: acquiring at least one two-dimensional image of a three-dimensional scene; for each two-dimensional image, inputting the two-dimensional image into a generative adversarial network to obtain a multi-view stylized image set, the multi-view stylized image set comprising stylized images of multiple different views corresponding to the two-dimensional image, the generative adversarial network being obtained by training an initial generative adversarial network based on a training set and a validation set, the training set comprising at least one training image, and the validation set comprising validation images corresponding to the training images, the validation images being obtained by performing three-dimensional consistent style processing on the training images; and performing image synthesis based on the multi-view stylized image set corresponding to each two-dimensional image to obtain a light field image of the three-dimensional scene. The trained generative adversarial network can obtain a multi-view stylized image set, and further obtain a light field image that takes into account both stylization and three-dimensional consistency.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for synthesizing light field images. Background Technology

[0002] Light field images can record the direction and spatial position information of light rays. They have broad application prospects in 3D reconstruction and virtual reality. Light field images are typically obtained by synthesizing multi-view images captured from a scene. Multi-view images are images acquired from multiple different perspectives within the scene, with each viewpoint providing a unique perspective of the scene.

[0003] In a light field image, the light field records the direction of light propagation in three-dimensional space, thus allowing for the synthesis of light field images from different perspectives. The consistency of light field images requires that images synthesized from different perspectives maintain coherence and consistency in content, structure, and visual appeal to accurately reflect the three-dimensional information of the real scene. Before synthesizing light field images, stylization processing can be performed on the multi-viewpoint images. Stylization processing refers to applying special effects to an image or its objects to give it a specific artistic style or visual effect.

[0004] Existing methods for synthesizing light field images based on multi-viewpoint images cannot achieve high 3D consistency of the light field image while simultaneously stylizing the multi-viewpoint images. Therefore, light field images synthesized using existing methods cannot simultaneously achieve stylization and 3D consistency. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for synthesizing light field images, which solves the shortcomings of existing technologies in synthesizing light field images that cannot simultaneously achieve stylization and three-dimensional consistency, thereby achieving the goal of simultaneously achieving stylization and three-dimensional consistency when synthesizing light field images.

[0006] This invention provides a method for synthesizing light field images, comprising:

[0007] Acquire at least one frame of a 2D image of a 3D scene;

[0008] For each of the two-dimensional images, the two-dimensional images are input into a generative adversarial network to obtain a multi-view stylized image set. The multi-view stylized image set includes stylized images from multiple different perspectives corresponding to the two-dimensional images. The generative adversarial network is obtained by training an initial generative adversarial network based on a training set and a validation set. The training set includes at least one training image, and the validation set includes a validation image corresponding to the training image. The validation image is obtained by performing three-dimensional consistent stylization processing on the training image.

[0009] Image synthesis is performed based on the multi-viewpoint stylized image set corresponding to each of the two-dimensional images to obtain the light field image of the three-dimensional scene.

[0010] According to a method for synthesizing light field images provided by the present invention, the step of inputting the two-dimensional image into a generative adversarial network to obtain a multi-view stylized image set includes:

[0011] The two-dimensional image is segmented into foreground and background instances to obtain a first pixel region corresponding to the foreground and a second pixel region corresponding to the background;

[0012] The first pixel region and the second pixel region are input into the generative adversarial network, and the first pixel region is stylized by the rendering module in the generative adversarial network to obtain the stylized first pixel region.

[0013] The stylized first pixel region and the second pixel region are merged to generate the stylized image;

[0014] The multi-viewpoint stylized image set is obtained based on the stylized images from multiple different perspectives.

[0015] According to a method for synthesizing a light field image provided by the present invention, the step of merging the stylized first pixel region and the second pixel region to generate the stylized image includes:

[0016] Extract style information representing the foreground style from the stylized first pixel region;

[0017] The style information is transferred to the second pixel region to obtain the stylized second pixel region;

[0018] The stylized first pixel region and the stylized second pixel region are merged to generate the stylized image.

[0019] According to a method for synthesizing light field images provided by the present invention, the multi-viewpoint stylized image set is obtained based on stylized images from multiple different perspectives, comprising:

[0020] Geometric correction and pixel-level alignment are performed on the pixels in each of the stylized images to obtain a consistent stylized image corresponding to each stylized image;

[0021] Each of the aforementioned consistent stylized images is identified as an image in the multi-viewpoint stylized image set.

[0022] According to the method for synthesizing light field images provided by the present invention, the generative adversarial network is trained based on the following method:

[0023] For each of the training images, the training image is input into the generator of the initial generative adversarial network to obtain at least one generated image corresponding to the training image;

[0024] For each generated image, the generated image, the training image, and the verification image corresponding to the training image are all input into the discriminator of the initial generative adversarial network to obtain the overfitting result corresponding to the generated image;

[0025] Based on the overfitting results corresponding to each of the training images, the model parameters of the generator and the discriminator are determined to obtain the generative adversarial network.

[0026] According to a method for synthesizing a light field image provided by the present invention, the step of inputting the generated image, the training image, and the verification image corresponding to the training image into the discriminator of the initial generative adversarial network to obtain the overfitting result corresponding to the generated image includes:

[0027] The generated image, the training image, and the verification image corresponding to the training image are all input into the discriminator, and the overfitting result corresponding to the generated image is calculated using the following formula (1):

[0028]

[0029] Among them, D train D represents the output of the discriminator on the training image. validation D represents the output of the discriminator to the verification image. generated This represents the output of the discriminator to the generated image. r represents the mean of the corresponding output over N consecutive minibatches. v r represents the overfitting result. v =0 indicates no overfitting, r v =1 indicates overfitting.

[0030] According to a method for synthesizing light field images provided by the present invention, the step of acquiring at least one frame of a two-dimensional image of a three-dimensional scene includes:

[0031] When the three-dimensional scene is a real three-dimensional scene, at least one disparity map is acquired by image acquisition of the real three-dimensional scene, each disparity map being an image obtained based on scene information recorded from different angles by a set of camera arrays, and each disparity map is determined as the two-dimensional image; and / or,

[0032] In the case that the three-dimensional scene is a virtual three-dimensional scene, at least one virtual image is obtained by image acquisition of the virtual three-dimensional scene, each virtual image is an image obtained by recording scene information from different angles by a virtual camera, and each virtual image is determined as the two-dimensional image.

[0033] The present invention also provides an apparatus for synthesizing light field images, comprising:

[0034] The acquisition module is used to acquire at least one frame of a 2D image of a 3D scene;

[0035] The processing module is used to input the two-dimensional images into a generative adversarial network for each of the two-dimensional images to obtain a multi-view stylized image set. The multi-view stylized image set includes stylized images from multiple different perspectives corresponding to the two-dimensional images. The generative adversarial network is obtained by training an initial generative adversarial network based on a training set and a validation set. The training set includes at least one training image, and the validation set includes a validation image corresponding to the training image. The validation image is obtained by performing three-dimensional consistent stylization processing on the training image.

[0036] The synthesis module is used to synthesize images based on the multi-viewpoint stylized image sets corresponding to each of the two-dimensional images to obtain the light field image of the three-dimensional scene.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for synthesizing light field images as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for synthesizing light field images as described above.

[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for synthesizing light field images as described above.

[0040] This invention provides a method, apparatus, electronic device, and storage medium for synthesizing light field images. The method involves acquiring at least one frame of a two-dimensional image of a three-dimensional scene. For each two-dimensional image, the image is input into a generative adversarial network (GAN) to obtain a multi-viewpoint stylized image set. The multi-viewpoint stylized image set includes stylized images from multiple different perspectives corresponding to the two-dimensional images. The GAN is obtained by training an initial GAN ​​based on a training set and a validation set. The training set includes at least one training image, and the validation set includes validation images corresponding to the training images. The validation images are obtained by performing three-dimensional consistent stylization processing on the training images. Image synthesis is performed based on the multi-viewpoint stylized image sets corresponding to each two-dimensional image to obtain a light field image of the three-dimensional scene. Since the GAN includes a generator and a discriminator, the generator strives to generate images that are as realistic as possible, while the discriminator tries to distinguish between real and generated images. This competitive mechanism allows the trained GAN to continuously optimize the quality of the generated images, making them closer to real images. The generative adversarial network trained in this invention is obtained by training a training set and a validation set. The training set includes at least one training image, and the validation set includes a validation image corresponding to the training image. The validation image is obtained by performing three-dimensional consistent stylization on the training image. Therefore, by inputting at least one frame of two-dimensional image into the trained generative adversarial network, a multi-view stylized image set that takes into account both stylization and three-dimensional consistency can be obtained. When image synthesis is performed based on multiple stylized images from different perspectives in this multi-view stylized image set, a light field image that takes into account both stylization and three-dimensional consistency can be obtained. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a schematic flowchart of the light field image synthesis method provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of the grating-type light field display device provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the parameters of the grating-type light field display device provided in an embodiment of the present invention;

[0045] Figure 4 This is a flowchart of the synthesized light field image provided in the embodiments of the present invention;

[0046] Figure 5 This is a schematic diagram of the structure of the light field image synthesis device provided in an embodiment of the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] It should be noted that the serial numbers assigned to the objects described in this invention, such as "first" and "second", are only used to distinguish the objects being described and do not have any sequential or technical meaning.

[0050] Light field communication is a communication method based on light field technology, capable of transmitting images and videos with a sense of three-dimensional depth. By capturing and transmitting light field information, light field communication systems can provide a more realistic and immersive viewing experience. Stylization is a crucial technical requirement in light field communication. For example, stylizing portraits involves artistically processing the human figure in an image to give it a specific style and effect. Such processing can enhance the light field communication experience, allowing the receiver to obtain a richer and more personalized visual experience when viewing light field images.

[0051] The inventors of this application have discovered that achieving image stylization while maintaining 3D consistency in light field communication systems presents certain difficulties. In the generation of multi-view images and the synthesis of light field images, achieving stylization requires steps such as target identification, stylization processing, and background processing for the target to be stylized, such as portraits. Furthermore, the stylized multi-view images need to be synthesized into light field images to maintain 3D consistency and realism when viewing the synthesized light field image. However, maintaining 3D consistency is difficult when synthesizing stylized multi-view images into light field images, making it challenging to simultaneously achieve stylization and 3D consistency in the synthesized light field image. Therefore, this invention aims to provide a method for synthesizing light field images that balances stylization and 3D consistency in the context of light field communication.

[0052] Based on this, embodiments of the present invention provide a method for synthesizing light field images. This method involves acquiring at least one frame of a two-dimensional image of a three-dimensional scene; for each two-dimensional image, inputting it into a generative adversarial network (GAN) to obtain a multi-view stylized image set. The multi-view stylized image set includes stylized images from multiple different perspectives corresponding to the two-dimensional images. The GAN is obtained by training an initial GAN ​​based on a training set and a validation set. The training set includes at least one training image, and the validation set includes validation images corresponding to the training images. The validation images are obtained by performing three-dimensional consistent stylization processing on the training images. Image synthesis is performed based on the multi-view stylized image sets corresponding to each two-dimensional image to obtain a light field image of the three-dimensional scene. Since the GAN includes a generator and a discriminator, the generator strives to generate images that are as realistic as possible, while the discriminator tries to distinguish between real and generated images. This competitive mechanism allows the trained GAN to continuously optimize the quality of the generated images, making them closer to real images. The generative adversarial network trained in this invention is obtained by training a training set and a validation set. The training set includes at least one training image, and the validation set includes a validation image corresponding to the training image. The validation image is obtained by performing three-dimensional consistent stylization on the training image. Therefore, by inputting at least one frame of two-dimensional image into the trained generative adversarial network, a multi-view stylized image set that takes into account both stylization and three-dimensional consistency can be obtained. When image synthesis is performed based on multiple stylized images from different perspectives in this multi-view stylized image set, a light field image that takes into account both stylization and three-dimensional consistency can be obtained.

[0053] The following is combined with Figures 1 to 4 The method for synthesizing light field images provided in the embodiments of the present invention will be described. Figure 1 This is a flowchart illustrating the light field image synthesis method provided in this embodiment of the invention. The light field image synthesis method provided in this embodiment is applicable to stylized light field image synthesis of two-dimensional images in various scenes, resulting in stylized light field images with high three-dimensional consistency. The execution subject of this method can be an electronic device such as a tablet computer, mobile phone, computer, server, server cluster, or a specially designed light field image synthesis device, or it can be a light field image synthesis apparatus installed in such an electronic device. This light field image synthesis apparatus can be implemented through software, hardware, or a combination of both. Figure 1 As shown, the method for synthesizing the light field image includes steps 110 to 130.

[0054] Step 110: Obtain at least one frame of two-dimensional image of the three-dimensional scene.

[0055] In this step, the 3D scene can be of any form, such as a real-world 3D scene or a simulated 3D scene. By acquiring images of the 3D scene using an image acquisition device, at least one frame of a 2D image can be obtained. Alternatively, a 2D image of the simulated 3D scene can be acquired using a simulation camera. The acquired 2D image can be an RGB color image with three channels: red, green, and blue (RGB).

[0056] Step 120: For each two-dimensional image, input the two-dimensional image into the generative adversarial network to obtain a multi-view stylized image set. The multi-view stylized image set includes stylized images from multiple different perspectives corresponding to the two-dimensional image. The generative adversarial network is obtained by training the initial generative adversarial network based on the training set and the validation set. The training set includes at least one training image, and the validation set includes a validation image corresponding to the training image. The validation image is obtained by performing three-dimensional consistent stylization processing on the training image.

[0057] In this step, the Generative Adversarial Network (GAN) is a network model obtained by training the initial GAN ​​on the training set and validation set.

[0058] For example, the initial generative adversarial network includes a generator and a discriminator. The generator strives to produce images that are as realistic as possible, while the discriminator tries to distinguish between real and generated images. This competitive mechanism allows the trained generative adversarial network to continuously optimize the quality of the generated images, making them closer to real images. The training set includes at least one training image, and the validation set includes validation images corresponding to the training images. The validation images are obtained by performing 3D consistent stylization on the training images.

[0059] After training the initial generative adversarial network on the training and validation sets, the initial generative adversarial network can learn how to generate multi-view stylized image sets that take into account both stylization and 3D consistency. Thus, in application, it can generate multi-view stylized image sets that take into account both stylization and 3D consistency for input single-frame 2D images or multi-frame 2D graphics.

[0060] Among them, the multi-viewpoint stylized image set can be understood as an image set that includes multiple frames of multi-viewpoint stylized images. The multi-viewpoint stylized images are multiple images obtained after stylizing and multi-view processing based on the input two-dimensional image, that is, multiple stylized images with different perspectives corresponding to the two-dimensional image.

[0061] Step 130: Based on the multi-viewpoint stylized image set corresponding to each two-dimensional image, perform image synthesis to obtain the light field image of the three-dimensional scene.

[0062] In this step, existing multi-view image synthesis methods can be used to synthesize the multi-view stylized image sets corresponding to each two-dimensional image to obtain the light field image of the three-dimensional scene.

[0063] For example, pixel-level fusion strategies such as weighted averaging and Laplacian pyramid fusion can be used to fuse image information from different viewpoints to generate a light field image. When generating the light field image, angle and depth information from the multi-viewpoint images can be used to reconstruct the light field data. Optionally, post-processing, such as tone mapping and contrast enhancement, can be performed on the synthesized light field image to improve the visual effect. In specific implementations, the image synthesis method can be adaptively determined according to different application scenarios, hardware conditions, and data characteristics.

[0064] The method for synthesizing light field images provided in this invention involves acquiring at least one frame of a two-dimensional image of a three-dimensional scene; inputting each two-dimensional image into a generative adversarial network (GAN) to obtain a multi-view stylized image set, which includes stylized images from multiple different perspectives corresponding to the two-dimensional images; training the GAN on an initial GAN ​​based on a training set and a validation set; the training set including at least one training image and the validation set including validation images corresponding to the training images, which are obtained by performing three-dimensional consistent stylization on the training images; and synthesizing images based on the multi-view stylized image sets corresponding to each two-dimensional image to obtain a light field image of the three-dimensional scene. Since the GAN includes a generator and a discriminator, the generator strives to generate images that are as realistic as possible, while the discriminator tries to distinguish between real and generated images. This competitive mechanism allows the trained GAN to continuously optimize the quality of the generated images, making them closer to real images. The generative adversarial network trained in this invention is obtained by training a training set and a validation set. The training set includes at least one training image, and the validation set includes a validation image corresponding to the training image. The validation image is obtained by performing three-dimensional consistent stylization on the training image. Therefore, by inputting at least one frame of two-dimensional image into the trained generative adversarial network, a multi-view stylized image set that takes into account both stylization and three-dimensional consistency can be obtained. When image synthesis is performed based on multiple stylized images from different perspectives in this multi-view stylized image set, a light field image that takes into account both stylization and three-dimensional consistency can be obtained.

[0065] In practical applications, image stylization mainly involves stylizing portraits, animals, cartoons, or other targets within the image. Therefore, in order to perform fast, accurate, and high-quality stylization on targets, we can first segment the two-dimensional image into foreground and background instances. The foreground is the target that needs to be stylized. Stylization is then performed on the foreground to improve the efficiency and quality of image stylization.

[0066] In one embodiment, when inputting a two-dimensional image into a generative adversarial network to obtain a multi-view stylized image set, this can be achieved in the following way:

[0067] The two-dimensional image is segmented into foreground and background instances to obtain a first pixel region corresponding to the foreground and a second pixel region corresponding to the background. The first and second pixel regions are input into a generative adversarial network (GAN). The first pixel region is stylized by the rendering module of the GAN to obtain a stylized first pixel region. The stylized first and second pixel regions are merged to generate a stylized image. Based on the stylized images from multiple different perspectives, a multi-viewpoint stylized image set is obtained.

[0068] Specifically, the foreground can include target objects in images such as portraits, animals, and cartoons, while the background can be understood as the part of a two-dimensional image excluding the foreground. Accurately separating the foreground and background can provide a clear foreground mask for subsequent stylization processing, better achieving three-dimensional consistency and stylization of the light field image.

[0069] For example, an instance segmentation network can be used to accurately segment the foreground in an image. This instance segmentation network could be, for example, a trained YOLO-based network model. During training, the model is trained using a labeled dataset. The preprocessed image is then input into the trained instance segmentation network for forward inference to obtain the segmentation result for each pixel. The final post-processing steps can further improve the accuracy and quality of the segmentation, such as noise removal, hole filling, and edge smoothing.

[0070] For example, the powerful parallel computing capabilities of a Graphics Processing Unit (GPU) can be utilized to extract image features using backbone networks such as VGGNet and ResNet after capturing a 2D image. Based on features such as grayscale, color, and geometric shape, the image is divided into several non-overlapping regions. The location and category of the target are detected within the image. Then, the selected regions are labeled for instance segmentation, and finally, the instance segmentation results are output, yielding the first and second pixel regions. Optionally, the parallel computing capabilities of the GPU can be fully utilized to process large-scale 3D light field data at near real-time speeds, enabling real-time portrait stylization and light field image synthesis.

[0071] For example, a generative adversarial network may include a rendering module for stylization processing, through which the target object can be stylized.

[0072] Using a portrait as an example, we can illustrate stylization processing. Portrait stylization involves applying stylistic treatments to extracted areas of the face, including but not limited to color adjustments, texture enhancement, and the addition of special effects. By artistically processing these areas, specific styles and effects are given to enhance the communication experience and personalization. Stylization includes sketch portrait stylization, cartoon character (animation) stylization, and oil painting stylization. Sketch portrait style focuses on preserving the facial contours and key features, removing irrelevant colors and textures, and mimicking a sketch style, resulting in a relatively serious look. Cartoon character (animation) style emphasizes simplicity, unified color and texture styles, and appropriate enhancement of facial features, such as adjustments to the eyes and nose. Oil painting style focuses on rendering artistically impactful colors and textures, significantly altering the overall content. Stylization can often be described mathematically, for example, using a Gram matrix as follows:

[0073]

[0074] The matrix is ​​defined as the matrix formed by the inner product of any k vectors in n-dimensional Euclidean space.

[0075] For example, after obtaining the stylized first pixel region, the stylized first pixel region and the second pixel region can be merged to generate a stylized image, and then a multi-viewpoint stylized image set can be obtained based on stylized images from multiple different perspectives.

[0076] In this embodiment, by performing instance segmentation of the foreground and background on the two-dimensional image, the pixel region of the foreground can be stylized quickly, accurately, and with high quality, thereby improving the efficiency and quality of stylization processing.

[0077] In practical applications, pasting a stylized foreground onto a background can generate an artistically styled image. However, this can lead to inconsistencies in color, lighting, and other statistical information between the foreground and background, making the overall stylized image appear unrealistic and disharmonious. Therefore, it is necessary to transfer the style information, such as texture, brushstrokes, patterns, and colors, from the first pixel region of the stylized image to the corresponding second pixel region of the background to ensure consistency in style between the foreground and background pixels in the stylized image.

[0078] In one embodiment, merging the stylized first pixel region and the second pixel region to generate a stylized image can be achieved in the following way:

[0079] Extract style information representing the foreground style from the stylized first pixel region; transfer the style information to the second pixel region to obtain the stylized second pixel region; merge the stylized first pixel region and the stylized second pixel region to generate the stylized image.

[0080] Specifically, style information representing the foreground style in the stylized first pixel region can be extracted using the Fast Fourier Transform (FFT) in the frequency domain. Style information includes, but is not limited to, texture, brushstrokes, patterns, and colors.

[0081] For example, a dual-domain style image harmonization network, which includes a dual-domain generator and a dual-domain discriminator, can be used to extract style information representing the foreground style in the first stylized pixel region and transfer this style information to the second pixel region to obtain the stylized second pixel region. This completes the harmonization process of the synthesized image and improves the overall stylization quality of the generated stylized image.

[0082] In this embodiment, style information representing the foreground style is extracted from the stylized first pixel region; this style information is then transferred to the second pixel region to obtain a stylized second pixel region; and the stylized first and second pixel regions are merged to generate a stylized image. This avoids the problem of inconsistent style in the overall stylized image caused by only stylizing the first pixel region corresponding to the foreground, and prevents unnatural transitions between the foreground and background regions when directly merging the stylized first and second pixel regions. Therefore, the method of this embodiment can improve the overall stylization quality of the stylized image.

[0083] For example, in order to further improve the three-dimensional consistency of the light field image, pixel-level three-dimensional position alignment processing can be performed on each stylized image in the multi-viewpoint stylized image set to calibrate the position of each pixel before synthesizing the light field image.

[0084] In one embodiment, when obtaining a multi-viewpoint stylized image set based on stylized images from multiple different perspectives, it can be specifically achieved in the following way:

[0085] Geometric correction and pixel-level alignment are performed on the pixels in each stylized image to obtain a consistent stylized image corresponding to each stylized image; each consistent stylized image is then identified as an image in the multi-viewpoint stylized image set.

[0086] Specifically, pixel-level geometric correction mainly eliminates image geometric distortion caused by factors such as camera position and lens distortion. For example, geometric correction can be performed through camera calibration, distortion correction, and perspective transformation.

[0087] For example, pixel-level alignment can be achieved through feature point detection and matching, transform model estimation, and image resampling and alignment. For instance, feature point detection algorithms, such as Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF), are used to extract feature points from the image at each viewpoint, and matching algorithms are used to find corresponding points between different images. Based on the matched feature point pairs, transform models, such as affine transformations and homography transformations, are set to describe the geometric relationships between different images. The set transform model is then applied to resample and transform one of the multi-viewpoint images, aligning it pixel-level with another multi-viewpoint image.

[0088] For example, stylized images synthesized from different angles can define a multi-viewpoint stylized image set. Each viewpoint of each multi-viewpoint stylized image provides a unique perspective on the scene, capturing more depth and detail information of the human figure. By synthesizing images from different viewpoints, a light field image is generated under light field communication, and the stylized light field image can be displayed in real time and dynamically on a light field display.

[0089] In this embodiment, when obtaining a multi-viewpoint stylized image set based on multiple stylized images from different perspectives, geometric correction and pixel-level alignment are performed on the pixels in each stylized image to obtain a consistent stylized image corresponding to each stylized image; each consistent stylized image is then identified as an image in the multi-viewpoint stylized image set. Based on this, by performing geometric correction and pixel-level alignment on the images, the consistency of pixels in the foreground and background across different perspectives can be ensured, eliminating parallax and inconsistency issues, and further improving the three-dimensional consistency of the synthesized light field image.

[0090] In one embodiment, the generative adversarial network is trained based on the following method:

[0091] For each training image, the training image is input into the generator of the initial generative adversarial network to obtain at least one generated image corresponding to the training image; for each generated image, the generated image, the training image, and the verification image corresponding to the training image are all input into the discriminator of the initial generative adversarial network to obtain the overfitting result corresponding to the generated image; based on the overfitting result corresponding to each training image, the model parameters of the generator and the discriminator are determined to obtain the generative adversarial network.

[0092] Specifically, the initial generative adversarial network can be an Efficient Geometry-aware 3D (EG3D) model. The EG3D model is a deep learning model that can be used to generate high-quality 3D images. It combines generative adversarial networks (GANs) and 3D geometry-aware technology to generate 3D images with realistic details and high realism. The explicit-implicit hybrid neural network architecture of the EG3D model includes a feature generator and mapping network based on StyleGAN2, a lightweight feature decoder, a neural rendering module, and a StyleGAN2 discriminator that can identify locations. The stylization processing in the rendering module includes steps such as color adjustment, texture enhancement, and special effects addition, making the image artistically attractive and visually appealing while maintaining depth and three-dimensionality.

[0093] For example, during training, disparity estimation can be performed using a disparity estimation module to obtain the positional offset of the same object in an image. Disparity estimation based on light field data aims to determine the disparity values ​​between images from different viewpoints, i.e., the positional offset of the same object in an image from adjacent viewpoints. This can be achieved by analyzing the displacement, texture information, and brightness changes of objects from different viewpoints, ensuring the accuracy of positional relationships in light field communication. Further, a depth map can be generated using a depth map generation module. Based on the disparity map, a depth map of the scene can be generated, i.e., the depth value of each pixel. The depth map represents the distance relationships of objects in the scene, describing the position of objects in three-dimensional space. Optionally, smaller disparity values ​​can be set to correspond to farther objects, and larger disparity values ​​to correspond to closer objects, to ensure the accuracy of the depth relationships in the synthesized 3D images.

[0094] In one embodiment, the generated image, the training image, and the verification image corresponding to the training image can all be input into the discriminator, and the overfitting result corresponding to the generated image can be calculated using the following formula (1):

[0095]

[0096] Among them, D train D represents the output of the discriminator on the training images. validation D represents the output of the discriminator to the verification image. generated This represents the discriminator's output on the generated image. r represents the mean of the corresponding output over N consecutive minibatches. v r represents the overfitting result. v =0 indicates no overfitting, r v =1 indicates overfitting.

[0097] Specifically, processing the received multi-view images using a generator and discriminator can improve the 3D consistency of the images. The discriminator provides discriminative feedback, a crucial step in achieving high 3D consistency in the light field image. Discriminative feedback aims to perform fidget analysis (FID) calculations on the generated and real images in the multi-view image to maintain the 3D consistency of the synthesized light field image during viewing. After obtaining the generated image, the image data, after portrait stylization and background processing, are synthesized from multiple viewpoints to generate a 3D consistent light field image. During the synthesis process, a smooth transition between the portrait and the background needs to be considered to maintain the naturalness and realism of the overall image. The discriminative feedback ensures that the synthesized light field image maintains 3D consistency during viewing, guaranteeing that the synthesized portrait and background meet the requirements of visual perception and the real scene. The specific methods and algorithms for rendering the image can be selected and adjusted according to the needs of the actual application to obtain the best processing results.

[0098] For example, D is used for the training set, validation set, and generated images, respectively. train D validation and D generated The discriminator output represents the training set, validation set, and generated images, and is expressed using... This represents their mean across N consecutive minibatches. In practical applications, N = 4 can be used, corresponding to 4 × 64 = 256 generated images. The overfitting result for each generated image can be determined using the formula (1) above.

[0099] For example, when determining the model parameters of the generator and discriminator based on the overfitting results corresponding to each training image, the proportion of the overfitting results among all results can be used to determine whether the initial generative adversarial network needs to continue training. For instance, if the proportion of the overfitting results among all results is less than a threshold, it can be determined that further training is not necessary, and the model parameters of the generator and discriminator can be set as the final trained parameters to obtain the generative adversarial network.

[0100] In this embodiment, for each training image, the training image is input into the generator of the initial generative adversarial network (GAN) to obtain at least one generated image corresponding to the training image. For each generated image, the generated image, the training image, and the corresponding validation image are all input into the discriminator of the initial GAN ​​to obtain the overfitting result corresponding to the generated image. Based on the overfitting result corresponding to each training image, the model parameters of the generator and the discriminator are determined to obtain the GAN. Based on this, the GAN can be effectively trained, improving the efficiency of model training.

[0101] In practical applications, optical field communication is used for real-world 3D scenes and virtual 3D scenes. For different scenarios, at least one frame of 2D image of the 3D scene can be obtained in different ways.

[0102] In one embodiment, acquiring at least one frame of a two-dimensional image of a three-dimensional scene can be achieved in the following way:

[0103] When the 3D scene is a real 3D scene, acquire at least one frame of disparity map obtained by image acquisition of the real 3D scene, each disparity map is an image obtained by recording scene information from different angles based on a set of camera arrays, and each disparity map is determined as a 2D image; and / or, when the 3D scene is a virtual 3D scene, acquire at least one frame of virtual image obtained by image acquisition of the virtual 3D scene, each virtual image is an image obtained by recording scene information from different angles based on a virtual camera, and each virtual image is determined as a 2D image.

[0104] Specifically, acquiring 2D images of a real 3D scene involves using a camera array to record information from different angles of the real-world 3D scene. Each image captured from a different angle is called a disparity map. Camera arrays include color camera arrays, depth camera arrays, and light field camera arrays. Acquiring 2D images of a virtual 3D scene involves using a virtual camera array in modeling software to capture images of a computer-generated 3D model or scene from multiple angles. Because virtual cameras are not constrained by camera size, the camera spacing can be infinitely small. Modeling software includes 3DS MAX, MAYA, Unity 3D, and Blender. The process of acquiring 2D images involves using multiple cameras to simultaneously record the propagation and interaction of light in the scene to obtain more comprehensive and three-dimensional data, thereby achieving a detailed record of the scene's depth, shape, and lighting.

[0105] For example, the electronic device executing the action may include an image acquisition module for acquiring two-dimensional images. For instance, the image acquisition module can acquire images of a real scene or a virtual scene. The image acquisition module provides a data source for the three-dimensional display device to obtain stereoscopic information, which is manifested by using a set of camera arrays to acquire multi-angle three-dimensional information of the scene.

[0106] For example, a digital camera can have a main lens that captures light, focuses it onto a photosensitive plane behind the lens, and then converts it into a digital image signal through the digital camera's imaging element. A depth camera can be a camera that uses infrared light or other technologies to achieve depth perception. It can calculate the distance to an object by measuring the time it takes for light to reflect back from the object's surface, thereby achieving depth recognition of a 3D scene. Depth cameras are generally used in conjunction with digital cameras. A light field camera can be a camera based on a light field sensor. By simultaneously recording light information from multiple directions in a scene, it can achieve post-processing adjustments to depth of field, viewpoint, and focal length, thereby improving the flexibility and editability of images. Optionally, a digital camera, such as a mobile phone camera, can be used to capture real 3D scenes.

[0107] In this embodiment, for both real and virtual 3D scenes, at least one frame of 2D image can be obtained through different methods to obtain the necessary materials for light field image synthesis. Based on this, the applicability of the present invention can be improved and the scope of application scenarios can be expanded.

[0108] For example, image synthesis can be performed based on the multi-viewpoint stylized image set corresponding to each two-dimensional image to obtain the light field image of the three-dimensional scene.

[0109] Specifically, multi-viewpoint image synthesis can be performed on image data that has undergone portrait stylization and background processing to generate a 3D light field image. The image data, containing the stylized portrait area and the processed background area, is used as input. The synthesis is performed using the encoding algorithm of a raster light field display device, and finally displayed on the device.

[0110] Figure 2 This is a schematic diagram of the structure of the grating-type light field display device provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the grating-type light field display device includes an LCD or LED display panel, a transparent pad, and a cylindrical grating. Information displayed on the 2D LED panel can be observed from different viewpoints 1, 2, 3, and 4 through the cylindrical grating. Figure 3 This is a schematic diagram of the parameters of the grating-type light field display device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, with the top left corner of the screen as the origin (0,0), the parameters of the raster light field display device are... Figure 3As shown in the figure. The number of viewpoints N can be calculated using the following formula (2). kl :

[0111]

[0112] Where (k,l) is the sub-pixel index; α is the tilt angle; D x N is the number of viewpoints within the period of the horizontal slit; tot k is the total number of viewpoints. off It is the horizontal distance from the top left corner of the sub-pixel to the point (0,0).

[0113] Given the sub-pixel index (k,l), the tilt angle α and the number of viewpoints D within the horizontal slit period can be used to determine the optimal viewing angle. x Total number of viewpoints N tot and k off The corresponding number of viewpoints N is calculated. kl The V here shift This value can be applied to stereoscopic displays based on human eye tracking. Optionally, to improve the display effect, a weighted encoding operation can be introduced, that is, mixing information from multiple viewpoints within a sub-pixel according to different proportions. Based on this, a final light field image can be generated from the multi-viewpoint image synthesized from the viewpoints, and this light field image can be displayed through a grating-type light field display device to represent the propagation of light and changes in viewpoint in the scene, thereby achieving three-dimensional information display and a richer viewing experience.

[0114] The method of the present invention will now be described in a specific implementation. Figure 4 This is a flowchart of the synthesized light field image provided in an embodiment of the present invention, such as... Figure 4 As shown, a camera array is used to acquire light field information of a real or virtual 3D scene, obtaining at least one frame of 2D image. A small number of 2D images are used to generate stylized images from various angles to give the images an artistic style; that is, a multi-viewpoint stylized image set is generated based on the 2D images. Using a discriminative feedback mechanism in a GAN network, loss is calculated and fed back through constraints of a loss function, allowing the network to learn to generate harmonized images, resulting in multi-viewpoint stylized images with high stylization and 3D consistency. Finally, a light field image can be synthesized based on the multi-viewpoint stylized images with higher stylization and 3D consistency.

[0115] For example, the execution body of this method may include an image acquisition module, a portrait segmentation module, a stylization processing module, a background processing module, and a multi-viewpoint image synthesis module. First, the image acquisition module acquires at least one frame of a two-dimensional image. Then, the portrait segmentation module accurately separates the person from the background in the two-dimensional image, providing precise regions for subsequent stylization processing. Next, the stylization processing module performs stylization processing on the portrait, including color adjustment, texture enhancement, and special effects addition, to enhance the image's artistic feel and personalized effect. Then, the background processing module stylizes the background and performs background synthesis, ensuring the realism and visual consistency of the synthesized image. Finally, the multi-viewpoint image synthesis module merges the stylized portrait with the synthesized background, achieving three-dimensional consistency when viewed in the light field system.

[0116] The light field image synthesis method provided in this invention includes a complete set of methods for segmentation, stylization, simulation, and display of light field stylization. By stylizing a human figure in a three-dimensional light field, it can achieve human figure stylization and three-dimensional image synthesis at near real-time speed. This method can achieve high-quality, real-time three-dimensional stylization effects in light field communication systems, providing an effective solution and a new methodological approach for the application of light field communication systems.

[0117] For example, a camera can be used to photograph a person, with uniform and soft lighting. At least one camera is placed on a plane 1.5 meters away, with a total placement length of 2.5 meters, outputting a single-viewpoint image with a resolution of 2560×1440. Utilizing the powerful parallel computing capabilities of the GPU, after photographing the person, the pixels of the foreground person and background are identified and classified for accurate separation. Single or a small number of 2D single-viewpoint images are used to generate stylized images from various angles. The image generation network performs deep learning based on the input person and background, using a GAN network for image generation and discrimination, iteratively training to generate images. Based on the input camera position parameters, a 2D person image is generated every 10 degrees, preparing for subsequent 3D image synthesis. Through the constraints of the loss function, the GAN network learns to generate images with 3D consistency. Based on the discriminative feedback of the GAN network, the loss value of the multi-viewpoint 2D images generated in the previous step is calculated and fed back to the image generation network, dynamically adjusting the learning strategy of the generation network to obtain images with 3D consistency. The aforementioned discriminator feedback unit is the discriminator in a generative adversarial network. The generated, three-dimensional consistent multi-viewpoint 2D images are synthesized into a light field image using a light field synthesis algorithm and displayed on a 65-inch cylindrical lens grating 3D display. A viewer positioned 2 meters from the display can see the synthesized, three-dimensional consistent light field image by moving no more than 1.25 meters laterally and no more than 0.5 meters forward or backward from the center of the display. This invention enables portrait stylization processing of the received multi-viewpoint images and ensures that the generated light field image maintains realism and three-dimensional consistency during the multi-viewpoint image synthesis process.

[0118] The apparatus for synthesizing light field images provided in the embodiments of the present invention will be described below. The apparatus for synthesizing light field images described below can be referred to in correspondence with the method for synthesizing light field images described above.

[0119] Figure 5 This is a schematic diagram of the structure of the light field image synthesis device provided in an embodiment of the present invention, with reference to... Figure 5 As shown, the light field image synthesis apparatus 500 includes:

[0120] The acquisition module 510 is used to acquire at least one frame of two-dimensional image of the three-dimensional scene;

[0121] The processing module 520 is used to input the two-dimensional images into the generative adversarial network for each two-dimensional image to obtain a multi-view stylized image set. The multi-view stylized image set includes stylized images from multiple different perspectives corresponding to the two-dimensional images. The generative adversarial network is obtained by training the initial generative adversarial network based on the training set and the validation set. The training set includes at least one training image, and the validation set includes a validation image corresponding to the training image. The validation image is obtained by performing three-dimensional consistent stylization processing on the training image.

[0122] The synthesis module 530 is used to synthesize images based on the multi-viewpoint stylized image set corresponding to each two-dimensional image to obtain the light field image of the three-dimensional scene.

[0123] In one example embodiment, the processing module 520 is specifically used for:

[0124] Instance segmentation of the two-dimensional image into foreground and background is performed to obtain the first pixel region corresponding to the foreground and the second pixel region corresponding to the background.

[0125] The first pixel region and the second pixel region are input into the generative adversarial network. The first pixel region is stylized by the rendering module in the generative adversarial network to obtain the stylized first pixel region.

[0126] The first and second pixel regions are merged to generate a stylized image;

[0127] A multi-viewpoint stylized image set is obtained based on stylized images from multiple different perspectives.

[0128] In one example embodiment, the processing module 520 is specifically used for:

[0129] Extract style information representing the foreground style from the stylized first pixel region;

[0130] The style information is transferred to the second pixel region to obtain the stylized second pixel region;

[0131] The stylized first pixel region and the stylized second pixel region are merged to generate a stylized image.

[0132] In one example embodiment, the processing module 520 is specifically used for:

[0133] Geometric correction and pixel-level alignment are performed on the pixels in each stylized image to obtain a consistent stylized image corresponding to each stylized image;

[0134] Each consistent stylized image is identified as an image in the multi-viewpoint stylized image set.

[0135] In one example embodiment, the generative adversarial network is trained based on the following method:

[0136] For each training image, the training image is input into the generator of the initial generative adversarial network to obtain at least one generated image corresponding to the training image;

[0137] For each generated image, the generated image, the training image, and the verification image corresponding to the training image are all input into the discriminator of the initial generative adversarial network to obtain the overfitting result corresponding to the generated image.

[0138] Based on the overfitting results corresponding to each training image, the model parameters of the generator and discriminator are determined, and a generative adversarial network is obtained.

[0139] In one example embodiment, the generated image, the training image, and the verification image corresponding to the training image are all input into the discriminator, and the overfitting result corresponding to the generated image is calculated using the following formula (1):

[0140]

[0141] Among them, D train D represents the output of the discriminator on the training images. validation D represents the output of the discriminator to the verification image. generated This represents the discriminator's output on the generated image. r represents the mean of the corresponding output over N consecutive minibatches. v r represents the overfitting result. v =0 indicates no overfitting, r v =1 indicates overfitting.

[0142] In one example embodiment, the acquisition module 510 is specifically used for:

[0143] In the case that the 3D scene is a real 3D scene, acquire at least one disparity map obtained by image acquisition of the real 3D scene. Each disparity map is an image obtained based on scene information recorded from different angles by a set of camera arrays, and each disparity map is defined as a 2D image; and / or,

[0144] In the case of a virtual 3D scene, at least one virtual image is acquired by image acquisition of the virtual 3D scene. Each virtual image is an image obtained by recording scene information from different angles using a virtual camera, and each virtual image is defined as a 2D image.

[0145] The apparatus of this embodiment can be used to execute the method of any embodiment in the side embodiment of the method for synthesizing light field images. Its specific implementation process and technical effects are similar to those in the side embodiment of the method for synthesizing light field images. For details, please refer to the detailed description in the side embodiment of the method for synthesizing light field images, which will not be repeated here.

[0146] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communication interface 620, a memory 630, and a communication bus 640. The processor 610, communication interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a method for synthesizing light field images. This method includes: acquiring at least one frame of a two-dimensional image of a three-dimensional scene; for each two-dimensional image, inputting the two-dimensional image into a generative adversarial network (GAN) to obtain a multi-view stylized image set, the multi-view stylized image set including stylized images from multiple different perspectives corresponding to the two-dimensional images; the GAN being trained on an initial GAN ​​based on a training set and a validation set; the training set including at least one training image; the validation set including validation images corresponding to the training images; and the validation images being obtained by performing three-dimensional consistent stylization processing on the training images; and synthesizing images based on the multi-view stylized image set corresponding to each two-dimensional image to obtain a light field image of the three-dimensional scene.

[0147] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for synthesizing light field images provided by the above methods. The method includes: acquiring at least one frame of a two-dimensional image of a three-dimensional scene; for each two-dimensional image, inputting the two-dimensional image into a generative adversarial network to obtain a multi-view stylized image set, the multi-view stylized image set including stylized images from multiple different perspectives corresponding to the two-dimensional images, the generative adversarial network being obtained after training an initial generative adversarial network based on a training set and a validation set, the training set including at least one training image, the validation set including a validation image corresponding to the training image, the validation image being obtained after performing three-dimensional consistent stylization processing on the training image; and synthesizing images based on the multi-view stylized image set corresponding to each two-dimensional image to obtain a light field image of the three-dimensional scene.

[0149] On the other hand, embodiments of the present invention also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the light field image synthesis method provided by the above methods. The method includes: acquiring at least one frame of a two-dimensional image of a three-dimensional scene; for each two-dimensional image, inputting the two-dimensional image into a generative adversarial network to obtain a multi-view stylized image set, the multi-view stylized image set including stylized images from multiple different perspectives corresponding to the two-dimensional images, the generative adversarial network being obtained after training an initial generative adversarial network based on a training set and a validation set, the training set including at least one training image, the validation set including a validation image corresponding to the training image, the validation image being obtained after performing three-dimensional consistent stylization processing on the training image; and synthesizing images based on the multi-view stylized image set corresponding to each two-dimensional image to obtain a light field image of the three-dimensional scene.

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synthesizing light field images, characterized in that, include: Acquire at least one frame of a 2D image of a 3D scene; For each of the two-dimensional images, the two-dimensional images are segmented into foreground and background instances to obtain a first pixel region corresponding to the foreground and a second pixel region corresponding to the background; The first pixel region and the second pixel region are input into a generative adversarial network. The first pixel region is stylized by the rendering module in the generative adversarial network to obtain a stylized first pixel region. The stylized first pixel region and the second pixel region are merged to generate the stylized image; A multi-viewpoint stylized image set is obtained based on the stylized images from multiple different perspectives; the multi-viewpoint stylized image set includes stylized images from multiple different perspectives corresponding to the two-dimensional image; the generative adversarial network is obtained by training an initial generative adversarial network based on a training set and a validation set; the training set includes at least one training image; the validation set includes a validation image corresponding to the training image; the validation image is obtained by performing three-dimensional consistent stylization processing on the training image. Image synthesis is performed based on the multi-viewpoint stylized image set corresponding to each of the two-dimensional images to obtain the light field image of the three-dimensional scene; The initial generative adversarial network is a three-dimensional generative adversarial network EG3D model. The EG3D model includes a feature generator and mapping network based on StyleGAN2, a lightweight feature decoder, a neural rendering module, and a StyleGAN2 discriminator with identifiable location.

2. The method for synthesizing light field images according to claim 1, characterized in that, The step of merging the stylized first pixel region and the second pixel region to generate the stylized image includes: Extract style information representing the foreground style from the stylized first pixel region; The style information is transferred to the second pixel region to obtain the stylized second pixel region; The stylized first pixel region and the stylized second pixel region are merged to generate the stylized image.

3. The method for synthesizing light field images according to claim 1, characterized in that, The stylized images based on multiple different perspectives are used to obtain the multi-viewpoint stylized image set, which includes: Geometric correction and pixel-level alignment are performed on the pixels in each of the stylized images to obtain a consistent stylized image corresponding to each stylized image; Each of the aforementioned consistent stylized images is identified as an image in the multi-viewpoint stylized image set.

4. The method for synthesizing light field images according to any one of claims 1-3, characterized in that, The generative adversarial network is trained using the following method: For each of the training images, the training image is input into the generator of the initial generative adversarial network to obtain at least one generated image corresponding to the training image; For each generated image, the generated image, the training image, and the verification image corresponding to the training image are all input into the discriminator of the initial generative adversarial network to obtain the overfitting result corresponding to the generated image; Based on the overfitting results corresponding to each of the training images, the model parameters of the generator and the discriminator are determined to obtain the generative adversarial network.

5. The method for synthesizing light field images according to claim 4, characterized in that, The step of inputting the generated image, the training image, and the verification image corresponding to the training image into the discriminator of the initial generative adversarial network to obtain the overfitting result corresponding to the generated image includes: The generated image, the training image, and the verification image corresponding to the training image are all input into the discriminator, and the overfitting result corresponding to the generated image is calculated using the following formula (1): （1） in, This represents the output of the discriminator on the training image. This represents the output of the discriminator to the verification image. This represents the output of the discriminator to the generated image. This represents the average of the corresponding outputs over N consecutive minibatches. This indicates the overfitting result. This indicates that there is no overfitting. This indicates overfitting.

6. The method for synthesizing light field images according to any one of claims 1-3, characterized in that, The acquisition of at least one frame of two-dimensional image of the three-dimensional scene includes: When the three-dimensional scene is a real three-dimensional scene, at least one disparity map is acquired by image acquisition of the real three-dimensional scene, each disparity map being an image obtained based on scene information recorded from different angles by a set of camera arrays, and each disparity map is determined as the two-dimensional image; and / or, In the case that the three-dimensional scene is a virtual three-dimensional scene, at least one virtual image is obtained by image acquisition of the virtual three-dimensional scene, each virtual image is an image obtained by recording scene information from different angles by a virtual camera, and each virtual image is determined as the two-dimensional image.

7. A device for synthesizing light field images, characterized in that, include: The acquisition module is used to acquire at least one frame of a 2D image of a 3D scene; The processing module is used to perform instance segmentation of the two-dimensional image into foreground and background for each of the two-dimensional images, to obtain a first pixel region corresponding to the foreground and a second pixel region corresponding to the background; The first pixel region and the second pixel region are input into a generative adversarial network. The first pixel region is stylized by the rendering module in the generative adversarial network to obtain a stylized first pixel region. The stylized first pixel region and the second pixel region are merged to generate the stylized image; A multi-viewpoint stylized image set is obtained based on the stylized images from multiple different perspectives; the multi-viewpoint stylized image set includes stylized images from multiple different perspectives corresponding to the two-dimensional image; the generative adversarial network is obtained by training an initial generative adversarial network based on a training set and a validation set; the training set includes at least one training image; the validation set includes a validation image corresponding to the training image; the validation image is obtained by performing three-dimensional consistent stylization processing on the training image. The synthesis module is used to synthesize images based on the multi-viewpoint stylized image set corresponding to each of the two-dimensional images to obtain the light field image of the three-dimensional scene. The initial generative adversarial network is a three-dimensional generative adversarial network EG3D model. The EG3D model includes a feature generator and mapping network based on StyleGAN2, a lightweight feature decoder, a neural rendering module, and a StyleGAN2 discriminator with identifiable location.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for synthesizing light field images as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for synthesizing light field images as described in any one of claims 1 to 6.

Images