Image generation method, model training method, display method, device, and medium

Abstract

This disclosure provides an image generation method, a model training method, a display method, an apparatus, and a medium, relating to the fields of generative artificial intelligence, image processing, and computer vision. The image generation method includes: stitching a noisy image with a first viewpoint image to obtain a noisy image; inputting the noisy image and depth features obtained based on the first viewpoint image into an image generation model, so that, guided by the depth features, noise in the noisy image is gradually removed through N time steps to obtain a second viewpoint image, where N is an integer greater than 1; the depth features represent the relative depth between multiple positions in the first viewpoint image in a semantically encoded form; and the viewpoint directions of the first and second viewpoint images are different.

Description

Technical Field

[0001] This disclosure relates to the fields of generative artificial intelligence, image processing, computer vision, and more specifically, to an image generation method, a model training method, a display method, an apparatus, and a medium. Background Technology

[0002] In the process of creating glasses-free 3D content, a set of left and right binocular views with controllable parallax accuracy needs to be output. By fusing these left and right binocular views, the human eye can perceive the depth and stereoscopic effect of the image. Therefore, how to accurately generate a matching view from a single original view is an important research direction for glasses-free 3D content generation. Summary of the Invention

[0003] In view of this, the present disclosure provides an image generation method, a model training method, a display method, an apparatus, and a medium.

[0004] According to one aspect of this disclosure, an image generation method is provided, comprising: stitching a noisy image with a first viewpoint image to obtain a noisy image; and inputting the noisy image and depth features obtained based on the first viewpoint image into an image generation model, so as to gradually remove noise from the noisy image through N time steps under the guidance of the depth features, thereby obtaining a second viewpoint image, where N is an integer greater than 1, wherein the depth features characterize the relative depth between multiple positions in the first viewpoint image in the form of semantic encoding, and the viewpoint directions of the first viewpoint image and the second viewpoint image are different.

[0005] According to another aspect of this disclosure, a method for training an image generation model is provided, comprising: acquiring a training sample pair, wherein the training sample pair includes a sample first viewpoint image and a sample second viewpoint image, the viewpoint directions of the sample first viewpoint image and the sample second viewpoint image being different; concatenating the noisy sample second viewpoint image with the sample first viewpoint image to obtain a noisy sample image; inputting the noisy sample image and sample depth features obtained based on the sample first viewpoint image into a model to be trained, so that the model to be trained, guided by the sample depth features, gradually removes noise from the sample noisy sample image through N time steps to obtain an output second viewpoint image, where N is an integer greater than 1; and training the parameters of the model to be trained based on the difference between the output second viewpoint image and the sample second viewpoint image to obtain an image generation model.

[0006] According to another aspect of this disclosure, a display method is provided, comprising: displaying a viewpoint image pair based on an original viewpoint image and a target viewpoint image, wherein the target viewpoint image is obtained by processing the original viewpoint image using the image generation method described above.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more instructions, wherein, when the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to perform the methods as disclosed herein.

[0008] According to another aspect of this disclosure, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the methods of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, which includes computer-executable instructions that, when executed, are used to implement the methods as disclosed herein. Attached Figure Description

[0010] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0011] Figure 1 This illustration schematically shows a system architecture to which an image generation method can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 A flowchart illustrating an image generation method according to an embodiment of the present disclosure is shown schematically;

[0013] Figure 3a A flowchart illustrating an image generation method according to another embodiment of the present disclosure is shown schematically;

[0014] Figure 3b Schematic illustration based on Figure 3a The diagram illustrates the process of performing N time steps on a noisy image using the method shown.

[0015] Figure 4a The flowchart illustrating the noise reduction module performing noise reduction according to the first embodiment of the present disclosure is shown in the schematic diagram.

[0016] Figure 4b The flowchart illustrating the noise reduction module performing noise reduction according to the second embodiment of the present disclosure is shown in the schematic diagram.

[0017] Figure 4c The flowchart illustrating the noise reduction module performing noise reduction according to the third embodiment of the present disclosure is shown in the schematic diagram.

[0018] Figure 4d The flowchart illustrating the noise reduction module performing noise reduction according to the fourth embodiment of the present disclosure is shown in the schematic diagram.

[0019] Figure 5A flowchart illustrating a training method for an image generation model according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 6 A flowchart illustrating a training method for an image generation model according to an embodiment of the present disclosure is shown schematically.

[0021] Figure 7 A flowchart illustrating a display method according to an embodiment of the present disclosure is shown schematically;

[0022] Figure 8 A block diagram of an image generation apparatus according to an embodiment of the present disclosure is shown schematically;

[0023] Figure 9 A block diagram schematically illustrates a training apparatus for an image generation model according to an embodiment of the present disclosure;

[0024] Figure 10 A block diagram of a display device according to an embodiment of the present disclosure is shown schematically;

[0025] Figure 11 A block diagram of an electronic device suitable for implementing an image generation method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0026] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0031] For naked-eye 3D content generation, a sequential process of depth estimation, viewpoint warping, and hole inpainting can be adopted to generate another view from a single original view. Specifically, this method first uses a monocular depth estimation algorithm to calculate a pixel-level depth map from a 2D image; then, it uses geometric projection principles to perform 3D warping, remapping the pixels of the left view to the right eye's perspective; finally, for foreground occlusion areas exposed due to the viewpoint switch, an image inpainting algorithm is used to complete the content, thus synthesizing a complete right view.

[0032] However, the above method has the following problems: First, the process is lengthy and complex. The chaining of the three processes not only increases computational latency and system integration complexity, but also causes errors to accumulate at each stage of the chain. Second, since this process relies on the depth map of the display output, this displayed depth map is prone to failure when facing complex scenes such as transparent materials, highly reflective surfaces, or missing textures. Once the depth map deviates, it will directly cause geometric distortion errors, resulting in distortion of the final generated 3D image, making the reliability and accuracy of 3D content generation low.

[0033] To this end, this disclosure provides an image generation method, which includes: stitching a noisy image with a first viewpoint image to obtain a noisy image; inputting the noisy image and depth features obtained based on the first viewpoint image into an image generation model, so as to gradually remove noise from the noisy image through N time steps under the guidance of the depth features, to obtain a second viewpoint image, where N is an integer greater than 1, wherein the depth features characterize the relative depth between multiple positions in the first viewpoint image in the form of semantic encoding, and the viewpoint directions of the first viewpoint image and the second viewpoint image are different.

[0034] Figure 1 The illustration schematically depicts a system architecture to which an image generation method can be applied according to embodiments of the present disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0035] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between different devices.

[0036] It should be noted that the image generation method provided in this embodiment can generally be executed by the server 105. Accordingly, the image generation apparatus provided in this embodiment can generally be located in the server 105.

[0037] Alternatively, the image generation method provided in this embodiment of the present disclosure can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the image generation apparatus provided in this embodiment of the present disclosure can also be disposed in the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0038] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0039] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0040] Figure 2 A flowchart illustrating an image generation method according to an embodiment of the present disclosure is shown schematically.

[0041] like Figure 2 As shown, the image generation method includes operations S210~S220.

[0042] In operation S210, the noisy image is stitched together with the first viewpoint image to obtain the noisy image.

[0043] In operation S210, the noisy image and the depth features obtained based on the first viewpoint image are input into the image generation model. Guided by the depth features, the noise in the noisy image is gradually removed through N time steps to obtain the second viewpoint image. The depth features represent the relative depth between multiple positions in the first viewpoint image in the form of semantic encoding. The viewpoint directions of the first viewpoint image and the second viewpoint image are different, and N is an integer greater than 1.

[0044] The noisy image can be a random noise feature map generated based on Gaussian noise, but this disclosure is not limited to this. The noisy image can also be a random noise feature map generated based on salt and pepper noise, Poisson noise, etc.

[0045] The first viewpoint image is acquired from the first viewpoint direction and contains complete effective pixel information such as color, contour, and depth geometry. The first viewpoint can be, for example, a left viewpoint, a center viewpoint, a right viewpoint, etc.

[0046] A noisy image is a composite feature map obtained by concatenating a noisy image and a first-viewpoint image along the feature channel dimension. The noisy image generated by this channel concatenation stores the first-viewpoint image and the noisy image separately. The pixels, depth texture, object edges, and occlusion relationships of the first-viewpoint image are fully preserved. The first-viewpoint image serves as a disparity reference, constraining the noisy image to generate a matching second-viewpoint image.

[0047] In one example, the first viewpoint image can be encoded using a VAE (Variational Auto-Encoder) to compress the high-resolution first viewpoint image into a low-dimensional latent feature map in a low-dimensional latent space. Simultaneously, the noisy image is compressed into a noisy latent image of the same size. Then, the low-dimensional latent feature map and the noisy latent image are concatenated along the channel dimension to obtain a noisy image. After denoising the noisy image, it is decoded back to a pixel image from the low-dimensional latent space. This method, due to compression, causes pixel blurring, which may result in a blurry second viewpoint image, potentially causing artifacts when viewed by the user.

[0048] In another example, instead of performing VAE encoding on the first viewpoint image and the noisy image, the first viewpoint image and the noisy image can be directly stitched together in pixel space by channel to obtain the noisy image. This method can ensure the clarity of the newly generated second viewpoint image, but it will result in higher computational power consumption.

[0049] In another example, to simultaneously address the issues of sharpness and excessive computational cost, after directly stitching the first-viewpoint image and the noisy image along channels in pixel space to obtain the noisy image, the loaded image can be segmented into image blocks of corresponding block sizes based on the current denoising stage. Smaller block sizes result in more blocks and stronger detail capture, but increase computational cost; larger block sizes result in fewer blocks, with each block containing a large area, reducing computational cost, but potentially losing subtle details and edge features. Therefore, for shallower feature layers, where attention is focused on high-frequency information such as details and textures, smaller block sizes can be used to ensure the learning of high-frequency information; correspondingly, for deeper feature layers, where attention is focused on understanding the global content, larger block sizes can be used to save computational cost and reduce computational complexity.

[0050] Since depth features represent the relative depth between multiple locations in a first-viewpoint image in a semantic encoding form, they can express the depth semantic information of multiple locations in the first-viewpoint image. For example, depth features can provide information on the distance hierarchy of pixels, occlusion boundaries, spatial perspective, and the three-dimensional structure of objects. Therefore, in complex occlusion and perspective-layered scenes, depth features can accurately distinguish the relative distance of each pixel.

[0051] In the process of denoising a noisy image, the first viewpoint image can be used as the basic constraint information, combined with depth features as additional guiding information, to denoise the noisy image in the noisy image, resulting in a second viewpoint image of the scene in the first viewpoint image from the second viewpoint. Specifically, the first viewpoint image provides basic constraint information such as scene texture, color, and object contours; depth features correct the offset of foreground objects, locate the void areas exposed after the viewpoint switch, and improve depth distortion issues in glass and multi-layered occlusion scenes, ensuring that the generated second viewpoint image has a reasonable spatial structure and coherent three-dimensional relationships.

[0052] According to embodiments of this disclosure, processing a noisy image to obtain a second viewpoint image based on depth features obtained from a first viewpoint image may include: inputting the noisy image and depth features into an image generation model, so that, guided by the depth features, noise in the noisy image is gradually removed through N time steps to obtain the second viewpoint image, where N is an integer greater than 1.

[0053] Image generation models can employ diffusion models. For example, an image generation model can be a diffusion model based on the DiT (Diffusion Transformer) architecture of Transformer; however, this disclosure is not limited thereto, and an image generation model can also employ a diffusion model based on the U-Net (U-shaped Network) architecture of convolution.

[0054] The denoising process can be broken down into multiple rounds of iterative denoising. Each time step represents one round of denoising operation, and each round removes a portion of the noise from the image. The total number of iterations N is set, starting from the maximum time step t=N and looping until t=1. The value of N can be from 4 to 200 steps.

[0055] For each iteration of denoising, the current time step can be encoded into a temporal vector, fused with depth features, and then input into the image generation model along with the noisy image. The image generation model can combine depth features and the noisy image of the current iteration to predict the noise of the current iteration, remove the predicted noise from the noisy image to obtain a denoised intermediate noise image, and then proceed to the next iteration of denoising. This process continues until N time-step denoising iterations are completed, at which point the second viewpoint image is output.

[0056] According to embodiments of this disclosure, by inputting a noisy image and depth features into an image generation model, and using the image generation model to perform denoising processing over N time steps, a second viewpoint image can be directly obtained. This enables end-to-end generation of new viewpoint images, simplifies the image generation process, and reduces the complexity of image generation.

[0057] According to embodiments of this disclosure, a noisy image is obtained by stitching together a first viewpoint image and noise. The depth features extracted from the first viewpoint image are then used as a guide to denoise the noisy image. Since the depth features represent the relative positional relationships of each position in the image through semantics, rather than just representing the depth information of pixel positions, this method of using implicit depth semantic features instead of explicit pixel-by-pixel depth estimation as a generation guide makes the generation of 3D content in complex scenes more tolerant of errors, avoids geometric distortion and image distortion caused by depth estimation errors, and improves the reliability and accuracy of image generation.

[0058] The following combination Figures 3a to 4d The image generation method of the present disclosure will be described in more detail below.

[0059] According to an embodiment of this disclosure, depth features are obtained by inputting a first viewpoint image into a depth backbone model, and using multiple feature extraction modules in the depth backbone model to perform multi-level encoding on the first viewpoint image to obtain depth features.

[0060] A deep backbone model can be an image coding backbone network composed of multiple layers of convolutional or Transformer feature extraction units stacked together, used to extract high-dimensional implicit information such as spatial distance, occlusion boundaries, and object semantics corresponding to pixels from a first-viewpoint image.

[0061] The original first-viewpoint image is input into a pre-trained deep backbone model. The deep backbone model contains multiple feature extraction modules arranged in series, which are used to extract deep features of the multi-scale semantic encoding of the first-viewpoint image.

[0062] For example, shallow encoding outputs high-resolution feature maps to capture pixel-level low-level information, including basic visual features such as edges, contours, brightness gradients, and subtle textures; intermediate-level encoding outputs medium-resolution feature maps to extract local contours, planar boundaries, and object block structures, which can characterize the local relative depth changes of objects; deep encoding outputs low-resolution global feature maps to extract global semantic information, identify the overall layout of the scene, object categories, and foreground-background hierarchical relationships, and output globally unified depth distribution semantics; the feature maps of different resolutions output by each level of encoder are combined to form the depth features of multi-scale semantic encoding.

[0063] According to embodiments of this disclosure, the first viewpoint image is encoded from shallow to deep using multiple feature extraction modules built into the deep backbone model. This allows for the layered extraction of image texture, object contours, and global depth semantic information, ultimately resulting in depth features that simultaneously carry scene semantics and three-dimensional spatial geometric constraints. This multi-level encoding method takes into account both local detail boundaries and global perspective levels, enabling precise constraints on pixel mapping logic for transparent media and multi-layered occluded areas during subsequent denoising iterations. It also allows for the early location of void areas caused by viewpoint switching, suppressing image stretching and ghosting defects caused by depth estimation distortion.

[0064] Figure 3a A flowchart illustrating an image generation method according to another embodiment of the present disclosure is shown schematically.

[0065] like Figure 3a As shown, the noisy image 301 is stitched together with the first viewpoint image 302 to obtain a noisy image. The first viewpoint image 302 is input into the depth backbone model M303, where multiple feature extraction modules are used to perform multi-level encoding on the first viewpoint image to obtain depth features. The noisy image and the depth features are then input into the image generation model M304. Guided by the depth features, the noise in the noisy image is gradually removed through N time steps to obtain the second viewpoint image 305.

[0066] Figure 3b Schematic illustration based on Figure 3aThe diagram illustrates the process of performing N time steps on a noisy image.

[0067] like Figure 3b As shown, in this embodiment, the time step is 4. By performing iterative denoising on the noisy image for 4 time steps, the noise in the noisy image is gradually removed, and a clear second viewpoint image can be obtained.

[0068] The above describes the overall method of image generation. The following section provides a detailed description of the specific process of denoising noisy images guided by depth features in the image generation model.

[0069] According to embodiments of this disclosure, the image generation model includes M denoising modules, where M is an integer greater than 1. For the nth time step, n=1, ..., N, guided by depth features, noise in the noisy image is gradually removed through N time steps to obtain a second viewpoint image. This may include: determining the nth guiding feature based on the depth features and the embedding features obtained based on the time step information of the nth time step; and using the M denoising modules to process the noisy image and the nth guiding feature to obtain the nth intermediate viewpoint image, wherein the second viewpoint image is the intermediate viewpoint image obtained at the Nth time step.

[0070] Time step information can characterize the residual noise intensity at the current denoising stage. A larger n value indicates a higher proportion of image noise and more incomplete image structure information, while a smaller n value indicates a higher degree of noise removal and more complete image details.

[0071] For a time step number n in discrete integer form, the time step can be mapped to a time series vector of uniform dimension through sinusoidal position coding or learnable embedding lookup table. Then, through multi-layer linear mapping and activation function, feature dimensionality enhancement and nonlinear transformation are completed to obtain the embedded features representing the time series.

[0072] The embedded features and depth features can be processed by feature concatenation or channel-weighted fusion to obtain the nth guiding feature. The embedded feature corresponding to the nth time step serves as a temporal control factor, while the depth feature output from the first viewpoint image represents spatial geometric constraints. On one hand, the depth feature fixes the perspective and occlusion relationships of objects; on the other hand, the time-step embedded feature controls the noise removal intensity of that round, avoiding viewpoint distortion, hole repair, and texture misalignment.

[0073] M denoising modules represent a multi-level feature processing unit stacked sequentially; each denoising module may include multiple network layers for feature extraction, conditional fusion, and noise prediction and removal. The denoising modules may be U-Net residual blocks, DiT Transformer Blocks, etc.

[0074] The stitched, noisy image can be used as the main input feature, and together with the nth guiding feature as an additional constraint, it can be input into M denoising modules arranged in series. The M denoising modules will gradually complete the denoising.

[0075] After M denoising modules complete the stepwise denoising, they output the nth intermediate viewpoint image corresponding to the current iteration step. This intermediate viewpoint image retains only the noise matching the current time step n, and is sent to the next iteration step to repeat the above guidance and denoising process until all N iterations are completed, resulting in a complete, clear, and hole-repaired second viewpoint image.

[0076] According to embodiments of this disclosure, by fusing depth features and temporal step embedding features into guiding features, each denoising module can directly capture and fuse deep semantic information, and dynamically match the denoising intensity of different iteration stages in combination with temporal step embedding, thereby improving the spatial consistency of intermediate viewpoint images output by each iteration step of the denoising stage, and generating high-quality second viewpoint images with coherent structure and no tearing holes.

[0077] According to an embodiment of this disclosure, for the m-th denoising module, the following operations are performed, m=1,...,M: the (m-1)-th first intermediate feature and the n-th guiding feature are processed to obtain the m-th first intermediate feature, wherein the (m-1)-th first intermediate feature is the feature output by the (m-1)-th denoising module, and the 0th first intermediate feature is the image after adding noise.

[0078] M denoising modules are connected in series, with the output of the previous denoising module serving as the input of the next. In the first time step, the denoising process of the M denoising modules is as follows: Noisy image → 1st denoising module → 1st first intermediate feature → 2nd denoising module → 2nd first intermediate feature → ... → Mth denoising module → Mth first intermediate feature (i.e., the intermediate viewpoint image). In the next time step, the intermediate viewpoint image is used as the noisy image and input into the M denoising modules, continuing the denoising process until N time steps are completed, resulting in the second viewpoint image.

[0079] The front-end denoising module is mainly responsible for processing high-noise input. It captures the global contour, overall color layout and large-scale structure of the image with a large receptive field, initially corrects the large spatial shift caused by the viewpoint change, eliminates high-intensity coarse noise, and outputs intermediate features with global structure correction and initial stripping of coarse noise.

[0080] The mid-level denoising module primarily focuses on mid-scale details such as object edges and texture blocks, balancing the overall structural integrity and filtering residual medium-intensity noise to optimize the transition between image layers. Each layer combines depth features to gradually reduce the medium-intensity noise left over from the previous layer; each layer outputs more optimized transition features, which are then passed on to the next denoising module.

[0081] The subsequent denoising module targets fine pixel textures, tiny lines, and light and shadow gradients, etc. Based on the deep semantic information in the nth guiding feature, it accurately filters out subtle noise in the image, corrects minor local perspective deviations, and unifies the color and spatial hierarchy of the image; it outputs the nth intermediate viewpoint image corresponding to the current iteration step and flows to the iteration process of the next time step.

[0082] According to embodiments of this disclosure, the M denoising modules follow a progressive logic of first global coarse correction, then mid-level detail optimization, and finally fine micro-texture repair. By using the nth guiding feature as a constraint, denoising is performed layer by layer from level 1 to level M, thereby improving the spatial coherence and image integrity of the intermediate viewpoint image at a single time step.

[0083] The following combination Figures 4a-4d Various feasible implementations of the noise reduction module of the present disclosure will be further described.

[0084] According to embodiments of this disclosure, processing the (m-1)th first intermediate feature and the nth guiding feature to obtain the mth first intermediate feature may include: concatenating the (m-1)th first intermediate feature and the nth guiding feature to obtain the mth second intermediate feature; and performing global self-attention calculation on the mth second intermediate feature to obtain the mth first intermediate feature.

[0085] The noisy image can be segmented and serialized to obtain an image token sequence, which is then used as the 0th first intermediate feature. Simultaneously, depth features and temporal embedding features can be fused into a conditional token sequence, which is used as the nth guiding feature.

[0086] The (m-1)th first intermediate feature and the nth guiding feature can be concatenated along the sequence length dimension to obtain the mth second intermediate feature. This mth second intermediate feature can then be input into an attention network unit, allowing the image token and conditional token to participate in information interaction and perform global attention calculations to obtain the mth first intermediate feature.

[0087] Figure 4a The flowchart illustrating the noise reduction module performing noise reduction according to the first embodiment of the present disclosure is shown.

[0088] like Figure 4a As shown, the (m-1)th first intermediate feature 410 and the nth guiding feature 420 are concatenated to obtain the mth second intermediate feature 430. The mth second intermediate feature 430 is then input into the attention network unit 440, which outputs the mth first intermediate feature 450.

[0089] In one example, the m-th second intermediate feature 430 obtained by splicing can be standardized to remove feature amplitude fluctuations, and then input into the attention network unit 440 for global self-attention calculation.

[0090] In one example, such as Figure 4a As shown, the attention network unit 440 may include a multi-head self-attention subunit 441 and a feedforward network subunit 442. The m-th second intermediate feature 430 can be first input into the multi-head self-attention subunit 441 for global coarse denoising. Then it can be input into the feedforward network subunit 442 for fine-grained denoising of local details.

[0091] The multi-head self-attention subunit 441 can calculate the correlation similarity between each token in the m-th second intermediate feature 430, identify large-scale structures such as global contours of the image, spatial location of objects, and large areas of color blocks, and remove global coarse noise.

[0092] The feedforward network subunit 442 can independently perform non-linear dimensional transformation on each token in the m-th second intermediate feature 403, focusing on the texture, lighting, and minor noise of a single image, and repairing local details and textures that were not processed by self-attention.

[0093] like Figure 4a As shown, the m-th second intermediate feature 430 can first be input into the multi-head self-attention subunit 441 for global coarse denoising; then, the m-th second intermediate feature 430 and the output of the multi-head self-attention subunit 441 are added together and input into the feedforward network subunit 442 for fine-grained denoising of local details. After local detail denoising, the output of the feedforward network subunit 442, the output of the multi-head self-attention subunit 441, and the m-th second intermediate feature can be added together to obtain the m-th first intermediate feature 450. Thus, through the residual jump connection of the self-attention unit and the feedforward network subunit, deep gradient vanishing can be avoided when multiple layers of M modules are stacked, the basic information of the original image color and contour can be continuously preserved, and repeated calculations can be avoided to prevent image blurring and tearing of object structure.

[0094] According to the embodiments of this disclosure, by concatenating intermediate features and guiding features, global attention can be directly calculated. This allows reuse of the multi-head self-attention and feedforward network residual structure of the Transformer encoder to complete the global association and interaction between condition tokens and image tokens without modifying the basic network module architecture, thus reducing additional computational overhead.

[0095] According to embodiments of this disclosure, processing the (m-1)th first intermediate feature and the nth guiding feature to obtain the mth first intermediate feature may further include: performing global self-attention calculation on the (m-1)th first intermediate feature to obtain the mth second intermediate feature; determining a query matrix based on the mth second intermediate feature, and determining a key matrix and a value matrix based on the nth guiding feature; and performing cross-attention fusion on the query matrix, key matrix, and value matrix to obtain the mth first intermediate feature.

[0096] Figure 4b The flowchart illustrating the noise reduction module performing noise reduction according to the second embodiment of the present disclosure is shown.

[0097] like Figure 4b As shown, the (m-1)th first intermediate feature 410 can be input into the multi-head self-attention subunit 431 for global self-attention calculation, thereby completing the denoising of global coarse noise. In one example, the (m-1)th first intermediate feature 410 can be standardized before being input into the multi-head self-attention subunit 431 for global self-attention calculation.

[0098] like Figure 4b As shown, the output of the multi-head attention unit can be added to the (m-1)th first intermediate feature to obtain the mth second intermediate feature, thus preserving the original features of the first intermediate feature.

[0099] The m-th second intermediate feature can be standardized, and the standardized m-th second intermediate feature is used as the query matrix. The query matrix and the n-th guiding feature 420 determine the key matrix and value matrix, which are then input into the cross-attention subunit 433. This allows the image token and the condition token to interact, thereby explicitly injecting the condition token information into the image features, guiding the denoising of noisy images, and achieving targeted correction of occluded, transparent, and other regions.

[0100] like Figure 4b As shown, further, the output of the cross-attention subunit and the m-th second intermediate feature can be added together and then input into the feedforward network subunit 432 to denoise the local details. After denoising the local details, the output of the feedforward network subunit, the feature obtained by adding the output of the cross-attention subunit 433 and the m-th second intermediate feature can be added together again to obtain the m-th first intermediate feature 450.

[0101] According to the embodiments of this disclosure, coarse noise is first removed by performing global self-attention calculation on the (m-1)th first intermediate feature. Then, depth features and temporal guidance conditions are introduced by cross-attention orientation. This does not change the length of the image token sequence, reduces the amount of global attention calculation, and enables on-demand reading of depth and temporal information by image pixels, making the condition constraints more precise and controllable.

[0102] According to embodiments of this disclosure, processing the (m-1)th first intermediate feature and the nth guiding feature to obtain the mth first intermediate feature may further include: normalizing the (m-1)th first intermediate feature according to a first adjustment parameter determined based on the nth guiding feature to obtain the mth second intermediate feature; performing global self-attention calculation on the mth second intermediate feature to obtain the mth third intermediate feature; and normalizing the mth third intermediate feature according to a second adjustment parameter determined based on the nth guiding feature to obtain the mth first intermediate feature.

[0103] The nth guiding feature can be input into the multilayer perceptron to obtain the first and second adjustment parameters. Specifically, the multilayer perceptron has a built-in shared feature extraction layer and two independent first and second linear projection branches. The nth guiding feature is first extracted by the feature extraction layer to obtain shared features. These shared features are then fed into the first and second linear projection branches. The first linear projection branch performs a first linear transformation on the shared features and outputs the first adjustment parameter. The second linear projection branch performs a second linear transformation on the shared features and outputs the second adjustment parameter.

[0104] The first and second adjustment parameters can be obtained during the training of the image generation model. At different time steps, the first and second adjustment parameters for the corresponding time step can be generated based on the guiding features of that time step.

[0105] Figure 4c The flowchart illustrating the noise reduction module performing noise reduction according to the third embodiment of the present disclosure is shown.

[0106] like Figure 4c As shown, the nth guiding feature 420 can be input into the multilayer perceptron 460, and the first adjustment parameter is output through the first linear projection branch of the multilayer perceptron 460. The first adjustment parameter is used to normalize the (m-1)th first intermediate feature 410 to obtain the mth second intermediate feature.

[0107] The first adjustment parameter may include a first scaling parameter and a first offset parameter. The (m-1)th first intermediate feature 410 can be standardized firstly to obtain a standardized (m-1)th first intermediate feature. Then, the standardized (m-1)th first intermediate feature is scaled and normalized sequentially using the first scaling parameter and the first offset parameter, and offset normalized to obtain the m-th second intermediate feature 430.

[0108] like Figure 4cAs shown, the m-th second intermediate feature 430 can be input into the multi-head self-attention sub-unit for global self-attention calculation to obtain the m-th third intermediate feature. Alternatively, the output of the multi-head self-attention unit can be added to the (m-1)-th first intermediate feature to obtain the m-th third intermediate feature.

[0109] The second adjustment parameter may include a second scaling parameter and a second offset parameter. First, the m-th third intermediate feature can be standardized to obtain a standardized m-th third intermediate feature. Then, the standardized m-th third intermediate feature can be scaled and normalized using the second scaling parameter and the second offset parameter, respectively, to obtain the m-th first intermediate feature.

[0110] like Figure 4c As shown, further, the m-th third intermediate feature after scaling and offset normalization can be input into the feedforward network subunit 432 for local detail denoising. After local detail denoising, the output of the feedforward network subunit 432 and the m-th third intermediate feature can be added together to obtain the m-th first intermediate feature.

[0111] According to embodiments of this disclosure, unlike the conditional fusion methods of sequence splicing and cross-attention, by transforming guiding features into adjustment parameters and embedding them into each layer of computation in an adaptive normalized form, there is no need to extend the length of the image token sequence, which greatly reduces the computational overhead of global self-attention.

[0112] According to embodiments of this disclosure, performing global self-attention calculation on the m-th second intermediate feature to obtain the m-th third intermediate feature may include: performing global self-attention calculation on the m-th second intermediate feature to obtain the m-th fourth intermediate feature; and normalizing the m-th fourth intermediate feature according to a third adjustment parameter determined based on the n-th guiding feature to obtain the m-th third intermediate feature.

[0113] Figure 4d The flowchart illustrating the noise reduction module performing noise reduction according to the fourth embodiment of the present disclosure is shown.

[0114] A third linear projection branch can be set in the multilayer perceptron. This branch is used to transform the shared features output by the shared feature extraction module in the multilayer perceptron, resulting in a third adjustment parameter. The third adjustment parameter represents the parameter used to scale and normalize the self-attention output.

[0115] like Figure 4d As shown, the m-th second intermediate feature 430 can be input into the multi-head self-attention subunit 431 to perform global self-attention calculation and obtain the m-th fourth intermediate feature.

[0116] The third adjustment parameter can include a third scaling parameter. The m-th fourth intermediate feature can be scaled and normalized using this third scaling parameter to obtain the scaled m-th fourth intermediate feature. The scaled and normalized m-th fourth intermediate feature is then added to the (m-1)-th first intermediate feature to obtain the m-th third intermediate feature.

[0117] like Figure 4d As shown, a fourth linear projection branch can also be set in the multilayer perceptron. The fourth linear projection branch is used to transform the shared features output by the shared feature extraction module in the multilayer perceptron to obtain the fourth adjustment parameter.

[0118] The fourth adjustment parameter includes the fourth scaling parameter, which is used to scale and normalize the output features of the feedforward network sub-units. The scaled and normalized features are added to the m-th third intermediate feature to obtain the m-th first intermediate feature.

[0119] The third and fourth adjustment parameters can be obtained during the training of the image generation model. At different time steps, the third and fourth adjustment parameters can be generated based on the guiding features of the corresponding time step.

[0120] According to embodiments of this disclosure, a hierarchical and temporally segmented two-level conditional constraint can be achieved through a first adjustment parameter, a second adjustment parameter, a third adjustment parameter, and a fourth adjustment parameter. By setting the first adjustment parameter before the multi-head attention subunit and the second adjustment parameter before the feedforward network subunit, depth features and time-step features can be pre-embedded into normalized features, ensuring that the global self-attention and local feedforward networks are constrained by the guided features throughout the computation. Furthermore, setting the third adjustment parameter after the multi-head attention unit and the fourth adjustment parameter after the feedforward network subunit allows for dynamic adjustment of the residual weights for global spatial correction and local detail restoration, improving the stability of the model denoising process.

[0121] According to embodiments of this disclosure, after processing at the nth time step, the process may further include: performing a quality assessment on the nth intermediate viewpoint image based on an image quality assessment metric to obtain a quality assessment result; and adjusting the number of time steps based on the quality assessment result.

[0122] The first viewpoint image can be used as a reference benchmark, and the peak signal-to-noise ratio (PSNR), structural similarity index Measure (SSIM), and learned perceptual image patch similarity (LPIPS) between the original first viewpoint image and the nth intermediate viewpoint image can be used as complete image quality evaluation indicators.

[0123] In some examples, a minimum inference time step can be preset. When the minimum inference time step is reached at the nth time step, metrics such as PSNR, SSIM, and LPIPS between the nth intermediate viewpoint image and the first viewpoint image are calculated. A comprehensive quality score can be obtained through weighted fusion with preset weights, thus yielding the quality assessment result.

[0124] A threshold for the quality assessment score can be preset. If the overall quality score is greater than the threshold, it indicates that the current denoising iteration has sufficiently removed noise and corrected spatial distortion, so there is no need to execute the remaining time steps completely. The total number of time steps is reduced to the current value of n, and the subsequent iteration process is terminated early. The current nth intermediate viewpoint image is directly output as the final second viewpoint image, reducing redundant computational overhead. If the overall quality score is less than the threshold, it indicates that the image denoising is not complete. The number of additional time steps that need to be added can be estimated based on the remaining time steps and the current quality score, and the time steps can be adjusted accordingly.

[0125] In some embodiments, after denoising is performed at each time step, a quality assessment can be performed on the intermediate viewpoint image. If the quality assessment result does not meet a preset threshold, the time step can be increased until the quality assessment result meets the preset threshold.

[0126] By conducting quality assessments on intermediate viewpoint images and dynamically adjusting the total number of inference time steps based on the assessment results, the number of iteration steps can be reduced and redundant calculations terminated for intermediate images with low distortion and images close to the acceptable standard. For low-quality intermediate images with large-area holes, glass reflection distortion, or severe perspective misalignment, the number of iteration steps can be automatically expanded to repair various image defects through more rounds of noise reduction. This achieves a dynamic balance between inference efficiency and the stereoscopic quality of the generated images.

[0127] Figure 5 A flowchart illustrating a training method for an image generation model according to an embodiment of the present disclosure is shown schematically.

[0128] like Figure 5 As shown, the training method for this image generation model includes operations S510~S540.

[0129] In operation S510, training sample pairs are acquired. The training sample pairs include a first viewpoint image and a second viewpoint image, and the viewpoint directions of the first viewpoint image and the second viewpoint image are different.

[0130] In operation S520, the second viewpoint image of the sample after noise addition is stitched together with the first viewpoint image of the sample to obtain the sample image after noise addition.

[0131] In operation S530, the sample image with added noise and the sample depth features obtained based on the first viewpoint image of the sample are input into the model to be trained. Under the guidance of the sample depth features, the model to be trained gradually removes the noise in the sample image with added noise through N time steps to obtain the output second viewpoint image.

[0132] In operation S540, the parameters of the model to be trained are trained based on the difference between the output second viewpoint image and the sample second viewpoint image to obtain the image generation model.

[0133] Multiple training sample pairs can be obtained from publicly available image or video resource libraries. These training sample pairs can include complex scenes such as regular real-world scenes, transparent glass occlusion, and multi-layered object occlusion.

[0134] Noise can be added to the sample second-viewpoint image using at least one or more of Gaussian noise, Poisson noise, and salt-and-pepper noise in a mixed superposition. Noise intensity parameters such as noise variance and noise amplitude can be randomly set to generate a noise matrix of the same size as the sample second-viewpoint image. The noise matrix is ​​then superimposed pixel-by-pixel on the sample second-viewpoint image to obtain the noise-added sample second-viewpoint image. Different noise intensities can be used in each round of training to improve the model's generalization ability.

[0135] The image channels can be used as the stitching dimension. The first viewpoint image of the sample is used as the first channel group, and the second viewpoint image of the sample after noise processing is used as the second channel group. Channel stitching is then performed to obtain the sample noisy image. The stitching process only stacks the pixel matrix along the channel dimension, without changing the width and height spatial dimensions of the image, thus preserving the complete spatial texture and pixel correspondence of each of the two images.

[0136] For example, if the training samples are three-channel RGB images, the total number of channels in the input image after stitching is 6. The first 3 channels store the RGB pixel information of the first viewpoint image of the sample, and the last 3 channels store the RGB pixel information of the second viewpoint image of the sample after noise addition. If it is a single-channel grayscale image, the stitching will form a 2-channel composite image, where channel 1 is the grayscale data of the first viewpoint image of the sample, and channel 2 is the grayscale data of the second viewpoint image of the sample after noise addition.

[0137] This allows the sample image to simultaneously carry the complete baseline texture of the scene and the noise interference information to be repaired, so as to support the model to be trained to complete the denoising and disparity matching of the sample second viewpoint image by combining the sample first viewpoint image.

[0138] In one example, pixel value range normalization can also be performed on the sample noisy image to map the pixels of the left and right views to the same value range (such as 0 to 1 or 0 to 255), eliminating the network training bias caused by the difference in brightness and value range between the two images, and ensuring that the information from the binocular images participates in feature extraction in a balanced manner.

[0139] The first viewpoint image of the sample can be input into a pre-trained deep backbone model. Multiple feature extraction modules in the deep backbone model can be used to encode the first viewpoint image of the sample at multiple levels to obtain the sample depth features. The sample input features are used as auxiliary inputs to provide 3D depth semantic information of the scene.

[0140] The model to be trained can be set with two input branches, which receive the noisy image of the sample and the sample depth features respectively. Guided by the depth features, the model to be trained gradually removes the noise in the noisy image of the sample through N time steps and obtains the output second viewpoint image.

[0141] In one example, a joint loss can be constructed based on the differences in pixels, structure, and disparity between the output second viewpoint image and the sample second viewpoint image. Specifically, pixel loss can be determined based on the differences in pixel grayscale and color between the sample second viewpoint image and the output second viewpoint image; structure loss can be determined based on the differences in edges, textures, and local structure between the sample second viewpoint image and the output second viewpoint image; and disparity consistency loss can be determined based on the difference between the true disparity determined by the sample first viewpoint image and the sample second viewpoint image and the disparity determined by the sample first viewpoint image and the output second viewpoint image. The joint loss corresponding to this sample iteration is obtained by weighted summing of the pixel loss, structure loss, and disparity consistency loss.

[0142] The parameters of the model to be trained can be updated based on the joint loss, and iterative training can be performed until the joint loss meets the target loss setting, thus obtaining the trained image generation model.

[0143] According to embodiments of this disclosure, by using the noisy sample second viewpoint image and the sample first viewpoint image stitched together as the input, and using the sample depth features of the sample first viewpoint image as guidance, the denoising model can continuously learn the fitting bias of structural distortion and texture misalignment during cross-viewpoint transformation based on the deep semantic information provided by the sample depth features. This improves the accuracy of the model in learning the mapping rules from the first viewpoint image to the second viewpoint image and enhances the robustness of model training.

[0144] According to embodiments of this disclosure, obtaining training sample pairs may include: performing frame extraction processing on a three-dimensional video at a preset frame interval to obtain an initial first viewpoint image and an initial second viewpoint image; and determining a sample first viewpoint image based on the initial first viewpoint image and a sample second viewpoint image based on the initial second viewpoint image to obtain training sample pairs.

[0145] A 3D video sequence containing multi-view stereoscopic images can be selected, and a fixed frame extraction interval parameter can be pre-configured. All video frames of the 3D video are traversed, and one frame image is extracted at every preset frame interval. The initial first viewpoint image of the corresponding reference viewpoint and the initial second viewpoint image of the target generation viewpoint are extracted respectively in the same spatial scene to form a pair of initial image groups, so as to avoid the training sample redundancy caused by the high similarity of consecutive frames.

[0146] Standardized preprocessing operations can be performed on the extracted initial first viewpoint image, including uniform scaling, color gamut normalization, invalid black border cropping, and illumination equalization correction, to obtain a sample first viewpoint image with uniform specifications; at the same time, the paired initial second viewpoint images are preprocessed with the same specifications to complete the alignment of size, color gamut, and image boundaries, to obtain the corresponding sample second viewpoint image.

[0147] The preprocessed and image-specification matched sample first-viewpoint images are paired and bound one-to-one with sample second-viewpoint images. A set of sample first-viewpoint images and a set of sample second-viewpoint images constitute a training sample pair. Multiple sets of sample pairs are summarized to form a complete multi-viewpoint training dataset, which is used for supervised training of the model to be trained later.

[0148] According to embodiments of this disclosure, initial images of dual viewpoints are obtained by extracting frames from 3D video at fixed frame intervals, and then standardized sample images are obtained through unified preprocessing to form training sample pairs. This not only leverages the inherent stereoscopic space and occlusion perspective information of 3D video to enrich the diversity of sample scenes, but also reduces sample data redundancy through frame extraction at intervals. Unified preprocessing eliminates training interference caused by differences in image size, lighting, and boundaries, ensuring that the spatial scenes of each training sample pair correspond one-to-one, and stabilizing the model's learning of cross-viewpoint mapping rules from the first viewpoint to the second viewpoint.

[0149] Figure 6 A flowchart illustrating a training method for an image generation model according to an embodiment of the present disclosure is shown schematically.

[0150] like Figure 6As shown, the sample second viewpoint image 601 is denoised to obtain the denoised sample second viewpoint image 602. The sample first viewpoint image 603 is input into the depth backbone model M604 to extract the depth features of the sample first viewpoint image 603. The denoised sample second viewpoint image 602 is concatenated with the sample first viewpoint image 603 to obtain the sample denoised image. The sample denoised image and the depth features output by the depth backbone model M604 are input into the model to be trained M605, so that the model to be trained, guided by the sample depth features obtained from the sample first viewpoint image, performs denoising on the sample denoised image to obtain the output second viewpoint image 606. Based on the difference 607 between the output second viewpoint image 606 and the sample second viewpoint image 601, the parameters of the model to be trained are trained to obtain the image generation model.

[0151] Figure 7 A flowchart illustrating a display method according to an embodiment of the present disclosure is shown schematically.

[0152] like Figure 7 As shown, the display method includes operation S710.

[0153] In operation S710, a pair of viewpoint images obtained based on the original viewpoint image and the target viewpoint image is displayed.

[0154] The target viewpoint image is obtained by processing the original viewpoint image using the image generation method described above.

[0155] The original viewpoint image corresponds to the first viewpoint image described above, and the target viewpoint image corresponds to the second viewpoint image described above. Viewpoint image pairs derived from the original and target viewpoint images can be displayed on a display device. The display device can be terminal hardware equipped with glasses-free 3D display functionality, used to receive, render, and output viewpoint image pairs for direct human visual observation of the stereoscopic image. This includes devices such as VR headsets, glasses-free 3D advertising screens, glasses-free 3D displays, and polarized 3D display devices.

[0156] A single baseline view, i.e., the original viewpoint image, can be obtained from the same scene through methods such as live-action camera capture and 3D software rendering. The image generation method described above can be executed on a display device to generate a target viewpoint image based on the original viewpoint image, and then the viewpoint image pair can be displayed. Alternatively, the original viewpoint image can be sent to a target electronic device, which can then execute the image generation method described above to generate the target viewpoint image based on the original viewpoint image. Finally, both the original viewpoint image and the target viewpoint image are sent to the display device to display the viewpoint image pair.

[0157] By displaying viewpoint image pairs composed of target viewpoint images obtained by processing the original viewpoint images through the aforementioned image generation method, the binocular view matching effect output by the generation method can be intuitively presented.

[0158] Based on the above image generation method, the present invention also provides an image generation apparatus 800. The following will be combined with... Figure 8 The device is described in detail.

[0159] Figure 8 A block diagram of an image generation apparatus according to an embodiment of the present disclosure is shown schematically.

[0160] like Figure 8 As shown, the image generation device 800 includes a stitching module 810 and a noise reduction module 820.

[0161] The stitching module 810 is used to stitch the noisy image with the first viewpoint image to obtain a noisy image.

[0162] The denoising module 820 is used to input the noisy image and the depth features obtained based on the first viewpoint image into the image generation model. Guided by the depth features, the noise in the noisy image is gradually removed through N time steps to obtain the second viewpoint image. The depth features represent the relative depth between multiple positions in the first viewpoint image in the form of semantic encoding. The viewpoint directions of the first viewpoint image and the second viewpoint image are different, and N is an integer greater than 1.

[0163] Based on the above model training method, the present invention also provides a training device 900 for an image generation model. The following will be combined with... Figure 9 The device is described in detail.

[0164] Figure 9 A block diagram of a training apparatus for an image generation model according to an embodiment of the present disclosure is shown schematically.

[0165] like Figure 9 As shown, the training device for the image generation model includes an acquisition module 910, a sample stitching module 920, a sample denoising module 930, and a sample training module 940.

[0166] The acquisition module 910 is used to acquire training sample pairs, wherein the training sample pairs include a first viewpoint image and a second viewpoint image, and the first viewpoint image and the second viewpoint image have different viewpoint directions.

[0167] The sample stitching module 920 is used to stitch the second viewpoint image of the sample after noise addition with the first viewpoint image of the sample to obtain the sample after noise addition image.

[0168] The sample denoising module 930 is used to input the noisy sample image and the sample depth features obtained based on the first viewpoint image of the sample into the model to be trained, so that the model to be trained can gradually remove the noise in the noisy sample image through N time steps under the guidance of the sample depth features, and obtain the output second viewpoint image.

[0169] The sample training module 940 is used to train the parameters of the model to be trained based on the difference between the output second viewpoint image and the sample second viewpoint image, so as to obtain the image generation model.

[0170] Based on the above-described generation method, the present invention also provides a display device 1000. The following will be combined with... Figure 9 The device is described in detail.

[0171] Figure 10 A block diagram of a display device according to an embodiment of the present disclosure is shown schematically.

[0172] like Figure 10 As shown, the display device 1000 includes a display module 1010.

[0173] The display module is used to display viewpoint image pairs obtained based on the original viewpoint image and the target viewpoint image.

[0174] The target viewpoint image is obtained by processing the original viewpoint image using the image generation method described above.

[0175] Figure 11 A block diagram of an electronic device suitable for implementing an image generation method according to an embodiment of the present disclosure is shown schematically. Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0176] like Figure 11 As shown, a computer electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1109 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0177] RAM 1103 stores various programs and data required for the operation of electronic device 1100. Processor 1101, ROM 1102 and RAM 1103 are interconnected via bus 1104.

[0178] According to embodiments of this disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to a bus 1104. The electronic device 1100 may also include one or more of the following components connected to the input / output (I / O) interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the input / output (I / O) interface 1105 as needed. A removable medium 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1110 as needed so that computer programs read from it can be installed into the storage section 1108 as needed.

[0179] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the image generation method according to the embodiments of this disclosure.

[0180] In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0181] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the object detection method provided in the embodiments of this disclosure.

[0182] When the computer program is executed by the processor 1101, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0183] According to embodiments of this disclosure, program code for executing computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages.

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. It should also be noted that in some alternative implementations, the functions indicated in the boxes may occur in a different order than those shown in the drawings.

[0185] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An image generation method, comprising: The noisy image is stitched together with the first viewpoint image to obtain the noisy image; as well as The noisy image and the depth features obtained based on the first viewpoint image are input into the image generation model. Guided by the depth features, the noise in the noisy image is gradually removed through N time steps to obtain the second viewpoint image, where N is an integer greater than 1. The depth features represent the relative depth between multiple locations in the first viewpoint image in the form of semantic encoding, and the viewpoint directions of the first viewpoint image and the second viewpoint image are different.

2. The method according to claim 1, wherein, The image generation model includes M denoising modules, where M is an integer greater than 1; For the nth time step, n=1, ..., N, guided by the depth features, the noise in the noisy image is gradually removed through N time steps to obtain the second viewpoint image, including: Based on the depth features and the embedding features obtained based on the time step information of the nth time step, the nth guiding feature is determined; and Using M denoising modules, the denoised image and the nth guiding feature are processed to obtain the nth intermediate viewpoint image, wherein the second viewpoint image is the intermediate viewpoint image obtained at the Nth time step.

3. The method according to claim 2, wherein, For the m-th denoising module, perform the following operations, where m = 1, ..., M: The (m-1)th first intermediate feature and the nth guiding feature are processed to obtain the m-th first intermediate feature, wherein the (m-1)th first intermediate feature is the feature output by the (m-1)th denoising module, and the 0th first intermediate feature is the noisy image.

4. The method according to claim 3, wherein, The process of processing the (m-1)th first intermediate feature and the nth guiding feature to obtain the mth first intermediate feature includes: The (m-1)th first intermediate feature and the nth guiding feature are concatenated to obtain the mth second intermediate feature; and global self-attention is calculated on the mth second intermediate feature to obtain the mth first intermediate feature; or... Global self-attention is calculated on the (m-1)th first intermediate feature to obtain the mth second intermediate feature; a query matrix is ​​determined based on the mth second intermediate feature, and a key matrix and a value matrix are determined based on the nth guiding feature; and the query matrix, the key matrix, and the value matrix are fused through cross-attention to obtain the mth first intermediate feature; or... Based on the first adjustment parameter determined based on the nth guiding feature, the (m-1)th first intermediate feature is normalized to obtain the mth second intermediate feature; global self-attention is calculated on the mth second intermediate feature to obtain the mth third intermediate feature; and based on the second adjustment parameter determined based on the nth guiding feature, the mth third intermediate feature is normalized to obtain the mth first intermediate feature.

5. The method according to claim 4, wherein, The step of performing global self-attention calculation on the m-th second intermediate feature to obtain the m-th third intermediate feature includes: Global self-attention is calculated on the m-th second intermediate feature to obtain the m-th fourth intermediate feature; and Based on the third adjustment parameter determined by the nth guiding feature, the mth fourth intermediate feature is normalized to obtain the mth third intermediate feature.

6. The method according to any one of claims 2 to 5, further comprising, after processing at the nth time step: Based on image quality assessment metrics, the quality of the nth intermediate viewpoint image is assessed to obtain the quality assessment result; and The number of time steps is adjusted based on the quality assessment results.

7. The method according to claim 1, wherein, The depth features are obtained through the following methods: The first viewpoint image is input into the depth backbone model, and multiple feature extraction modules in the depth backbone model are used to perform multi-level encoding on the first viewpoint image to obtain the depth features.

8. A training method for an image generation model, comprising: Obtain training sample pairs, wherein the training sample pairs include a first viewpoint image and a second viewpoint image, and the first viewpoint image and the second viewpoint image have different viewpoint directions; The noisy sample second viewpoint image is stitched together with the sample first viewpoint image to obtain the noisy sample image; The noisy image of the sample and the sample depth features obtained based on the first viewpoint image of the sample are input into the model to be trained. Guided by the sample depth features, the model gradually removes noise from the noisy image of the sample through N time steps, resulting in an output second viewpoint image, where N is an integer greater than 1. Based on the difference between the output second viewpoint image and the sample second viewpoint image, the parameters of the model to be trained are trained to obtain the image generation model.

9. The method according to claim 8, wherein, The acquisition of training sample pairs includes: The 3D video is processed by frame extraction according to a preset frame interval to obtain an initial first viewpoint image and an initial second viewpoint image; and The training sample pair is obtained by determining the first viewpoint image of the sample based on the initial first viewpoint image and determining the second viewpoint image of the sample based on the initial second viewpoint image.

10. A display method, comprising: Displays a pair of viewpoint images based on an original viewpoint image and a target viewpoint image, wherein the target viewpoint image is obtained by processing the original viewpoint image using the method described in claims 1 to 7.

11. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.

13. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.

Images