Image generation method, electronic equipment and computer readable storage medium

Abstract

The invention discloses an image generation method, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring an original view angle image, an original view angle point cloud corresponding to the original view angle image, and a target view angle; constructing a three-dimensional scene graph based on the original view angle image and the original view angle point cloud, and projecting the three-dimensional scene graph to a target view angle to obtain a first point cloud projection image under the target view angle; and inputting the original view angle image and the first point cloud projection image into an image generation model for geometric projection hole completion and / or occlusion completion, and outputting a target image under the target view angle. In this way, geometric consistency and uniformity of image quality are improved.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and in particular to image generation methods, electronic devices, and computer-readable storage media. Background Technology

[0002] Current intelligent driver assistance systems (ADAS) heavily rely on the collaboration of multi-view perception data and radar data in several core modules, including perception, environmental mapping, and behavior prediction. This data is typically collected synchronously by multiple onboard cameras and radar systems to cover a panoramic view of the vehicle's surroundings. However, due to issues such as occlusion, blind spots, and sensor position differences in the real environment, relying solely on information from existing observation perspectives is insufficient to fully reconstruct the complete scene, thus limiting the system's robustness and generalization capabilities in complex traffic environments.

[0003] To compensate for perception blind spots and enhance scene representation capabilities, researchers have proposed NovelView Synthesis (NVS) technology, which aims to infer and synthesize image information from unobserved perspectives from images of one or more original viewpoints. This technology is significant for tasks in autonomous driving such as scene completion, occluded region prediction, cross-view semantic consistency analysis, and multi-camera fusion, effectively improving the system's understanding and prediction capabilities of the 3D environment. However, current NovelView Synthesis methods all have some limitations. Summary of the Invention

[0004] This application provides an image generation method, an electronic device, and a computer-readable storage medium that can improve the uniformity of geometric consistency and image quality.

[0005] To address the aforementioned technical problems, this application provides an image generation method comprising: acquiring an original viewpoint image, an original viewpoint point cloud corresponding to the original viewpoint image, and a target viewpoint; constructing a three-dimensional scene map based on the original viewpoint image and the original viewpoint point cloud, and projecting the three-dimensional scene map onto the target viewpoint to obtain a first point cloud projection image under the target viewpoint; inputting the original viewpoint image and the first point cloud projection image into an image generation model for geometric projection hole completion and / or occlusion completion, and outputting a target image under the target viewpoint.

[0006] Before inputting the original viewpoint image and the first point cloud projection image into the image generation model for geometric projection hole completion and / or occlusion completion, and outputting the target image under the target viewpoint, the process includes: adding random noise to the first point cloud projection image to obtain the second point cloud projection image, and using the second point cloud projection image as the first point cloud projection image.

[0007] The image generation model includes an encoder, a hybrid attention network, a decoder, and a sampler. The original viewpoint image and a first point cloud projection image are input into the image generation model for geometric projection hole completion and / or occlusion completion, outputting a target image from the target viewpoint. This includes: inputting the original viewpoint image and the first point cloud projection image into the encoder to obtain a first encoded feature corresponding to the original viewpoint image and a second encoded feature corresponding to the first point cloud projection image; inputting the first and second encoded features into the hybrid attention network for geometric projection hole completion and / or occlusion completion to obtain a third encoded feature corresponding to the first encoded feature and a fourth encoded feature corresponding to the second encoded feature; inputting the third and fourth encoded features into the decoder so that the decoder combines the hierarchical encoded features provided by the encoder to obtain a first decoded feature and a second decoded feature; and inputting the first and second decoded features into the sampler to obtain the target image from the target viewpoint and the original viewpoint image.

[0008] The hybrid attention network includes a first Transformer layer, a fusion layer, and a second Transformer layer. The first and second encoded features are input into the hybrid attention network for geometric projection hole completion and / or occlusion completion to obtain a third encoded feature corresponding to the first encoded feature and a fourth encoded feature corresponding to the second encoded feature. This includes: inputting the first and second encoded features into the first Transformer layer to obtain a first image feature corresponding to the first encoded feature and a second image feature corresponding to the second encoded feature; inputting the first and second image features into the fusion layer for concatenation to obtain a third image feature; inputting the third image feature into the second Transformer layer to obtain a fourth image feature; and separating and reshaping the fourth image feature to obtain the third encoded feature corresponding to the first encoded feature and the fourth encoded feature corresponding to the second encoded feature.

[0009] The encoder is a variational autoencoder, the decoder is a variational autodecoder, and the first Transformer layer consists of a Vision Transformer.

[0010] The process of constructing a 3D scene map based on the original viewpoint image and the original viewpoint point cloud includes: matching and coloring the pixels in the original viewpoint image and the original viewpoint point cloud to construct the 3D scene map.

[0011] The process involves matching and coloring pixels in the original viewpoint image with the original viewpoint point cloud to construct a 3D scene map. This includes: performing a dilation operation on the 3D points in the original viewpoint point cloud; projecting the dilated 3D points onto the original viewpoint image to obtain the pixels and pixel values ​​corresponding to each 3D point, thereby constructing the 3D scene map; wherein the 3D scene map contains texture information.

[0012] Obtaining the target perspective includes: calculating the target perspective based on changes in vehicle trajectory, camera pose, or adjustments to camera intrinsic and extrinsic parameters.

[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method provided by any of the above technical solutions.

[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing a computer program, which, when executed by a processor, is used to implement the method provided by any of the above technical solutions.

[0015] The image generation method, electronic device, and computer-readable storage medium provided in this application acquire an original viewpoint image, the corresponding original viewpoint point cloud, and a target viewpoint; construct a 3D scene map based on the original viewpoint image and the original viewpoint point cloud, and project the 3D scene map onto the target viewpoint to obtain a first point cloud projection image under the target viewpoint; input the original viewpoint image and the first point cloud projection image into an image generation model for geometric projection hole completion and / or occlusion completion, and output the target image under the target viewpoint. This method integrates the advantages of geometric projection and image generation model learning generation. Through the collaborative mechanism of geometric projection structure guidance and image generation model perception completion, it achieves joint modeling of image content and geometric structure, realizes the structured generation of new viewpoint images in complex urban traffic scenes, improves the uniformity of geometric consistency and image quality, and effectively restores texture and reconstructs content in occluded areas by geometric projection structure guidance and learning generation when there are occlusions, perception blind spots, or incomplete multi-view information, thereby improving the system's ability to identify hidden elements in dynamic traffic environments. The method of this application is applicable to simulating the viewpoint shift caused by changes in vehicle trajectory, camera intrinsic and extrinsic parameters, and sensor redeployment during autonomous driving. It ensures that the generated images meet the requirements of the actual system in terms of visual quality and geometric consistency, thereby effectively supporting the accurate implementation of multi-view data fusion and mapping perception tasks in autonomous driving systems. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1This is a flowchart illustrating an embodiment of the image generation method provided in this application; Figure 2 This is a schematic flowchart of another embodiment of the image generation method provided in this application; Figure 3 This is a schematic flowchart of another embodiment of the image generation method provided in this application; Figure 4 yes Figure 3 A flowchart illustrating an embodiment of step 34; Figure 5 This is a schematic flowchart of another embodiment of the image generation method provided in this application; Figure 6 yes Figure 5 A flowchart illustrating an embodiment of step 52; Figure 7 This is a schematic diagram illustrating an application scenario of the image generation method provided in this application; Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application; Figure 9 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0018] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0019] Current intelligent driver assistance systems (ADAS) heavily rely on the collaboration of multi-view perception data and radar data in several core modules, including perception, environmental mapping, and behavior prediction. This data is typically collected synchronously by multiple onboard cameras and radar systems to cover a panoramic view of the vehicle's surroundings. However, due to issues such as occlusion, blind spots, and sensor position differences in the real environment, relying solely on information from existing observation perspectives is insufficient to fully reconstruct the complete scene, thus limiting the system's robustness and generalization capabilities in complex traffic environments.

[0020] To compensate for perception blind spots and enhance scene representation capabilities, researchers have proposed NovelView Synthesis (NVS) technology, which aims to infer and synthesize image information from unobserved perspectives from images of one or more original viewpoints. This technology is significant for tasks in autonomous driving such as scene completion, occlusion region prediction, cross-view semantic consistency analysis, and multi-camera fusion, effectively improving the system's understanding and prediction capabilities of the 3D environment. However, current NovelView Synthesis methods all have some limitations, failing to address the issue of pairing novel view image data with radar data.

[0021] Currently, image new perspective synthesis methods can be broadly classified into two categories: geometry-based view synthesis methods and learning-based generative methods.

[0022] On the one hand, geometry-based view synthesis methods mainly rely on dense point clouds or high-quality depth information, utilizing extrinsic and intrinsic parameters of the camera for geometric reprojection of the image. These methods offer good structural fidelity and are particularly suitable for tasks requiring alignment with LiDAR point clouds, such as 3D sensing or high-precision mapping. However, they have extremely high requirements for the accuracy and registration quality of the depth data, resulting in poor robustness. When the input data is sparse or measurement errors exist, the synthesized image often exhibits severe geometric distortion or holes.

[0023] On the other hand, deep learning-based image generation methods (such as NeRF and 3D Gaussian Splatting) implicitly model the color and density distribution of a scene from an image using neural networks, thereby generating images from new perspectives. While these methods possess strong texture synthesis and perspective generalization capabilities, and can generate high-quality, continuously varying images to a certain extent, they often suffer from structural inconsistencies and geometric offsets due to overfitting of the original perspective image. This results in the generated image not matching the real point cloud. In autonomous driving scenarios, geometric consistency is not only crucial for image quality but also directly impacts the accuracy of downstream tasks.

[0024] Based on this, this application proposes a method that integrates the advantages of geometric projection and image generation model learning. This method involves acquiring the original viewpoint image, the corresponding point cloud, and the target viewpoint; constructing a 3D scene map based on the original viewpoint image and point cloud, and projecting the 3D scene map onto the target viewpoint to obtain the first point cloud projection image under the target viewpoint; inputting the original viewpoint image and the first point cloud projection image into an image generation model for geometric projection hole completion and / or occlusion completion, and outputting the target image under the target viewpoint. Through the collaborative mechanism of structural guidance from geometric projection and perceptual completion from the image generation model, joint modeling of image content and geometric structure is achieved. This enables structured generation of new viewpoint images in complex urban traffic scenes, improving the uniformity of geometric consistency and image quality. Furthermore, in cases of occlusion, perceptual blind spots, or incomplete multi-view information, the image generation model effectively restores texture and reconstructs content in occluded areas through structural guidance and learning generation of geometric projection, enhancing the system's ability to identify hidden elements in dynamic traffic environments. The method of this application is applicable to simulating viewpoint shifts caused by changes in vehicle trajectory, variations in camera intrinsic and extrinsic parameters, and sensor redeployment during autonomous driving. It ensures that the generated images meet the requirements of real-world systems in terms of visual quality and geometric consistency, thereby effectively supporting the accurate implementation of multi-view data fusion and mapping perception tasks in autonomous driving systems. See any of the following embodiments for details.

[0025] See Figure 1 , Figure 1 This is a schematic flowchart of an embodiment of the image generation method provided in this application. The method includes: Step 11: Obtain the original view image, the original view point cloud corresponding to the original view image, and the target view.

[0026] In some embodiments, this application can be applied to driving simulation scenarios and to actual driving scenarios.

[0027] In some embodiments, the original viewpoint image can be acquired using a camera with a fixed viewpoint, and the original viewpoint point cloud can be acquired using a device such as a lidar with a fixed viewpoint.

[0028] In some embodiments, the target viewing angle can be calculated based on changes in vehicle trajectory, camera pose, or adjustments to camera intrinsic and extrinsic parameters. That is, the target viewing angle is a different viewpoint from the original viewing angle.

[0029] Step 12: Construct a 3D scene map based on the original viewpoint image and the original viewpoint point cloud, and project the 3D scene map onto the target viewpoint to obtain the first point cloud projection image under the target viewpoint.

[0030] In some embodiments, based on the original viewpoint image and the original viewpoint point cloud, the point cloud is matched, registered, and textured to construct a 3D scene map with texture information.

[0031] In some embodiments, matching, registration, and texture mapping need to be performed between the original view image and the original view point cloud.

[0032] Matching, registering, and mapping point clouds with images requires multiple steps, mainly involving feature extraction, spatial transformation calculation, and color information mapping. The specific methods are as follows: First, feature extraction and matching operations are performed.

[0033] Feature point extraction mainly involves selecting key points (such as corner points and edge points) in the point cloud and calculating their geometric features such as normal vectors and curvature; and extracting feature points (such as SIFT and SURF features) from corresponding locations in the image. Correspondence establishment: By matching image and point cloud feature point pairs using feature similarity (such as Euclidean distance and angular similarity), a preliminary correspondence is determined. Then spatial transformation calculations are performed, such as coarse registration and fine registration.

[0034] Coarse registration: Based on the initial correspondence, the initial values ​​of the rotation matrix and translation vector are quickly estimated using the 4PC algorithm (finding four coplanar points) or RANSAC (random sample consensus). Fine registration: Using the coarse registration result as the initial value, the transformation parameters are optimized through ICP (Iterative Closest Point) or NDT (Normal Distance Transform) to maximize the overlap between the two sets of data. Then, texture mapping is performed, including color filling and global optimization.

[0035] Color filling: Based on the registered correspondence, the RGB information of the image is mapped to the registered point cloud to generate a colored textured point cloud. Global optimization: By iteratively adjusting the transformation parameters and combining data distribution and local features, the color mapping effect is further optimized to ensure spatial location and color consistency.

[0036] In some embodiments, texture mapping, also known as texture mapping, is a technique in computer graphics that maps two-dimensional texture images onto the surface of three-dimensional objects, primarily used to enhance the visual realism of three-dimensional graphics. Its core process involves establishing a correspondence between points on the object's surface and points in texture space, projecting bitmap or raster images onto the surface of a geometric model, encompassing various types such as diffuse maps, normal maps, and specular maps.

[0037] In some embodiments, it is assumed that the point cloud coordinates under the original view are The camera pose under the original lane view is If other lanes deviate, we can assume the deviation pose increment is... Then the camera pose under the new perspective (target perspective) can be calculated as follows: Finally, the point cloud is projected onto the camera with the new perspective, resulting in a point cloud projection image from the new perspective: , where K is the camera's intrinsic parameter.

[0038] Step 13: Input the original view image and the first point cloud projection image into the image generation model to perform geometric projection hole completion and / or occlusion completion, and output the target image under the target view.

[0039] The image generation model can be trained using real images from the target's perspective and images from the original perspective. The trained image generation model can then predict the target image from the target's perspective based on the original perspective image and the first point cloud projection image.

[0040] The final generated target image from the target's perspective has the ability to be geometrically aligned with the original LiDAR point cloud (original viewpoint point cloud) in terms of structure, and can be used for cross-modal fusion between the image and the point cloud.

[0041] It is understandable that the target image from the target's perspective is obtained from the projection image of the first point cloud, which in turn is obtained from the original lidar point cloud (original viewpoint point cloud). Based on this, the target image generated from the target's perspective has the ability to be geometrically aligned with the original lidar point cloud (original viewpoint point cloud) in terms of structure.

[0042] In this embodiment, the method involves acquiring the original viewpoint image, the corresponding point cloud, and the target viewpoint; constructing a 3D scene map based on the original viewpoint image and the point cloud, and projecting the 3D scene map onto the target viewpoint to obtain the first point cloud projection image under the target viewpoint; inputting the original viewpoint image and the first point cloud projection image into the image generation model for geometric projection hole completion and / or occlusion completion, and outputting the target image under the target viewpoint. This method integrates the advantages of geometric projection and image generation model learning generation. Through the collaborative mechanism of structural guidance of geometric projection and perceptual completion of image generation model, joint modeling of image content and geometric structure is achieved, enabling structured generation of new viewpoint images in complex urban traffic scenes, improving the uniformity of geometric consistency and image quality, and effectively restoring texture and reconstructing content in occluded areas by using the structural guidance and learning generation method of geometric projection when there are occlusions, perceptual blind spots, or incomplete multi-view information. This enhances the system's ability to identify hidden elements in dynamic traffic environments. The method of this application is applicable to simulating the viewpoint shift caused by changes in vehicle trajectory, camera intrinsic and extrinsic parameters, and sensor redeployment during autonomous driving. It ensures that the generated images meet the requirements of the actual system in terms of visual quality and geometric consistency, thereby effectively supporting the accurate implementation of multi-view data fusion and mapping perception tasks in autonomous driving systems.

[0043] See Figure 2 , Figure 2 This is a schematic flowchart of another embodiment of the image generation method provided in this application. The method includes: Step 21: Obtain the original view image, the original view point cloud corresponding to the original view image, and the target view.

[0044] Step 22: Construct a 3D scene map based on the original viewpoint image and the original viewpoint point cloud, and project the 3D scene map onto the target viewpoint to obtain the first point cloud projection image under the target viewpoint.

[0045] In some embodiments, steps 21 to 22 have the same or similar technical solutions as any embodiment of this application, and will not be described in detail here.

[0046] Step 23: Add random noise to the first point cloud projection image to obtain the second point cloud projection image, and use the second point cloud projection image as the first point cloud projection image.

[0047] In some embodiments, adding random noise to the first point cloud projection image can enhance the robustness of the image generation model.

[0048] In some embodiments, adding random noise to the first point cloud projection image is equivalent to introducing random noise perturbation into the second point cloud projection image from the target viewpoint, thereby enhancing the robustness of the model.

[0049] Step 24: Input the original view image and the first point cloud projection image into the image generation model to perform geometric projection hole completion and / or occlusion completion, and output the target image under the target view.

[0050] In some embodiments, since the second point cloud projection image is used as the first point cloud projection image in step 23, the first point cloud projection image in step 24 is essentially the second point cloud projection image mentioned in step 23.

[0051] In some embodiments, step 24 has the same or similar technical solution as any embodiment of this application, and will not be described in detail here.

[0052] In this embodiment, the following steps are taken: acquiring the original viewpoint image, the corresponding point cloud of the original viewpoint image, and the target viewpoint; constructing a 3D scene map based on the original viewpoint image and the original viewpoint point cloud, and projecting the 3D scene map onto the target viewpoint to obtain the first point cloud projection image under the target viewpoint; adding random noise to the first point cloud projection image to obtain the second point cloud projection image, and using the second point cloud projection image as the first point cloud projection image; inputting the original viewpoint image and the first point cloud projection image into the image generation model for geometric projection hole completion and / or occlusion completion, and outputting the target image under the target viewpoint. This approach combines the advantages of geometric projection and image generation model learning generation. Through the collaborative mechanism of the structural guidance of geometric projection and the perceptual completion of the image generation model, joint modeling of image content and geometric structure is achieved, enabling the structured generation of new viewpoint images in complex urban traffic scenes, improving the uniformity of geometric consistency and image quality, and effectively restoring texture and reconstructing content in occluded areas by using the structural guidance and learning generation method of geometric projection when there are occlusions, perceptual blind spots, or incomplete multi-view information, thereby improving the system's ability to identify hidden elements in dynamic traffic environments. The method of this application is applicable to simulating the viewpoint shift caused by changes in vehicle trajectory, camera intrinsic and extrinsic parameters, and sensor redeployment during autonomous driving. It ensures that the generated images meet the requirements of the actual system in terms of visual quality and geometric consistency, thereby effectively supporting the accurate implementation of multi-view data fusion and mapping perception tasks in autonomous driving systems.

[0053] See Figure 3 , Figure 3 This is a schematic flowchart of another embodiment of the image generation method provided in this application. The method includes: Step 31: Obtain the original view image, the original view point cloud corresponding to the original view image, and the target view.

[0054] Step 32: Construct a 3D scene map based on the original viewpoint image and the original viewpoint point cloud, and project the 3D scene map onto the target viewpoint to obtain the first point cloud projection image under the target viewpoint.

[0055] In some embodiments, steps 31 to 32 have the same or similar technical solutions as any embodiment of this application, and will not be described in detail here.

[0056] In some embodiments, random noise can be added to the first point cloud projection image to obtain a second point cloud projection image, and the second point cloud projection image can be used as the first point cloud projection image.

[0057] In this embodiment, the image generation model includes an encoder, a hybrid attention network, a decoder, and a sampler.

[0058] Step 33: Input the original view image and the first point cloud projection image into the encoder to obtain the first coded feature corresponding to the original view image and the second coded feature corresponding to the first point cloud projection image.

[0059] In some embodiments, the encoder is a variational autoencoder.

[0060] A Variational Autoencoder (VAE) is a powerful deep learning model, a deep generative model based on an autoencoder structure. An autoencoder is a neural network consisting of an encoder and a decoder. The encoder compresses the input data into a low-dimensional representation, the latent variable, extracting core information from complex data; the decoder then transforms the latent variable back into the original input. The goal is for the encoder and decoder networks to learn a compact representation of the data by minimizing the reconstruction error between the input and output through training.

[0061] Variational autoencoders model latent variables by introducing a probability model, so that the latent variables are no longer a fixed value, but a probability distribution. In this way, variational autoencoders can not only compress and reconstruct data, but also generate new data samples similar to the training data.

[0062] In some embodiments, the original view image and the first point cloud projection image are input to the encoder, which can process the original view image and the first point cloud projection image respectively to obtain the first coded feature corresponding to the original view image and the second coded feature corresponding to the first point cloud projection image.

[0063] Step 34: Input the first and second coding features into the hybrid attention network to perform geometric projection hole completion and / or occlusion completion, to obtain the third coding feature corresponding to the first coding feature and the fourth coding feature corresponding to the second coding feature.

[0064] In some embodiments, the hybrid attention network can utilize a multi-image attention completion mechanism to learn from the first encoded features, thereby filling in holes or occluded areas in the first point cloud projection image caused by projection. The hybrid attention network is used to enhance geometric consistency and texture detail reconstruction capabilities.

[0065] In some embodiments, the first and second encoded features are input into a hybrid attention network and processed separately to obtain a first image feature corresponding to the first encoded feature and a second image feature corresponding to the second encoded feature. Then, the first and second image features are concatenated to obtain a third image feature. The third image feature is then processed to obtain a fourth image feature. Finally, the fourth image feature is input into a separation and reshaping network to obtain a third encoded feature corresponding to the first encoded feature and a fourth encoded feature corresponding to the second encoded feature.

[0066] Perform geometric projection hole completion and / or occlusion completion In some embodiments, the hybrid attention network includes: a first Transformer layer, a fusion layer, and a second Transformer layer, see below. Figure 4 Step 34 can be the following process: Step 341: Input the first encoding feature and the second encoding feature into the first Transformer layer to obtain the first image feature corresponding to the first encoding feature and the second image feature corresponding to the second encoding feature.

[0067] In some embodiments, a first encoded feature and a second encoded feature are input to a first Transformer layer, so that the first Transformer layer processes the first encoded feature and the second encoded feature respectively. This results in a first image feature corresponding to the first encoded feature and a second image feature corresponding to the second encoded feature.

[0068] In some embodiments, the first Transformer layer consists of a Vision Transformer.

[0069] Vision Transformer (ViT) is an innovative model that applies the Transformer architecture to computer vision tasks, learning global information and long-range dependencies in images through a self-attention mechanism.

[0070] Core Architecture ViT segments images into fixed-size image blocks (visual tokens), converts them into sequences through linear mapping, and adds positional encoding to preserve spatial information. The core of the model is a Transformer encoder, incorporating a multi-head attention mechanism and a feedforward neural network, enabling parallel processing of the entire image sequence.

[0071] The model has the following characteristics: Global dependency: Self-attention mechanisms can capture long-distance dependencies between different regions of an image, improving feature extraction efficiency.

[0072] Data dependency: Best performance requires pre-training on large datasets (such as JFT-300M), but its performance on open-source datasets such as ImageNet is still close to that of convolutional neural networks.

[0073] Scalability: Supports large-scale parameter training (over 10 billion parameters), showing potential in fields such as generative modeling.

[0074] Step 342: Input the first image features and the second image features into the fusion layer for splicing to obtain the third image features.

[0075] Step 343: Input the third image feature into the second Transformer layer to obtain the fourth image feature.

[0076] In some embodiments, the second Transformer layer may have multiple cascaded Transformers. The attention mechanism in the multiple cascaded Transformers is used to further perform geometric projection hole completion and / or occlusion completion on the third image features.

[0077] In some embodiments, the first image feature may correspond to information such as batch size, feature length, and feature dimension. The second image feature may also correspond to information such as batch size, feature length, and feature dimension. Based on this, the stitched third image feature corresponds to information such as batch size, twice the feature length, and feature dimension. For example, the first image feature X is shaped as (B, L, D), the second image feature Y is shaped as (B, L, D), and the third image feature is shaped as (B, 2L, D). Where B represents the batch size, L represents the feature length, and D represents the feature dimension.

[0078] Step 344: Separate and reshape the fourth image feature to obtain the third coded feature corresponding to the first coded feature and the fourth coded feature corresponding to the second coded feature.

[0079] In some embodiments, since the fourth image feature includes features from both the original viewpoint image and the first point cloud projection image, the fourth image feature is separated and reconstructed to obtain the third coded feature corresponding to the first coded feature and the fourth coded feature corresponding to the second coded feature. In this case, the third coded feature corresponds to the original viewpoint image.

[0080] In some embodiments, the fourth image feature is obtained from the third image feature, which is obtained by concatenating the first and second image features. Therefore, the fourth image feature is also equivalent to a concatenated image feature (vector). Thus, it is necessary to separate and reshape the fourth image feature. For example, the fourth image feature can be bisected (e.g., divided equally) to obtain a first part feature and a second part feature. The first part feature is the third encoded feature, and the second part is the fourth encoded feature.

[0081] Step 35: Input the third and fourth encoded features into the decoder so that the decoder combines the hierarchical encoded features provided by the encoder to obtain the first and second decoded features.

[0082] In some embodiments, the decoder is a variational self-decoder. The decoder is used to reconstruct the third coded feature, the fourth coded feature, and the hierarchical coded feature to obtain the first decoded feature and the second decoded feature.

[0083] In some embodiments, the encoder consists of several coding layers. Each coding layer can generate a hierarchical coding feature. Similarly, the decoder consists of several decoding layers. Based on this, the coding layers of the encoder and the decoding layers of the decoder are connected one by one according to a hierarchical relationship. For example, the encoder consists of a first coding layer, a second coding layer, and a third coding layer. The decoder consists of a first decoding layer, a second decoding layer, and a third decoding layer. Then, the output of the first coding layer can be connected to the input of the first decoding layer, the output of the second coding layer can be connected to the input of the second decoding layer, and the output of the third coding layer can be connected to the input of the third decoding layer. In this way, the hierarchical coding features output by the coding layers are provided to the corresponding decoding layers, so that the decoding layers can use the hierarchical coding features to reconstruct the third coding feature to obtain the first decoded feature, and reconstruct the fourth coding feature to obtain the second decoded feature.

[0084] In other embodiments, the specific number of encoding layers of the encoder can be set according to actual conditions, such as 4 encoding layers, 6 encoding layers, 10 encoding layers, 15 encoding layers, etc. The specific number of decoding layers of the decoder can be set according to actual conditions, such as 4 decoding layers, 6 decoding layers, 10 decoding layers, 15 decoding layers, etc.

[0085] In other embodiments, the encoder's partial encoding layers and the decoder's partial decoding layers can be connected one-to-one according to a hierarchical relationship. Alternatively, all encoding layers of the encoder and all decoding layers of the decoder can be connected one-to-one according to a hierarchical relationship.

[0086] In some embodiments, the third and fourth coded features are input to the decoder. The decoder reconstructs the third coded feature using the hierarchical coded features corresponding to the original viewpoint image to obtain the first decoded feature. The decoder also reconstructs the fourth coded feature using the hierarchical coded features corresponding to the first point cloud projection image to obtain the second decoded feature.

[0087] Step 36: Input the first and second decoded features into the sampler to obtain the target image and the original view image from the target's perspective.

[0088] In some embodiments, a first decoding feature and a second decoding feature are input to a sampler, so that the sampler samples the first decoding feature to obtain the original viewpoint image and samples the second decoding feature to obtain the target image from the target viewpoint.

[0089] In this embodiment, the original viewpoint image, the corresponding point cloud of the original viewpoint image, and the target viewpoint are acquired. A three-dimensional scene map is constructed based on the original viewpoint image and the original viewpoint point cloud, and the three-dimensional scene map is projected onto the target viewpoint to obtain the first point cloud projection image under the target viewpoint. The original viewpoint image and the first point cloud projection image are input into the image generation model. The encoder, hybrid attention network, decoder, and sampler in the image generation model are used to perform geometric projection hole completion and / or occlusion completion, and the target image under the target viewpoint is output. This method integrates the advantages of geometric projection and image generation model learning generation. Through the collaborative mechanism of geometric projection structure guidance and image generation model perception completion, joint modeling of image content and geometric structure is achieved. This enables the structured generation of new viewpoint images in complex urban traffic scenes, improves the uniformity of geometric consistency and image quality, and effectively restores texture and reconstructs content in occluded areas by geometric projection structure guidance and learning generation when there are occlusions, perception blind spots, or incomplete multi-view information. This improves the system's ability to identify hidden elements in dynamic traffic environments. The method of this application is applicable to simulating the viewpoint shift caused by changes in vehicle trajectory, camera intrinsic and extrinsic parameters, and sensor redeployment during autonomous driving. It ensures that the generated images meet the requirements of the actual system in terms of visual quality and geometric consistency, thereby effectively supporting the accurate implementation of multi-view data fusion and mapping perception tasks in autonomous driving systems.

[0090] See Figure 5 , Figure 5 This is a schematic flowchart of another embodiment of the image generation method provided in this application. The method includes: Step 51: Obtain the original view image, the original view point cloud corresponding to the original view image, and the target view.

[0091] Step 52: Match and color the pixels in the original view image with the original view point cloud to construct a 3D scene map, and project the 3D scene map onto the target view to obtain the first point cloud projection image under the target view.

[0092] In some embodiments, steps 51 to 52 have the same or similar technical solutions as any embodiment of this application, and will not be described in detail here.

[0093] In some embodiments, see Figure 6 Step 52 can be the following process: Step 521: Perform a dilation operation on the 3D points in the original viewpoint point cloud.

[0094] Step 522: Project the 3D points after the dilation operation onto the original viewpoint image to obtain the pixels and pixel values ​​corresponding to each 3D point, thereby constructing a 3D scene map; wherein, the 3D scene map contains texture information.

[0095] In some embodiments, pixels in the original viewpoint image and the original viewpoint point cloud are matched and colored. Since the point cloud is colorless and sparse, matching and coloring of the point cloud and image are necessary. Specifically, for each frame of scanned point cloud and image, the point cloud needs to be inflated into a small sphere. Assuming the small sphere, when projected onto the image plane, occupies one pixel radius, the radius of the inflated sphere for each point cloud is calculated. The specific calculation formula is as follows: assuming the coordinates of the 3D point are (x, y, z), the inflated radius is... The camera's intrinsic focal length is The following formula is given: .

[0096] By inflating the point cloud into small spheres before projecting it onto the image, misprojection and incorrect coloring of the point cloud due to its sparsity are reduced. For example, the point cloud may be so sparse that it projects points that should be occluded.

[0097] Through the above operations, all point clouds were successfully colored, and a sparse point cloud 3D structure map (3D scene map) was constructed.

[0098] Step 53: Input the original view image and the first point cloud projection image into the image generation model to perform geometric projection hole completion and / or occlusion completion, and output the target image under the target view.

[0099] In some embodiments, step 53 has the same or similar technical solution as any embodiment of this application, and will not be described in detail here.

[0100] In this embodiment, the method involves acquiring the original viewpoint image, the corresponding point cloud, and the target viewpoint; constructing a 3D scene map based on the original viewpoint image and the point cloud, and projecting the 3D scene map onto the target viewpoint to obtain the first point cloud projection image under the target viewpoint; inputting the original viewpoint image and the first point cloud projection image into the image generation model for geometric projection hole completion and / or occlusion completion, and outputting the target image under the target viewpoint. This method integrates the advantages of geometric projection and image generation model learning generation. Through the collaborative mechanism of structural guidance of geometric projection and perceptual completion of image generation model, joint modeling of image content and geometric structure is achieved, enabling structured generation of new viewpoint images in complex urban traffic scenes, improving the uniformity of geometric consistency and image quality, and effectively restoring texture and reconstructing content in occluded areas by using the structural guidance and learning generation method of geometric projection when there are occlusions, perceptual blind spots, or incomplete multi-view information. This enhances the system's ability to identify hidden elements in dynamic traffic environments. The method of this application is applicable to simulating the viewpoint shift caused by changes in vehicle trajectory, camera intrinsic and extrinsic parameters, and sensor redeployment during autonomous driving. It ensures that the generated images meet the requirements of the actual system in terms of visual quality and geometric consistency, thereby effectively supporting the accurate implementation of multi-view data fusion and mapping perception tasks in autonomous driving systems.

[0101] In some embodiments, this application can be applied to image generation scenarios during new perspective shifts in autonomous driving scenarios. Combined with Figure 7 The process is as follows: 1. First, input the original camera view image (original view image) as the initial data source, and then perform point cloud matching and coloring operations to construct a textured 3D scene geometry (3D scene graph).

[0102] 2. Next, lane-level spatial transformation or new perspective transformation is performed. The point cloud is projected onto the target perspective plane through geometric reprojection to simulate the perspective change map generated after the vehicle crosses lanes or the camera position changes, and a new perspective point cloud projection image (first point cloud projection image) is obtained.

[0103] 3. Overlay a random noise image onto the point cloud projection image from this new perspective.

[0104] 4. Input the image with noise superimposed on the original viewpoint image and the new viewpoint point cloud projection image into the variational autoencoder for encoding. The variational autoencoder compresses the image features and generates latent variables containing scene structure and texture information.

[0105] 5. Inject the features (latent variables) into the hybrid attention network to perform feature fusion. By fusing features, the geometric projection holes and occluded parts under the new perspective are filled in, and the complete high-definition high-dimensional features of the new perspective image are obtained.

[0106] 6. The optimized features (high-dimensional features of the new perspective image) are jointly reconstructed into high-dimensional image latent variables by variational autoencoder and decoder. The sampler then restores the high-dimensional image latent variables into a high-definition image, including the view after lane offset (target image) and the original perspective view.

[0107] In this application, the geometric consistency of rigid objects such as vehicles and road topology is ensured under the structural constraints of geometric projection, including cross-view geometric consistency between the new and original viewpoints. Simultaneously, visual blind spots are repaired through generative completion technology (image generation model). This method, through a collaborative mechanism of geometric projection and neural network (image generation model) generation, solves problems such as geometric offset, geometric distortion, and texture distortion during viewpoint shifts in autonomous driving scenarios while maintaining real-time performance.

[0108] See Figure 8 , Figure 8 This is a schematic diagram of an embodiment of the electronic device provided in this application. The electronic device 80 includes a processor 81 and a memory 82; wherein the memory 82 is used to store a computer program, and the processor 81 is used to execute the computer program to implement the following method: Acquire the original view image, the corresponding point cloud of the original view image, and the target view; construct a 3D scene map based on the original view image and the original view point cloud, and project the 3D scene map onto the target view to obtain the first point cloud projection image under the target view; input the original view image and the first point cloud projection image into the image generation model to perform geometric projection hole completion and / or occlusion completion, and output the target image under the target view.

[0109] In some embodiments, before inputting the original viewpoint image and the first point cloud projection image into the image generation model for geometric projection hole completion and / or occlusion completion, and outputting the target image under the target viewpoint, the processor 81 is further configured to execute a computer program to implement the following method: adding random noise to the first point cloud projection image to obtain a second point cloud projection image, and using the second point cloud projection image as the first point cloud projection image.

[0110] In some embodiments, the image generation model includes: an encoder, a hybrid attention network, a decoder, and a sampler; the processor 81 is further configured to execute a computer program to implement the following method: inputting the original viewpoint image and the first point cloud projection image into the encoder to obtain a first coded feature corresponding to the original viewpoint image and a second coded feature corresponding to the first point cloud projection image; inputting the first coded feature and the second coded feature into the hybrid attention network to perform geometric projection hole completion and / or occlusion completion to obtain a third coded feature corresponding to the first coded feature and a fourth coded feature corresponding to the second coded feature; inputting the first coded feature, the second coded feature, the third coded feature, and the fourth coded feature into the decoder to obtain a first decoded feature and a second decoded feature; inputting the first decoded feature and the second decoded feature into the sampler to obtain a target image under the target viewpoint and the original viewpoint image.

[0111] In some embodiments, the hybrid attention network includes: a first Transformer layer, a fusion layer, a second Transformer layer, and a separation and reshaping layer; the processor 81 is further configured to execute a computer program to implement the following method: inputting a first encoded feature and a second encoded feature into the first Transformer layer to obtain a first image feature corresponding to the first encoded feature and a second image feature corresponding to the second encoded feature; inputting the first image feature and the second image feature into the fusion layer for concatenation to obtain a third image feature; inputting the third image feature into the second Transformer layer to obtain a fourth image feature; inputting the fourth image feature into the separation and reshaping layer to obtain a third encoded feature corresponding to the first encoded feature and a fourth encoded feature corresponding to the second encoded feature.

[0112] In some embodiments, the encoder is a variational autoencoder, the decoder is a variational autodecoder, and the first Transformer layer consists of a Vision Transformer.

[0113] In some embodiments, the processor 81 is further configured to execute a computer program to implement the following method: matching and coloring pixels in the original viewpoint image and the original viewpoint point cloud to construct a three-dimensional scene map.

[0114] In some embodiments, the processor 81 is further configured to execute a computer program to implement the following method: performing a dilation operation on three-dimensional points in the original viewpoint point cloud; projecting the dilated three-dimensional points onto the original viewpoint image to obtain the pixel and pixel value corresponding to each three-dimensional point, thereby constructing a three-dimensional scene map; wherein the three-dimensional scene map contains texture information.

[0115] In some embodiments, the processor 81 is further configured to execute a computer program to implement the method of calculating the target viewpoint based on changes in vehicle trajectory, camera pose, or adjustments to camera intrinsic and extrinsic parameters.

[0116] In some embodiments, the processor 81 is further configured to execute a computer program to implement the method of any embodiment of this application.

[0117] See Figure 9 , Figure 9 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. The computer-readable storage medium 90 is used to store a computer program 91, which, when executed by a processor, implements the following method: Acquire the original view image, the corresponding point cloud of the original view image, and the target view; construct a 3D scene map based on the original view image and the original view point cloud, and project the 3D scene map onto the target view to obtain the first point cloud projection image under the target view; input the original view image and the first point cloud projection image into the image generation model to perform geometric projection hole completion and / or occlusion completion, and output the target image under the target view.

[0118] In some embodiments, before inputting the original viewpoint image and the first point cloud projection image into the image generation model for geometric projection hole completion and / or occlusion completion, and outputting the target image under the target viewpoint, the computer program 91, when executed by the processor, is also used to implement the following method: adding random noise to the first point cloud projection image to obtain a second point cloud projection image, and using the second point cloud projection image as the first point cloud projection image.

[0119] In some embodiments, the image generation model includes: an encoder, a hybrid attention network, a decoder, and a sampler; when executed by a processor, the computer program 91 is further configured to implement the following method: inputting the original viewpoint image and the first point cloud projection image into the encoder to obtain a first coded feature corresponding to the original viewpoint image and a second coded feature corresponding to the first point cloud projection image; inputting the first coded feature and the second coded feature into the hybrid attention network for geometric projection hole completion and / or occlusion completion to obtain a third coded feature corresponding to the first coded feature and a fourth coded feature corresponding to the second coded feature; inputting the third coded feature and the fourth coded feature into the decoder so that the decoder combines the hierarchical coded features provided by the encoder to obtain a first decoded feature and a second decoded feature; inputting the first decoded feature and the second decoded feature into the sampler to obtain a target image and an original viewpoint image under the target viewpoint.

[0120] In some embodiments, the hybrid attention network includes: a first Transformer layer, a fusion layer, and a second Transformer layer; when executed by a processor, the computer program 91 is further configured to implement the following method: inputting a first coded feature and a second coded feature into the first Transformer layer to obtain a first image feature corresponding to the first coded feature and a second image feature corresponding to the second coded feature; inputting the first image feature and the second image feature into the fusion layer for concatenation to obtain a third image feature; inputting the third image feature into the second Transformer layer to obtain a fourth image feature; and separating and reshaping the fourth image feature to obtain a third coded feature corresponding to the first coded feature and a fourth coded feature corresponding to the second coded feature.

[0121] In some embodiments, the encoder is a variational autoencoder, the decoder is a variational autodecoder, and the first Transformer layer consists of a Vision Transformer.

[0122] In some embodiments, when the computer program 91 is executed by the processor, it is also used to implement the following method: matching and coloring pixels in the original viewpoint image and the original viewpoint point cloud to construct a three-dimensional scene map.

[0123] In some embodiments, when the computer program 91 is executed by the processor, it is further configured to implement the following method: performing a dilation operation on the three-dimensional points in the original viewpoint point cloud; projecting the dilated three-dimensional points onto the original viewpoint image to obtain the pixel and pixel value corresponding to each three-dimensional point, thereby constructing a three-dimensional scene map; wherein the three-dimensional scene map contains texture information.

[0124] In some embodiments, when executed by a processor, the computer program 91 is also used to implement the following method: calculating the target viewing angle based on changes in vehicle trajectory, camera pose, or adjustments to camera intrinsic and extrinsic parameters.

[0125] In some embodiments, when executed by a processor, computer program 91 is also used to implement the method of any embodiment of this application.

[0126] In summary, the image generation method, electronic device, and computer-readable storage medium provided in this application acquire an original viewpoint image, the corresponding original viewpoint point cloud, and a target viewpoint; construct a 3D scene map based on the original viewpoint image and the original viewpoint point cloud, and project the 3D scene map onto the target viewpoint to obtain a first point cloud projection image under the target viewpoint; input the original viewpoint image and the first point cloud projection image into an image generation model for geometric projection hole completion and / or occlusion completion, and output the target image under the target viewpoint. This method integrates the advantages of geometric projection and image generation model learning generation. Through the collaborative mechanism of structural guidance of geometric projection and perceptual completion of image generation model, it achieves joint modeling of image content and geometric structure, realizes the structured generation of new viewpoint images in complex urban traffic scenes, improves the uniformity of geometric consistency and image quality, and effectively restores texture and reconstructs content in occluded areas by using the structural guidance and learning generation method of geometric projection when there are occlusions, perceptual blind spots, or incomplete multi-view information, thereby improving the system's ability to identify hidden elements in dynamic traffic environments. The method of this application is applicable to simulating the viewpoint shift caused by changes in vehicle trajectory, camera intrinsic and extrinsic parameters, and sensor redeployment during autonomous driving. It ensures that the generated images meet the requirements of the actual system in terms of visual quality and geometric consistency, thereby effectively supporting the accurate implementation of multi-view data fusion and mapping perception tasks in autonomous driving systems.

[0127] Furthermore, this application also has the following advantages: 1. It combines the structural fidelity of traditional geometric reconstruction with the texture synthesis capabilities of deep learning methods. By working together with explicit geometric modeling (such as reprojection based on sparse or dense point clouds) and a neural network-driven image completion module, it achieves the structured generation of new perspective images in complex urban traffic scenes, improving the consistency of geometric uniformity and image quality.

[0128] 2. Particularly suitable for mild to moderate viewpoint shifts caused by minor changes in vehicle trajectory, camera installation position adjustments, or calibration parameter updates in autonomous driving scenarios. Compared to fixed-view or full-view modeling methods, this method has higher adaptability and generalization ability, and can quickly synthesize high-fidelity viewpoint images that match the current vehicle state while maintaining real-time performance. For example, it is suitable for simulating viewpoint changes caused by mild to moderate vehicle deviations, sensor replacements, or recalibrations during autonomous driving, demonstrating strong adaptability and high generalization ability.

[0129] 3. In the presence of occlusion, perception blind spots, or incomplete multi-view information, the hybrid generation mechanism effectively restores texture and reconstructs content in occluded areas through a combination of structural priors and learning-based generation, thereby enhancing the system's ability to identify hidden elements in dynamic traffic environments.

[0130] 4. The design takes into account the data alignment requirements between multimodal sensors, including images and point clouds. By using explicit modeling that preserves geometric information, the generated images can be used for data fusion with LiDAR and other sensors, avoiding the geometric distortion problems in traditional NeRF-based or image generation models, and enhancing the consistency and reliability of cross-modal sensing and spatiotemporal mapping.

[0131] 5. The image generation model adopts an end-to-end architecture, avoiding the cumbersome two-step approach of mainstream generation algorithms: training the noise estimation module and then iteratively sampling. The image generation model only requires one sampling step to directly generate the result. Compared to mainstream generation models, the image generation model achieves an inference speed of 5 FPS on the A100 graphics card, representing a speed improvement of over 100 times.

[0132] Furthermore, the entire new perspective image generation process is an end-to-end inference process, requiring only one sampling step to complete image generation, which improves inference efficiency by more than 100 times compared to the two-step diffusion model.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of circuits or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image generation method, characterized in that, The method includes: Acquire the original view image, the original view point cloud corresponding to the original view image, and the target view; A 3D scene map is constructed based on the original viewpoint image and the original viewpoint point cloud, and the 3D scene map is projected onto the target viewpoint to obtain the first point cloud projection image under the target viewpoint. The original view image and the first point cloud projection image are input into the image generation model to perform geometric projection hole completion and / or occlusion completion, and the target image under the target view is output.

2. The image generation method according to claim 1, characterized in that, Before inputting the original viewpoint image and the first point cloud projection image into the image generation model for geometric projection hole completion and / or occlusion completion, and outputting the target image under the target viewpoint, the process includes: Random noise is added to the first point cloud projection image to obtain a second point cloud projection image, and the second point cloud projection image is used as the first point cloud projection image.

3. The image generation method according to claim 1 or 2, characterized in that, The image generation model includes: an encoder, a hybrid attention network, a decoder, and a sampler; The step of inputting the original viewpoint image and the first point cloud projection image into an image generation model for geometric projection hole completion and / or occlusion completion, and outputting the target image under the target viewpoint, includes: The original view image and the first point cloud projection image are input into the encoder to obtain the first coded feature corresponding to the original view image and the second coded feature corresponding to the first point cloud projection image. The first and second encoded features are input into the hybrid attention network for geometric projection hole completion and / or occlusion completion to obtain the third encoded feature corresponding to the first encoded feature and the fourth encoded feature corresponding to the second encoded feature. The third and fourth coding features are input to the decoder so that the decoder combines the hierarchical coding features provided by the encoder to obtain the first and second decoding features. The first decoding feature and the second decoding feature are input into the sampler to obtain the target image from the target viewpoint and the original viewpoint image.

4. The image generation method according to claim 3, characterized in that, The hybrid attention network includes: a first Transformer layer, a fusion layer, and a second Transformer layer; the step of inputting the first encoded feature and the second encoded feature into the hybrid attention network for geometric projection hole completion and / or occlusion completion to obtain a third encoded feature corresponding to the first encoded feature and a fourth encoded feature corresponding to the second encoded feature includes: The first encoded feature and the second encoded feature are input into the first Transformer layer to obtain the first image feature corresponding to the first encoded feature and the second image feature corresponding to the second encoded feature; The first image feature and the second image feature are input into the fusion layer and stitched together to obtain the third image feature; The third image feature is input into the second Transformer layer to obtain the fourth image feature; The fourth image feature is separated and reconstructed to obtain the third coded feature corresponding to the first coded feature and the fourth coded feature corresponding to the second coded feature.

5. The image generation method according to claim 4, characterized in that, The encoder is a variational autoencoder, the decoder is a variational autodecoder, and the first Transformer layer is composed of Vision Transformers.

6. The image generation method according to claim 1, characterized in that, The construction of a 3D scene map based on the original viewpoint image and the original viewpoint point cloud includes: The pixels in the original view image and the original view point cloud are matched and colored to construct the three-dimensional scene map.

7. The image generation method according to claim 6, characterized in that, The process of matching and coloring pixels in the original viewpoint image and the original viewpoint point cloud to construct the 3D scene map includes: Perform a dilation operation on the three-dimensional points in the original viewpoint point cloud; The three-dimensional points after the dilation operation are projected onto the original viewpoint image to obtain the pixels and pixel values ​​corresponding to each three-dimensional point, thereby constructing the three-dimensional scene map; wherein, the three-dimensional scene map contains texture information.

8. The image generation method according to claim 1, characterized in that, The acquisition of the target viewpoint includes: The target viewing angle is calculated based on changes in vehicle trajectory, camera pose, or adjustments to camera intrinsic and extrinsic parameters.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed by a processor, is used to implement the method as described in any one of claims 1-8.

Images