Panoramic streetscape image generation method and system
By constructing data sets and extracting multi-scale coding features, combining conditional attention mechanisms and classifier-free training strategies, panoramic street scene images with real distortions are directly generated, which solves the problem of insufficient consistency in the generation of three-dimensional street scene images in the existing technology, and significantly improves the authenticity of the image and the performance of the autonomous driving system.
Patent Information
- Application Number
- CN202510323125.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-19
- Publication Date
- 2025-04-18
- Estimated Expiration
- 2045-03-19
AI Technical Summary
In the prior art, the three-dimensional street scene image generation is multi-view stitching, resulting in a lack of consistency in key dimensions such as height, distance, and depth, affecting the authenticity of the image and the performance of the autonomous driving system.
By constructing the data set, multi-scale coding features are extracted, and a conditional attention mechanism is used to generate a corrected panoramic image, combined with a training strategy without classifier guidance, a panoramic street scene image with real distortion is directly generated.
The global brightness consistency and seamless connection of panoramic images are achieved, the authenticity and consistency of images are improved, the generation process is simplified, and the computing efficiency is improved, and the perception ability of the autonomous driving system is significantly improved.
Smart Images

Figure SMS_11 
Figure SMS_20 
Figure SMS_24
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a method and system for generating a panoramic street view image. Background Art
[0002] In the field of autonomous driving, especially in the tasks of autonomous driving dataset generation and enhancement, generating high-fidelity three-dimensional street scene images with complex geometric control has become a key requirement. The generation quality of three-dimensional street scene images directly affects the autonomous driving system's understanding of the environment and decision-making capabilities.
[0003] In the related technology, the generation of 3D street view images is mainly achieved by splicing, that is, first generating images from multiple perspectives (such as front view, left view, right view, etc.), then performing 2D annotations on the images from each perspective, such as 2D borders or 2D semantic segmentation, and then splicing the images from multiple perspectives to form a complete 3D street view image. Since 2D annotations cannot effectively capture 3D geometric information, the images generated from different perspectives lack consistency in key dimensions such as height, distance, and depth, resulting in the generated 3D street view images being prone to uneven brightness, discontinuous edge connections, and other problems, which affect the authenticity of the 3D street view images, and thus affect the performance of the autonomous driving system in perception tasks, and also limit the precise positioning of objects in the scene, thereby reducing the autonomous driving system's ability to understand and make decisions about the environment.
[0004] Therefore, it is necessary to provide a method and system for generating a panoramic street view image to solve the above problems. Summary of the invention
[0005] The present invention provides a method and system for generating a panoramic street view image, which directly generates a complete panoramic image with real distortion, rather than generating an unreal distorted panoramic image by splicing multiple perspective images. Through this method, it is possible to generate a high-fidelity panoramic street view image with global brightness consistency and seamless connection and real distortion on the basis of retaining precise geometric control, and the authenticity and consistency of the image are improved, the complexity of multi-perspective generation is reduced, and the applicability and real-time performance in the autonomous driving scenario are significantly improved, which can effectively solve at least one technical problem involved in the background technology.
[0006] In order to solve the above-mentioned technical problems, the present invention is achieved as follows: A method for generating a panoramic street view image comprises the following steps: Step S1, constructing a data set, wherein the data set includes original panoramic images of streets in multiple autonomous driving scenarios and road BEV maps, and annotating scene conditions for each panoramic image and road BEV map, wherein the scene conditions include a 3D target box, a camera pose, and a text description; Step S2, respectively extracting image features of the original panoramic image, BEV map and 3D target frame, geometric features of the camera posture and text features of the text description; fusing the extracted image features with the geometric features and text features after being modulated by distortion noise to generate multi-scale coding features; Step S3, decoding the multi-scale encoded features, embedding the scene condition information into each layer of the decoding process through the conditional attention mechanism, and generating a corrected panoramic image that can accurately reflect the scene conditions; Step S4, optimizing the generated corrected panoramic image based on a training strategy without classifier guidance, randomly discarding some scene conditions during the training process, improving the consistency between the corrected panoramic image and the scene condition information, and avoiding excessive dependence on the scene conditions.
[0007] As a preferred improvement, the autonomous driving scenarios include urban roads, highways and rural roads.
[0008] As a preferred improvement, the following step is further included before step S2: dividing the data set into different categories according to different contents of the text description, wherein: The text description of the first type of data set is related to environmental elements and is used to describe the overall environment in the image; the text description of the second type of data set is related to dynamic targets in the scene; the text description of the third type of data set is related to static targets in the scene.
[0009] As a preferred improvement, the extracted image features are expressed as , where Indicates the number of layers of the extraction network; R represents the real number space; Indicates the height of the feature map; Indicates the width of the feature map; Represents the number of channels; the extracted geometric features are expressed as: ; The extracted text features are expressed as: , where Indicates the number of words in the text description; Represents the embedding dimension of text features.
[0010] As a preferred improvement, in step S2, the distortion noise modulation process of the image feature specifically includes the following steps: adding noise to the image feature through a diffusion model and then removing the noise, introducing distortion noise in the noise adding process to simulate the geometric distortion that may exist in the panoramic image, adjusting the noise characteristics of the distortion noise through a control function to make it conform to the characteristics of the road geometry and the environmental optical distortion, and the distortion noise is adjusted at the time step The formula for generating is: In the formula, Represents the time step Distortion noise generated when Indicates the distortion noise intensity; represents the distortion mapping function; Represents the time step Image feature input at the time; Indicates the intensity of controlling random noise; represents standard Gaussian noise.
[0011] As a preferred improvement, the distortion mapping function is specifically selected as a radial distortion mapping function, which is expressed as: In the formula, Represents the weight coefficient for controlling radial distortion; Represents the normalized radial distance from the pixel to the optical center of the image.
[0012] As a preferred improvement, multi-scale encoding features It is expressed as: In the formula, represents the activation function with batch normalization; Represents element-wise multiplication operation; and Represent the trainable weights of text features and geometric features respectively.
[0013] As a preferred improvement, step S3 specifically includes the following steps: Step S31: Multi-scale encoding features Perform upsampling operation, the upsampling process is expressed as: In the formula, Represents the upsampled feature map; Represents an upsampling operation; Step S32: Construct a noise filter gate to remove noise and supplement missing details to avoid information loss during upsampling. The noise filter gate is implemented through a convolutional layer with batch normalization, which is expressed as: In the formula, Represents the convolution weight of the noise filtering gate; Represents the filtered feature map; Step S33, the missing detail information in the noise filtering process is supplemented by convolution projection operation, and the supplementation process is expressed as: In the formula, It is a convolution projection operation, which is used to supplement the useful information that has been filtered out; represents the decoded features after supplementation; Step S34, embedding the scene condition information into each layer of the decoding process through the conditional attention mechanism to generate a panoramic image that can accurately reflect the scene conditions. The process is expressed as: In the formula, Represents the conditional attention mechanism; Represents text features; Represents the camera posture information; Represents 3D target frame information; Represents the feature map after the conditional attention mechanism; Step S35: restore the multi-scale coding features to the original resolution by stepwise upsampling, and enhance the image details and quality by bilinear interpolation algorithm, and output the final corrected panoramic image. , expressed as: In the formula, Represents a bilinear interpolation upsampling operation. As a preferred improvement, the objective function formed in the training phase of the strategy without classifier guidance is expressed as: In the formula, represents the prediction error of the model; Indicates the real distortion; Represents the weight coefficient of distortion loss; Represents the weight coefficient of the time step.
[0014] A system for executing the above-mentioned panoramic street view image generation method comprises: A data set construction module is used to construct a data set, wherein the data set includes original panoramic images of streets in multiple autonomous driving scenarios and road BEV maps, and annotates scene conditions for each panoramic image and road BEV map, wherein the scene conditions include a 3D target box, a camera pose, and a text description; The multi-scale feature coding network is used to extract the image features of the original panoramic image, BEV map and 3D target box, the geometric features of the camera posture and the text features of the text description respectively; the extracted image features are modulated by distortion noise and fused with the geometric features and text features to generate multi-scale coding features; The multi-scale feature decoding network is used to decode the multi-scale encoded features and embed the scene condition information into each layer of the decoding process through the conditional attention mechanism to generate a corrected panoramic image that can accurately reflect the scene conditions; The optimization module optimizes the generated corrected panoramic image based on a training strategy without classifier guidance. During the training process, some scene conditions are randomly discarded to improve the consistency between the corrected panoramic image and the scene condition information and avoid excessive dependence on the scene conditions.
[0015] The beneficial effects of the present invention are: (1) It abandons the traditional multi-view stitching method and directly generates a complete panoramic image, effectively avoiding the problems of uneven brightness and discontinuous edges between multi-view images in the existing technology. At the same time, it solves the unrealistic distortion problems existing in the existing direct panoramic image generation method, such as excessive edge stretching and depth distortion, thereby significantly improving the overall consistency, realism and visual quality of the generated image; (2) Introducing technology based on dynamic distortion noise control, in the process of adding noise to the diffusion model, by adding radial distortion, a geometric deformation noise, to simulate the geometric distortion characteristics of the real panorama, not only retaining the detailed information in the image, but also generating high-fidelity panoramic images that conform to the real optical distortion, significantly improving the consistency of street view images in edge connection and depth perception, and providing more accurate perception data support for the autonomous driving system. At the same time, the combination of distortion noise and conditional input (such as camera posture, environmental factors) enables the generated image to more realistically reflect the geometric characteristics and visual effects in complex environments; (3) The introduction of multi-scale geometric control technology, combined with the pre-trained diffusion model, can accurately generate geometric details such as road elevation and target object height. This not only improves the authenticity of the image, but also significantly improves the training effect of 3D perception tasks such as BEV map segmentation and 3D object detection, and enhances the performance of the autonomous driving model; (4) By combining multiple conditional inputs such as camera pose, text description, 3D target box, and road BEV map, it provides flexible image generation control capabilities. The text description can include additional conditions such as weather and time, and street view images with different environmental scenes can be generated to meet diverse application needs. (5) Different from existing solutions that rely on multi-view geometric transformation, high-quality panoramic images are generated directly through conditional coding, without relying on cumbersome view transformation or geometric stitching, which simplifies the generation process and improves computational efficiency. DETAILED DESCRIPTION
[0016] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0017] This embodiment provides a method for generating a panoramic street view image, comprising the following steps: Step S1, constructing a data set, wherein the data set includes original panoramic images of streets in multiple autonomous driving scenarios and road BEV maps, and annotating scene conditions for each panoramic image and road BEV map, wherein the scene conditions include a 3D target box, a camera pose, and a text description.
[0018] The autonomous driving scenarios include urban roads, highways and rural roads. According to the different contents of the text description, the data set is divided into different categories, and the dynamic effect and detailed generation effect of the model are improved through round-by-round training.
[0019] The dataset is divided into three categories: The text descriptions in the first category of datasets are related to environmental factors, such as weather, time, street type, etc. This type of text is relatively concise and is used to describe the overall environment in the image; The text descriptions of the second type of dataset are related to dynamic objects in the scene, such as the movement status, position, and interaction of pedestrians and animals with the surrounding environment; The text descriptions of the third category of datasets are related to static objects in the scene, such as architectural style, floor height, building materials and other details.
[0020] Step S2, respectively extracting the image features of the original panoramic image, BEV map and 3D target frame, the geometric features of the camera posture and the text features of the text description; the extracted image features are modulated by distortion noise and then fused with the geometric features and text features to generate multi-scale coding features.
[0021] The extracted image features are expressed as , where Indicates the number of layers of the extraction network; R represents the real number space; Indicates the height of the feature map; Indicates the width of the feature map; Indicates the number of channels. Image features are extracted through multi-scale convolutional neural networks.
[0022] The extracted geometric features are expressed as: The geometric feature extraction process can capture key information such as road elevation, height and position of the target object, and then embed the extracted geometric features into the same dimension as the image features through the geometric mapping function. The geometric features are extracted through Camera Pose Auto-Encoders.
[0023] The extracted text features are expressed as: , where Indicates the number of words in the text description; Represents the embedding dimension of text features. Text features are extracted by text encoders (such as BiGRU or Transformer).
[0024] The distortion noise modulation process of image features specifically includes the following steps: removing noise from image features through a diffusion model, introducing distortion noise during the noise addition process, and simulating radial distortion that may exist in the original panoramic image. The distortion noise is adjusted through a control function to make it conform to the characteristics of road geometry and environmental optical distortion. The distortion noise is modulated at the time step The formula for generating is: In the formula, Represents the time step Distortion noise generated when Indicates the distortion noise intensity; represents the distortion mapping function; Represents the time step Image feature input at the time; Indicates the intensity of controlling random noise; represents standard Gaussian noise.
[0025] In this implementation, the distortion mapping function is specifically selected as a radial distortion mapping function, which is expressed as: In the formula, Represents the weight coefficient for controlling radial distortion; Represents the normalized radial distance from the pixel to the optical center of the image.
[0026] The final multi-scale encoding feature It is expressed as: In the formula, represents the activation function with batch normalization; Represents element-wise multiplication operation; and Represent the trainable weights of text features and geometric features respectively.
[0027] In the noise addition stage of the diffusion model, the distortion noise affects the image generation process through dynamic adjustment, thereby ensuring that the generated image not only meets the geometric consistency of the panorama, but also retains the real optical distortion characteristics. By introducing dynamic distortion noise, the present invention further improves the authenticity and visual consistency of the generated panoramic image on the basis of fusing geometric features and text features, especially the consistency in edge connection and depth perception. At the same time, the distortion perception mechanism significantly enhances the diversity and environmental adaptability of the generated image, making it more valuable in autonomous driving scenarios.
[0028] Step S3, decode the multi-scale encoded features, embed the scene condition information into each layer of the decoding process through the conditional attention mechanism, and generate a corrected panoramic image that can accurately reflect the scene conditions.
[0029] Step S3 specifically includes the following steps: Step S31: Multi-scale encoding features Perform upsampling operation, the upsampling process is expressed as: In the formula, Represents the upsampled feature map; Represents an upsampling operation; Step S32: Construct a noise filter gate to remove noise and supplement missing details to avoid information loss during upsampling. The noise filter gate is implemented through a convolutional layer with batch normalization, which is expressed as: In the formula, Represents the convolution weight of the noise filtering gate; Represents the feature map after noise filtering; Step S33, the missing detail information in the noise filtering process is supplemented by convolution projection operation, and the supplementation process is expressed as: In the formula, It is a convolution projection operation, which is used to supplement the useful information that has been filtered out; Represents the decoded features after supplementation.
[0030] Step S34, embedding the scene condition information into each layer of the decoding process through the conditional attention mechanism to generate a panoramic image that can accurately reflect the scene conditions. The process is expressed as: In the formula, Represents the conditional attention mechanism; Represents the camera posture information; Represents 3D target frame information; Represents the feature map after the conditional attention mechanism; The text description, camera pose information and 3D target frame information are used to conditionally constrain the image to ensure the consistency of the generated panoramic image in the geometric and semantic dimensions, so that the generated image can accurately reflect the road elevation, object height and other scene condition information in a specific scene.
[0031] Step S35: restore the multi-scale coding features to the original resolution by stepwise upsampling, and enhance the image details and quality by bilinear interpolation algorithm, and output the final corrected panoramic image. , expressed as: In the formula, Represents a bilinear interpolation upsampling operation. After decoding, high-fidelity and detail-rich panoramic images can be obtained, avoiding the shortcomings of traditional multi-view stitching methods and effectively improving the quality and consistency of the generated images.
[0032] Step S4, based on the training strategy without classifier guidance, the generated corrected panoramic image Optimize and randomly discard some scene conditions during training to improve the corrected panoramic image Consistency with scene condition information to avoid over-dependence on scene conditions.
[0033] The strategy without classifier guidance is based on the existing diffusion model. In the training phase, randomness is introduced to increase the generalization ability of the model. The resulting objective function is expressed as: In the formula, represents the prediction error of the model; Indicates the real distortion; Represents the weight coefficient of distortion loss; Represents the weight coefficient of the time step.
[0034] The key to classifier-free guidance is to introduce "null" placeholders for missing scene conditions during the inference phase, so that the model can still maintain the generation effect when facing the lack of real data, thereby achieving consistency in panoramic street view images. Through this random discarding strategy, the situation where the model is overly dependent on certain specific scene conditions is effectively solved. Especially in the scenario of autonomous driving, this classifier-free training strategy can improve the robustness and applicability of the model under different environmental conditions, thereby improving the global consistency of the generated panoramic street view images under multiple perspectives.
[0035] The present invention directly generates a complete panoramic street view image with real distortion through multiple condition inputs such as road BEV map, 3D target frame, camera posture and text description, combined with a diffusion model with distortion noise, without the need for the traditional multi-view image stitching process, thus avoiding the problems of brightness differences between viewpoints and incoherent edge connections. During the encoding process, the road geometry and text description information are extracted simultaneously, and through precise geometric control, accurate generation of geometric details such as road elevation and target object height is achieved, thereby significantly improving the global consistency and visual quality of the generated image. In addition, the text description can include additional conditions such as weather and time, so that the generated street view image has higher adaptability and realism in different environments, meeting the needs of complex application scenarios.
[0036] In order to further improve the training effect of the system in the generation of panoramic street view images, the present invention also proposes a training strategy based on non-classifier guidance. In this strategy, some scene-level conditions (such as camera posture, text description, etc.) are randomly discarded during the training process. By introducing a non-classifier guidance mechanism, the model can still generate high-quality street view images without specific conditions. This method effectively avoids the model's excessive dependence on certain conditions and increases the generalization ability of the model. In the inference stage, the system introduces "null" placeholders for missing conditions, and adjusts the generation process through the conditional denoising network to ensure that the generated images still have consistency and high-quality performance. Through this strategy, the present invention not only improves the global consistency of street view images, but also reduces the impact of noise interference on image generation quality during training, so that the system can adapt to the requirements of more scenarios.
[0037] This embodiment also provides a panoramic street view image generation system, including: A data set construction module is used to construct a data set, wherein the data set includes original panoramic images of streets in multiple autonomous driving scenarios and road BEV maps, and annotates scene conditions for each panoramic image and road BEV map, wherein the scene conditions include a 3D target box, a camera pose, and a text description; The multi-scale feature coding network is used to extract the image features of the original panoramic image, BEV map and 3D target box, the geometric features of the camera posture and the text features of the text description respectively; the extracted image features are modulated by distortion noise and fused with the geometric features and text features to generate multi-scale coding features; The multi-scale feature decoding network is used to decode the multi-scale encoded features and embed the scene condition information into each layer of the decoding process through the conditional attention mechanism to generate a corrected panoramic image that can accurately reflect the scene conditions.
[0038] The optimization module optimizes the generated corrected panoramic image based on a training strategy without classifier guidance. During the training process, some scene conditions are randomly discarded to improve the consistency between the corrected panoramic image and the scene condition information and avoid excessive dependence on the scene conditions.
[0039] The embodiments of the present invention are described above, but the present invention is not limited to the above-mentioned specific implementation modes. The above-mentioned specific implementation modes are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, all of which are within the protection of the present invention.
Claims
1. A method for generating a panoramic street view image, characterized in that: The steps include: Step S1, constructing a data set, wherein the data set includes original panoramic images of streets in multiple autonomous driving scenarios and road BEV maps, and annotating scene conditions for each panoramic image and road BEV map, wherein the scene conditions include a 3D target box, a camera pose, and a text description; Step S2, respectively extracting image features of the original panoramic image, BEV map and 3D target frame, geometric features of the camera posture and text features of the text description; fusing the extracted image features with the geometric features and text features after being modulated by distortion noise to generate multi-scale coding features; Step S3, decoding the multi-scale encoded features, embedding the scene condition information into each layer of the decoding process through the conditional attention mechanism, and generating a corrected panoramic image that can accurately reflect the scene conditions; Step S4, optimizing the generated corrected panoramic image based on a training strategy without classifier guidance, randomly discarding some scene conditions during the training process, improving the consistency between the corrected panoramic image and the scene condition information, and avoiding excessive dependence on the scene conditions.
2. The method for generating a panoramic street view image according to claim 1, characterized in that: The autonomous driving scenarios include urban roads, highways and rural roads.
3. The method for generating a panoramic street view image according to claim 1, characterized in that: Before step S2, the following steps are also included: dividing the data set into different categories according to different contents of the text description, wherein: The text description of the first type of data set is related to environmental elements and is used to describe the overall environment in the image; the text description of the second type of data set is related to dynamic targets in the scene; the text description of the third type of data set is related to static targets in the scene.
4. The method for generating a panoramic street view image according to claim 1, characterized in that: The extracted image features are expressed as , where Indicates the number of layers of the extraction network; R represents the real number space; Indicates the height of the feature map; Indicates the width of the feature map; Represents the number of channels; the extracted geometric features are expressed as: ; The extracted text features are expressed as: , where Indicates the number of words in the text description; Represents the embedding dimension of text features.
5. The method for generating a panoramic street view image according to claim 4, characterized in that: In step S2, the distortion noise modulation process of the image feature specifically includes the following steps: adding noise to the image feature through the diffusion model and then removing the noise, introducing distortion noise in the noise adding process to simulate the geometric distortion that may exist in the panoramic image, and adjusting the noise characteristics of the distortion noise through the control function to make it conform to the characteristics of the road geometry and the environmental optical distortion. The distortion noise is adjusted at the time step The formula for generating is: In the formula, Represents the time step Distortion noise generated when Indicates the distortion noise intensity; represents the distortion mapping function; Represents the time step Image feature input at the time; Indicates the intensity of controlling random noise; represents standard Gaussian noise.
6. The method for generating a panoramic street view image according to claim 5, characterized in that: The distortion mapping function is specifically selected as a radial distortion mapping function, which is expressed as: In the formula, Represents the weight coefficient for controlling radial distortion; Represents the normalized radial distance from the pixel to the optical center of the image.
7. The method for generating a panoramic street view image according to claim 6, characterized in that: Multi-scale encoding features It is expressed as: In the formula, represents the activation function with batch normalization; Represents element-wise multiplication operation; and Represent the trainable weights of text features and geometric features respectively.
8. The method for generating a panoramic street view image according to claim 7, characterized in that: Step S3 specifically includes the following steps: Step S31: Multi-scale encoding features Perform upsampling operation, the upsampling process is expressed as: In the formula, Represents the upsampled feature map; Represents an upsampling operation; Step S32: Construct a noise filter gate to remove noise and supplement missing details to avoid information loss during upsampling. The noise filter gate is implemented through a convolutional layer with batch normalization, which is expressed as: In the formula, Represents the convolution weight of the noise filtering gate; Represents the filtered feature map; Step S33, the missing detail information in the noise filtering process is supplemented by convolution projection operation, and the supplementation process is expressed as: In the formula, It is a convolution projection operation, which is used to supplement the useful information that has been filtered out; represents the decoded features after supplementation; Step S34, embedding the scene condition information into each layer of the decoding process through the conditional attention mechanism to generate a panoramic image that can accurately reflect the scene conditions. The process is expressed as: In the formula, Represents the conditional attention mechanism; Represents text features; Represents the camera posture information; Represents 3D target frame information; Represents the feature map after the conditional attention mechanism; Step S35: restore the multi-scale coding features to the original resolution by stepwise upsampling, and enhance the image details and quality by bilinear interpolation algorithm, and output the final corrected panoramic image. , expressed as: In the formula, Represents a bilinear interpolation upsampling operation.
9. The method for generating a panoramic street view image according to claim 8, characterized in that: The objective function formed in the training phase of the strategy without classifier guidance is expressed as: In the formula, represents the prediction error of the model; Indicates the real distortion; Represents the weight coefficient of distortion loss; Represents the weight coefficient of the time step.
10. A system for executing the method for generating a panoramic street view image according to any one of claims 1 to 9, characterized in that: include: A data set construction module is used to construct a data set, wherein the data set includes original panoramic images of streets in multiple autonomous driving scenarios and road BEV maps, and annotates scene conditions for each panoramic image and road BEV map, wherein the scene conditions include a 3D target box, a camera pose, and a text description; The multi-scale feature coding network is used to extract the image features of the original panoramic image, BEV map and 3D target box, the geometric features of the camera posture and the text features of the text description respectively; the extracted image features are modulated by distortion noise and fused with the geometric features and text features to generate multi-scale coding features; The multi-scale feature decoding network is used to decode the multi-scale encoded features and embed the scene condition information into each layer of the decoding process through the conditional attention mechanism to generate a corrected panoramic image that can accurately reflect the scene conditions; The optimization module optimizes the generated corrected panoramic image based on a training strategy without classifier guidance. During the training process, some scene conditions are randomly discarded to improve the consistency between the corrected panoramic image and the scene condition information and avoid excessive dependence on the scene conditions.
Citation Information
Patent Citations
Street view map production method, control server, unmanned aerial vehicle and production device
CN105022405A
Around view image splicing method and system based on improved distortion correction and mark splicing
CN113362228A
Multi-source information fusion method and device, electronic equipment and storage medium
CN113378605A
Panoramic image splicing method based on deformable attention mechanism
CN116245714A
Fusion-based object tracker using lidar point cloud and surrounding cameras for autonomous vehicles
US20230334673A1
Cited By
Panoramic picture generation method and device based on structured prompt, equipment and medium
CN120852576A
Method, apparatus and medium for generating panoramic picture based on structured prompt
CN120852576B
