Severe weather scene simulation method for automatic driving
By using semantic space and visual feature vector generators in autonomous driving scenario simulation, the problems of distortion in severe weather scenarios and difficulties in data acquisition are solved, high-fidelity severe weather images are generated, the test scenario coverage is enriched, and the cost is reduced.
Patent Information
- Application Number
- CN202511878864.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-03-20
AI Technical Summary
Existing autonomous driving scenario simulation methods are prone to distorting key objects when generating severe weather conditions, resulting in limited realism. Furthermore, data acquisition is difficult and expensive, leading to poor simulation effects and high costs.
By establishing a semantic space and visual feature vectors, a pre-trained zero-shot weather generalization generator is used to generate simulated images with superimposed severe weather effects. The images are encoded using CLIP Text and Image Encoder, and combined with a diffusion model and perceptual consistency loss to ensure the semantic consistency and realism of the generated images.
It enables the generation of high-fidelity severe weather scenarios without real data, enriches the test coverage, reduces simulation costs and improves efficiency, and avoids dangerous data acquisition processes.
Smart Images
Figure CN121706566A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of autonomous driving scenario simulation, and more specifically, to a method for simulating severe weather scenarios for autonomous driving. Background Technology
[0002] The development and testing of autonomous driving systems heavily rely on simulation environments. The core value of simulation lies in its ability to safely and efficiently simulate various extreme and dangerous scenarios, especially severe weather conditions such as heavy fog, torrential rain, sandstorms, and hail. Traditional methods for generating severe weather require paired data of "clear image - specific weather image" to learn the rendering model, with weather effects coupled to scene content during training. However, this method has the following drawbacks in severe weather scene generation: 1. It easily leads to distortion of key objects in the scene when generating new weather, such as rendering vehicle taillights as fog lights. 2. Traditional simulation-rendered weather effects are based on manually defined rules and physical models, resulting in limited realism and significant differences from weather images collected in the real world. 3. For many extreme weather conditions, a large number of high-quality real sensors (cameras, LiDAR) are needed for data collection, a process that is extremely difficult, expensive, and dangerous. This leads to a lack of training data to build or calibrate the simulator's weather rendering model. Therefore, how to achieve high-fidelity simulation of unknown and unexperienced severe weather scenarios in autonomous driving simulation systems without real data is of great significance. Summary of the Invention
[0003] This invention provides a method for simulating severe weather scenarios for autonomous driving, which solves the problems of high data acquisition costs, incomplete data, and poor simulation results in existing vehicle autonomous driving scenario simulations. It can realize rare severe weather combinations, greatly enrich the coverage of test scenarios, improve simulation testing efficiency, and reduce simulation costs.
[0004] To achieve the above objectives, the present invention provides the following technical solution:
[0005] A method for simulating severe weather scenarios for autonomous driving, comprising:
[0006] Establish a semantic space to receive clear images of simulated scene scenarios and text descriptions of severe weather conditions.
[0007] Set a text encoder to encode the text description into a corresponding semantic feature vector;
[0008] Configure the image encoder to encode clear images of the simulated scene into visual feature vectors;
[0009] Using a pre-trained zero-shot weather generalization generator, a generalized simulated image with superimposed target severe weather effects is generated based on the semantic feature vector and the visual feature vector.
[0010] Preferably, the establishment of the semantic space includes:
[0011] Collect text descriptions of various weather conditions to form weather texts, including: fog, heavy rain, sandstorm, hail, and rain / snow.
[0012] Preferably, the pre-trained zero-sample weather generalization generator includes:
[0013] A visual-semantic mapping model is established and trained using clear weather images and weather text to generate target severe weather images;
[0014] The generated target severe weather image is verified by perceptual consistency check. If it fails, the text description or model is adjusted. If it passes, it is injected into the autonomous driving simulation platform for simulation testing.
[0015] Preferred options also include:
[0016] The performance of the zero-sample weather generalization generator in generating simulated weather conditions is tested through an autonomous driving system. If a failure case is found, the text description is automatically or manually corrected and then re-entered into the zero-sample weather generalization generator to generate simulated images for reinforcement testing and subsequent model training.
[0017] Preferably, the training of the visual-semantic mapping model employs a conditional generation network with a diffusion model as its backbone.
[0018] Preferably, a perceptual consistency loss is introduced into the training visual-semantic mapping model to ensure that the structure of key objects in the generated image remains consistent with the clear image and is not destroyed by weather effects.
[0019] Preferably, generating a generalized simulated image superimposed with the target severe weather effect based on the semantic feature vector and the visual feature vector includes:
[0020] The semantic feature vector and the visual feature vector of the new weather are input into the zero-sample weather generalization generator, and the fusion of the content of the clear scene and the visual effect of the new weather is decoupled through the mapping relationship of the visual-semantic mapping model, thereby generating the final photorealistic severe weather image.
[0021] Preferred options also include:
[0022] The generated image and the original sharp image are processed separately using a semantic segmentation model pre-trained on a sharp image, and their similarity at the feature level is compared to ensure that the generated image does not destroy the semantic information of the scene.
[0023] Preferably, the text encoder is CLIP Text Encoder.
[0024] Preferably, the image encoder is a CLIP Image Encoder.
[0025] This invention provides a method for simulating severe weather scenarios for autonomous driving. The method encodes the simulated scene image and the textual description of the target severe weather into semantic feature vectors and visual feature vectors, respectively, and inputs them into a pre-trained zero-shot weather generalization generator to generate a generalized simulated image that superimposes the effects of the target severe weather. This addresses the problems of high data acquisition costs, incomplete data, and poor simulation results in existing autonomous driving scenario simulations. It enables the simulation of rare severe weather combinations, greatly enriching the coverage of test scenarios, improving simulation testing efficiency, and reducing simulation costs. Attached Figure Description
[0026] To more clearly illustrate the specific embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below.
[0027] Figure 1 This is a schematic diagram of a severe weather scenario simulation method for autonomous driving provided by the present invention.
[0028] Figure 2 This is a flowchart illustrating a simulation of severe weather scenarios for autonomous driving, provided by an embodiment of the present invention. Detailed Implementation
[0029] To enable those skilled in the art to better understand the embodiments of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and implementation methods.
[0030] To address the issues of high data acquisition costs, incomplete data, and poor simulation results in current autonomous driving scenario simulations for existing vehicles, this invention provides a method for simulating severe weather scenarios for autonomous driving. This method solves the problems of high data acquisition costs, incomplete data, and poor simulation results in current autonomous driving scenario simulations for existing vehicles, and can realize rare severe weather combinations, greatly enriching the coverage of test scenarios, improving simulation testing efficiency, and reducing simulation costs.
[0031] like Figure 1 and Figure 2 As shown, a method for simulating severe weather scenarios for autonomous driving includes:
[0032] S1 establishes a semantic space to receive clear images of the simulated scene and textual descriptions of the target's severe weather.
[0033] S2: Set the text encoder to encode the text description into a corresponding semantic feature vector.
[0034] S3: Set the image encoder to encode clear simulation scene images into visual feature vectors.
[0035] S4: Using a pre-trained zero-sample weather generalization generator, a generalized simulated image with superimposed target severe weather effects is generated based on the semantic feature vector and the visual feature vector.
[0036] Specifically, by combining text descriptions with simulated scene images, any imaginable, even rare in reality, combination of severe weather, such as hail clouds mixed with rain, can be generated, greatly enriching the coverage of test scenarios. Simultaneously, the generated images are not only visually realistic but, more importantly, maintain the semantic consistency required for autonomous driving perception tasks, making simulation test results more reliable. This method requires no real data on the target weather, generating data solely from descriptions, thus solving the fundamental problem of scarce extreme weather data. This method avoids the time-consuming, labor-intensive, and dangerous process of collecting real data, accelerating the development and verification cycle of autonomous driving systems.
[0037] The establishment of the semantic space includes: collecting text descriptions corresponding to various weather conditions to form weather texts, including: heavy fog, rainstorms, sandstorms, hail, and rain and snow.
[0038] In practical applications, text descriptions of various weather conditions are collected. These descriptions can be in natural language, such as: "Dense fog with visibility less than 50 meters, appearing grayish-white"; structured parameters: {"weather_type": "fog", "visibility": 50, "density": 0.8, "color": [R, G, B]}; all descriptions are encoded into high-dimensional semantic feature vectors using a pre-trained text encoder. This constitutes a "weather dictionary". The visual foundation is prepared with a large number of high-quality, clear images of simulated weather scenes acquired under various lighting and environmental conditions, along with corresponding semantic segmentation maps; visual feature vectors are extracted from these images using a pre-trained image encoder.
[0039] Furthermore, the pre-trained zero-shot weather generalization generator includes:
[0040] A visual-semantic mapping model is established and trained using clear weather images and weather text to generate target severe weather images.
[0041] The generated target severe weather image is verified by perceptual consistency check. If it fails, the text description or model is adjusted. If it passes, it is injected into the autonomous driving simulation platform for simulation testing.
[0042] The method also includes: testing the performance of the zero-sample weather generalization generator in generating simulated weather conditions through an autonomous driving system; if a failure case is found, automatically or manually correcting the text description and re-entering it into the zero-sample weather generalization generator to generate simulated images for reinforcement testing and subsequent model training.
[0043] Furthermore, the training of the visual-semantic mapping model employs a conditional generative network with a diffusion model as its backbone.
[0044] Furthermore, a perceptual consistency loss is introduced into the trained visual-semantic mapping model to ensure that the structure of key objects in the generated image remains consistent with the clear image and is not corrupted by weather effects.
[0045] Furthermore, the step of generating a generalized simulated image superimposed with the target severe weather effect based on the semantic feature vector and the visual feature vector includes:
[0046] The semantic feature vector and the visual feature vector of the new weather are input into the zero-sample weather generalization generator, and the fusion of the content of the clear scene and the visual effect of the new weather is decoupled through the mapping relationship of the visual-semantic mapping model, thereby generating the final photorealistic severe weather image.
[0047] The method also includes:
[0048] The generated image and the original sharp image are processed separately using a semantic segmentation model pre-trained on a sharp image, and their similarity at the feature level is compared to ensure that the generated image does not destroy the semantic information of the scene.
[0049] Furthermore, the text encoder employs CLIP Text Encoder.
[0050] Furthermore, the image encoder employs CLIP Image Encoder.
[0051] As can be seen, this invention provides a method for simulating severe weather scenarios for autonomous driving. The method encodes the simulated scene image and the textual description of the target severe weather into semantic feature vectors and visual feature vectors, respectively, and inputs them into a pre-trained zero-sample weather generalization generator to generate a generalized simulated image superimposed with the effects of the target severe weather. This addresses the problems of high data acquisition costs, incomplete data, and poor simulation results in existing vehicle autonomous driving scenario simulations. It can achieve rare combinations of severe weather, greatly enriching the coverage of test scenarios, improving simulation testing efficiency, and reducing simulation costs.
[0052] The structure, features, and effects of the present invention have been described in detail above with reference to the embodiments shown in the figures. The above description is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or equivalent embodiments modified to have equivalent changes, shall be within the protection scope of the present invention as long as they do not exceed the spirit covered by the specification and figures.
Claims
1. A method for simulating severe weather scenarios for autonomous driving, characterized in that, include: Establish a semantic space to receive clear images of simulated scene scenarios and text descriptions of severe weather conditions. Set a text encoder to encode the text description into a corresponding semantic feature vector; Configure the image encoder to encode clear images of the simulated scene into visual feature vectors; Using a pre-trained zero-shot weather generalization generator, a generalized simulated image with superimposed target severe weather effects is generated based on the semantic feature vector and the visual feature vector.
2. The method for simulating severe weather scenarios for autonomous driving according to claim 1, characterized in that, The establishment of the semantic space includes: Collect text descriptions of various weather conditions to form weather texts, including: fog, heavy rain, sandstorm, hail, and rain / snow.
3. The method for simulating severe weather scenarios for autonomous driving according to claim 2, characterized in that, The pre-trained zero-shot weather generalization generator includes: A visual-semantic mapping model is established and trained using clear weather images and weather text to generate target severe weather images; The generated target severe weather image is verified by perceptual consistency check. If it fails, the text description or model is adjusted. If it passes, it is injected into the autonomous driving simulation platform for simulation testing.
4. The method for simulating severe weather scenarios for autonomous driving according to claim 3, characterized in that, Also includes: The performance of the zero-sample weather generalization generator in generating simulated weather conditions is tested through an autonomous driving system. If a failure case is found, the text description is automatically or manually corrected and then re-entered into the zero-sample weather generalization generator to generate simulated images for reinforcement testing and subsequent model training.
5. The method for simulating severe weather scenarios for autonomous driving according to claim 4, characterized in that, The training visual-semantic mapping model employs a conditional generative network with a diffusion model as its backbone.
6. The method for simulating severe weather scenarios for autonomous driving according to claim 5, characterized in that, A perceptual consistency loss is introduced into the trained visual-semantic mapping model to ensure that the structure of key objects in the generated image remains consistent with the clear image and is not destroyed by weather effects.
7. The method for simulating severe weather scenarios for autonomous driving according to claim 6, characterized in that, The process of generating a generalized simulated image superimposed with the target severe weather effect based on the semantic feature vector and the visual feature vector includes: The semantic feature vector and the visual feature vector of the new weather are input into the zero-sample weather generalization generator, and the fusion of the content of the clear scene and the visual effect of the new weather is decoupled through the mapping relationship of the visual-semantic mapping model, thereby generating the final photorealistic severe weather image.
8. The method for simulating severe weather scenarios for autonomous driving according to claim 7, characterized in that, Also includes: The generated image and the original sharp image are processed separately using a semantic segmentation model pre-trained on a sharp image, and their similarity at the feature level is compared to ensure that the generated image does not destroy the semantic information of the scene.
9. The method for simulating severe weather scenarios for autonomous driving according to claim 8, characterized in that, The text encoder used is CLIP Text Encoder.
10. The method for simulating severe weather scenarios for autonomous driving according to claim 9, characterized in that, The image encoder uses CLIP Image Encoder.