Obstacle sensing method of automatic driving system in rainy and foggy weather and storage medium
By simulating an image degradation model under rainy and foggy weather, images from sunny days are synthesized to generate composite images under rainy and foggy weather. This solves the problem of low obstacle perception accuracy of autonomous driving systems under rainy and foggy weather, improves obstacle detection performance under rainy and foggy weather, and saves costs.
Patent Information
- Application Number
- CN202510830359.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-20
- Publication Date
- 2025-10-17
AI Technical Summary
The existing autonomous driving systems have low obstacle perception accuracy in rainy and foggy weather, making it difficult to meet the accuracy requirements of the autonomous driving system in rainy and foggy weather.
By simulating image degradation models under rainy and foggy weather, and using blurring, noise, scaling, JPEG compression, and ringing degradation models, sunny images are synthesized to generate synthetic images under rainy and foggy weather. The BEV detection algorithm is then used for training to obtain obstacle perception.
It improves obstacle perception accuracy in rainy and foggy weather, reduces costs, avoids additional dataset preparation and annotation, and keeps detection performance unaffected in sunny weather.
Smart Images
Figure CN120808307A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of automatic driving, in particular to a method for obstacle perception of an automatic driving system in rainy and foggy weather and a storage medium. BACKGROUND
[0002] Target detection is the core task of obstacle perception of automatic driving, and the automatic driving system needs to obtain high-precision positioning and classification of surrounding obstacles in sunny and extreme weather. Extreme weather, especially common rainy and foggy weather, can change the environmental state of automatic driving, affect the perception ability of automatic driving sensors, and further increase the perception difficulty of the automatic driving system algorithm. Raindrops in rainy weather can hinder the field of view, causing image blur or distortion. The distorted area may be misinterpreted by the automatic driving system as an obstacle, affecting the accuracy of obstacle detection. Rain marks in rainy weather can cause the camera to introduce stripes or lines, causing noise in the image and interfering with the ability of the automatic driving system to accurately interpret the scene, which may affect the performance of algorithms used to detect and classify objects. In foggy weather, the visibility of road users is reduced, making it difficult for the camera to clearly capture the details of the road and the surrounding environment, making it difficult for the automatic driving system to correctly perceive the road and obstacles.
[0003] The existing target detection perception algorithm in the automatic driving system is basically based on the BEV deep learning method for obstacle perception. The deep learning-based method needs to build a data set and perform labeling, but the existing public data set does not cover rainy and foggy scenes. For self-collected data, rainy and foggy weather rarely occurs at the same time, and high labeling fees are required for obstacle labeling for model learning and training. Therefore, the existing method has low accuracy for obstacle perception in rainy and foggy weather and cannot meet the accuracy requirements of the automatic driving system in rainy and foggy weather. SUMMARY
[0004] In view of the deficiencies of the prior art, the present application provides an obstacle perception method for an automatic driving system in rainy and foggy weather and a storage medium, which solves the problem of low accuracy of the existing method for obstacle perception in rainy and foggy weather and the difficulty of meeting the accuracy requirements of the automatic driving system in rainy and foggy weather.
[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:
[0006] An obstacle perception method for an automatic driving system in rainy and foggy weather, the perception method comprising the following steps:
[0007] S1, obtaining a sunny image containing obstacles collected in sunny weather;
[0008] S2, set a plurality of first degradation models and corresponding selection ratios, and set a plurality of second degradation models, each second degradation model being composed of a plurality of first degradation models selected according to the selection ratios;
[0009] S3, performing image degradation on the sunny day image according to the second degradation model to obtain a synthetic image under rain and fog weather;
[0010] S4, training the BEV detection algorithm through the sunny day image and the synthetic image, and obtaining a to-be-identified image for obstacle perception.
[0011] Preferably, in step S2, the first degradation model includes a blur degradation model, a noise degradation model, a scaling degradation model, a JPEG compression degradation model, and a ringing degradation model.
[0012] Preferably, the blur degradation model selected by the first-order second degradation model is Gaussian blur and sinc blur selected with a probability of {0.9, 0.1}, for Gaussian blur, a probability of {0.7, 0.15, 0.15} is adopted for Gaussian kernel, generalized Gaussian kernel, and plateau-shaped kernel, a blur kernel size is randomly selected from {7, 9,..., 21}, a standard deviation sigma is sampled from [0.2, 3], for the generalized Gaussian kernel and the plateau-shaped kernel, a shape parameter beta is selected from [0.5, 4] and [1, 2] respectively, a noise degradation model adopts Gaussian noise or Poisson noise and grayscale noise selected with a probability of {0.6, 0.4}, Gaussian noise and Poisson noise selected with a probability of {0.5, 0.5}, a Gaussian noise sigma range is [1, 30], a Poisson noise scale is set to [0.05, 3], a JPEG compression degradation model sets a JPEG compression quality factor to [30, 95], and a ringing degradation model adopts a sinc filter with a selection probability of 0.8.
[0013] As preferred, the second degradation model selected by the second order of the second degradation model selects the Gaussian blur and the sine blur with the probability of {0.9, 0.1}, for the Gaussian blur, the Gaussian kernel, the generalized Gaussian kernel and the plateau shape kernel with the probability of {0.7, 0.15, 0.15}, the blur kernel size is randomly selected from {7, 9,..., 21}, the standard deviation sigma is sampled from [0.2, 1.5], for the generalized Gaussian kernel and the plateau shape kernel, the shape parameter beta is selected from [0.5, 4] and [1, 2] respectively, the noise degradation model selects the Gaussian noise or the Poisson noise and the gray noise with the probability of {0.6, 0.4}, selects the Gaussian noise and the Poisson noise with the probability of {0.5, 0.5}, the Gaussian noise sigma range is [1, 25], the Poisson noise scale is set to [0.05, 2.5], the JPEG compression quality factor of the JPEG compression degradation model is set to [30, 95], and the ringing degradation model adopts the sine filter with the selection probability of 0.8.
[0014] As preferred, the expression of the Gaussian blur is as follows:
[0015]
[0016]
[0017]
[0018] As preferred, in the above formula, For the Gaussian blur kernel k with the kernel size of 2t + 1, the (i, j) element in [−t, t] is sampled from the Gaussian distribution, (i, j) represents the coefficient coordinates of the Gaussian blur kernel, Sigma is the covariance matrix, C is the spatial coordinate, N is the normalization constant, and Sigma is the standard deviation along the two principal axes, that is, the eigenvalue of the covariance matrix, is the rotation degree, when = , k is an isotropic Gaussian blur kernel, otherwise k is an anisotropic kernel.
[0019] As preferred, the probability density function of the generalized Gaussian kernel is:
[0020]
[0021] The probability density function of the plateau shape kernel is:
[0022]
[0023] In the above formula, beta is the shape parameter.
[0024] As preferred, the expression of the sine filter kernel is:
[0025]
[0026] In the above formula, is a kernel of the sinc filter at the core coordinate (i, j), is a cut-off frequency.
[0027] As preferred, in step S3, the expression of the synthetic image is:
[0028]
[0029] In the above formula, represents the synthetic image, D represents the degradation model, n represents the order of the degradation model, the second-order degradation model of the application, and y represents the collected sunny day image.
[0030] The technical scheme further provides a computer storage medium, which is used for storing program data, and the program data, when executed by a computer, realizes the obstacle perception method of the automatic driving system in rainy and foggy weather.
[0031] Compared with the prior art, the application provides an obstacle perception method of an automatic driving system in rainy and foggy weather and a storage medium, which has the following beneficial effects:
[0032] 1. According to the image collected in rainy and foggy weather, the image may appear fuzzy, false, ringing, color change and other problems, and in view of the above situation, a degradation model is established by using a computer image algorithm, a first degradation model is used to simulate the image blur problem in rainy and foggy weather, a second degradation model is used to realize repeated scaling and noise addition to simulate the image false problem in rainy and foggy weather, and JPEG compression and ringing are used to simulate the color change and ringing problem in rainy and foggy weather, so as to obtain a synthetic image in rainy and foggy weather.
[0033] 2. The application further adopts a generalized Gaussian blur kernel and a platform-shaped kernel to contain more different kernel shapes, so as to well approximate the real camera blur, so as to better obtain a real sample closer to rainy and foggy weather, that is, a synthetic image. BRIEF DESCRIPTION OF DRAWINGS
[0034] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the application without imposing undue limitation thereon. In the drawings:
[0035] Figure 1 is a flowchart of the obstacle perception method of the automatic driving system in rainy and foggy weather of the application;
[0036] Figure 2 is a schematic diagram of the sunny day image of the application;
[0037] Figure 3 A schematic diagram of a synthetic image of the present application. DETAILED DESCRIPTION
[0038] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments. Thus, the implementation process of how to apply technical means to solve technical problems and achieve technical effects can be fully understood and implemented.
[0039] Those of ordinary skill in the art can understand that all or part of the steps of the following embodiments can be completed by programs instructing relevant hardware, therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of being implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes.
[0040] In order to solve the problem that the existing method has low obstacle perception accuracy in rainy and foggy weather and is difficult to meet the accuracy requirement of the automatic driving system in rainy and foggy weather, the present application provides an obstacle perception method of an automatic driving system in rainy and foggy weather, which synthesizes rainy and foggy weather data from the existing data set without sacrificing the performance of sunny weather, without the need for additional data preparation and secondary labeling, and can greatly save costs, such as Figure 1 As shown in the figure, the perception method comprises the following steps:
[0041] S1, obtaining a sunny image containing an obstacle collected in sunny weather; the sunny image is obtained from the nuScenes data set, and can also be self-shot. From the image collected by the camera in sunny weather to the sunny image collected by the camera in rainy and foggy weather, it is called image degradation. The present application synthesizes a synthetic image in rainy and foggy weather from the sunny image collected by the camera in sunny weather by modeling the degradation model.
[0042] S2, setting a plurality of first degradation models and corresponding selection ratios, and setting a plurality of second degradation models, each second degradation model being composed of a plurality of first degradation models selected according to the selection ratio. The first degradation model includes a blur degradation model, a noise degradation model, a scaling degradation model, a JPEG compression degradation model and a ringing degradation model.
[0043] Image scaling of the scaling degradation model is generally divided into two kinds of down-sampling and up-sampling, several commonly used scaling algorithms are nearest neighbor interpolation, area scaling, bilinear interpolation and bicubic interpolation, different scaling algorithms will produce different effects - some will produce blurred images, some will produce over-sharpened images, in order to include more and complex resizing effects, the present application randomly adjusts the image size operation from the above selection.
[0044] PEG compression of the JPEG compression degradation model is a commonly used digital image lossy compression technique, which first converts the image to YCbCr color space, then down-samples the chroma channel, divides the image into 8 × 8 blocks, and transforms each block with two-dimensional discrete cosine transform (DCT), and finally quantizes the DCT coefficients, JPEG compression usually introduces unacceptable block artifacts, the quality of the compressed image is determined by the quality factor q ∈ [0, 100], where lower q indicates higher compression ratio and poorer quality. The present application uses PyTorch to implement the DiffJPEG algorithm.
[0045] The noise degradation model mainly uses two commonly used noise types, additive Gaussian noise and Poisson noise. The probability density function of additive Gaussian noise is equal to the probability density function of Gaussian distribution, and the noise intensity is determined by the standard deviation (i.e. σ value). When each channel of the RGB image has independent sampling noise, the synthesized noise is color noise. The present application synthesizes gray noise by using the same sampling noise for all three channels. Poisson noise follows Poisson distribution and is usually used to approximately simulate sensor noise caused by statistical quantum fluctuations. The intensity of Poisson noise is proportional to the intensity of the image, and the noise at different pixels is independent.
[0046] The first-order second degradation model selects a fuzzy degradation model as Gaussian blur and sinc blur selected by a probability of {0.9, 0.1}, for Gaussian blur, Gaussian kernel, generalized Gaussian kernel and plateau-shaped kernel are selected by a probability of {0.7, 0.15, 0.15}, the size of the blur kernel is randomly selected from {7, 9,..., 21}, the standard deviation sigma is sampled from [0.2, 3], for the generalized Gaussian kernel and the plateau-shaped kernel, the shape parameter beta is selected from [0.5, 4] and [1, 2] respectively, the noise degradation model selects Gaussian noise or Poisson noise and grayscale noise by a probability of {0.6, 0.4}, Gaussian noise and Poisson noise are selected by a probability of {0.5, 0.5}, the range of Gaussian noise sigma is [1, 30], the scale of Poisson noise is set to [0.05, 3], the JPEG compression quality factor of the JPEG compression degradation model is set to [30, 95], the ringing degradation model adopts a sinc filter, and the selection probability is 0.8. The parameter range value selected by the application can maintain sufficient generalization ability of the degradation model, so the parameter selection range is wide enough and has sufficient randomness. It is verified by experiments that the parameter range given by the application can meet the performance requirements of obstacle detection in rainy and foggy weather.
[0047] The second-order second degradation model selects a fuzzy degradation model as Gaussian blur and sinc blur selected by a probability of {0.9, 0.1}, for Gaussian blur, Gaussian kernel, generalized Gaussian kernel and plateau-shaped kernel are selected by a probability of {0.7, 0.15, 0.15}, the size of the blur kernel is randomly selected from {7, 9,..., 21}, the standard deviation sigma is sampled from [0.2, 1.5], for the generalized Gaussian kernel and the plateau-shaped kernel, the shape parameter beta is selected from [0.5, 4] and [1, 2] respectively, the noise degradation model selects Gaussian noise or Poisson noise and grayscale noise by a probability of {0.6, 0.4}, Gaussian noise and Poisson noise are selected by a probability of {0.5, 0.5}, the range of Gaussian noise sigma is [1, 25], the scale of Poisson noise is set to [0.05, 2.5], the JPEG compression quality factor of the JPEG compression degradation model is set to [30, 95], the ringing degradation model adopts a sinc filter, and the selection probability is 0.8.
[0048] The expression of Gaussian blur is as follows:
[0049]
[0050]
[0051]
[0052] Although Gaussian blur kernels are widely used to model blur degradation, they can not approximate real camera blur well, in order to include more different kernel shapes, the present application further adopts generalized Gaussian blur kernels and plateau kernels, wherein the probability density function of the generalized Gaussian kernel is:
[0053]
[0054] The probability density function of the plateau kernel is:
[0055]
[0056] In the above formula, Gaussian blur, for a Gaussian blur kernel k with kernel size 2t + 1, the (i, j) ∈ [-t, t] elements are sampled from a Gaussian distribution, (i, j) represents the coefficient coordinates of the Gaussian blur kernel, Σ is the covariance matrix, C is the spatial coordinate, N is the normalization constant, and σ x and σ y are the standard deviations along the two principal axes, i.e. the eigenvalues of the covariance matrix, θ is the rotation degree, when , k is an isotropic Gaussian blur kernel, otherwise k is an anisotropic kernel, and β is a shape parameter.
[0057] The ringing artifact usually appears as a pseudo-edge near a sharp transition in the image, the ringing artifact appears as an increased jump at the edge transition, the main reason for these artifacts is that the signal is band-limited and has no high frequency, the present application adopts a sinc filter, an idealized filter that cuts off high frequencies, to synthesize ringing and overshoot artifacts training pairs, the expression of the sinc filter kernel is:
[0058]
[0059] In the above formula, is the kernel of the sinc filter at the kernel coordinates (i, j), is the cutoff frequency.
[0060] S3, according to the second degradation model, the sunny day image is degraded several times to obtain a synthetic image under rainy and foggy weather;
[0061] The present application initially models the degradation model using a method commonly used in computer graphics, such as formula (1), the image y collected under sunny weather is first convolved with the blur kernel k, then a downsampling operation with a scale factor r is performed, noise m is added, and finally an upsampling operation with a scale factor r is performed and JPEG compression is used to obtain a synthetic image under rainy and foggy weather.
[0062] (1)
[0063] where D represents a degradation model, but this method cannot solve unknown noise and complex artifacts in rainy and foggy weather, so it is improved again and a new high-order degradation model is proposed as shown in Figure 1 As shown in the formula (2), the n-order model involves the process of n repeated degradation models, and the degradation model of each stage is modeled by using the degradation model commonly used in computer graphics shown in the formula (1), but the hyperparameters are different, and it is verified through experiments that the performance of the second-order degradation model used in the application is better.
[0064] The expression of the synthetic image is:
[0065] (2)
[0066] In the above formula, The synthetic image is represented by D, the degradation model is represented by n, the order of the degradation model, the second-order degradation model of the application, y represents the collected sunny day image, n=2 of the second-order degradation model, and the permutation and combination of blur, noise, scaling, JPEG compression and ringing are used to model the synthetic image in rainy and foggy weather.
[0067] S4, train the BEV detection algorithm by using the sunny day image and the synthetic image, and obtain the to-be-identified image to perform obstacle perception.
[0068] The training method and parameters in the BEVFormer, PolarFormer, PETRv2, BEVDepth and BEVStereo papers are used to train in the pytorch framework on the Nvidia V100, the sunny day inference verification is performed on the dataset nuScenes as shown in Tables 1 and 2, the rainy and foggy weather inference verification is performed on the extreme weather dataset RADIATE, and the training is performed for 30 epcho, the application can achieve the balance of training and inference performance in the experiment, the trained pytorch model is converted into a TensorRT (a special forward inference framework provided by Nvidia) model, the specific conversion process is that the pytorch model is converted into a standard model onnx, the onnx model is converted into a tensorRT model, and the inference is performed on the Nvidia 4090 to obtain the final obstacle perception result.
[0069] Table 1 below is a comparison of the detection performance of the application and the existing BEV algorithm in the RADIATE dataset under rainy and foggy weather, and it can be seen from the table that the detection performance of the application greatly surpasses the performance of the existing BEV detection algorithm under rainy and foggy weather.
[0070]
[0071] The following Table 2 is a comparison of the detection performance between the present application and the existing BEV algorithm under sunny weather on the nuScenes dataset. As can be seen from the table, the present application does not affect the detection performance under sunny weather.
[0072]
[0073] The above embodiments have been described in detail, and the principles and embodiments of the present application have been described by applying specific examples. The above description of the embodiments is only intended to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific embodiments and application scope will be changed, and the above description should not be understood as a limitation of the present application.
Claims
1. The obstacle perception method of the autonomous driving system in rainy and foggy weather is characterized by: The perception method includes the following steps: S1. Obtain a clear sky image containing obstacles collected on a clear day; S2. Setting a number of first degradation models and corresponding selection ratios, and setting a number of second-order degradation models, where each second-order degradation model is composed of a number of first degradation models selected according to the selection ratios; S3. performing image degradation on the sunny image several times according to the second degradation model to obtain a composite image under rainy and foggy weather conditions; S4. Train the BEV detection algorithm using sunny day images and synthetic images, and obtain images to be identified for obstacle perception.
2. The obstacle sensing method according to claim 1, characterized in that: In step S2, the first degradation model includes a blur degradation model, a noise degradation model, a scaling degradation model, a JPEG compression degradation model, and a ringing degradation model.
3. The obstacle sensing method according to claim 2, characterized in that: The blur degradation models selected for the first-order and second-order degradation models are Gaussian blur and sinc blur selected with a probability of {0.9, 0.1}. For Gaussian blur, Gaussian kernel, generalized Gaussian kernel and plateau kernel with probabilities of {0.7, 0.15, 0.15} are used. The blur kernel size is randomly selected from {7, 9, ... 21}, and the standard deviation σ is sampled from [0.2, 3]. For the generalized Gaussian kernel and plateau kernel, the shape parameter β is sampled from [0.5, 4] and [1, 2], the noise degradation model uses Gaussian noise or Poisson noise and grayscale noise with a probability of {0.6, 0.4}, and Gaussian noise and Poisson noise with a probability of {0.5, 0.5}. The Gaussian noise sigma range is [1, 30], the Poisson noise scale is set to [0.05, 3], the JPEG compression quality factor of the JPEG compression degradation model is set to [30, 95], and the ringing degradation model uses a sinc filter with a selection probability of 0.
8.
4. The obstacle sensing method according to claim 2, wherein: The blur degradation model selected by the second degradation model of the second order adopts Gaussian blur and sinc blur selected with a probability of {0.9, 0.1}. For Gaussian blur, Gaussian kernel, generalized Gaussian kernel and plateau shape kernel with a probability of {0.7, 0.15, 0.15} are used. The blur kernel size is randomly selected from {7, 9, ... 21}, and the standard deviation σ is sampled from [0.2, 1.5]. For the generalized Gaussian kernel and plateau kernel, the shape parameter β is sampled from [0.5, 4] and [1, 2 ], the noise degradation model adopts the probability of {0.6, 0.4} to select Gaussian noise or Poisson noise and grayscale noise, and the probability of {0.5, 0.5} to select Gaussian noise and Poisson noise, the Gaussian noise sigma range is [1, 25], the Poisson noise scale is set to [0.05, 2.5], the JPEG compression quality factor of the JPEG compression degradation model is set to [30, 95], and the ringing degradation model adopts the sinc filter with a selection probability of 0.
8.
5. The obstacle sensing method according to claim 3, characterized in that: The expression of Gaussian blur is as follows:
6. The obstacle sensing method according to claim 1, characterized in that: In the above formula, Gaussian blur, for a Gaussian blur kernel k with a kernel size of 2t + 1, its (i, j) ∈ [−t, t] element is sampled from a Gaussian distribution, (i, j) represents the coefficient coordinates of the Gaussian blur kernel, Σ is the covariance matrix, C is the spatial coordinate, N is the normalization constant, and is the standard deviation along the two principal axes, that is, the eigenvalue of the covariance matrix, is the degree of rotation, when = When , k is the isotropic Gaussian blur kernel, otherwise k is the anisotropic kernel.
7. The obstacle sensing method according to claim 3, characterized in that: The probability density function of the generalized Gaussian kernel is: The probability density function of platform nucleation is: In the above formula, β is the shape parameter.
8. The obstacle sensing method according to claim 3, characterized in that: The expression for the sinc filter kernel is: In the above formula, is the kernel of the sinc filter at the kernel coordinate (i, j), is the cutoff frequency.
9. The obstacle sensing method according to claim 1, characterized in that: In step S3, the expression of the synthesized image is: In the above formula, Denotes a synthetic image, D expresses a degradation model, n denotes the order of the degradation model, the second-order degradation model of the present invention, and y denotes a collected sunny image.
10. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by a computer, it implements the obstacle perception method of the automatic driving system in rainy and foggy weather according to any one of claims 1 to 9.