An influence analysis method for a visual sensor of an intelligent automobile in a rain and fog environment

Abstract

The application relates to a visual sensor influence analysis method, in particular to a visual sensor influence analysis method for intelligent automobile visual sensors in rain and fog environments, steps of which comprise the following: a test scene is constructed based on a rain and fog weather environment simulation platform of a closed site; [rain and fog weather parameters-target object distance-image information affected by the weather] data is collected; the influence of the rain and fog environment on the intelligent automobile visual sensors is analyzed through three levels of original data, image features and identification functions; the application realizes full-dimension quantitative analysis of the influence of the rain and fog environment on the intelligent automobile visual sensors, so that the performance evaluation of the visual sensors in the rain and fog environment is more comprehensive and accurate; the distance parameter is introduced into the influence evaluation range, so as to better guide the defect analysis and optimization design of the visual sensor algorithm. The application discloses the attenuation effect of the rain and fog environment on the performance of the visual sensor, and provides technical support for the robustness optimization and test verification of the visual sensing system.

Description

TECHNICAL FIELD

[0001] The application relates to an influence analysis method of a visual sensor, in particular to an influence analysis method of a visual sensor of an intelligent automobile in a rain and fog environment. BACKGROUND

[0002] As an important information receiving medium between an intelligent automobile and a driving environment, the robustness and reliability of a visual sensor are directly related to the safety of the intelligent automobile in a driving process. However, a complex environment formed by coupling of a severe weather environment and a complex traffic flow is prone to cause degradation of a visual sensor system, and thus causes problems of functional safety or expected functional safety. As a very common severe weather environment in daily driving of an intelligent automobile, how to quantitatively analyze the influence of a rain and fog environment on a visual sensor before the intelligent automobile is actually put into use is of great significance to improving the component level, system level and whole vehicle level safety of the intelligent automobile. Most of existing analysis methods consider an image quality dimension, that is, the degradation of a same target object in different rain or fog environments compared with a fine weather is used to analyze the influence. The existing analysis methods have the following problems: the existing methods only analyze the influence of rain and fog in a single dimension, it is difficult to comprehensively evaluate the influence of the rain and fog environment on the visual sensor, and the existing methods cannot provide effective contribution for development and application of the intelligent automobile; in addition, the existing methods ignore the distance parameter between the target object and the automobile, and only using different weather parameters cannot provide help for building an application test scene of the intelligent automobile. SUMMARY

[0003] In order to solve the above technical problems, the application provides an influence analysis method of a visual sensor of an intelligent automobile in a rain and fog environment, and the steps include:

[0004] Step 1, a test scene is constructed based on a rain and fog weather environment simulation platform of a closed field, and a test scene construction and rain and fog environment visual sensor data collection process is as follows: a static vehicle target object is placed, a vehicle carrying a visual sensor gradually approaches the target object at a certain distance and at a certain speed, no-rain and no-fog influence images corresponding to different vehicle target object distances are collected through the visual sensor, then a rain or fog simulation device is started, rain and fog weather parameters are recorded, and rain and fog image data corresponding to different vehicle target object distances are collected through the visual sensor; the distance between the vehicle target object and the automobile in the calibration image is recorded during the collection process; and a [rain and fog weather parameter-target object distance-weather-influenced image information] data pair is repeatedly collected by changing different rain and fog weather parameters.

[0005] Further, the collected rain and fog image data Im s includes a rain image, denoted as Im sr , and a fog image, denoted as Im sf .

[0006] Step 2: Analyze the impact of rain and fog on the vision sensors of intelligent vehicles through three different levels: raw data, image features, and recognition functions. The input is the image Im at different target distances without the influence of rain and fog. r and rain and fog image data Im s A combination of images.

[0007] Among them, the raw data hierarchical impact analysis is used to quantify the impact of rain and fog environments on the raw image information output by the vision sensors of intelligent vehicles, and is divided into full-reference indicators and no-reference indicators; the full-reference indicators require an image Im unaffected by rain and fog. r As a benchmark, two methods were used for calculation: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM).

[0008] The no-reference metric uses the Tenengrad gradient evaluation metric, which consists of the horizontal and vertical gradient values ​​of a pixel, and a threshold is set for the gradient. The sensitivity of the adjustment function;

[0009] Furthermore, the PSNR is defined based on the image mean square error (MSE), given an M*N rain- and fog-free image Im. r and rain and fog image data Im s Its MSE is defined as:

[0010]

[0011]

[0012] In the formula, MAX represents the maximum pixel value in the image, and SSIM is defined as:

[0013]

[0014] In the formula, x and y are Im and y, respectively. r and Im s pixel intensity, These represent the means of x and y, respectively. These represent the variances of x and y, respectively. It is the covariance of x and y; is a constant that maintains the stability of the SSIM formula; Q is the dynamic range of the image; k1 and k2 are constants.

[0015] Furthermore, the expression for the Tenengrad gradient evaluation index is as follows:

[0016]

[0017] where, is the gradient of the pixel point , and the specific expression is:

[0018]

[0019] where, and are the gradient values of the pixel point in horizontal and vertical directions, respectively, and the specific expression is:

[0020]

[0021] where, is the input image, and * represents a two-dimensional convolution operation, S x and S y represent the Sobel operator values in horizontal and vertical directions, respectively:

[0022]

[0023]

[0024] The larger the Tenengrad value of the whole image, the higher the overall edge contrast of the image, the clearer the details, and the more accurate the focus.

[0025] Under the condition of the same target distance, the peak signal-to-noise ratio (PSNR), the structural similarity (SSIM), and the Tenengrad gradient evaluation index of each group of Im r and Im s images were calculated, and the original data level influence analysis results were constructed using linear interpolation with the target distance and meteorological parameters as independent variables and PSNR, SSIM, and Tenengrad as dependent variables. The degradation rules of pixel-level differences, structure fidelity, and edge sharpness of images under different target distances and different meteorological parameters in rain and fog environments were revealed intuitively.

[0026] In the image feature level influence analysis, two local feature extraction and description algorithms, SIFT and ORB, were introduced, and the collected rain and fog image data were used to quantitatively evaluate the influence of rain and fog meteorological conditions on the feature detection and matching performance of intelligent vehicle vision sensors.

[0027] To quantitatively evaluate the degradation effect of the feature extraction algorithm based on corner detection, the ORB algorithm was used to describe the key point number decay rate. First, the same target distance rain and fog-free image Im r and rain and fog image data Im s were selected, and the ORB and SITF algorithms were used to extract the key points, respectively. Then, the rain and fog-free image Imr The total number of key points detected in Im s The total number of key points detected in Im The ORB key point number retention rate under different target distances and meteorological parameters is calculated as follows:

[0028]

[0029] Under the same experimental framework, the SIFT algorithm is used to calculate the key point number retention rate caused by rain and fog weather, to evaluate the influence of rain and fog environment on the scale and rotation invariant feature detection performance, and the expression is as follows:

[0030]

[0031] In the formula, GT and D represent the SIFT key point recognition results of Im r and Im s , respectively.

[0032] In the analysis of the influence of the recognition function level, the target recognition and distance estimation functions commonly used by intelligent vehicle vision sensors are selected to analyze the influence of rain and fog.

[0033] Among them, the YOLO series algorithm is selected as the object for analyzing the influence of rain and fog on the target recognition function. The ground truth GT of the bounding box of each target in the image Im r without the influence of rain and fog is obtained first; the YOLO algorithm is used to detect the rain and fog image data Im s , to obtain the detection bounding box D, and the IOU method is used to evaluate the target recognition algorithm:

[0034]

[0035] The accuracy evaluation index of the target recognition algorithm is defined as the value between 0 and 1, and the larger the value, the more accurate the target recognition algorithm. Im r and Im s images are used as the input of YOLO-V5, and the corresponding are output. GT and D are defined as the output values of the image without the influence of rain and fog and the output values of the rain and fog image data, respectively; the recognition reduction degree is used to analyze the influence value of rain and fog environment on the target recognition function, and the expression is as follows:

[0036]

[0037] ​​​​​​​In the distance estimation function impact analysis, in order to analyze the influence of rain and fog environment on the distance estimation function, the distance true value of the cone bucket calibration is used And the distance estimation function output Difference comparison is adopted The influence value of rain and fog environment on the distance estimation function is analyzed The expression is:

[0038]

[0039] After obtaining the rain and fog environment influence analysis results of the original data layer, the image feature layer and the recognition function layer of the intelligent vehicle vision sensor, firstly, the analysis results obtained by each layer are normalized, and the result range is normalized to Then, the three layer influence results are added together to obtain the analysis results considering the three influence factors, in order to make the expression more clear, the final rain and fog environment multi-layer influence analysis results of the intelligent vehicle vision sensor are obtained by using the normalization processing method again.

[0040] The beneficial effects of the present application are:

[0041] The present application proposes a new intelligent vehicle vision sensor influence analysis method for rain and fog environment. The method firstly splits the analysis method into original data layer, image feature layer and recognition function layer, and then realizes the full-dimensional quantification of the influence of rain and fog environment on the intelligent vehicle vision sensor, so that the performance evaluation of the vision sensor in the rain and fog environment is more comprehensive and accurate. In addition, the distance parameter is introduced into the influence evaluation range, and the introduction of the parameter can better guide the defect analysis and optimization design of the vision sensor algorithm. The multi-layer influence analysis results of the present application clearly reveal the attenuation effect of the rain and fog environment on the performance of the vision sensor, and provide data basis and method support for the robustness optimization and test verification of the vision sensing system. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 It is the overall flowchart of the method of the present application.

[0043] Figure 2 It is the test scene schematic diagram of the rain and fog environment impact analysis of the present application.

[0044] Figure 3 It is the multi-layer influence analysis method flowchart of the present application.

[0045] Figure 4 It is the image acquisition result schematic diagram of the embodiment part of the present application.

[0046] Figure 5 It is the image PSNR value result in the rain scene of the embodiment of the present application.

[0047] Figure 6 Image SSIM value results in a rain scenario for embodiments of the present application.

[0048] Figure 7 Image Tenengrad value results in a rain scenario for embodiments of the present application.

[0049] Figure 8 Image PSNR value results in a fog scenario for embodiments of the present application.

[0050] Figure 9 Image SSIM value results in a fog scenario for embodiments of the present application.

[0051] Figure 10 Image Tenengrad value results in a fog scenario for embodiments of the present application.

[0052] Figure 11 Image PSNR value results in a rain scenario for embodiments of the present application.

[0053] Figure 12 Image PSNR value results in a rain scenario for embodiments of the present application.

[0054] Figure 13 Image PSNR value results in a fog scenario for embodiments of the present application.

[0055] Figure 14 Image PSNR value results in a fog scenario for embodiments of the present application.

[0056] Figure 15 Image PSNR value results in a rain scenario for embodiments of the present application.

[0057] Image PSNR value results in a rain scenario for embodiments of the present application. Figure 16 Image PSNR value results in a fog scenario for embodiments of the present application.

[0058] Figure 17 Image PSNR value results in a fog scenario for embodiments of the present application.

[0059] Figure 18 Image PSNR value results in a fog scenario for embodiments of the present application.

[0060] Figure 19 Multi-level impact analysis results in a rain scenario for embodiments of the present application.

[0061] Figure 20 Multi-level impact analysis results in a fog scenario for embodiments of the present application. DETAILED DESCRIPTION​​​​​​​

[0062] As Figure 1 shown, the embodiment provides an influence analysis method for a visual sensor of an intelligent vehicle in a rain and fog environment, including the following steps:

[0063] (1) Test scene design for rain and fog environment influence analysis:

[0064] In order to accurately analyze the influence of the rain and fog environment on the visual sensor of the intelligent vehicle, the present application constructs a test scene based on a closed site rain and fog meteorological environment simulation platform, as Figure 2 shown, the rain and fog meteorological environment simulation platform is an existing facility, which has a closed site, a simulated road, and a rain and fog meteorological environment simulation device, etc. The specific test scene construction and rain and fog environment visual sensor data collection process is as follows: first, place the vehicle target object statically, and turn on the corresponding rain or fog simulation device; then, the host vehicle carrying the visual sensor starts to gradually approach the target object at a speed of 5km / h from a distance of 50 meters from the target object, and records the image information of the vehicle target object collected by the visual sensor in the process; the distance between the vehicle target object and the host vehicle in the image is calibrated by using an equidistance cone bucket with an interval of 10 meters; finally, the above process is repeated by changing different rain and fog meteorological parameters to obtain a [rain and fog meteorological parameter-target object distance-affected image information] data pair, which is used for the influence analysis of the rain and fog environment on the visual sensor of the intelligent vehicle.

[0065] During the collection of rain and fog environment visual sensor data, the image area other than the vehicle target object in the image range remains unchanged during the collection process, eliminating the interference of other image areas on the collection results. In addition, the influence analysis method proposed by the present application needs to collect the rain and fog-free image Im r as a reference for analysis, so it is necessary to collect the Im s corresponding to different vehicle target object distances before collecting the rain and fog image data Im r . The collected rain and fog image data Im s includes rain images, denoted as Im sr , and fog images, denoted as Im sf .

[0066] (2) Multi-level influence analysis method of rain and fog environment on intelligent vehicle visual sensor

[0067] The influence analysis of the rain and fog environment on the visual sensor of the intelligent vehicle constructed by the present application is divided into three different levels, including the original data, image features, and recognition function levels, and the input is the image combination of the rain and fog-free image Im r and the rain and fog image data Im s under different target distances.

[0068] Firstly, three specific influence analysis methods are introduced respectively, and then the multi-level influence analysis results are integrated by using normalization method.

[0069] (2.1) Raw data level influence analysis

[0070] Raw data level influence analysis is used to quantify the influence of rain and fog environment on the output raw image information of vision sensor of intelligent vehicle, which is divided into full-reference index and no-reference index; the full-reference index needs the rain and fog free image Im r As a benchmark, the methods used include Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). PSNR is defined based on Mean Square Error (MSE) of image, given the rain and fog free image Im r and the rain and fog image data Im s , the MSE is defined as:

[0071]

[0072] In the formula, MAX represents the maximum value of pixel value in the image, which is 255 in this embodiment. Assuming that the pixel intensity of given Im r and Im s is x and y respectively, the corresponding SSIM is defined as:

[0073]

[0074] In the formula, μx and μy represent the mean value of x and y respectively; σx2 and σy2 represent the variance of x and y respectively; Cov(x, y) is the covariance of x and y; is a constant to maintain the stability of formula (3); Q is the dynamic range of image, which is Q=255 in this embodiment; k1 and k2 are constants, which are k1=0.01 and k2=0.02 in this embodiment. The no-reference index uses Tenengrad gradient evaluation index. Tenengrad is composed of gradient values of pixel points in horizontal and vertical directions extracted by Sobel operator, and a threshold is set for the gradient

[0075] to adjust the sensitivity of the function, the expression is as follows:

[0076]

[0077] In the formula, Gx and Gy are the gradients of pixel points , the specific expression is as follows:

[0078] ​​

[0079] In the formula, and These are the gradient values ​​in the horizontal and vertical directions of the pixel, respectively, and their specific expressions are as follows:

[0080]

[0081] In the formula, The input image is *, which represents a two-dimensional convolution operation, and S... x and S y These represent the values ​​of the Sobel operator in the horizontal and vertical directions, respectively.

[0082]

[0083] The higher the Tenengrad value of the entire image, the higher the overall edge contrast, the clearer the details, and the more accurate the focus.

[0084] Under the condition of the same target distance, the present invention applies different methods to each group of Im. r and Im s The three evaluation indices, namely Peak Signal-to-Noise Ratio (PSNR), Image Structure Similarity (SSIM), and Tenengrad, were calculated. With target distance and meteorological parameters as independent variables and PSNR, SSIM, and Tenengrad as dependent variables, a linear interpolation method was used to construct the original data hierarchical influence analysis results. The original data hierarchical influence analysis method proposed in this invention can intuitively reveal the degradation laws of images under different target distances and meteorological parameter conditions in rain and fog environments in terms of pixel-level differences, structural fidelity, and edge sharpness.

[0085] (2.2) Analysis of the influence of image feature hierarchy

[0086] In the image feature hierarchy impact analysis, this invention introduces two local feature extraction and description algorithms, SIFT and ORB, and combines them with the collected rain and fog image data to quantitatively evaluate the impact of rain and fog meteorological conditions on the feature detection and matching performance of intelligent vehicle vision sensors.

[0087] Under rainy and foggy weather conditions, the imaging contrast of visual sensors typically decreases significantly, leading to a decline in the performance of feature extraction algorithms based on corner detection. To quantitatively assess this degradation effect, this invention applies the ORB algorithm to describe the rate of keypoint attenuation. The specific process is as follows: First, select an image Im at the same target distance that is unaffected by rain and fog. r and rain and fog image data Im s Keypoints were extracted using both ORB and SITF algorithms; then, Im was recorded. r The total number of key points detected in the middle is Ims The total number of key points detected in the image is . By calculating:

[0088]

[0089] The ORB key point number retention rate under different target distances and weather parameters can be obtained . Under the same experimental framework, the SIFT algorithm can also be used to calculate the key point number retention rate to evaluate the influence of rain and fog environment on the detection performance of scale and rotation invariant features. The specific expression is:

[0090]

[0091] In the formula, is the key point number retention rate caused by rain and fog weather obtained by using the SIFT algorithm; respectively represent the Im r and Im s corresponding SIFT key point recognition results.

[0092] (2.3) Analysis of the impact of the recognition function level

[0093] In the analysis of the impact of the recognition function level, the present application selects two commonly used functions of intelligent vehicle vision sensors, target recognition and distance estimation, to analyze the impact of rain and fog.

[0094] (2.3.1) Analysis of the impact of the target recognition function

[0095] The present application selects the YOLO series algorithm as the object for analyzing the impact of rain and fog on the target recognition function. The GroundTruthLabeler tool package of MATLAB is used to obtain the true value GT of the outer bounding box of each frame of image target in the image Im r without rain and fog influence after manual correction; the YOLO algorithm is used to detect the rain and fog image data Im s , to obtain the detection outer bounding box D, and the IOU method is used to evaluate the target recognition algorithm:

[0096]

[0097] is the accuracy evaluation index of the target recognition algorithm, and the output is a value between 0 and 1. The larger the value, the more accurate the target recognition algorithm. Im r and Im s images are used as the input of YOLO-V5, and the corresponding are defined as and respectively. The present application uses the recognition reduction degree to analyze the impact of rain and fog environment on the target recognition function , the specific expression is:

[0098]

[0099] (2.3.2) Distance estimation function impact analysis

[0100] In order to analyze the influence of rain and fog environment on distance estimation function, the present application uses the distance true value of cone bucket calibration and the distance estimation function output to compare the differences, the present application uses and to analyze the influence value of rain and fog environment on distance estimation function , the expression is:

[0101]

[0102] (2.4) Multi-level impact analysis method

[0103] After getting the rain and fog environment impact analysis results of the original data layer, image feature layer and recognition function layer of intelligent vehicle vision sensor, first, the analysis results obtained by each level are normalized, and the result range is normalized to , then the three level impact results are added together to get the analysis results considering three influence factors at the same time, in order to make the expression more clear, the multi-level impact analysis results of rain and fog environment on intelligent vehicle vision sensor are obtained by using normalization processing method again, the method flow is shown in Figure 3 .

[0104] Impact analysis experiment and result description

[0105] The rain and fog weather environment simulation platform is used to collect rain and fog free images Im r at different distances of [10 m, 350 m], and different rain intensities or fog visibility are set to collect rain and fog image data Im s at different target distances, part of the image collection results are shown in Figure 4 . Then Im r and Im s are input into the impact analysis method of the present application, and the analysis results of each level are shown in Figures 5-20 .

[0106] For the original data level impact analysis, PSNR, SSIM and Tenengrad three indexes comprehensively quantify the visual image quality degradation trend under rain and fog environment, as shown in Figures 5-10As shown, in the rainfall scenario, as the rainfall intensity increased from 30 mm / h to 90 mm / h, and the target distance increased from 10 m to 50 m, the image PSNR value decreased from approximately 16.3 dB to a minimum of around 12.6 dB, indicating a significant deterioration in the image signal-to-noise ratio; the SSIM value gradually decreased from 0.7582 to 0.6686, reflecting a gradual loss of image structural consistency; and the Tenengrad value plummeted from approximately 160,000 to below 70,000, indicating severe degradation of image edge details and sharpness. In the fog scenario, the impact on image quality was even more pronounced. As fog visibility decreased from 200 m to 30 m, and the target distance increased, the PSNR value decreased from approximately 20.0 dB to 12.3 dB, the SSIM value decreased from 0.7965 to 0.6806, and the Tenengrad value decreased from approximately 160,000 to below 60,000. When visibility in fog is less than 100 m, all image quality indicators drop rapidly, especially at the distance of distant targets where image quality degrades drastically.

[0107] In image feature level analysis, rain and fog environments significantly interfere with the local feature extraction capabilities of intelligent vehicle vision sensors, such as... Figures 11-14 As shown. This embodiment uses... and The performance changes of two typical feature extraction algorithms under severe weather conditions were quantified. In the rainfall scenario, as the rainfall intensity increased from 30 mm / h to 90 mm / h, and the target distance increased, The percentage dropped rapidly from a high of approximately 62% to a low of less than 20%, exhibiting a clear non-linear decay trend, indicating that raindrop interference has a significant impact on corner-based fast feature extraction methods; meanwhile, The percentage also dropped from over 40% to approximately 21%, indicating that even the SIFT algorithm, which possesses scale invariance and rotation robustness, struggles to cope with image blurring and reduced contrast caused by heavy rain. In foggy scenes, feature extraction capabilities are even more significantly affected. The retention rate dropped from over 60% to about 22%, and there was a precipitous decline when fog visibility was below 100 m; The similar decrease in accuracy from 41% to around 20% indicates that under dense fog conditions, image contrast and texture information are severely lost, making feature point extraction ineffective. Overall, both feature extraction algorithms exhibit a significant performance degradation trend in rainy and foggy environments, with the degree of degradation intensifying with increasing weather intensity and target distance. These results demonstrate that rainy and foggy environments significantly weaken the stability of local image features, impacting the performance of visual perception systems in specific intelligent vehicle driving tasks.

[0108] In the identification function level analysis, the rain and fog environment has a particularly significant impact on the target identification and distance estimation functions of the intelligent vehicle vision sensor. Figures 15-18 It is shown that With the change of meteorological intensity and target distance. In the rain scenario, as the rainfall intensity increases from 30 mm / h to 90 mm / h, from 0.91 to a minimum of about 0.49, showing a double attenuation trend with the increase of target distance and rainfall. This shows that under the conditions of heavy rain and long distance, the target recognition ability is significantly weakened. At the same time, a non-linear fluctuation upward trend, the error value has increased significantly in the moderate rain stage, and under the conditions of heavy rain and long distance target, the maximum error is close to 0.9, indicating that the vision sensor is difficult to provide reliable distance information. In the fog scenario, as the fog visibility decreases from 200 m to 30 m, sharp decline, from 0.31 to about 0.07, and the decline trend is particularly steep when the fog visibility is less than 100 m, indicating that thick fog has a suppressive effect on target detection ability. At the same time, the error rises rapidly, especially under the combined conditions of low visibility and target distance greater than 30 m, the error value exceeds 0.5. In summary, the rain and fog environment will seriously affect the function of the vision sensor. The analysis results show that in the actual driving scenario, rain and fog weather will directly threaten the functional safety of the vision perception system, and must be compensated and optimized through multi-sensor fusion or robust algorithm.

[0109] Figures 19-20 It is shown that the multi-level influence analysis results of the intelligent vehicle vision sensor in two typical environments of rain and fog. The figure fully reflects the comprehensive performance degradation trend of the vision sensor under different meteorological intensity and target distance conditions. In the rain scenario, the performance score decreases significantly as the rainfall intensity and target distance increase. The normalized index quickly drops from the ideal state of close to 1.0 as the rain intensifies, and under the conditions of heavy rain and target distance exceeding 40 m, the comprehensive performance has fallen below 0.1. In the fog scenario, the comprehensive performance is highly related to the visibility, and when the fog visibility decreases from 200 m to 50 m, the comprehensive index gradually decreases from 0.9 to below 0.25; when the target distance is further extended to more than 50 m, the performance drops to about 0.05. The overall surface presents a steep slope from the upper right to the lower left, showing the combined attenuation effect of thick fog and long distance on the vision sensor.

[0110] Overall, the multi-level influence analysis results of the present application clearly reveal the attenuation effect of rain and fog environment on the performance of visual sensor. The method effectively integrates the three dimensions of raw data, image features and specific functions, making the performance evaluation of visual sensor in rain and fog environment more comprehensive and accurate, and providing data basis and method support for the robustness optimization and test verification of visual perception system.

Claims

1. A method for analyzing the impact of rain and fog on intelligent vehicle vision sensors, characterized by the following steps: include: Step 1: Construct a test scenario based on a closed-site rain and fog meteorological environment simulation platform. Place a static vehicle target object. A vehicle equipped with a vision sensor gradually approaches the target object from a certain distance at a certain speed. Collect rain and fog images and rain and fog images corresponding to different vehicle target object distances through the vision sensor. At the same time, record the distance between the vehicle target object and the vehicle in the calibration image. Repeat the collection of data pairs [rain and fog meteorological parameters - target object distance - image information affected by meteorological conditions] by changing different rain and fog meteorological parameters. Step 2: Quantify the impact of rain and fog environment on the raw image information output by intelligent vehicle vision sensor through raw data hierarchical impact analysis, including full reference index and no reference index; The full reference index uses images free from rain and fog as a benchmark and is calculated using two methods: peak signal-to-noise ratio (PSNR) and image structure similarity index (SSIM). The no-reference index uses the Tenengrad gradient evaluation index, which consists of the horizontal and vertical gradient values ​​of pixels. Linear interpolation is used to construct the original data hierarchical impact analysis results. Then, through image feature hierarchy influence analysis, the ORB algorithm is applied to describe the keypoint number decay rate, and the ORB keypoint number retention rate under different target object distances and meteorological parameters is calculated; the SIFT algorithm is used to calculate the keypoint number retention rate caused by rain and fog. Then, through the hierarchical impact analysis of the identification function, the impact of rain and fog on the target identification function is analyzed by selecting two functions: target identification and distance estimation. Among them, the YOLO series algorithm is used as the object of the impact analysis of rain and fog on the target identification function. The impact analysis of the distance estimation function uses the difference between the true distance value calibrated by the cone and the output value of the distance estimation function to analyze the impact of the rain and fog environment on the distance estimation function. After obtaining the impact analysis results of rain and fog environment on the raw data layer, image feature layer and recognition function layer of intelligent vehicle vision sensor, the analysis results obtained separately for each layer are first normalized. Then, the impact results of the three layers are added together to obtain the analysis result that considers the three influencing factors at the same time. The normalization method is used again to obtain the final multi-level impact analysis results of rain and fog environment on intelligent vehicle vision sensor.

2. The method for analyzing the impact of rain and fog on intelligent vehicle vision sensors according to claim 1, characterized in that: In step 1, the collected rain and fog image data includes rainfall images and fog images.

3. The method for analyzing the impact of rain and fog on intelligent vehicle vision sensors according to claim 1, characterized in that: In step 2, the original data hierarchy influence analysis, the PSNR is defined based on the image mean square error (MSE). Given a rain- and fog-free image Im of size M*N... r and rain and fog image data Im s Its MSE is defined as: In the formula, MAX represents the maximum pixel value in the image, and SSIM is defined as: In the formula, x and y are Im and y, respectively. r and Im s pixel intensity, , These represent the means of x and y, respectively. and These represent the variances of x and y, respectively. It is the covariance of x and y; is a constant that maintains the stability of the SSIM formula; Q is the dynamic range of the image; k1 and k2 are constants.

4. The method for analyzing the impact of rain and fog on intelligent vehicle vision sensors according to claim 1, characterized in that: In step 2, the original data hierarchical impact analysis is performed, given a rain-free and fog-free impact image Im of size M*N. r and rain and fog image data Im s The Tenengrad gradient evaluation index is expressed as follows: In the formula, For pixels The gradient at point is specifically expressed as: In the formula, and These are the gradient values ​​in the horizontal and vertical directions of the pixel, respectively, and their specific expressions are as follows: In the formula, For the input image, S represents a two-dimensional convolution operation. x and S y These represent the values ​​of the Sobel operator in the horizontal and vertical directions, respectively. The higher the Tenengrad value of the entire image, the higher the overall edge contrast, the clearer the details, and the more accurate the focus.

5. The method for analyzing the impact of rain and fog on intelligent vehicle vision sensors according to claim 1, characterized in that: In step 2, the hierarchical impact analysis of the original data, under the condition of the same target distance, the three indices PSNR, SSIM, and Tenengrad are calculated for each image combination. The target distance and meteorological parameters are used as independent variables, and PSNR, SSIM, and Tenengrad are used as dependent variables. The results of the hierarchical impact analysis of the original data are constructed by linear interpolation.

6. The method for analyzing the impact of rain and fog on intelligent vehicle vision sensors according to claim 1, characterized in that: In step 2, the image feature hierarchy influence analysis first selects an image Im with no rain or fog influence at the same distance from the target object. r and rain and fog image data Im s Key points were extracted using ORB and SITF algorithms, respectively. Then, record the image Im without the influence of rain or fog. r The total number of key points detected in the middle is Im s The total number of key points detected in the middle is The ORB keypoint retention rate was calculated under different target distances and meteorological parameters. : Within the same experimental framework, the keypoint retention rate due to rain and fog weather was calculated using the SIFT algorithm. To evaluate the impact of rain and fog environments on the detection performance of scale- and rotation-invariant features, the expression is: In the formula, and Representing Im r and Im s The corresponding SIFT key point recognition results.

7. The method for analyzing the impact of rain and fog on intelligent vehicle vision sensors according to claim 1, characterized in that: In step 2, the impact analysis of the target recognition function first obtains the image Im, which is free from the influence of rain and fog. r The ground truth value (GT) of the bounding box of the target in each frame of the image; using the YOLO algorithm to analyze the rain and fog image data. s The detection is performed to obtain the detection bounding box D, and the target recognition algorithm is evaluated using the IOU method: This is a metric for evaluating the accuracy of target recognition algorithms. The output is a value between 0 and 1; a larger value indicates a more accurate target recognition algorithm. r and Im s The image is used as input to YOLO-V5, and the corresponding output is... Defined as the image output value without the influence of rain or fog. Rain and fog image data output values ; The impact of rain and fog on target recognition function was analyzed using the degree of recognition reduction. The expression is: In the impact analysis of the distance estimation function, the difference between the true distance value calibrated by the cone and the output value of the distance estimation function was compared. and Analysis of the impact of rain and fog on distance estimation function Its expression is: in, The true value of the distance calibrated for the cone. Output value for the distance estimation function.

Images