Snowy day digital twinning scene fusion construction method in automatic driving sample library

Abstract

The invention discloses a snowy day digital twinborn scene fusion construction method in an automatic driving sample library. The method comprises the following steps: acquiring an unprocessed image in an original picture data set in clear weather; an accumulated snow plane construction algorithm based on spatial perception is obtained; obtaining a snowy day scene construction algorithm based on a random snow line model; and fusing an accumulated snow plane construction algorithm based on spatial perception and a snowy day scene construction algorithm based on a random snow line model, and applying the fused algorithm to the obtained image to obtain a result image after snow adding. According to the invention, the snow accumulation effect and the snowflake falling effect can be effectively simulated, scene construction from a sunny condition to a snowy condition is realized, and a high-quality test sample library is constructed.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a method for fusion construction of snowy digital twin scenes in an autonomous driving sample library. Background Art

[0002] Snowy scenes are different from other scenes. Rainy and foggy weather causes the loss of scene information through refraction or reflection of water mist particles in the air, while snowy scenes directly block the propagation of light through snowflakes. At the same time, they will directly cover the plane, directly hindering the observation of the target. At present, there are few relevant literatures on snowy sample libraries. The existing snowy sample library research focuses on the snowline model. Liu Yunfu et al. proposed the Snow-100k sample library, which includes synthetic sample data and real data sets. It provides 100,000 synthetic snowy scene sample pairs and 1,329 real snowy scene samples. Chen Weiting et al. proposed the SRRS (Snow Removal in Realistic Scenario) sample library, which contains 15,000 synthetic data and 1,000 real data from the Internet. The synthetic original data set is taken from the RESIDE data set. Chen Weiting et al. proposed the CSD (Comprehensive Snow Dataset) sample library, which contains 10,000 synthetic images, also selected from the RESIDE data set, and synthesized the snowline data through tools such as PhotoShop. Chen Weiting et al. proposed the RVSD (Realistic Video DeSnowing Dataset) video sample library, which contains 110 sets of snowy video data, each of which contains snow, fog, and clear scene data. It uses a rendering engine and various enhancement techniques to generate snow with multiple realistic physical properties. Zhang Kaihao et al. used Photoshop software to establish the SnowKITTI2012 dataset based on the public dataset KITTI2012, which contains 1,500 training images and 1,000 test images. In addition, they also proposed the SnowCityScapes dataset based on the Cityscapes dataset, which contains 2,000 training and test images. The above sample libraries have poor control over the amount of snow, or the image generation process is more complicated and takes a long time to generate. Summary of the invention

[0003] The purpose of the present invention is to provide a method for fusion construction of snowy digital twin scenes in an autonomous driving sample library to solve the problems existing in the prior art. The present invention studies the snowy scenes in the sample library of an autonomous driving visual perception test system, and proposes a foggy scene construction algorithm based on depth of field estimation that combines a snow area with a snow line model. The algorithm can effectively simulate the effects of snow accumulation and falling snowflakes, realize scene construction from clear conditions to snowy conditions, and construct a high-quality test sample library.

[0004] In order to achieve the above object, the present invention adopts the following technical scheme:

[0005] The method for fusion construction of snowy digital twin scenes in the autonomous driving sample library includes the following steps:

[0006] Step 1: Obtain the images in the original image dataset under unprocessed sunny weather;

[0007] Step 2: Obtain a snow plane construction algorithm based on spatial perception;

[0008] Step 3: Obtain a snow scene construction algorithm based on a random snow line model;

[0009] Step 4: The algorithms obtained in step 3 and step 4 are combined and applied to the image obtained in step 1 to obtain the result after adding snow (Figure I). s .

[0010] Furthermore, in step 2, the snow plane construction algorithm based on spatial perception is obtained, including the following steps:

[0011] Step 2.1: Sense the area where snow can accumulate and obtain the area where snow scene construction is required;

[0012] Step 2.2: Refine the region according to the snow-covered area and construct the snow scene to obtain a snow image.

[0013] Furthermore, in step 2.1, the area where snow can accumulate is perceived, and the area where snow accumulation scene needs to be constructed is obtained, as follows:

[0014] Extract the area with smaller value fluctuation in the image as the snow accumulation area. First, obtain the image gradient information:

[0015]

[0016] in, are the gradient amplitudes of the c channel image in the x and y directions respectively; G is the gradient value;

[0017] Select the area with smaller fluctuations, expressed as:

[0018] Map s =|G| <Gl

[0019] Among them, Map s For areas where snow can accumulate; G l is the fluctuation threshold;

[0020] Eliminate the influence of the sky area to simplify the solution process, Map s It is expressed as:

[0021] Map s =(G <G l )∩(I B <(I R +I G ) / 2+ω)

[0022] Among them, I B , I G , I R They are the image RGB channels respectively; is the color deviation parameter.

[0023] Furthermore, the color deviation parameter ω takes a value of 0.1.

[0024] Furthermore, in step 2.2, the region is refined according to the snow-covered area and a snow scene is constructed to obtain a snow image, including:

[0025] (1) Calculate Map s Connected domain, and sort them in descending order according to the connected domain, and record the coordinates in the connected domain as O c , the number of pixels is N c ;

[0026] (2) Calculate the area utilization rate within each connected domain, v i =N c / (max(O c )-min(O c ));

[0027] (3) Take the utilization rate greater than v i The area is used as the refined image Map' s , which is the refined snow accumulation area;

[0028] Get the refined map s 'After the image is obtained, the snow area is fused with the original image to obtain the snow scene image. The fusion method is as follows:

[0029] I S_G =J×Map s ′+(1-Map s ′)

[0030] Among them, I S_GFor snow image, Map s ' is the refined snow area.

[0031] Furthermore, the utilization rate v i The value of is 0.5.

[0032] Furthermore, in step 3, a snow scene construction algorithm based on a random snow line model is obtained, including the following steps:

[0033] Step 3.1: Design a single-layer random snowline generation algorithm;

[0034] Step 3.2: Merge multiple single-layer random snow lines to form a multi-layer random snow line scene.

[0035] Furthermore, a single-layer random snowline generation algorithm is designed in step 3.1, as follows:

[0036] First, the snow line distribution position is determined by the snow line mask, and then the snow line filter is designed as follows:

[0037] S S_i =SnowFilter(R s ,Map)

[0038] Among them, S S_i is the snow line layer; SnowFilter is the snow line filter; R s is the filtered snow core and the basic snow line shape; Map is the snow line distribution map.

[0039] Furthermore, in step 3.2, multiple single-layer random snow lines are fused to form a multi-layer random snow line scene:

[0040]

[0041] Among them, S S This is the snow line effect after multi-source fusion. is the cumulative effect of multiple layers of snow lines, θ r is a random angle, δ si is the snow line density of the i-th layer, n δ The number of layers.

[0042] Furthermore, in step 4, the obtained algorithm is applied to the image, specifically:

[0043]

[0044] Among them, I s is the image after adding snow, J is the original image, J×Map' s +(1+Map' s ) is the spatially aware snow plane construction algorithm in step 3, Build an algorithm for the snowy scenario for the random snowline model in step 4.

[0045] Compared with the prior art, the present invention has the following beneficial technical effects:

[0046] The present invention can effectively achieve the effect of adding snow lines of different concentrations to the picture, complete the mapping process in the digital twin, and construct the corresponding scene. The main idea of ​​the present invention is to construct the snow scene based on the detection and refinement of the snow-covered area. First, the image gradient information is obtained, and the area with small gradient fluctuations is screened out. The influence of the sky area is eliminated in combination with the color information, and the area is refined to obtain the snow-covered area. Then, a multi-source and multi-layer fused snow line layer is used to simulate the real snowy conditions. The two are combined to obtain a snow-added image. The present invention can achieve the effect of adding different amounts of snow to the image. The image after snow addition gradually blurs as the snow line density increases, and the snow-added effect obtained is no different from the actual one. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings in the specification are used to provide further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0048] Figure 1 It is a schematic diagram of the process of the present invention;

[0049] Figure 2(a) is the 11th frame image in the dataset, as the original image. Figure 2(b) to Figure 2(f) To set the snow line density δ for Figure 2(a) s =The processing results of 10, 20, 30, 40, 50;

[0050] Figure 3(a) is the 41st frame image in the dataset, as the original image. Figure 3(b) to Figure 3(f) To set the snow line density δ for Figure 3(a) s =The processing results of 10, 20, 30, 40, 50;

[0051] Figure 4(a) is the 71st frame image in the dataset, as the original image. Figure 4(b) to Figure 4(f) To set the snow line density δ for Figure 4(a) s =The processing results of 10, 20, 30, 40, 50;

[0052] Figure 5(a) is the 101st frame image in the dataset, which is the original image. Figure 5(b) to Figure 5(f) To set the snow line density δ for Figure 5(a) s =The processing results of 10, 20, 30, 40, 50;

[0053] Figure 6(a) is the 11th frame image in the dataset, which is used as the original image. Figure 6(b) to Figure 6(f)After adding snow to Figure 6(a), the snow line density δ is set s =The processing results of 10, 20, 30, 40, 50;

[0054] Figure 7(a) is the 41st frame image in the dataset, which is used as the original image. Figure 7(b) to Figure 7(f) After adding snow to Figure 7(a), the snow line density δ is set s =The processing results of 10, 20, 30, 40, 50;

[0055] Figure 8(a) is the 71st frame image in the dataset, which is used as the original image. Figure 8(b) to Figure 8(f) After adding snow to Figure 8(a), the snow line density δ is set s =The processing results of 10, 20, 30, 40, 50;

[0056] Figure 9(a) is the 101st frame image in the dataset, which is the original image. Figure 9(b) to Figure 9(f) After adding snow to Figure 9(a), the snow line density δ is set s =The processing results of 10, 20, 30, 40, 50;

[0057] Figure 10(a)-Figure 10(c) These are the corresponding information entropy, average gradient and FADE value curves for the non-parameter evaluation indicators, where the solid line is the result of the snow addition experiment and the dotted line is the result of the snow accumulation experiment.

[0058] Figure 10(d)-Figure 10(f) The following are the corresponding evaluation indicators PSNR, SSIM and fuzzy value curves. DETAILED DESCRIPTION

[0059] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0060] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0061] See also Figure 1 , the present invention designs a snow plane construction algorithm based on spatial perception and a snow scene construction algorithm based on a random snow line model, and proposes an algorithm for generating snow areas through perception and refinement of snow-covered areas. Secondly, according to the snow line model, a snow line model based on random snow line generation is designed, and the distribution positions of snowflakes are randomly distributed in the image. Applying the proposed snow accumulation model and snow line model to sunny images, it can be concluded that the image is close to the real situation in both subjective perception and objective evaluation, and the image quality is significantly reduced.

[0062] The specific steps include:

[0063] Step 1: Obtain the images in the original image dataset under unprocessed sunny weather;

[0064] Step 2: Obtain a snow plane construction algorithm based on spatial perception;

[0065] Step 3: Obtain a snow scene construction algorithm based on a random snow line model;

[0066] Step 4: The algorithms obtained in step 3 and step 4 are combined and applied to the image obtained in step 1 to obtain the result after adding snow (Figure I). s ;

[0067] Wherein, in step 2, a snow plane construction algorithm based on spatial perception is obtained to obtain an area where snow scene construction is required, including the following steps:

[0068] 2.1 Snow-covered area perception:

[0069] The area with smaller value fluctuation in the image is extracted as the snow accumulation area. First, obtain the image gradient information:

[0070]

[0071] in, are the gradient amplitudes of the c channel image in the x and y directions respectively; G is the gradient value. Select the area with smaller fluctuations, that is:

[0072] Map s =|G| <G l

[0073] Among them, Map s For areas where snow can accumulate; G l is the fluctuation threshold.

[0074] Eliminating the influence of the sky area simplifies the solution process, and Maps can be expressed as:

[0075] Map s =(G <G l )∩(I B <(I R +I G ) / 2+ω)

[0076] Among them, I B , I G , I R They are the image RGB channels respectively; is the color deviation parameter.

[0077] 2.2 Refine the area according to the snow-covered area and construct the snow scene to obtain the snow image:

[0078] Map s Further refinement to obtain the snow area:

[0079] (1) Calculate Map s Connected domain, and sort them in descending order according to the size of the connected domain, and record the coordinates in the connected domain as O c , the number of pixels is N c ;

[0080] (2) Calculate the area utilization rate within each connected domain, v i =N c / (max(O c )-min(O c ));

[0081] (3) Take the utilization rate greater than v i Area, as a refined Map' s The image is the refined snow-accumulating area.

[0082] Get the refined map s 'After the image is obtained, the snow area is fused with the original image to obtain the snow scene image. The fusion method is as follows:

[0083] I S_G =J×Maps ′+(1-Map s ′)

[0084] Among them, I S_G For snow image, Map s ' is the refined snow area.

[0085] The step 3 obtains a snow scene construction algorithm based on a random snow line model, including the following steps:

[0086] Step 3.1: Design a single-layer random snowline generation algorithm:

[0087] First, the snow line distribution position is determined by the snow line mask, and then the snow line filter is designed as follows:

[0088] S S_i =SnowFilter(R s ,Map)

[0089] Among them, S S_i is the snow line layer; SnowFilter is the snow line filter; R s is the filtered snow core and the basic snow line shape; Map is the snow line distribution map.

[0090] Step 3.2: Merge multiple single-layer random snow lines to form a multi-layer random snow line scene:

[0091] Simulate snow lines at different depths of field by merging multiple layers of snow lines:

[0092]

[0093] Among them, S S This is the snow line effect after multi-source fusion. is the cumulative effect of multiple layers of snow lines, θ r is a random angle, δ si is the snow line density of the i-th layer, n δ The number of layers.

[0094] Step 4: Blend the snow area with the multi-layer random snow line scene:

[0095] Apply the algorithm obtained in steps 2 and 3 to the images in the dataset in step 1:

[0096]

[0097] Among them, I s is the image after adding snow, and J is the original image. J×Map' s +(1+Map' s ) is the spatially aware snow plane construction algorithm in step 3, Build an algorithm for the snowy scenario for the random snowline model in step 4.

[0098] Figure 2(a), Figure 3(a), Figure 4(a), Figure 5(a), Figure 6(a), Figure 7(a), Figure 8(a), Figure 9(a) are all original images without processing. Figure 2(b)-Figure 2(f) , Figure 3(b)-Figure 3(f) , Figure 4(b)-Figure 4(f) , Figure 5(b)-Figure 5(f) By adding pictures with different snow line densities to different frames, it can be seen that as the snow line density increases, the information covered by snow in the image gradually increases, gradually approaching the real snowy scene. Figure 6(b)-Figure 6(f) , Figure 7(b)-Figure 7(f) , Figure 8(b)-Figure 8(f) , Figure 9(b)-Figure 9(f) The pictures with snow and different snow line densities added to different frames show that the road surface is almost covered by the added snow after adding snow, which is in line with the actual situation. After adding snow lines of different densities, the real snowy scene can be simulated well. Figure 10(a)-(c) shows the information entropy, average gradient and FADE value curves of the corresponding non-parametric evaluation indicators. Among them, the solid line is the result of the snow adding experiment, and the dotted line is the result of the snow accumulation experiment. Affected by the snow line, the image information entropy continues to decrease, and the FADE continues to increase. Due to the small overall coverage area of ​​the snow line, the average gradient does not decrease significantly. Since there is no effective information and fluctuation in the snow-covered area, the information entropy and FADE are significantly reduced, while the edge area changes greatly compared to the original image, resulting in a significant increase in the average gradient. Figure 10(d)-(f) shows the corresponding PSNR, SSIM and fuzzy value curves of the parameter evaluation indicators. Both PSNR and SSIM decrease with the increase of snow line density, and the fuzzy coefficient increases continuously. Due to the influence of snow accumulation, the influence of the snow line decreases, resulting in slow changes in PSNR, SSIM and fuzzy coefficient under snow conditions. It can be seen from Table 1 and Figure 10 that as δ s As the information entropy, average gradient, PSNR, and SSIM gradually decrease, the FADE and fuzzy coefficients continue to increase, which is consistent with the actual situation. Therefore, after the processing of the present invention, the snow scene effects under different snow amounts can be effectively obtained.

[0099] Table 1n d Objective indicators of change

[0100]

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims of the invention.

Claims

1. A method for fusion construction of snowy digital twin scenes in an autonomous driving sample library, characterized in that: The following steps are involved: Step 1: Obtain the images in the original image dataset under unprocessed sunny weather; Step 2: Obtain a snow plane construction algorithm based on spatial perception; Step 3: Obtain a snow scene construction algorithm based on a random snow line model; Step 4: The algorithms obtained in step 3 and step 4 are combined and applied to the image obtained in step 1 to obtain the result after adding snow (Figure I). s .

2. The method for fusion construction of snowy digital twin scenes in the autonomous driving sample library according to claim 1 is characterized in that: In step 2, the snow plane construction algorithm based on spatial perception is obtained, which includes the following steps: Step 2.1: Sense the area where snow can accumulate and obtain the area where snow scene construction is required; Step 2.2: Refine the region according to the snow-covered area and construct the snow scene to obtain a snow image.

3. The method for constructing snowy digital twin scenes in the autonomous driving sample library according to claim 2 is characterized in that: In step 2.1, the snow-covered area is perceived to obtain the area where the snow scene needs to be constructed, as follows: Extract the area with smaller value fluctuation in the image as the snow accumulation area. First, obtain the image gradient information: in, are the gradient amplitudes of the c channel image in the x and y directions respectively; G is the gradient value; Select the area with smaller fluctuations, expressed as: Map s =|G|<G l Among them, Map s For areas where snow can accumulate; G l is the fluctuation threshold; Eliminate the influence of the sky area to simplify the solution process, Map s It is expressed as: Map s =(G<G l )∩(I B <(I R +I G ) / 2+ω) Among them, I B , I G , I R They are the image RGB channels respectively; is the color deviation parameter.

4. The method for fusion construction of snowy digital twin scenes in the autonomous driving sample library according to claim 3 is characterized in that: The color deviation parameter The value is 0.

1.

5. The method for constructing snowy digital twin scenes in the autonomous driving sample library according to claim 3 is characterized in that: In step 2.2, the region is refined according to the snow-covered area and the snow scene is constructed to obtain a snow image, including: (1) Calculate Map s Connected domain, and sort them in descending order according to the connected domain, and record the coordinates in the connected domain as O c , the number of pixels is N c ; (2) Calculate the area utilization rate within each connected domain, v i =N c / (max(O c )-min(O c )); (3) Take the utilization rate greater than v i The area is used as the refined image Map' s , which is the refined snow accumulation area; Get the refined map s 'After the image is obtained, the snow area is fused with the original image to obtain the snow scene image. The fusion method is as follows: I S_G =J×Map′ s +(1-Map′ s ) Among them, I S_G For snow image, Map s ' is the refined snow area.

6. The method for fusion construction of snowy digital twin scenes in the autonomous driving sample library according to claim 5 is characterized in that: The utilization rate v i The value of is 0.

5.

7. The method for fusion construction of snowy digital twin scenes in the autonomous driving sample library according to claim 1 is characterized in that: In step 3, a snow scene construction algorithm based on a random snow line model is obtained, which includes the following steps: Step 3.1: Design a single-layer random snowline generation algorithm; Step 3.2: Merge multiple single-layer random snow lines to form a multi-layer random snow line scene.

8. The method for constructing a snowy digital twin scene fusion in an autonomous driving sample library according to claim 7, characterized in that: In step 3.1, a single-layer random snowline generation algorithm is designed as follows: First, the snow line distribution position is determined by the snow line mask, and then the snow line filter is designed as follows: S S_i =SnowFilter(R s ,Map) Among them, S S_i is the snow line layer; SnowFilter is the snow line filter; R s is the filtered snow core and the basic snow line shape; Map is the snow line distribution map.

9. The method for fusion construction of snowy digital twin scenes in the autonomous driving sample library according to claim 8 is characterized in that: In step 3.2, multiple single-layer random snow lines are merged to form a multi-layer random snow line scene: Among them, S S This is the snow line effect after multi-source fusion. is the cumulative effect of multiple layers of snow lines, θ r is a random angle, δ si is the snow line density of the i-th layer, n δ The number of layers.

10. The method for constructing snowy digital twin scenes in the autonomous driving sample library according to claim 9, characterized in that: In step 4, the obtained algorithm is applied to the image, specifically: Among them, I s is the image after adding snow, J is the original image, J×Map' s +(1+Map' s ) is the spatially aware snow plane construction algorithm in step 3, Build an algorithm for the snowy scenario for the random snowline model in step 4.

Images