Sand-dust weather digital twinborn scene fusion construction method in automatic driving sample library

Abstract

The invention discloses a sand and dust weather digital twinborn scene fusion construction method in an automatic driving sample library. The method comprises the following steps: acquiring an unprocessed image in an original picture data set in sunny weather; obtaining a sky dust scene reconstruction algorithm based on color deflection; obtaining an atmospheric dust model construction algorithm based on combined depth information; and fusing the sky dust scene reconstruction algorithm and the atmosphere dust model construction algorithm, and applying the fused algorithm to the obtained image to obtain an image added with dust. According to the invention, the severe environment of sand and dust weather is studied, the weather change degree is controlled, and the diversification of the database is realized.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a method for fusion construction of sandstorm weather digital twin scenes in an autonomous driving sample library. Background Art

[0002] Sandstorms have not received widespread attention. On the one hand, they occur mostly in inland arid areas, and are uncommon in developed coastal areas, where they are less affected. On the other hand, sandstorms are characterized by suddenness and short duration, and they often occur in spring and winter.

[0003] In 2022, Si Yazhong et al. proposed the SIRB (Sand-dust Image Reconstruction Benchmark) sample library, which contains 16,000 synthetic data and 230 real data, simulating images under different dust intensities. In 2023, Liu et al. proposed the RUSH (Real Underwater, Sand dust, and Hazy images) sample library, which collected 250 images from the Internet, including 50 underwater images, 100 foggy images, and 100 sand and dust images. However, only bad weather images were collected, and there were no corresponding control images, which made it difficult to use in deep learning network training.

[0004] With the emergence of large models, multi-task and multi-modal research has become a trend. Research on the clarity of severe weather has gradually developed from single-task to multi-task, from single-image processing to video processing, and from traditional methods to deep learning. In the face of increasingly complex task requirements, sufficient data support has become the primary condition.

[0005] At present, the dust model has the following problems: there are very few types of algorithms, that is, there is little research in this field, which leads to insufficient number of available samples, making it difficult to support the current controlled experiments in autonomous driving scenarios. At the same time, the existing algorithms are still low in actual restoration and have a large amount of calculations, which cannot truly reflect the actual situation. Summary of the invention

[0006] The purpose of the present invention is to provide a method for fusion construction of digital twin scenes of sandstorm weather in an autonomous driving sample library to solve the problems existing in the prior art. The present invention studies the harsh environment of sandstorm weather, controls the degree of weather changes, and realizes the diversification of the database.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme:

[0008] The method for fusion construction of sandstorm weather digital twin scenes in the autonomous driving sample library includes the following steps:

[0009] Step 1: Obtain the images in the original image dataset under unprocessed sunny weather;

[0010] Step 2: Obtain a sky dust scene reconstruction algorithm based on color deflection;

[0011] Step 3: Obtain an algorithm for constructing an atmospheric dust model based on combined depth information;

[0012] Step 4: The algorithm obtained in step 3 is combined with the algorithm obtained in step 4, and applied to the image obtained in step 1 to obtain an image with added dust.

[0013] Furthermore, the sky dust scene reconstruction algorithm based on color deflection is obtained in step 2, including the following steps:

[0014] Step 2.1: Extract the sky area;

[0015] Step 2.2: Perform regional reconstruction based on color deflection on the extracted sky area to obtain a sky dust scene reconstruction algorithm based on color deflection.

[0016] Furthermore, the sky area is extracted in step 2.1, and the specific process is as follows:

[0017] Extract the sky area according to the following rules:

[0018] Map sky =I B <(I R +I G ) / 2+ω

[0019] Among them, Map sky is the extracted sky area, I R , I B , I G are the red, blue and green channel images of the image respectively, and ω is the color deviation parameter;

[0020] The above rules are further refined by analyzing the graph and using the maximum connected domain method to map sky To refine:

[0021] Map sky =C max (I B <(I R +I G ) / 2+ω)

[0022] Among them, C max The maximum connected domain calculation method eliminates the influence of the sky area and simplifies the solution process. sky The corresponding image part is the sky area image I sky .

[0023] Furthermore, the value of the color deviation parameter ω is 0.1.

[0024] Furthermore, in step 2.2, the extracted sky area is reconstructed based on color deflection, specifically:

[0025] The sky area image is transformed from RGB to HSV. The formula for RGB to HSV color transformation is:

[0026] I V =max(I skyR ,I skyG ,I skyB )

[0027]

[0028] Among them I skyR , I skyB , I skyG are the red, blue and green channel images of the sky area image, I V As the image brightness channel, take the maximum value of the image RGB; I S As a saturation channel, it indicates the change between the pixel RGB values. The larger the value, the brighter the image. H is the hue channel;

[0029] By controlling I H Achieve color deflection and control I S The transformation controls the color change of the sky:

[0030]

[0031] Among them, k s To adjust the parameters and control the image saturation, I' S is the adjusted saturation;

[0032] The H channel is adjusted accordingly with reference to the saturation channel:

[0033] I′ H =I H -180°

[0034] I' H is the adjusted hue;

[0035] The adjusted hue channel, brightness channel and saturation channel are merged to obtain the converted sky reconstructed image I' sky .

[0036] Furthermore, in step 3, an atmospheric dust model construction algorithm based on combined class depth information is obtained, which specifically includes the following steps:

[0037] Step 3.1: Construct a dust scene model based on atmospheric light estimation;

[0038] Step 3.2: Perform image depth simulation on the dust scene model based on atmospheric light estimation.

[0039] Furthermore, in step 3.1, a dust scene model based on atmospheric light estimation is constructed, specifically:

[0040] Reconstructed image of the sky I' sky Perform global atmospheric light estimation and perform scene conversion by controlling the changes in each RGB channel, namely:

[0041] A c =r c ×A 1c

[0042] Among them, A c is the atmospheric light estimation corresponding to any channel, where c is any channel in RGB; A 1c is the global atmospheric light; r c is the conversion ratio, and its formula is as follows:

[0043]

[0044] Among them, I c_ave is the mean value of channel c; k c is the conversion coefficient, which is used to adjust the conversion ratio.

[0045] Furthermore, in step 3.2, image depth simulation is performed on the dust scene model based on atmospheric light estimation, specifically:

[0046] Correct the atmospheric light value through depth-like simulation:

[0047]

[0048] Among them, A c is the atmospheric light estimation corresponding to any channel, i, j are the pixel coordinate indices in the width and height directions of the image respectively, m is the image height, and A'' is the global atmospheric light estimation after transformation and correction.

[0049] Furthermore, in step 4, the algorithm obtained in step 3 and the algorithm obtained in step 4 are integrated and applied to the image obtained in step 1 to obtain an image after adding sand and dust:

[0050]

[0051] Among them, I d (x) is a dust scene image; I' sky The clear image after reconstructing the sky; α sis the dust coefficient; A” is the global atmospheric light estimate after conversion correction, t 0 (x) is the initial transmittance.

[0052] Compared with the prior art, the present invention has the following beneficial technical effects:

[0053] The present invention can effectively achieve the effect of adding different concentrations of dust to the picture and complete the mapping process in the digital twin. The main idea of ​​the present invention is to realize the construction of dust weather images based on sky area reconstruction and atmospheric light estimation conversion. First, the sky area is extracted, and the HSV channel features of the image are used to realize the reconstruction of the sky area. Then, the atmospheric light estimation under sandstorm weather is used to increase the dust effect, and the image-like depth estimation is used to roughly distinguish the near and far views of the image. The two are combined to obtain a dust image. The present invention can achieve the effect of adding different densities of dust to the image. The image after adding dust gradually blurs as the density of the dust increases, and the dust adding effect obtained is no different from the actual one. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings in the specification are used to provide further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0055] Figure 1 It is a schematic diagram of the process of the present invention;

[0056] Figure 2 This is the original picture under clear weather;

[0057] Figures 3 to 7 These are the treatment effect diagrams under different dust conditions;

[0058] in, Figure 3 Reconstruction for the sky area;

[0059] Figure 4 It is 0.1 density dust;

[0060] Figure 5 It is 0.2 density dust;

[0061] Figure 6 It is 0.3 density dust;

[0062] Figure 7 It is 0.4 density dust. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0065] See also Figure 1 The flowchart of the present invention is shown in FIG. The present invention designs a sky dust scene reconstruction model based on color deflection and an atmospheric dust model construction algorithm based on combining depth information, and proposes a sky region reconstruction model by reconstructing the sky region color. Secondly, an atmospheric dust model construction algorithm based on combining depth information is designed according to the class depth information. The proposed sky dust scene reconstruction model based on color deflection and the atmospheric dust model construction algorithm model based on combining depth information are applied to sunny images, and it can be concluded that the image is close to the real situation image in both subjective perception and objective evaluation, and the image quality is significantly reduced.

[0066] The specific steps are as follows:

[0067] Step 1: Obtain the images in the original image dataset under unprocessed sunny weather;

[0068] Step 2: Obtain a sky dust scene reconstruction algorithm based on color deflection;

[0069] Step 3: Obtain an algorithm for constructing an atmospheric dust model based on combined depth information;

[0070] Step 4: The algorithm obtained in step 3 is combined with the algorithm obtained in step 4 and applied to the image obtained in step 1, and the image after adding dust;

[0071] The step 2 of obtaining a sky dust scene reconstruction algorithm based on color deflection includes the following steps:

[0072] Step 2.1: Sky area extraction algorithm:

[0073] The sky area can be extracted according to the following rules:

[0074] Map sky =I B <(I R +I G ) / 2+ω

[0075] Among them, Map sky is the extracted sky area, I R , I B , I G are the red, blue and green channel images of the image respectively, and ω is the color deviation parameter;

[0076] The above rules are further refined. By analyzing the graph, the maximum connected domain method is used to map sky Refine

[0077] Map sky =C max (I B <(I R +I G ) / 2+ω)

[0078] Among them, C max The maximum connected domain calculation method eliminates the influence of the sky area and simplifies the solution process. sky The corresponding image part is the sky area image I sky .

[0079] Step 2.2: Perform color deflection-based regional reconstruction on the extracted sky area:

[0080] In the hue diagram, the blue area in the image mainly changes around 240°, and the yellow area mainly changes around 60°. The formula for RGB to HSV color transformation is:

[0081] I V =max(I skyR ,I skyG ,I skyB )

[0082]

[0083] Among them I skyR , I skyB , I skyG are the red, blue and green channel images of the sky area image, I V As the image brightness channel, take the maximum value of the image RGB; I SAs a saturation channel, it indicates the change between the pixel RGB values. The larger the value, the brighter the image. H The hue channel.

[0084] By controlling I H Achieve color deflection and control I S The transformation controls the color change of the sky:

[0085]

[0086] Among them, k s To adjust the parameters and control the image saturation, I' S is the adjusted saturation.

[0087] The H channel is also adjusted accordingly:

[0088] I′ H =I H -180°

[0089] I' H is the adjusted hue.

[0090] The adjusted hue channel, brightness channel and saturation channel are merged to obtain the converted sky reconstructed image I' sky .

[0091] The step 3 obtains an atmospheric dust model construction algorithm based on combining depth information, including the following steps:

[0092] Step 3.1: Dust scene construction algorithm based on atmospheric light estimation:

[0093] Reconstructed image of the sky I' sky Perform global atmospheric light estimation and perform scene conversion by controlling the changes in each RGB channel, namely:

[0094] A c =r c ×A 1c

[0095] Among them, A c is the atmospheric light estimation corresponding to any channel, where c is any channel in RGB; A 1c is the global atmospheric light; r c is the conversion ratio, and its formula is as follows:

[0096]

[0097] Among them, I c_ave is the mean value of channel c; k c is the conversion coefficient, which is used to adjust the conversion ratio.

[0098] Step 3.2: Perform image-like depth simulation on the dust scene model based on atmospheric light estimation:

[0099] Since the atmospheric light values ​​obtained in step 3.1 are relatively consistent in each part, the atmospheric light values ​​are corrected through quasi-depth simulation based on the actual situation:

[0100]

[0101] Among them, A c is the atmospheric light estimation corresponding to any channel, i, j are the pixel coordinate indices in the width and height directions of the image respectively, m is the image height, and A'' is the global atmospheric light estimation after transformation and correction.

[0102] Step 4: The algorithm obtained in step 3 is combined with the algorithm obtained in step 4 and applied to the image obtained in step 1 to obtain the image after adding dust:

[0103] The method is:

[0104]

[0105] Among them, I d (x) is a dust scene image; I' sky is the reconstructed image of the sky; α s is the dust coefficient; A” is the global atmospheric light estimate after conversion correction, t 0 (x) is the initial transmittance.

[0106] Figure 1 is the algorithm flow chart, Figure 2 Reconstruct the image for the sky area, Figure 3-Figure 7 These are dust pictures with dust densities of 0.1, 0.2, 0.3, and 0.4. It can be seen that after adding dust, the image has improved in terms of loss of texture details, the image has turned yellow overall, and the information loss is serious, which is consistent with actual dust weather.

[0107] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims of the invention.

Claims

1. A method for fusion construction of sandstorm weather digital twin scenes in an autonomous driving sample library, characterized in that: The following steps are involved: Step 1: Obtain the images in the original image dataset under unprocessed sunny weather; Step 2: Obtain a sky dust scene reconstruction algorithm based on color deflection; Step 3: Obtain an algorithm for constructing an atmospheric dust model based on combined depth information; Step 4: The algorithm obtained in step 3 is combined with the algorithm obtained in step 4, and applied to the image obtained in step 1 to obtain an image with added dust.

2. The method for fusion construction of sandstorm weather digital twin scenes in the autonomous driving sample library according to claim 1 is characterized in that: The sky dust scene reconstruction algorithm based on color deflection is obtained in step 2, including the following steps: Step 2.1: Extract the sky area; Step 2.2: Perform regional reconstruction based on color deflection on the extracted sky area to obtain a sky dust scene reconstruction algorithm based on color deflection.

3. The method for fusion construction of sandstorm weather digital twin scenes in the autonomous driving sample library according to claim 2 is characterized in that: Extract the sky area in step 2.

1. The specific process is as follows: Extract the sky area according to the following rules: Map sky =I B <(I R +I G ) / 2+ω Among them, Map sky is the extracted sky area, I R , I B , I G are the red, blue and green channel images of the image respectively, and ω is the color deviation parameter; To further refine the above rules, we use the maximum connected domain method to analyze the map. sky To refine: Map sky =C max (I B <(I R +I G ) / 2+ω) Among them, C max The maximum connected domain calculation method eliminates the influence of the sky area and simplifies the solution process. sky The corresponding image part is the sky area image I sky .

4. The method for fusion construction of sandstorm weather digital twin scenes in the autonomous driving sample library according to claim 3 is characterized in that: The value of the color deviation parameter ω is 0.

1.

5. The method for fusion construction of sandstorm weather digital twin scenes in the autonomous driving sample library according to claim 3 is characterized in that: In step 2.2, the extracted sky area is reconstructed based on color deflection, specifically: The sky area image is transformed from RGB to HSV. The formula for RGB to HSV color transformation is: I V =max(I skyR ,I skyG ,I skyB ) Among them I skyR , I skyB , I skyG are the red, blue and green channel images of the sky area image, I V As the image brightness channel, take the maximum value of the image RGB; I S As a saturation channel, it indicates the change between the pixel RGB values. The larger the value, the brighter the image. H is the hue channel; By controlling I H Achieve color deflection and control I S The transformation controls the color change of the sky: Among them, k s To adjust the parameters and control the image saturation, I' S is the adjusted saturation; The H channel is adjusted accordingly with reference to the saturation channel: I′ H =I H -180° I' H is the adjusted hue; The adjusted hue channel, brightness channel and saturation channel are merged to obtain the converted sky reconstructed image I' sky .

6. The method for fusion construction of sandstorm weather digital twin scenes in the autonomous driving sample library according to claim 1 is characterized in that: In step 3, an atmospheric dust model construction algorithm based on combined class depth information is obtained, which specifically includes the following steps: Step 3.1: Construct a dust scene model based on atmospheric light estimation; Step 3.2: Perform image depth simulation on the dust scene model based on atmospheric light estimation.

7. The method for fusion construction of sandstorm weather digital twin scenes in the autonomous driving sample library according to claim 6 is characterized in that: In step 3.1, a dust scene model based on atmospheric light estimation is constructed, specifically: Reconstructed image of the sky I' sky Perform global atmospheric light estimation and perform scene conversion by controlling the changes in each RGB channel, namely: A c =r c ×A 1c Among them, A c is the atmospheric light estimation corresponding to any channel, where c is any channel in RGB; A 1c is the global atmospheric light; r c is the conversion ratio, and its formula is as follows: Among them, I c_ave is the mean value of channel c; k c is the conversion coefficient, which is used to adjust the conversion ratio.

8. The method for fusion construction of sandstorm weather digital twin scenes in the autonomous driving sample library according to claim 7 is characterized in that: In step 3.2, the image depth simulation of the dust scene model based on atmospheric light estimation is performed as follows: Correct the atmospheric light value through depth-like simulation: Among them, A c is the atmospheric light estimation corresponding to any channel, i, j are the pixel coordinate indices in the width and height directions of the image respectively, m is the image height, and A'' is the global atmospheric light estimation after transformation and correction.

9. The method for fusion construction of sandstorm weather digital twin scenes in the autonomous driving sample library according to claim 1 is characterized in that: In step 4, the algorithm obtained in step 3 is combined with the algorithm obtained in step 4 and applied to the image obtained in step 1 to obtain the image after adding sand and dust: Among them, I d (x) is a dust scene image; I' sky The clear image after reconstructing the sky; α s is the dust coefficient; A” is the global atmospheric light estimate after conversion correction; t0(x) is the initial transmittance.

Images