Self-supervised blind image decomposition method for automatic driving scene

Abstract

The invention discloses a self-supervision blind image decomposition method for an automatic driving scene, relates to the technical field of automatic driving perception, and solves the problems that in the process that high-quality data is used for supervising and training normal noise, the data is difficult to collect in a real scene, and an efficient noise reduction method easy to train is lacked. A self-supervised blind image decomposition method for an automatic driving scene comprises the following steps: step 1, adding and superposing degraded noise for the automatic driving scene: setting three typical non-ideal images for the automatic driving scene, and carrying out self-supervised blind image decomposition on image blurring caused by sensor noise, rainfall and snowfall; according to the design, three different noises are adopted to simulate a scene, and the noises are added into an image. According to the method, a blind image decomposition mode is adopted, various types of degradation are uniformly removed at a time so as to improve the noise reduction efficiency, and a double-flow decoding mechanism is introduced so as to improve the noise reduction accuracy.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving perception technology, specifically a self-supervised blind image decomposition method for autonomous driving scenarios. Background Technology

[0002] As a strategic emerging industry, autonomous driving has entered a transitional phase from experimentation to widespread commercialization, which places higher demands on the robustness of autonomous driving perception technology. How to improve the robustness of this technology has become an increasingly important issue in recent years.

[0003] In complex traffic scenarios, the input images of autonomous driving perception systems often contain complex and diverse noise in a superimposed state, influenced by weather, lighting, and hardware issues. However, most existing autonomous driving perception systems lack effective means to deal with noise, resulting in insufficient robustness.

[0004] On the one hand, some existing studies have addressed this problem by changing the model design structure. However, due to limitations such as computing equipment conditions and inference speed requirements, the room for improvement in model performance is limited, and it cannot fundamentally solve the problem of low robustness caused by noise.

[0005] Therefore, existing research addresses this problem more directly by denoising the input data. However, current denoising methods often employ a process of "first identifying the noise type, then selectively removing the noise," which not only limits the efficiency of the denoising process but also requires high-quality (normal-noise) data pairs for supervised training. Such data is difficult to collect in real-world scenarios, and current research lacks an efficient and easily trainable denoising method. Therefore, to address these shortcomings, we propose a self-supervised blind image decomposition method for autonomous driving scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a self-supervised blind image decomposition method for autonomous driving scenarios, in order to solve the problem that existing methods mentioned in the background art mostly adopt the approach of "first identifying the noise type and then performing targeted noise reduction" for noise reduction. This not only limits the efficiency of the noise reduction process, but also requires high-quality data pairs (normal-noise) for supervised training on the data side. Such data is difficult to collect in real-world scenarios. Under the current research, there is a lack of an efficient and easy-to-train noise reduction method.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a self-supervised blind image decomposition method for autonomous driving scenarios, comprising the following steps:

[0008] Step 1: Adding and superimposing degradation noise for autonomous driving scenarios: Three typical non-idealized images for autonomous driving scenarios are set up: For image blurring caused by sensor noise, rain and snow, three different noises are designed to simulate the scenario, add noise to the image, and construct a linear selection superimposed image for training non-idealized images and masked noise images.

[0009] Step 2: Image information fusion for blind image decomposition: Input the non-idealized image and masked image obtained in Step 1 into two parallel encoders respectively, and use the features of the two encoders for feature fusion and refinement through multi-level feature interaction blocks, two feature thinning blocks, and one feature enhancement block;

[0010] Step 3: Blind Image Decomposition Structure Flow Prediction: Input the features obtained in Step 2 into the convolutional structure flow decoder, which includes the residual structure, to reconstruct pixels with low-level semantics;

[0011] Step 4: Blind Image Decomposition and Appearance Flow Prediction: Performed simultaneously with Step 3, the features obtained in Step 2 are input into the appearance flow decoder to assist in extracting local textures of the image and correcting the global structure;

[0012] Step 5: Overall model training strategy: A self-supervised strategy is adopted to globally optimize the two parallel encoders in Step 2, the convolutional structure stream decoder in Step 3, and the appearance stream decoder in Step 4.

[0013] Preferably, sensor noise processing includes the following steps:

[0014] D1: Simulated using salt and pepper noise, representing random white or black dots appearing in the image due to sensor malfunction.

[0015] D2: Set the number of image noise points to num1 = random(0, num...). pixel ), where num pixel This represents the number of pixels in the image, and random(i,j) means randomly selecting a value between i and j;

[0016] D3: Randomly sample num1 pixels from the image. For each sampled pixel (x, y), obtain a random floating-point number r. judge .

[0017] Preferably, the random floating-point number r judge =random(0,1), if r judge If r > 0.5, then change the pixel value of pixel (x,y) to 255. judge If the value is ≤0.5, then the pixel value of pixel (x,y) is modified to 0 to obtain the salt and pepper noise image I. ps (x,y).

[0018] Preferably, the processing of rainfall noise includes the following steps:

[0019] E1: Use a uniform distribution to generate random noise to simulate raindrop distribution, v(x,y)=random(0,255), where v(x,y) represents the pixel value at pixel (x,y);

[0020] E2: Using Gaussian blur as the blur kernel, motion blur is used to simulate the effect of raindrops falling. The image after blurring the raindrops is represented as follows: Where k represents the kernel radius, and w(i,j) represents the Gaussian blur kernel function, defined as: Where i and j are the coordinates of the distance from the kernel center, and σ is the standard deviation used to control the degree of ambiguity;

[0021] E3: Transform the blurred raindrop noise image I blurred_rain With the original image I ori (x,y) are superimposed to obtain an image I with rainfall noise. rain (x,y)=I ori (x,y)·(1-β)+I blurred_rain (x,y)·β, where β is the weight used to control the intensity of the rain effect.

[0022] Preferably, the processing of snowfall noise includes the following steps:

[0023] F1: Simulating snowflake density and size using Poisson distribution: Where threshold is the threshold value used to control snowflake density;

[0024] F2: Use the Gaussian blur function to blur the generated snowflakes to simulate the falling and accumulating effect of snowflakes:

[0025] F3: I blur the image with snowfall noise. blurred_snow Compared with the original image I ori (x,y) are superimposed to obtain an image I with snowfall noise. snow (x,y)=I ori (x,y)·(1-γ)+I blurred_rain (x,y)·γ, where γ is a weight used to control the intensity of the snowfall effect.

[0026] Preferably, constructing a linear selection superimposed image involves linearly selecting and superimposing three types of noise to obtain a non-idealized image I for training. nosie The non-idealized image I used for training nosie The formula is:

[0027]

[0028] in to I nosie Perform random masking to obtain the masked noise image I. mask .

[0029] Preferably, the two encoders include an encoder E built based on Informer for extracting features from noisy images. noise and the feature encoder E built based on Informer for predicting nearest neighbor masks mask ;

[0030] E noise With E mask The features are refined level by level. For the i-th level, by E noise The generated feature F ni With E mask The generated feature F mi Through feature interaction block B Ii F' is obtained by completing the information exchange between the two encoders. ni With F' mi , where F' ni With F' mi This will be used as the input for the next level encoder: B Ii (F ni ,F mi )=(C1(C3(C1(F ni +F mi ))))·Sigmoid(FC(FC(AvgPool(F ni +F mi )))),F' ni =C3(B Ii (F ni ,F mi )), F' mi =C3(B Ii (F ni ,F mi ), where Cj represents a convolution with a kernel of size j×j, and FC represents a fully connected layer;

[0031] The two feature refinement blocks include B Rn and B Rm B Rn and B Rm The structures are identical, both built upon two self-attention layers and one convolutional layer, for E noise Feature refinement block B Rn It consists of two self-attention layers. For the first layer, its query... key value The self-attention of this layer can be represented as:

[0032] For the second level, its query key value The self-attention of this layer can be represented as: Subsequently, convolution is used to fuse and refine the features, obtaining E. noise Refined features F Rn Similarly, based on B Rm Get E mask Refined features F Rm ;

[0033] Feature enhancement block is B E , will E noise The resulting refined feature F Rn With E mask The refined features F obtained Rm Fusion and feature enhancement are performed: it first integrates F Rn With F Rm The concatenation is then performed, followed by refinement through multiple convolutional layers to obtain enhanced features F = C1(C3(C1(Concat(F)). Rn ,F Rm )))).

[0034] Preferably, the pixel of the low-level semantics is F. structure F structure The formula for obtaining it is:

[0035] F structure =Decoder(F)=C3(C1((C3(F)+F)+(C3(F)+F)))+C1((C3(F)+F)+(C3(F)+F));

[0036] Simultaneously, a reconstruction loss function is designed to remove high-frequency texture information from the image while preserving global structural information. The reconstruction loss is the predicted structural information S. inf With truth structure information S gt Manhattan distance, expressed as

[0037] Preferably, the process of assisting in the extraction of local textures and correcting the global structure of an image includes the following steps:

[0038] S1: First, to further enhance the image's appearance features, E will be used... noise The refined features F obtained RnCross-attention is performed with the enhanced feature F, and the query Q = F, where both the key and value are E. noise The refined features F obtained Rn This can be represented as K = V = F Rn The attention score is calculated as follows:

[0039] in Indicates query Q i With key K j The dot product score, d k Indicates the dimension of the key vector;

[0040] S2: Output Features in

[0041] S3: F app The output appearance is further predicted using two convolutional layers, with the appearance loss being the predicted appearance A. inf With truth value appearance information A gt The Euclidean distance is expressed as

[0042] Preferably, a global loss function is constructed by combining steps three and four. global loss function The formula is:

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] 1. This invention constructs a blind image decomposition method for autonomous driving scenarios. It adopts a self-supervised training method, which greatly reduces the need for high-quality image pairs that are difficult to obtain, namely normal image and noisy image pairs. At the same time, this invention only uses real data, eliminating the need for synthetic data in the original method and improving the quality of blind image decomposition.

[0045] 2. Traditional methods focus on first identifying abnormal patterns and then performing targeted encoding and decoding, which significantly increases system consumption and makes it difficult to handle superimposed noise. This invention uses a unified encoder and decoder to achieve blind image decomposition, improving system efficiency. 3. The dual-stream decoding mechanism for structural appearance designed in this invention will effectively improve the accuracy of the model. Attached Figure Description

[0046] Figure 1 This is a flowchart of the self-supervised blind image decomposition method for autonomous driving scenarios according to the present invention;

[0047] Figure 2 This is a system diagram of feature fusion and refinement of the two encoders in this invention. Detailed Implementation

[0048] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0049] Please see Figures 1 to 2 The present invention provides an embodiment of a self-supervised blind image decomposition method for autonomous driving scenarios, comprising the following steps:

[0050] Step 1: Adding and superimposing degradation noise for autonomous driving scenarios: Three typical non-idealized images for autonomous driving scenarios are set up: For image blurring caused by sensor noise, rain and snow, three different noises are designed to simulate the scenario, add noise to the image, and construct a linear selection superimposed image for training non-idealized images and masked noise images.

[0051] Step 2: Image information fusion for blind image decomposition: Input the non-idealized image and masked image obtained in Step 1 into two parallel encoders respectively, and use the features of the two encoders for feature fusion and refinement through multi-level feature interaction blocks, two feature thinning blocks, and one feature enhancement block;

[0052] Step 3: Blind Image Decomposition Structure Flow Prediction: Input the features obtained in Step 2 into the convolutional structure flow decoder, which includes the residual structure, to reconstruct pixels with low-level semantics;

[0053] Step 4: Blind Image Decomposition and Appearance Flow Prediction: Performed simultaneously with Step 3, the features obtained in Step 2 are input into the appearance flow decoder to assist in extracting local textures of the image and correcting the global structure;

[0054] Step 5: Overall model training strategy: A self-supervised strategy is adopted to globally optimize the two parallel encoders in Step 2, the convolutional structure stream decoder in Step 3, and the appearance stream decoder in Step 4.

[0055] Sensor noise processing includes the following steps:

[0056] D1: Simulated using salt and pepper noise, representing random white or black dots appearing in the image due to sensor malfunction.

[0057] D2: Set the number of image noise points to num1 = random(0, num...). pixel ), where num pixel This represents the number of pixels in the image, and random(i,j) means randomly selecting a value between i and j;

[0058] D3: Randomly sample num1 pixels from the image. For each sampled pixel (x, y), obtain a random floating-point number r. judge random floating-point number r judge =random(0,1), if r judge If r > 0.5, then change the pixel value of pixel (x,y) to 255. judge If the value is ≤0.5, then the pixel value of pixel (x,y) is modified to 0 to obtain the salt and pepper noise image I. ps (x,y).

[0059] The following steps are involved in handling rain noise:

[0060] E1: Use a uniform distribution to generate random noise to simulate raindrop distribution, v(x,y)=random(0,255), where v(x,y) represents the pixel value at pixel (x,y);

[0061] E2: Using Gaussian blur as the blur kernel, motion blur is used to simulate the effect of raindrops falling. The image after blurring the raindrops is represented as follows: Where k represents the kernel radius, and w(i,j) represents the Gaussian blur kernel function, defined as: Where i and j are the coordinates of the distance from the kernel center, and σ is the standard deviation used to control the degree of ambiguity;

[0062] E3: Transform the blurred raindrop noise image I blurred_rain With the original image I ori (x,y) are superimposed to obtain an image I with rainfall noise. rain (x,y)=I ori (x,y)·(1-β)+I blurred_rain (x,y)·β, where β is the weight used to control the intensity of the rain effect.

[0063] The following steps are involved in handling snowfall noise:

[0064] F1: Simulating snowflake density and size using Poisson distribution: Where threshold is the threshold value used to control snowflake density;

[0065] F2: Use the Gaussian blur function to blur the generated snowflakes to simulate the falling and accumulating effect of snowflakes:

[0066] F3: I blur the image with snowfall noise. blurred_snow Compared with the original image I ori (x,y) are superimposed to obtain an image I with snowfall noise.snow (x,y)=I ori (x,y)·(1-γ)+I blurred_rain (x,y)·γ, where γ is a weight used to control the intensity of the snowfall effect.

[0067] Constructing a linear selection superimposed image involves linearly selecting and superimposing three types of noise to obtain a non-idealized image I for training. nosie The non-idealized image I used for training nosie The formula is:

[0068]

[0069] in to I nosie Perform random masking to obtain the masked noise image I. mask .

[0070] The two encoders include an encoder E built based on Informer for extracting features from noisy images. noise and the feature encoder E built based on Informer for predicting nearest neighbor masks mask ;

[0071] E noise With E mask The features are refined level by level. For the i-th level, by E noise The generated feature F ni With E mask The generated feature F mi Through feature interaction block B Ii F' is obtained by completing the information exchange between the two encoders. ni With F' mi , where F' ni With F' mi This will be used as the input for the next level encoder: B Ii (F ni ,F mi )=(C1(C3(C1(F ni +F mi ))))·Sigmoid(FC(FC(AvgPool(F ni +F mi )))),F' ni =C3(B Ii (F ni ,F mi )), F' mi =C3(B Ii (F ni ,F mi ), where Cj represents a convolution with a kernel of size j×j, and FC represents a fully connected layer;

[0072] The two feature refinement blocks include B Rn and B Rm B Rn and B Rm The structures are identical, both built upon two self-attention layers and one convolutional layer, for E noise Feature refinement block B Rn It consists of two self-attention layers. For the first layer, its query... key value The self-attention of this layer can be represented as:

[0073] For the second level, its query key value The self-attention of this layer can be represented as: Subsequently, convolution is used to fuse and refine the features, obtaining E. noise Refined features F Rn Similarly, based on B Rm Get E mask Refined features F Rm ;

[0074] Feature enhancement block is B E , will E noise The resulting refined feature F Rn With E mask The refined features F obtained Rm Perform fusion and feature enhancement:

[0075] It first will F Rn With F Rm The concatenation is then performed, followed by refinement through multiple convolutional layers to obtain enhanced features F = C1(C3(C1(Concat(F)). Rn ,F Rm )))).

[0076] The pixel with low-level semantics is F structure F structure The formula for obtaining it is:

[0077] F structure =Decoder(F)=C3(C1((C3(F)+F)+(C3(F)+F)))+C1((C3(F)+F)+(C3(F)+F));

[0078] Simultaneously, a reconstruction loss function is designed to remove high-frequency texture information from the image while preserving global structural information. The reconstruction loss is the predicted structural information S. infWith truth structure information S gt Manhattan distance, expressed as

[0079] The steps to assist in extracting local textures and correcting global structure in an image include the following:

[0080] S1: First, to further enhance the image's appearance features, E will be used... noise The refined features F obtained Rn Cross-attention is performed with the enhanced feature F, and the query Q = F, where both the key and value are E. noise The refined features F obtained Rn This can be represented as K = V = F Rn The attention score is calculated as follows:

[0081] in Indicates query Q i With key K j The dot product score, d k Indicates the dimension of the key vector;

[0082] S2: Output Features in

[0083] S3: F app The output appearance is further predicted using two convolutional layers, with the appearance loss being the predicted appearance A. inf With truth value appearance information A gt The Euclidean distance is expressed as

[0084] By combining steps three and four, a global loss function can be constructed.

[0085] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A self-supervised blind image decomposition method for autonomous driving scenarios, characterized in that: The steps include: Step 1: Adding and superimposing degraded noise for autonomous driving scenarios: Set up three typical non-ideal images for autonomous driving scenarios: for image blur caused by sensor noise, rain and snowfall, design three different noises to simulate the scene, add noise to the image, and construct a linear selection superimposer for training non-ideal images and mask noise images; Step 2: Image information fusion for blind image decomposition: The non-idealized image and mask image obtained in step 1 are input into two parallel encoders respectively, and the features of the two encoders are fused and refined through a multi-level feature interaction block, two feature refinement blocks, and a feature enhancement block; Step 3: Blind image decomposition structure flow prediction: The features obtained in step 2 are input into the convolutional structure flow decoder including the residual structure to reconstruct pixels with low-level semantics; Step 4: Blind image decomposition appearance flow prediction: This is performed simultaneously with step 3. The features obtained in step 2 are input into the appearance flow decoder to assist in extracting local texture of the image and correcting the global structure. Step 5: Overall model training strategy: Use a self-supervised strategy to globally optimize the two parallel encoders in step 2, the convolutional structure stream decoder in step 3, and the appearance stream decoder in step 4.

2. The self-supervised blind image decomposition method for autonomous driving scenarios according to claim 1, characterized in that: Sensor noise processing includes the following steps: D1: simulated using salt and pepper noise, which indicates random white or black dots that appear in the image due to sensor malfunction; D2: Set the number of image noise points to num1 = random (0, num pixel ), where num pixel Indicates the number of pixels in the image, random(i, j) means randomly using a value between i and j; D3: Randomly sample pixels from num1 images, and for each sampled pixel (x, y), get a random floating point number r judge .

3. The self-supervised blind image decomposition method for autonomous driving scenarios according to claim 2, characterized in that: Random floating point number r judge =random(0,1), if r judge > 0.5, then the pixel value of the pixel point (x, y) is modified to 255. judge ≤0.5, then the pixel value of the pixel point (x, y) is modified to 0 to obtain the salt and pepper noise image I ps (x, y).

4. The self-supervised blind image decomposition method for autonomous driving scenarios according to claim 3, characterized in that: The processing of rainfall noise includes the following steps: E1: Generate random noise using uniform distribution to simulate raindrop distribution, v(x, y) = random(0, 255), where v(x, y) represents the pixel value at the pixel point (x, y); E2: Use Gaussian blur as the blur kernel and motion blur to simulate the falling effect of raindrops. The image after the raindrop blur is represented as follows: Where k represents the kernel radius, w(i, j) represents the Gaussian blur kernel function, which is defined as: Where i and j are the coordinates from the center of the kernel, and σ is the standard deviation used to control the degree of blur; E3: The blurred raindrop noise image I blurred_rain With the original image I ori (x, y) are superimposed to obtain the image I with rainfall noise rain (x, y) = I ori (x, y)·(1-β)+I blurred_rain (x, y)·β, where β is a weight used to control the intensity of the rain effect.

5. The self-supervised blind image decomposition method for autonomous driving scenarios according to claim 4, characterized in that: The processing of snowfall noise includes the following steps: F1: Use Poisson distribution to simulate snowflake density and size: Where threshold is the threshold for controlling snowflake density; F2: Use the Gaussian blur function to blur the generated snowflakes to simulate the falling and accumulation effects of snowflakes: F3: The blurred image with snow noise I blurred_snow With the original image I ori (x, y) are superimposed to obtain image I with snowfall noise snow (x, y) = I ori (x, y)·(1-γ)+I blurred_rain (x, y)·γ, where γ is a weight used to control the intensity of the snow effect.

6. The self-supervised blind image decomposition method for autonomous driving scenarios according to claim 5, characterized in that: The linear selection superpositioner is constructed by linearly selecting and superposing three types of noise to obtain a non-idealized image I for training. nosie , the non-idealized image I used for training nosie The formula is: in to I nosie Perform random masking to obtain masked noise image I mask .

7. The self-supervised blind image decomposition method for autonomous driving scenarios according to claim 6, characterized in that: The two encoders include an encoder E based on Informer for extracting noise image features. noise And the feature encoder E built on Informer for predicting adjacent masks mask ; E noise With E mask The features are refined step by step. For the i-th level, E noise The resulting feature F ni With E mask The resulting feature F mi Through the feature interaction block B Ii Complete the information exchange between the two encoders to obtain F′ ni and F′ mi , where F′ ni and F′ mi Will be used as the input of the next level encoder: B Ii (F ni , F mi )=(C1(C3(C1(F ni +F mi ))))·Sigmoid(FC(FC(AvgPool(F ni +F mi )))),F′ ni =C3(B Ii (F ni , F mi )), F′ mi =C3(B Ii (F ni , F mi )), where Cj represents the convolution kernel j×j, and FC represents the fully connected layer; The two feature refinement blocks include B Rn and B Rm , B Rn and B Rm The structure is the same, both are based on two self-attention layers and one convolutional layer. noise Feature refinement block B Rn , which includes two self-attention layers. For the first layer, its query key value The self-attention of this layer can be expressed as For the second layer, the query key value The self-attention of this layer can be expressed as After that, convolution is used to fuse and refine the features to obtain E noise The refinement feature F Rn . Similarly, based on B Rm Get E mask The refinement feature F Rm ; The feature enhancement block is B E , E noise The resulting refined feature F Rn With E mask The obtained refined features F Rm Fusion and feature enhancement: First, F Rn With F Rm After concatenation, it is connected to multiple convolutional layers for refinement to obtain the enhanced feature F = C1(C3(C1(Concat(F Rn , F Rm )))).

8. The self-supervised blind image decomposition method for autonomous driving scenarios according to claim 7, characterized in that: The low-level semantic pixel is F structure , F structure The formula for obtaining is: F structure =Decoder(F)=C3(C1((C3(F)+F)+(C3(F)+F)))+C1((C3(F)+F)+(C3(F)+F)); At the same time, a reconstruction loss function is designed to remove high-frequency texture information in the image while retaining the global structural information. The reconstruction loss is the predicted structural information S inf and the truth structure information S gt The Manhattan distance of 9. The self-supervised blind image decomposition method for autonomous driving scenarios according to claim 8, characterized in that: Assisting in extracting local texture of an image and correcting the global structure includes the following steps: S1: First, to further enhance the image appearance features, the E noise The obtained refined features F Rn Perform cross attention with enhanced feature F, query Q = F, key and value are both E noise The obtained refined features F Rn Expressed as K = V = F Rn , the attention score is calculated as in Represents the query Q i With key K j The dot product score, d k represents the dimension of the key vector; S2: Output features in S3: F app The output appearance is further predicted through two convolutional layers, and the appearance loss is the predicted appearance A inf and the true appearance information A gt The Euclidean distance of 10. The self-supervised blind image decomposition method for autonomous driving scenarios according to claim 9, characterized in that: Combine steps 3 and 4 to construct a global loss function Global loss function The formula is:

Images