A multi-modal image defogging method and device, electronic equipment and storage medium

Abstract

The application provides a multi-modal image defogging method and device, electronic equipment and storage medium, and relates to the technical field of image processing, which comprises the following steps: performing feature extraction on an RGB image and radar point cloud data respectively to obtain RGB image features and radar point cloud features; inputting a first parallel convolution module and a second parallel convolution module for convolution processing to obtain a first feature fusion result and a second feature fusion result respectively; performing feature fusion through the first feature fusion result and the second feature fusion result to obtain a multi-modal image fusion result; inputting the multi-modal image fusion result into an image defogging model based on a visual processing improvement algorithm to obtain a fusion defogging image, wherein the image defogging model comprises a target encoder, a target decoder and a feature enhancement module, and the feature enhancement module is constructed based on a Vision Transformer. The application realizes the improvement of image defogging precision.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a multimodal image dehazing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image dehazing is of great significance in many fields, as it can solve image degradation problems under various complex atmospheric conditions such as non-uniform fog, dense fog, and haze-fog mixtures, and restore clear images. Traditional physical model-based methods usually require estimations of assumptions about the haze scene, which are often too idealistic to cope with the complex and changing haze environment in reality. While deep learning methods have partially alleviated these problems, they are still limited by training data bias, and the image dehazing accuracy cannot meet practical needs. Summary of the Invention

[0003] The problem addressed by this invention is how to improve the accuracy of image dehazing.

[0004] To address the above problems, the present invention provides a multimodal image dehazing method, apparatus, electronic device, and storage medium.

[0005] In a first aspect, the present invention provides a multimodal image dehazing method, comprising:

[0006] Feature extraction was performed on RGB images and radar point cloud data respectively to obtain RGB image features and radar point cloud features;

[0007] The RGB image features and the radar point cloud features are input into a first parallel convolution module and a second parallel convolution module for convolution processing to obtain a first feature fusion result and a second feature fusion result, respectively. The first parallel convolution module is used to perform matrix subtraction operations, and the second parallel convolution module is used to perform matrix addition operations.

[0008] By fusing the first feature fusion result and the second feature fusion result, a multimodal image fusion result is obtained;

[0009] Based on the improved visual processing algorithm, the multimodal image fusion result is input into the image dehazing model to obtain the fused dehazed image. The image dehazing model includes a target encoder, a target decoder, and a feature enhancement module, and the feature enhancement module is built based on Vision Transformer.

[0010] Optionally, the process of constructing the image dehazing model includes:

[0011] The fusion results of historical multimodal images are input into the initial encoder for feature enhancement to obtain the initial encoder image;

[0012] The feature enhancement module is trained using the initial encoder image to obtain a visually processed image;

[0013] The visually processed image is input into the initial decoder for image reconstruction to obtain the initial decoder image;

[0014] The total loss value is obtained by calculating the mean squared error loss function based on the initial encoder image and the initial decoder image;

[0015] The model parameters of the initial encoder and the initial decoder are adjusted according to the total loss value until the total loss value meets the preset conditions. The initial encoder and the initial decoder after parameter adjustment are then used as the target encoder and the target decoder, respectively.

[0016] Optionally, the initial encoder image includes an input image, a first convolutional layer processed image, and a second convolutional layer processed image; the initial decoder image includes a third convolutional layer processed image, a fourth convolutional layer processed image, and an output image; the step of calculating the total loss value based on the mean squared error loss function of the initial encoder image and the initial decoder image includes:

[0017] The main loss value is obtained by calculating the mean squared error loss function using the input image and the output image;

[0018] The mean squared error loss function is calculated by processing the image through the first convolutional layer and the image through the fourth convolutional layer to obtain the regularization term loss value;

[0019] The mean squared error loss function is calculated by processing the image through the second convolutional layer and the image through the third convolutional layer to obtain the constraint term loss value;

[0020] The total loss value is obtained based on the main loss value, the regularization term loss value, and the constraint term loss value.

[0021] The total loss value includes:

[0022] ,

[0023] in, The total loss value, The main loss value, The loss value of the regularization term. The loss value is the constraint term.

[0024] Optionally, training the feature enhancement module using the initial encoder image to obtain a visually processed image includes:

[0025] The decomposed image is obtained by decomposing the initial encoder image;

[0026] The visually processed image is obtained from the decomposed image based on the multi-head attention mechanism.

[0027] Optionally, the step of extracting features from the RGB image and radar point cloud data respectively to obtain RGB image features and radar point cloud features includes:

[0028] Acquire the RGB image and the radar point cloud data;

[0029] The RGB image is input into the image feature extraction module to obtain the RGB image features, wherein the image feature extraction module includes an image convolutional layer and an image ReLU activation function layer;

[0030] The radar point cloud data is input into the radar point cloud feature extraction module to obtain the radar point cloud features. The radar point cloud feature extraction module includes a radar point cloud convolutional layer, a radar point cloud normalization layer, and a radar point cloud ReLU activation function layer.

[0031] Optionally, the step of inputting the RGB image features and the radar point cloud features into a first parallel convolution module and a second parallel convolution module for convolution processing to obtain a first feature fusion result and a second feature fusion result, respectively, includes:

[0032] The RGB image features and the radar point cloud features are input into the first parallel convolution module for convolution processing to obtain the first feature fusion result. The first parallel convolution module includes a first RGB convolution branch and a first radar point cloud convolution branch.

[0033] The first feature fusion result includes:

[0034] ,

[0035] in, The result of the first feature fusion. The output result of the RGB image features after processing by the first RGB convolution branch. The output result of the radar point cloud features after processing by the first radar point cloud convolutional branch;

[0036] The RGB image features and the radar point cloud features are input into the second parallel convolution module for convolution processing to obtain the second feature fusion result. The second parallel convolution module includes a second RGB convolution branch and a second radar point cloud convolution branch.

[0037] The result of the second feature fusion is as follows:

[0038] ,

[0039] in, The result of the second feature fusion. The output result of the RGB image features after processing by the second RGB convolution branch. The output result is the radar point cloud features processed by the second radar point cloud convolution branch.

[0040] Optionally, the step of performing feature fusion using the first feature fusion result and the second feature fusion result to obtain a multimodal image fusion result includes:

[0041] The multimodal image fusion result is obtained by performing regularization processing on the first feature fusion result and the second feature fusion result;

[0042] The multimodal image fusion result is as follows:

[0043] ,

[0044] Wherein, Fusion_feature is the result of the multimodal image fusion. The result of the first feature fusion. The result of the second feature fusion is represented by Conv(), which performs convolution processing, BN(), which performs normalization processing, and ReLU(), which performs ReLU activation function processing.

[0045] In a second aspect, the present invention provides a multimodal image dehazing device, comprising: a feature extraction module, used to extract features from RGB images and radar point cloud data respectively, to obtain RGB image features and radar point cloud features;

[0046] The convolution processing module is used to input the RGB image features and the radar point cloud features into the first parallel convolution module and the second parallel convolution module for convolution processing, and obtain the first feature fusion result and the second feature fusion result respectively. The first parallel convolution module is used to perform matrix subtraction operation, and the second parallel convolution module is used to perform matrix addition operation.

[0047] The image fusion module is used to perform feature fusion using the first feature fusion result and the second feature fusion result to obtain a multimodal image fusion result;

[0048] The image dehazing module is used to input the multimodal image fusion result into the image dehazing model to obtain a fused dehazed image based on the improved visual processing algorithm. The image dehazing model includes a target encoder, a target decoder, and a feature enhancement module, and the feature enhancement module is built based on Vision Transformer.

[0049] Thirdly, the present invention provides an electronic device, including a memory and a processor;

[0050] The memory is used to store computer programs;

[0051] The processor is configured to implement the multimodal image dehazing method as described in the first aspect when executing the computer program.

[0052] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal image dehazing method as described in the first aspect.

[0053] The beneficial effects of the multimodal image dehazing method, apparatus, electronic device, and storage medium of the present invention are as follows: Feature extraction is performed on RGB images and radar point cloud data separately. By extracting features from these two modalities separately, the contour information of the image can be better obtained. A first parallel convolution module performs matrix subtraction after convolution processing of the features of the two modalities, thereby capturing the difference features. A second parallel convolution module performs matrix addition after convolution processing of the features of the two modalities, strengthening the common features. The features processed by the first and second parallel convolution modules are fused to obtain the multimodal image fusion result, enhancing the fused features and laying the foundation for subsequent image processing. Based on an improved visual processing algorithm, the multimodal image fusion result is input into the image dehazing model to obtain the fused dehazed image. Image dehazing is achieved through a target encoder, target decoder, and feature enhancement module. The introduction of a feature enhancement module based on Vision Transformer enables the image dehazing model to better adapt to scenes with different haze densities and non-uniform haze distributions, thereby improving the image dehazing accuracy. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating a multimodal image dehazing method according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the structure of an image dehazing model according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the structure of an image feature extraction module according to an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the structure of a radar point cloud feature extraction module according to an embodiment of the present invention;

[0058] Figure 5 This is a schematic diagram of the structure of a multimodal image dehazing device according to an embodiment of the present invention;

[0059] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0060] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0061] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0062] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0063] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0064] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0065] like Figure 1 As shown, an embodiment of the present invention provides a multimodal image dehazing method, comprising:

[0066] Step 110: Extract features from the RGB image and radar point cloud data respectively to obtain RGB image features and radar point cloud features.

[0067] Specifically, for input images of different sizes, a size adaptation process is first performed, adjusting them to 256×256 pixels through center cropping or random cropping, resulting in a final input RGB image of size 256×256×3. Then, the image data is normalized by dividing the original pixel values ​​in the 0-255 range by 255, standardizing all pixel values ​​to the [0,1] interval. This processing method satisfies the model input requirements and improves the numerical stability of the training process. For the accompanying radar point cloud information, X and Y coordinate data (corresponding to image pixel positions), as well as the two key parameters Intensity and Distance, are directly extracted from the Excel spreadsheet. This data is organized into a two-channel three-dimensional matrix, where the first channel stores intensity values ​​and the second channel stores distance values, with a matrix size of X×Y. We normalize the data for each channel separately by dividing each channel value by its maximum value, mapping all values ​​to the [0,1] range. Finally, to maintain consistency with the image data, the 3D matrix was also randomly cropped and its size was adjusted to 256×256×2 to ensure size matching of the multimodal data.

[0068] Step 120: Input the RGB image features and the radar point cloud features into the first parallel convolution module and the second parallel convolution module for convolution processing to obtain the first feature fusion result and the second feature fusion result, respectively. The first parallel convolution module is used to perform matrix subtraction operation, and the second parallel convolution module is used to perform matrix addition operation.

[0069] Specifically, since RGB images and radar point cloud information are different data, multimodal feature fusion is required. Therefore, in the implementation of the feature fusion module, a dual-channel network structure is adopted to process and cross-fuse the RGB image features and radar point cloud features during the fusion process. It is divided into a first parallel convolution module Diff (difference branch) and a second parallel convolution module Sum (sum branch), which respectively capture the difference features and enhance the common features.

[0070] Step 130: Perform feature fusion using the first feature fusion result and the second feature fusion result to obtain a multimodal image fusion result.

[0071] Specifically, the results from the two branches are convolved and regularized respectively, and then the two matrices of the same size are added together. In this way, the final multimodal image fusion result contains the pixel features of the original foggy RGB image and the intensity and distance information of the radar point cloud data. That is, the final result contains pixel features and contour features.

[0072] Step 140: Based on the improved visual processing algorithm, the multimodal image fusion result is input into the image dehazing model to obtain the fused dehazed image. The image dehazing model includes a target encoder, a target decoder, and a feature enhancement module. The feature enhancement module is built based on Vision Transformer.

[0073] Specifically, in the aforementioned steps, feature information extraction and enhancement have been achieved. The multimodal image fusion result is input into the image dehazing model to obtain the fused dehazed image. A single-image dehazing method based on detail-enhancing convolution and content-guided attention is employed. The image dehazing model is designed with a target encoder and a target decoder interspersed with a feature enhancement module built on Vision Transformer. The feature enhancement module uses a self-attention mechanism to achieve efficient local information aggregation. Vision Transformer (ViT) is a model structure that applies the classic Transformer architecture from natural language processing to computer vision tasks.

[0074] In this embodiment, feature extraction is performed on RGB images and radar point cloud data separately. By extracting features from these two modalities separately, the contour information of the image can be better obtained. A first parallel convolution module processes the features of the two modalities through convolution and then performs matrix subtraction to capture the differences in features. A second parallel convolution module processes the features of the two modalities through convolution and then performs matrix addition to enhance common features. The features processed by the first and second parallel convolution modules are fused to obtain a multimodal image fusion result, enhancing the fused features and laying the foundation for subsequent image processing. Based on an improved visual processing algorithm, the multimodal image fusion result is input into an image dehazing model to obtain a fused dehazed image. Image dehazing is achieved through a target encoder, target decoder, and feature enhancement module. The introduction of a feature enhancement module based on Vision Transformer enables the image dehazing model to better adapt to scenes with different fog densities and non-uniform fog distributions, thereby improving the image dehazing accuracy.

[0075] Optionally, the process of constructing the image dehazing model includes:

[0076] The fusion results of historical multimodal images are input into the initial encoder for feature enhancement to obtain the initial encoder image;

[0077] The feature enhancement module is trained using the initial encoder image to obtain a visually processed image;

[0078] The visually processed image is input into the initial decoder for image reconstruction to obtain the initial decoder image;

[0079] The total loss value is obtained by calculating the mean squared error loss function based on the initial encoder image and the initial decoder image;

[0080] The model parameters of the initial encoder and the initial decoder are adjusted according to the total loss value until the total loss value meets the preset conditions. The initial encoder and the initial decoder after parameter adjustment are then used as the target encoder and the target decoder, respectively.

[0081] Specifically, such as Figure 2 As shown, the initial encoder is responsible for extracting features from the input image and performing multi-scale feature representation. The initial decoder progressively restores the features extracted by the initial encoder to the original image size and generates the final output. The output of the initial encoder is sent to the feature enhancement module (Vision Transformer), which processes it and then sends it back to the initial decoder to obtain the final result.

[0082] In this optional embodiment, image dehazing is achieved through a feature enhancement module, thereby improving the accuracy of image dehazing. The loss function is calculated using mean squared error, which measures the average squared difference between the predicted and true values. This method is convenient for use with optimization algorithms such as gradient descent, and is particularly suitable for image reconstruction tasks such as image dehazing.

[0083] Optionally, the initial encoder image includes an input image, a first convolutional layer processed image, and a second convolutional layer processed image; the initial decoder image includes a third convolutional layer processed image, a fourth convolutional layer processed image, and an output image; the step of calculating the total loss value based on the mean squared error loss function of the initial encoder image and the initial decoder image includes:

[0084] The main loss value is obtained by calculating the mean squared error loss function using the input image and the output image;

[0085] The mean squared error loss function is calculated by processing the image through the first convolutional layer and the image through the fourth convolutional layer to obtain the regularization term loss value;

[0086] The mean squared error loss function is calculated by processing the image through the second convolutional layer and the image through the third convolutional layer to obtain the constraint term loss value;

[0087] The total loss value is obtained based on the main loss value, the regularization term loss value, and the constraint term loss value.

[0088] The total loss value includes:

[0089] ,

[0090] in, The total loss value, The main loss value, The loss value of the regularization term. The loss value is the constraint term.

[0091] In some more specific embodiments, such as Figure 2 As shown, in the initial encoder section, the fusion result of historical multimodal images is first input into the input layer (fusion_feature input) of the initial encoder to obtain the input image. The input image is then processed by the first convolutional layer (DEB N1) to obtain the first convolutional layer processed image. The first convolutional layer processed image is then downsampled, and the processed image is input into the second convolutional layer (DEB N2) to obtain the second convolutional layer processed image. The second convolutional layer processed image is then downsampled and input into the feature enhancement module.

[0092] In the initial decoder section, the image processed by the feature enhancement module is upsampled and then input into the third convolutional layer (DEB N3) to obtain the processed image. The Fusion module then fuses this processed image with the downsampled image from the first convolutional layer in the initial encoder section. This fused image is then upsampled and input into the fourth convolutional layer (DEB N4) to obtain the processed image. Finally, the processed image is input into the output layer of the initial decoder to obtain the output image. Each convolutional layer includes a convolutional block (Conv) and a feature enhancement module.

[0093] Specifically, three loss functions were designed to calculate the loss value in different processing steps. All three loss functions use Mean Square Error (MSE) for calculation. The calculation process of the loss function includes:

[0094] ,

[0095] in, for, The values ​​of the matrix obtained by the corresponding decoder. The value obtained by the corresponding encoder is N, where N is the number of current training samples.

[0096] Different weights are assigned to the three loss values: the main loss value Loss_1, the regularization term loss value Loss_2, and the constraint term loss value Loss_3 have weights of 0.6, 0.2, and 0.2, respectively. The corresponding total loss values ​​include:

[0097] ,

[0098] in, The total loss value, The main loss value, The loss value of the regularization term. The loss value is the constraint term.

[0099] Based on the aforementioned loss, gradient descent using the SGD optimizer is employed to adjust the parameters of the neural network. By continuously adjusting the network parameters according to the loss, the loss is gradually reduced. Finally, when the loss stabilizes at a very low value, the model training is considered to have converged. SGD (Stochastic Gradient Descent) is one of the most fundamental and widely used optimization algorithms in deep learning. It iteratively updates the model parameters to minimize the loss function. Regarding the setting of optimization parameters, the learning rate... Setting it to 0.001 allows for adjustment of the learning rate during training. A suitable learning rate is necessary. Too high a value can easily lead to the model failing to converge. If the value is too high, the model converges too slowly.

[0100] ,

[0101] in, These are the model parameters updated in the next iteration. For learning rate, These are the model parameters at the current iteration time. These are the model parameters.

[0102] In this optional embodiment, three loss functions are designed to calculate the error at different processing stages of the model. The three loss functions are applied to different levels of the encoder. This multi-level supervision mechanism can combat feature distortion and improve overall robustness. Multiple supervision signals prevent gradient vanishing and make deep networks easier to train.

[0103] Optionally, training the feature enhancement module using the initial encoder image to obtain a visually processed image includes:

[0104] The decomposed image is obtained by decomposing the initial encoder image;

[0105] The visually processed image is obtained from the decomposed image based on the multi-head attention mechanism.

[0106] In some more specific embodiments, a vision-based improvement is made to the traditional transformer model, using a Vision Transformer to process the fused data after encoder processing. The initial input encoder image, i.e., the feature matrix, is divided into several non-overlapping smaller matrices. For example, the input feature matrix is ​​64*64*3, which is decomposed into 64 smaller matrices (patches) of 8*8*3. Each patch is then flattened into a vector, the process of which includes:

[0107] ,

[0108] in, Let be the vector corresponding to the i-th small matrix. For the i-th image patch, `Linear()` is the linear transformation layer, `Flatten()` is the flattening operation, N is the number of image patches, and D is the number of elements in the dimension matrix (i.e., 8*8*3=192 dimensions). Because this processing is unordered, position information is added to each vector to form the final token. The position information records the position of the small matrix corresponding to the current vector in the original large matrix, which is used for subsequent small matrix concatenation. The process includes:

[0109] ,

[0110] in, The token corresponding to the i-th patch. This represents the position information corresponding to the i-th patch. The final token sequence is: After obtaining the token sequence, three independent weight matrices are randomly generated. Based on the token sequence, corresponding K, Q, and V are generated. The attention value for each token is then calculated based on the obtained K, Q, and V. The process includes:

[0111] ,

[0112] ,

[0113] in, Let K be the transpose of matrix K. is the scaling factor, and Softmax() is the Softmax function.

[0114] The model processes the patch block corresponding to the current token based on the token output from the encoder, ensuring that it only pays attention to the current input patch and is not affected by other patches. Then, it predicts the content of the next patch block, accelerating the processing of each patch. This process is implemented using Masked Multi-Head Self-Attention. The process includes:

[0115] ,

[0116] Here, M and V are the correlation matrices. After processing each patch, each token is transformed from a one-dimensional vector into its original three-dimensional tensor. Finally, based on the positional information added during encoding, these are concatenated to form the three-dimensional matrix initially input into the module for further processing. The loss is calculated using the cross-entropy loss function, including:

[0117] ,

[0118] in, For loss, Given the input feature matrix, The output feature matrix is ​​shown below after module processing, where c represents the number of different classes. After each batch of training, the model's parameters are backpropagated based on the loss value. Gradient descent is performed according to the chain rule to update the parameters. This includes:

[0119] ,

[0120] ,

[0121] ,

[0122] Through continuous updates These parameters continuously optimize the model's feature processing capabilities.

[0123] Optionally, the step of extracting features from the RGB image and radar point cloud data respectively to obtain RGB image features and radar point cloud features includes:

[0124] Acquire the RGB image and the radar point cloud data;

[0125] The RGB image is input into the image feature extraction module to obtain the RGB image features, wherein the image feature extraction module includes an image convolutional layer and an image ReLU activation function layer;

[0126] The radar point cloud data is input into the radar point cloud feature extraction module to obtain the radar point cloud features. The radar point cloud feature extraction module includes a radar point cloud convolutional layer, a radar point cloud normalization layer, and a radar point cloud ReLU activation function layer.

[0127] In some specific embodiments, such as Figure 3 As shown, the image feature extraction module uses convolution operations, including four convolutional layers of different sizes. An image ReLU activation function layer is interspersed after the last convolutional layer to increase the model's non-linearity. The ReLU activation function includes:

[0128] ,

[0129] Each convolutional layer uses the same parameters: kernel=3 (kernel size 3*3), stride=1 (kernel stride of 1), and padding='same' (automatic edge padding). By setting the same convolutional parameters, the extracted image feature information matrix has a size of 128*128*3. The specific formula for calculating the output size of the convolution operation is as follows:

[0130] ,

[0131] ,

[0132] in, This represents the height of the output after the convolution operation. The height of the input matrix for the convolution operation. This represents the width of the matrix output by the convolution operation. P is the width of the input matrix for the convolution operation, P is the number of padding pixels, K is the kernel size, and S is the kernel stride.

[0133] In some specific embodiments, such as Figure 4 As shown, similar to the image feature extraction module described above, the radar point cloud feature extraction module also primarily extracts feature information through convolution operations. It includes three convolutional layers of different sizes, and after each convolutional operation, a RescaleNorm layer is performed for layer normalization. An image ReLU activation function layer is interspersed after the last convolutional layer to prevent excessively large feature values ​​from affecting subsequent convolutional operations. The formula used for the RescaleNorm operation is as follows:

[0134] ,

[0135] Where x is the input tensor, For the set scalar, These are the parameters for the entire module model. These are the parameters of the upper convolutional layer. For setting parameter weights, This is the scaling value. This is the offset. Since the size of the input point cloud matrix information matrix is ​​the same as that of the foggy image matrix (except that it has one less channel), the parameters used in the convolution operation in this module are the same as those used in the convolution operation for feature extraction from the foggy image. The final output radar point cloud feature matrix size is 128*128*2.

[0136] In this optional embodiment, RGB images provide rich texture and color information, making them suitable for recognizing the appearance features of objects. Radar point cloud data provides high-precision spatial geometric information (such as distance and shape), is unaffected by illumination, and is suitable for detecting the position and contours of objects. ReLU is proposed as an activation function, making it easier to inversely recover images during image dehazing.

[0137] Optionally, the step of inputting the RGB image features and the radar point cloud features into a first parallel convolution module and a second parallel convolution module for convolution processing to obtain a first feature fusion result and a second feature fusion result, respectively, includes:

[0138] The RGB image features and the radar point cloud features are input into the first parallel convolution module for convolution processing to obtain the first feature fusion result. The first parallel convolution module includes a first RGB convolution branch and a first radar point cloud convolution branch.

[0139] The first feature fusion result includes:

[0140] ,

[0141] in, The result of the first feature fusion. The output result of the RGB image features after processing by the first RGB convolution branch. The output result of the radar point cloud features after processing by the first radar point cloud convolutional branch;

[0142] The RGB image features and the radar point cloud features are input into the second parallel convolution module for convolution processing to obtain the second feature fusion result. The second parallel convolution module includes a second RGB convolution branch and a second radar point cloud convolution branch.

[0143] The result of the second feature fusion is as follows:

[0144] ,

[0145] in, The result of the second feature fusion. The output result of the RGB image features after processing by the second RGB convolution branch. The output result is the radar point cloud features processed by the second radar point cloud convolution branch.

[0146] Optionally, the step of performing feature fusion using the first feature fusion result and the second feature fusion result to obtain a multimodal image fusion result includes:

[0147] The multimodal image fusion result is obtained by performing regularization processing on the first feature fusion result and the second feature fusion result;

[0148] The multimodal image fusion result is as follows:

[0149] ,

[0150] Wherein, Fusion_feature is the result of the multimodal image fusion. The result of the first feature fusion. The result of the second feature fusion is represented by Conv(), which performs convolution processing, BN(), which performs normalization processing, and ReLU(), which performs ReLU activation function processing.

[0151] In some more specific embodiments, the first parallel convolutional module Diff (difference branch) and the second parallel convolutional module Sum (sum branch) are used to process foggy image feature information and radar point cloud feature information, respectively. In the Diff branch, the preprocessed foggy image feature information and radar point cloud feature information are matrices of 128*128*3 and 128*128*2, respectively. The foggy image feature information data is input into the first sub-branch of the Diff branch for image feature enhancement, which is achieved through linear connection of convolution and ReLU modules. The result after each convolution operation is recorded. After the last convolution operation, the results of each convolution operation are summed to obtain the final enhanced image feature information, which is a 64*64*3 matrix. The radar point cloud feature information is input into the second sub-branch of the Diff branch and undergoes the same processing as the foggy image feature information, finally obtaining the enhanced radar point cloud features, which are matrices of 64*64*2. Then, a subtraction operation is performed on the obtained feature values. Specifically, the value of each channel of the image feature information matrix is ​​subtracted from the sum of the values ​​of the two channels of the radar point cloud feature matrix. This yields the final result of the Diff branch. The first parallel convolution module includes a first RGB convolution branch and a first radar point cloud convolution branch. The first RGB convolution branch includes:

[0152] ,

[0153] ,

[0154] ,

[0155] in, For the k-th convolutional layer, This represents the output of the i-th convolutional layer in the first RGB convolutional branch, where i is 1, 2, or 3. The first radar point cloud convolutional branch includes:

[0156] ,

[0157] ,

[0158] ,

[0159] in, The final difference calculation for the output of the i-th convolutional layer of the first radar point cloud convolutional branch includes:

[0160] .

[0161] The operations in the Sum branch are similar to those in the Diff branch, except that the final subtraction and absolute value operation is changed to an addition operation. The two results obtained from the convolution operations in the two sub-branches are denoted as follows: and The final result obtained from this branch includes:

[0162] The results from the two branches are convolved and regularized respectively. Then, the two matrices of the same size are added together. In this way, the final matrix contains the pixel features of the original foggy image as well as the intensity and distance information of the radar point cloud. That is, the final result contains the pixel features and contour features of the original image.

[0163] like Figure 5 As shown in the figure, an embodiment of the present invention provides a multimodal image dehazing device, comprising:

[0164] Feature extraction module 10 is used to extract features from RGB image and radar point cloud data respectively to obtain RGB image features and radar point cloud features;

[0165] The convolution processing module 20 is used to input the RGB image features and the radar point cloud features into the first parallel convolution module and the second parallel convolution module for convolution processing, and obtain the first feature fusion result and the second feature fusion result respectively. The first parallel convolution module is used to perform matrix subtraction operation, and the second parallel convolution module is used to perform matrix addition operation.

[0166] Image fusion module 30 is used to perform feature fusion using the first feature fusion result and the second feature fusion result to obtain a multimodal image fusion result;

[0167] The image dehazing module 40 is used to input the multimodal image fusion result into the image dehazing model based on the improved visual processing algorithm to obtain a fused dehazed image. The image dehazing model includes a target encoder, a target decoder, and a feature enhancement module, and the feature enhancement module is built based on Vision Transformer.

[0168] The multimodal image dehazing device of this embodiment is used to implement the multimodal image dehazing method described above. Its advantages over the prior art are the same as the advantages of the multimodal image dehazing method compared with the prior art, and will not be repeated here.

[0169] Optionally, the multimodal image dehazing device further includes a model building module, which is used to: input the historical multimodal image fusion result into the initial encoder for feature enhancement to obtain the initial encoder image;

[0170] The feature enhancement module is trained using the initial encoder image to obtain a visually processed image;

[0171] The visually processed image is input into the initial decoder for image reconstruction to obtain the initial decoder image;

[0172] The total loss value is obtained by calculating the mean squared error loss function based on the initial encoder image and the initial decoder image;

[0173] The model parameters of the initial encoder and the initial decoder are adjusted according to the total loss value until the total loss value meets the preset conditions. The initial encoder and the initial decoder after parameter adjustment are then used as the target encoder and the target decoder, respectively.

[0174] Optionally, the model building module is further configured to: calculate the main loss value by performing a mean squared error loss function on the input image and the output image;

[0175] The mean squared error loss function is calculated by processing the image through the first convolutional layer and the image through the fourth convolutional layer to obtain the regularization term loss value;

[0176] The mean squared error loss function is calculated by processing the image through the second convolutional layer and the image through the third convolutional layer to obtain the constraint term loss value;

[0177] The total loss value is obtained based on the main loss value, the regularization term loss value, and the constraint term loss value.

[0178] The total loss value includes:

[0179] ,

[0180] in, The total loss value, The main loss value, The loss value of the regularization term. The loss value is the constraint term.

[0181] Optionally, the model building module is further configured to: obtain a decomposed image by decomposing the initial encoder image;

[0182] The visually processed image is obtained from the decomposed image based on the multi-head attention mechanism.

[0183] Optionally, the feature extraction module 10 is specifically used to: acquire the RGB image and the radar point cloud data;

[0184] The RGB image is input into the image feature extraction module to obtain the RGB image features, wherein the image feature extraction module includes an image convolutional layer and an image ReLU activation function layer;

[0185] The radar point cloud data is input into the radar point cloud feature extraction module to obtain the radar point cloud features. The radar point cloud feature extraction module includes a radar point cloud convolutional layer, a radar point cloud normalization layer, and a radar point cloud ReLU activation function layer.

[0186] Optionally, the convolution processing module 20 is specifically used to: input the RGB image features and the radar point cloud features into the first parallel convolution module for convolution processing to obtain the first feature fusion result, wherein the first parallel convolution module includes a first RGB convolution branch and a first radar point cloud convolution branch;

[0187] The first feature fusion result includes:

[0188] ,

[0189] in, The result of the first feature fusion. The output result of the RGB image features after processing by the first RGB convolution branch. The output result of the radar point cloud features after processing by the first radar point cloud convolutional branch;

[0190] The RGB image features and the radar point cloud features are input into the second parallel convolution module for convolution processing to obtain the second feature fusion result. The second parallel convolution module includes a second RGB convolution branch and a second radar point cloud convolution branch.

[0191] The result of the second feature fusion is as follows:

[0192] ,

[0193] in, The result of the second feature fusion. The output result of the RGB image features after processing by the second RGB convolution branch. The output result is the radar point cloud features processed by the second radar point cloud convolution branch.

[0194] Optionally, the image fusion module 30 is specifically used to: perform regularization processing on the first feature fusion result and the second feature fusion result, and then fuse the features to obtain the multimodal image fusion result;

[0195] The multimodal image fusion result is as follows:

[0196] ,

[0197] Wherein, Fusion_feature is the result of the multimodal image fusion. The result of the first feature fusion. The result of the second feature fusion is represented by Conv(), which performs convolution processing, BN(), which performs normalization processing, and ReLU(), which performs ReLU activation function processing.

[0198] like Figure 6 As shown, an electronic device 600 provided in this embodiment of the invention includes a memory 610 and a processor 620; the memory 610 is used to store a computer program; the processor 620 is used to implement the multimodal image dehazing method as described above when the computer program is executed.

[0199] Alternatively, an electronic device 600 includes a memory 610 and a processor 620 coupled to the memory 610; the memory 610 is configured to store a computer program; and the processor 620 is configured to perform the following operations when the computer program is executed:

[0200] Feature extraction was performed on RGB images and radar point cloud data respectively to obtain RGB image features and radar point cloud features;

[0201] The RGB image features and the radar point cloud features are input into a first parallel convolution module and a second parallel convolution module for convolution processing to obtain a first feature fusion result and a second feature fusion result, respectively. The first parallel convolution module is used to perform matrix subtraction operations, and the second parallel convolution module is used to perform matrix addition operations.

[0202] By fusing the first feature fusion result and the second feature fusion result, a multimodal image fusion result is obtained;

[0203] Based on the improved visual processing algorithm, the multimodal image fusion result is input into the image dehazing model to obtain the fused dehazed image. The image dehazing model includes a target encoder, a target decoder, and a feature enhancement module, and the feature enhancement module is built based on Vision Transformer.

[0204] This invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the multimodal image dehazing method described above.

[0205] Alternatively, a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the following operations:

[0206] Feature extraction was performed on RGB images and radar point cloud data respectively to obtain RGB image features and radar point cloud features;

[0207] The RGB image features and the radar point cloud features are input into a first parallel convolution module and a second parallel convolution module for convolution processing to obtain a first feature fusion result and a second feature fusion result, respectively. The first parallel convolution module is used to perform matrix subtraction operations, and the second parallel convolution module is used to perform matrix addition operations.

[0208] By fusing the first feature fusion result and the second feature fusion result, a multimodal image fusion result is obtained;

[0209] Based on the improved visual processing algorithm, the multimodal image fusion result is input into the image dehazing model to obtain the fused dehazed image. The image dehazing model includes a target encoder, a target decoder, and a feature enhancement module, and the feature enhancement module is built based on Vision Transformer.

[0210] Electronic device 600, which can serve as a server or client of the present invention, is described below as an example of a hardware device applicable to various aspects of the present invention. Electronic device 600 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 600 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0211] Electronic device 600 includes a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or a computer program loaded from a storage unit into random access memory (RAM). The RAM may also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0212] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0213] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A multi-modal image defogging method, characterized in that, The method comprises the following steps: Feature extraction is performed on the RGB image and the radar point cloud data respectively to obtain RGB image features and radar point cloud features; The RGB image features and the radar point cloud features are input into a first parallel convolution module and a second parallel convolution module for convolution processing to obtain first feature fusion results and second feature fusion results respectively, wherein the first parallel convolution module is used for matrix subtraction operation, and the second parallel convolution module is used for matrix addition operation, comprising: The RGB image features and the radar point cloud features are input into the first parallel convolution module for convolution processing to obtain the first feature fusion results, wherein the first parallel convolution module comprises a first RGB convolution branch and a first radar point cloud convolution branch; The RGB image features and the radar point cloud features are input into the second parallel convolution module for convolution processing to obtain the second feature fusion results, wherein the second parallel convolution module comprises a second RGB convolution branch and a second radar point cloud convolution branch; Feature fusion is performed through the first feature fusion results and the second feature fusion results to obtain multi-modal image fusion results; The multi-modal image fusion results are input into an image defogging model based on a visual processing improvement algorithm to obtain a fusion defogging image, wherein the image defogging model comprises a target encoder, a target decoder and a feature enhancement module, the feature enhancement module is constructed based on a Vision Transformer, and a construction process of the image defogging model comprises: The historical multi-modal image fusion results are input into an initial encoder for feature enhancement to obtain an initial encoder image; The feature enhancement module is trained through the initial encoder image to obtain a visual processing image; The visual processing image is input into an initial decoder for image restoration to obtain an initial decoder image; Mean square error loss function calculation is performed according to the initial encoder image and the initial decoder image to obtain a total loss value; The model parameters of the initial encoder and the initial decoder are adjusted according to the total loss value until the total loss value meets a preset condition, and the initial encoder and the initial decoder after parameter adjustment are taken as the target encoder and the target decoder respectively.

2. The multi-modal image defogging method of claim 1, wherein, The initial encoder image comprises an input image, a first convolution layer processing image and a second convolution layer processing image, the initial decoder image comprises a third convolution layer processing image, a fourth convolution layer processing image and an output image, and the mean square error loss function calculation is performed according to the initial encoder image and the initial decoder image to obtain a total loss value, comprising: Mean square error loss function calculation is performed through the input image and the output image to obtain a main loss value; Mean square error loss function calculation is performed through the first convolution layer processing image and the fourth convolution layer processing image to obtain a regular term loss value; Mean square error loss function calculation is performed through the second convolution layer processing image and the third convolution layer processing image to obtain a constraint term loss value; The total loss value is obtained according to the main loss value, the regularization term loss value and the constraint term loss value; The total loss value includes: ， wherein, is the total loss value, is the main loss value, is the regularizer loss value, is the constraint loss value.

3. The multi-modal image defogging method of claim 1, wherein, The training of the feature enhancement module through the initial encoder image obtains a visual processing image, including: The initial encoder image is decomposed to obtain a decomposed image; Based on the multi-head attention mechanism, the visual processing image is obtained according to the decomposed image.

4. The multi-modal image defogging method of claim 1, wherein, The feature extraction module is used for respectively extracting features from the RGB image and the radar point cloud data to obtain RGB image features and radar point cloud features. The RGB image and the radar point cloud data are obtained; The RGB image is input into an image feature extraction module to obtain the RGB image features, wherein the image feature extraction module includes an image convolution layer and an image ReLU activation function layer. The radar point cloud data is input into a radar point cloud feature extraction module to obtain the radar point cloud features, wherein the radar point cloud feature extraction module includes a radar point cloud convolution layer, a radar point cloud normalization layer and a radar point cloud ReLU activation function layer.

5. The multi-modal image defogging method of claim 1, wherein, The RGB image features and the radar point cloud features are input into a first parallel convolution module and a second parallel convolution module for convolution processing to obtain a first feature fusion result and a second feature fusion result, respectively, including: The RGB image features and the radar point cloud features are input into the first parallel convolution module for convolution processing to obtain the first feature fusion result, wherein the first parallel convolution module includes a first RGB convolution branch and a first radar point cloud convolution branch. The first feature fusion result includes: ， wherein, is the first feature fusion result, is the output result of the RGB image feature after being processed by the first RGB convolution branch, is the output result of the radar point cloud feature after being processed by the first radar point cloud convolution branch; The RGB image features and the radar point cloud features are input into the second parallel convolution module for convolution processing to obtain the second feature fusion result, wherein the second parallel convolution module includes a second RGB convolution branch and a second radar point cloud convolution branch. The second feature fusion result is: ， wherein, is the second feature fusion result, is the output result of the RGB image feature after being processed by the second RGB convolution branch, is the output result of the radar point cloud feature after being processed by the second radar point cloud convolution branch.

6. The multi-modal image defogging method of claim 1, wherein, The feature fusion is performed through the first feature fusion result and the second feature fusion result to obtain a multi-modal image fusion result, including: The multi-modal image fusion result is obtained through the first feature fusion result and the second feature fusion result after regularization processing and feature fusion. The multi-modal image fusion result includes: ， Wherein, Fusion_feature is the multi-modal image fusion result, is the first feature fusion result, is the second feature fusion result, Conv() is convolution processing, BN() is normalization processing, and ReLU() is ReLU activation function processing.

7. A multi-modal image defogging apparatus, comprising: The feature extraction module is used for respectively extracting features from the RGB image and the radar point cloud data to obtain RGB image features and radar point cloud features. The convolution processing module is used for inputting the RGB image features and the radar point cloud features into a first parallel convolution module and a second parallel convolution module for convolution processing to obtain a first feature fusion result and a second feature fusion result, respectively, wherein the first parallel convolution module is used for matrix subtraction operation, and the second parallel convolution module is used for matrix addition operation, including: The RGB image features and the radar point cloud features are input into the first parallel convolution module for convolution processing to obtain the first feature fusion result, wherein the first parallel convolution module includes a first RGB convolution branch and a first radar point cloud convolution branch. ​ The RGB image features and the radar point cloud features are input into the second parallel convolution module for convolution processing to obtain the second feature fusion result, wherein the second parallel convolution module includes a second RGB convolution branch and a second radar point cloud convolution branch; The image fusion module is configured to perform feature fusion on the first feature fusion result and the second feature fusion result to obtain a multi-modal image fusion result. The image dehazing module is configured to input the multi-modal image fusion result into an image dehazing model based on a visual processing improvement algorithm to obtain a fusion dehazing image, wherein the image dehazing model includes a target encoder, a target decoder, and a feature enhancement module, the feature enhancement module is constructed based on a Vision Transformer, and the construction process of the image dehazing model includes: inputting a historical multi-modal image fusion result into an initial encoder for feature enhancement to obtain an initial encoder image; training the feature enhancement module based on the initial encoder image to obtain a visual processing image; inputting the visual processing image into an initial decoder for image restoration to obtain an initial decoder image; performing mean square error loss function calculation based on the initial encoder image and the initial decoder image to obtain a total loss value; adjusting model parameters of the initial encoder and the initial decoder based on the total loss value until the total loss value meets a preset condition, and taking the initial encoder and the initial decoder after parameter adjustment as the target encoder and the target decoder, respectively.

8. An electronic device, comprising: comprising a memory and a processor; the memory is configured to store a computer program; the processor is configured to implement the multi-modal image dehazing method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the multi-modal image dehazing method of any one of claims 1 to 6 is implemented.

Images