Cross-modal interaction conversion method and device for haze remote sensing image target detection

Abstract

The application discloses a cross-modal interaction conversion method and device for haze remote sensing image target detection. The method comprises the following steps: using an attention module of a cross-modal interaction conversion model to perform a sequential query step and a reverse sequential query step. The sequential query is performed by taking a one-dimensional haze feature sequence as a query, a one-dimensional haze-removed feature sequence as a key and a value. The reverse sequential query is a symmetric operation of the sequential query. The feature sequence is input into a feedforward network of the cross-modal interaction conversion model to obtain a first feature sequence and a second feature sequence. The feature sequence is reshaped into a two-dimensional feature map and input into the cross-modal interaction conversion model for fusion and feature extraction. The extracted feature map is used for target detection. The haze image and the haze-removed image obtained by haze removal are simultaneously input into the cross-modal interaction conversion model, bidirectional deep feature interaction is realized, and the accuracy and robustness of target detection under haze conditions are significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and target detection, and in particular to a cross-modal interactive conversion method and apparatus for target detection in haze remote sensing images. Background Technology

[0002] Remote sensing images play a crucial role in a variety of applications. However, real-world remote sensing images are often affected by severe weather (especially fog and haze), resulting in reduced image contrast, blurred boundaries, and loss of details, which seriously affects the accuracy of target detection.

[0003] Existing remote sensing image target detection methods are typically developed and evaluated under ideal imaging conditions. When these target detection methods are applied to hazy images, their performance degrades significantly. To address the aforementioned problems, existing solutions mainly include: (1) dehazing before detection; (2) training the detector directly on hazy images; and (3) joint detection of clear images.

[0004] However, existing solutions also have corresponding problems. For the dehazing-then-detection approach, although dehazing can improve image clarity, the dehazing process may introduce artifacts or lose high-level semantic information, limiting detection accuracy. For the approach of training the detector directly on the hazy image, it is still limited by the inherent reduced visibility and blurred details of the hazy image. For joint clear image detection, paired hazy images and clear images are required for supervised training, but paired hazy images and clear images are very scarce in real-world scenarios. Summary of the Invention

[0005] Purpose of the invention: This invention provides a cross-modal interactive conversion method and apparatus for target detection in hazy remote sensing images, aiming to solve the technical problem of low accuracy and robustness of target detection in remote sensing images under hazy conditions in the prior art.

[0006] Technical Solution: This invention provides a cross-modal interactive transformation method for target detection in haze remote sensing images, comprising: acquiring haze remote sensing images; performing dehazing processing on the haze remote sensing images to obtain corresponding dehaze remote sensing images; extracting features from the haze remote sensing images and the dehaze remote sensing images respectively to obtain corresponding haze feature patches and dehaze feature patches; converting the haze feature patches and the dehaze feature patches into one-dimensional haze feature sequences and one-dimensional dehaze feature sequences respectively; using the attention module of a cross-modal interactive transformation model built with a Transformer neural network to execute sequential query steps and reverse query steps, wherein: in the sequential query step: using the one-dimensional haze feature sequence as the query and the one-dimensional dehaze feature sequence as the key and value, calculating the corresponding one-dimensional dehaze feature sequence based on the query of the one-dimensional haze feature sequence. The reverse query step involves: using the one-dimensional dehazing feature sequence as the query and the one-dimensional haze feature sequence as the key and value; calculating the corresponding one-dimensional haze feature sequence based on the query; superimposing the one-dimensional haze feature sequence with the corresponding one-dimensional dehazing feature sequence calculated in the sequential query step; inputting this superimposed ...

[0007] Specifically, dimensionality reduction is performed through pooling, converting the two-dimensional haze feature map and dehaze feature map into one-dimensional haze feature sequences and one-dimensional dehaze feature sequences, respectively; learnable location information is then embedded into the one-dimensional haze feature sequences and one-dimensional dehaze feature sequences, respectively.

[0008] Specifically, in the sequential query step and the reverse query step: based on the number h of attention heads in the attention module, the one-dimensional haze feature sequence and the one-dimensional defogging feature sequence are dimensionally split, and the feature sequence of 1 / h dimension corresponding to each attention head is calculated; after each attention head performs query-based calculation of the corresponding key and value for the 1 / h dimension feature sequence, the 1 / h dimension feature sequence is concatenated to obtain the original one-dimensional haze feature sequence and the one-dimensional defogging feature sequence.

[0009] Specifically, the execution sequence query step uses the following formula:

[0010] MCA h-d =softmax(Q h K d T / dk 1 / 2 V d ,

[0011] Among them, MCA h-d Q represents the function that calculates the corresponding key and value based on the query; softmax represents the normalized exponential function; Q represents the normalized exponential function. h K represents a one-dimensional haze feature sequence used as a query parameter. d Let V represent the one-dimensional dehazed feature sequence used as the key parameter, T denote the matrix transpose, and V... d Let d represent the one-dimensional dehazing feature sequence used as a value parameter. k This represents the dimension of the feature sequence corresponding to each attention head.

[0012] Specifically, the reverse query step adopts the same formula structure as the sequential query step, wherein the query parameters are replaced with a one-dimensional defogging feature sequence, and the key parameters and value parameters are replaced with a one-dimensional haze feature sequence.

[0013] Specifically, the feedforward network of the cross-modal interaction conversion model adopts the following formula:

[0014] FFN(x) = ReLU(xW1 + b1)W2 + b2,

[0015] Where FFN(x) represents the feedforward network, x represents the input feature sequence, ReLU represents the activation function, W1 and W2 represent the first and second fully connected weight matrices, respectively, and b1 and b2 represent the first and second bias terms, respectively.

[0016] Specifically, after the first or second feature sequence is reshaped into a two-dimensional feature map, during the process of outputting the two-dimensional feature map by the cross-modal interactive conversion model, nearest neighbor interpolation is used during the training period and bilinear interpolation is used during the inference period; the two-dimensional feature map is superimposed with the corresponding haze feature map block or dehaze feature map block to obtain an optimized two-dimensional feature map; the optimized two-dimensional feature map is then input into the cross-modal interactive conversion model.

[0017] This invention also provides a cross-modal interactive conversion device for target detection in haze remote sensing images, comprising: a dehazing unit, a feature extraction unit, a cross-modal interaction unit, a reshaping unit, and a fusion unit, wherein: the dehazing unit is used to acquire haze remote sensing images, perform dehazing processing on the haze remote sensing images, and obtain corresponding dehaze remote sensing images; the feature extraction unit is used to extract features from the haze remote sensing images and the dehaze remote sensing images respectively, to obtain corresponding haze feature patches and dehaze feature patches, and convert the haze feature patches and the dehaze feature patches into one-dimensional haze feature sequences and one-dimensional dehaze feature sequences respectively; the cross-modal interaction unit is used to execute a sequential query step and a reverse query step using the attention module of a cross-modal interactive conversion model established with a Transformer neural network, wherein: the sequential query step uses the one-dimensional haze feature sequence as the query and the one-dimensional dehaze feature sequence as the key and value, based on... The system calculates a corresponding one-dimensional dehazing feature sequence based on a one-dimensional haze feature sequence query. A reverse query step is performed: using the one-dimensional dehazing feature as the query and the one-dimensional haze feature as the key and value, the corresponding one-dimensional haze feature sequence is calculated based on the one-dimensional dehazing feature sequence query. The reshaping unit is used to superimpose the one-dimensional haze feature sequence with the corresponding one-dimensional dehazing feature sequence calculated in the sequential query step, input it into the feedforward network of the cross-modal interactive transformation model, and output a first feature sequence after passing through a fully connected layer. The system also superimposes the one-dimensional dehazing feature sequence with the corresponding one-dimensional haze feature sequence calculated in the reverse query step, inputs it into the feedforward network of the cross-modal interactive transformation model, and outputs a second feature sequence after passing through a fully connected layer. The fusion unit is used to reshape the first feature sequence and the second feature sequence into two-dimensional feature maps, input them into the cross-modal interactive transformation model for fusion and feature extraction, and use the extracted feature maps for target detection in haze remote sensing images.

[0018] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: By simultaneously inputting haze remote sensing images and dehaze-processed remote sensing images into the cross-modal interactive conversion model, fine-grained, bidirectional deep feature interaction between haze features and dehaze features is achieved, thereby enhancing the discriminative ability of features and suppressing noise and redundant information, thus significantly improving the accuracy and robustness of target detection under haze conditions. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the cross-modal interactive conversion method for target detection in haze remote sensing images provided by the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0021] In existing technologies, target detection using remote sensing images under hazy conditions, whether directly based on training from the original hazy image or relying solely on dehazed images, fails to fully leverage the complementary advantages of the two modalities. While the original hazy image retains key global scene clues such as object location and overall layout, it suffers from low visibility and blurred details. On the other hand, while the dehazed image can restore local details and contrast, it may be accompanied by the introduction of artifacts or the loss of key contextual information.

[0022] In this invention, by using a cross-modal interactive transformation model, the features of both hazy and dehazed images are utilized and cross-modal interactive transformation is performed. This enables deep bidirectional feature mining between hazy and dehazed images, and the fusion of hazy and dehazed images as "full text" is used for feature extraction. This effectively enhances the discriminative ability of features and significantly improves the target detection performance.

[0023] See Figure 1 This is a flowchart illustrating the cross-modal interactive conversion method for target detection in haze remote sensing images provided by the present invention.

[0024] In this embodiment of the invention, haze remote sensing images are acquired, and the haze remote sensing images are defogging processed to obtain corresponding defogging remote sensing images.

[0025] In practice, the process can be performed on three synthetic haze remote sensing datasets: DOTA-Haze, SODA-A-Haze, and DroneVehicle-Haze. For each dataset, dehazing methods such as DCP, AOD-Net, GridDehaze, C2PNet, and RIDCP are introduced to obtain dehazed images.

[0026] In this embodiment of the invention, features are extracted from the haze remote sensing image and the defogging remote sensing image respectively to obtain corresponding haze feature blocks and defogging feature blocks. The haze feature blocks and the defogging feature blocks are then converted into one-dimensional haze feature sequences and one-dimensional defogging feature sequences respectively.

[0027] In practice, features are extracted to obtain corresponding haze feature patches and dehaze feature patches. Rotated RetinaNet can be used as the baseline detector, and ResNet-50 pre-trained on ImageNet can be used as the backbone network. Multi-scale feature extraction is performed at the P3 to P7 levels of the feature pyramid network to obtain haze feature patches and dehaze feature patches.

[0028] In practice, the cross-modal interactive transformation model receives haze feature maps and dehaze feature maps. Since the standard Transformer needs to run on a one-dimensional token sequence, the two-dimensional feature maps must be converted into such sequences.

[0029] Therefore, in this embodiment of the invention, dimensionality reduction is performed by pooling to convert the two-dimensional haze feature blocks and defogging feature blocks into one-dimensional haze feature (token) sequences and one-dimensional defogging feature (token) sequences, respectively.

[0030] In specific implementation, the one-dimensional token sequence originally does not contain location information. In order for the cross-modal interactive conversion model to learn and extract location information from the one-dimensional feature sequence, learnable location information is embedded in the one-dimensional haze feature sequence and the one-dimensional defogging feature sequence, respectively, in this embodiment of the invention.

[0031] In this embodiment of the invention, the attention module of a cross-modal interactive transformation model built with a Transformer neural network is used to perform a sequential query step and a reverse query step, wherein: the sequential query step is performed by using a one-dimensional haze feature sequence as the query and a one-dimensional dehazing feature sequence as the key and value, and calculating the corresponding one-dimensional dehazing feature sequence based on the query of the one-dimensional haze feature sequence; the reverse query step is performed by using a one-dimensional dehazing feature as the query and a one-dimensional haze feature as the key and value, and calculating the corresponding one-dimensional haze feature sequence based on the query of the one-dimensional dehazing feature sequence.

[0032] In practical implementation, cross-modal attention from haze to dehaze is calculated, as well as symmetrical cross-modal attention from dehaze to haze, to achieve bidirectional cross-modal feature exchange. This allows for information complementarity between haze-to-dehaze and dehaze-to-haze aspects, taking into account both the overall layout in haze images and the local details in dehaze images. Furthermore, through two symmetrical calculations, the semantic alignment of the one-dimensional feature sequences and two-dimensional feature maps output by the two branches of the cross-modal interactive transformation model is achieved, facilitating the subsequent fusion of the two two-dimensional feature maps.

[0033] In practical implementation, the cross-modal interaction conversion model of the present invention promotes intermodal feature interaction by introducing a multi-head cross-modal attention mechanism, rather than the traditional self-attention mechanism which only captures intramodal relationships.

[0034] In practice, a one-dimensional haze feature sequence or a one-dimensional defogging feature sequence is used as the query, and the corresponding key value is calculated. Specifically, the query outputs the one-dimensional defogging feature sequence or one-dimensional haze feature sequence that best matches the query one-dimensional haze feature sequence or one-dimensional defogging feature sequence.

[0035] In this embodiment of the invention, based on the number h of attention heads in the attention module, the one-dimensional haze feature sequence and the one-dimensional defogging feature sequence are dimensionally split, and a feature sequence of dimension 1 / h corresponding to each attention head is calculated; after each attention head performs query-based calculation of the corresponding key and value on the 1 / h dimension feature sequence, the 1 / h dimension feature sequence is concatenated to obtain the original one-dimensional haze feature sequence and the one-dimensional defogging feature sequence.

[0036] In practical implementation, the attention head decomposes the C-dimensional vector (one-dimensional in this specific implementation) into h parts, each C / h-dimensional, independently calculating the key value based on query matching, and then concatenating the results back together. This allows the model to focus on cross-modal information at the same location, learning matching rules from haze to dehaze and from dehaze to haze, and then concatenating them to obtain the original feature vector, using parallel computation to obtain a richer feature subspace representation.

[0037] In this embodiment of the invention, the execution sequence query step adopts the following formula:

[0038] MCA h-d =softmax(Q h K d T / d k 1 / 2 V d ,

[0039] Among them, MCA h-d Q represents the function that calculates the corresponding key and value based on the query; softmax represents the normalized exponential function; Q represents the normalized exponential function. h K represents a one-dimensional haze feature sequence used as a query parameter. d Let V represent the one-dimensional dehazed feature sequence used as the key parameter, T denote the matrix transpose, and V... d Let d represent the one-dimensional dehazing feature sequence used as a value parameter. k This represents the dimension of the feature sequence corresponding to each attention head.

[0040] In practice, Q h =X h W Q h K d = X d W K d V d =X d W V d , where X h and X d W represents the one-dimensional haze feature sequence and the one-dimensional defogging feature sequence, respectively. Q h WK d and W V d These represent the learnable projection matrices.

[0041] In this embodiment of the invention, the reverse query step adopts the same formula structure as the sequential query step, wherein the query parameters are replaced with a one-dimensional defogging feature sequence, and the key parameters and value parameters are replaced with a one-dimensional haze feature sequence.

[0042] In this embodiment of the invention, a one-dimensional haze feature sequence is superimposed with the corresponding one-dimensional dehazing feature sequence calculated in the sequential query step, and input into the feedforward network of the cross-modal interactive conversion model. After passing through a fully connected layer, a first feature sequence is output. The one-dimensional dehazing feature sequence is superimposed with the corresponding one-dimensional haze feature sequence calculated in the reverse query step, and input into the feedforward network of the cross-modal interactive conversion model. After passing through a fully connected layer, a second feature sequence is output.

[0043] In practice, to further refine the feature representation and ensure training stability, the token sequence after cross-attention processing is input into the feedforward network for further processing. First, the interactive feature sequence is normalized layer by layer to stabilize the gradient distribution during training. Then, it is output through a two-layer fully connected network. This involves a non-linear transformation of the normalized features to enhance their expressive power.

[0044] In practice, during the above-mentioned layer normalization operation, the original feature sequence and the cross-modal attention feature sequence can be weighted and superimposed to adjust the fusion ratio between the two, so as to avoid the interaction features from overly covering the original modal feature sequence.

[0045] In this embodiment of the invention, after layer normalization, the feedforward network of the cross-modal interaction transformation model adopts the following formula:

[0046] FFN(x) = ReLU(xW1 + b1)W2 + b2,

[0047] Where FFN(x) represents the feedforward network, x represents the input feature sequence, ReLU represents the activation function, W1 and W2 represent the first and second fully connected weight matrices, respectively, and b1 and b2 represent the first and second bias terms, respectively.

[0048] In this embodiment of the invention, the first feature sequence and the second feature sequence are respectively reshaped into two-dimensional feature maps, and input into a cross-modal interactive transformation model for fusion (fusion of two two-dimensional feature maps) and feature extraction. The extracted feature maps are then used for target detection in haze remote sensing images.

[0049] In this embodiment of the invention, after the first feature sequence or the second feature sequence is reshaped into a two-dimensional feature map, during the process of outputting the two-dimensional feature map by the cross-modal interactive conversion model, nearest neighbor interpolation is used during the training period and bilinear interpolation is used during the inference period; the two-dimensional feature map is superimposed with the corresponding haze feature map block or dehaze feature map block to obtain an optimized two-dimensional feature map; the optimized two-dimensional feature map is input into the cross-modal interactive conversion model.

[0050] In practice, when the residuals of the two-dimensional feature map reconstructed from the first feature sequence (original query input is haze feature) are added together, the superimposed feature map should be haze feature map blocks; when the residuals of the two-dimensional feature map reconstructed from the second feature sequence (original query input is defogging feature) are added together, the superimposed feature map should be defogging feature map blocks.

[0051] In the specific implementation, during the process of cross-modal interactive transformation model outputting two-dimensional feature maps, a "nearest neighbor interpolation during training + bilinear interpolation during inference" scheme is adopted. Nearest neighbor interpolation is used during training to improve training speed and reduce computational overhead, while bilinear interpolation is used during inference to improve the smoothness of feature maps and reduce edge jagged artifacts. Finally, the refined features after upsampling are added to the residuals of the input original branch features haze feature map patch and dehaze feature map patch to retain the low-order spatial information (such as edges and textures) of the original features, while incorporating the high-order semantic information after cross-modal interaction.

[0052] In practical implementation, introducing a cross-modal interaction transformation model can promote semantic-level interaction between the two feature branches. This improvement significantly enhances detection accuracy, with mAP reaching 58.4%, highlighting the importance of deep cross-modal feature alignment. This also significantly enhances the robustness of target detection in remote sensing images under blurred conditions.

[0053] This invention also provides a cross-modal interactive conversion device for target detection in haze remote sensing images, comprising: a dehazing unit, a feature extraction unit, a cross-modal interaction unit, a reshaping unit, and a fusion unit, wherein: the dehazing unit is used to acquire haze remote sensing images, perform dehazing processing on the haze remote sensing images, and obtain corresponding dehaze remote sensing images; the feature extraction unit is used to extract features from the haze remote sensing images and the dehaze remote sensing images respectively, to obtain corresponding haze feature patches and dehaze feature patches, and convert the haze feature patches and the dehaze feature patches into one-dimensional haze feature sequences and one-dimensional dehaze feature sequences respectively; the cross-modal interaction unit is used to execute a sequential query step and a reverse query step using the attention module of a cross-modal interactive conversion model established with a Transformer neural network, wherein: the sequential query step uses the one-dimensional haze feature sequence as the query and the one-dimensional dehaze feature sequence as the key and value, based on... The system calculates a corresponding one-dimensional dehazing feature sequence based on a one-dimensional haze feature sequence query. A reverse query step is performed: using the one-dimensional dehazing feature as the query and the one-dimensional haze feature as the key and value, the corresponding one-dimensional haze feature sequence is calculated based on the one-dimensional dehazing feature sequence query. The reshaping unit is used to superimpose the one-dimensional haze feature sequence with the corresponding one-dimensional dehazing feature sequence calculated in the sequential query step, input it into the feedforward network of the cross-modal interactive transformation model, and output a first feature sequence after passing through a fully connected layer. The system also superimposes the one-dimensional dehazing feature sequence with the corresponding one-dimensional haze feature sequence calculated in the reverse query step, inputs it into the feedforward network of the cross-modal interactive transformation model, and outputs a second feature sequence after passing through a fully connected layer. The fusion unit is used to reshape the first feature sequence and the second feature sequence into two-dimensional feature maps, input them into the cross-modal interactive transformation model for fusion and feature extraction, and use the extracted feature maps for target detection in haze remote sensing images.

[0054] In specific implementations, the cross-modal interactive conversion device for target detection in haze remote sensing images provided by the present invention, wherein the methods, steps or functions executed by each execution unit can refer to the cross-modal interactive conversion method for target detection in haze remote sensing images provided by the present invention.

Claims

1. A cross-modal interactive conversion method for haze remote sensing image target detection, characterized in that, The method comprises the following steps: acquiring a haze remote sensing image, performing haze removal processing on the haze remote sensing image, and acquiring a corresponding haze-removed remote sensing image; performing feature extraction on the haze remote sensing image and the haze-removed remote sensing image respectively to obtain corresponding haze feature blocks and haze-removed feature blocks, and converting the haze feature blocks and the haze-removed feature blocks into one-dimensional haze feature sequences and one-dimensional haze-removed feature sequences respectively; using an attention module of a cross-modal interaction conversion model established by a Transformer neural network to perform a sequential query step and a reverse sequential query step, wherein: in the sequential query step, the one-dimensional haze feature sequence is taken as a query, the one-dimensional haze-removed feature sequence is taken as a key and a value, and the corresponding one-dimensional haze-removed feature sequence is calculated based on the one-dimensional haze feature sequence query; in the reverse sequential query step, the one-dimensional haze-removed feature sequence is taken as a query, the one-dimensional haze feature sequence is taken as a key and a value, and the corresponding one-dimensional haze feature sequence is calculated based on the one-dimensional haze-removed feature sequence query; stacking the one-dimensional haze feature sequence and the corresponding one-dimensional haze-removed feature sequence calculated in the sequential query step, inputting the stacked sequence into a feedforward network of the cross-modal interaction conversion model, and outputting a first feature sequence through a full connection layer; stacking the one-dimensional haze-removed feature sequence and the corresponding one-dimensional haze feature sequence calculated in the reverse sequential query step, inputting the stacked sequence into the feedforward network of the cross-modal interaction conversion model, and outputting a second feature sequence through the full connection layer; reshaping the first feature sequence and the second feature sequence into two-dimensional feature maps respectively, and inputting the two-dimensional feature maps into the cross-modal interaction conversion model for fusion and feature extraction, and using the extracted feature map for haze remote sensing image target detection. 2.The cross-modal interaction conversion method for haze remote sensing image target detection of claim 1, wherein, The conversion of the haze feature blocks and the haze-removed feature blocks into one-dimensional haze feature sequences and one-dimensional haze-removed feature sequences respectively comprises: performing dimension reduction processing through pooling to convert the two-dimensional haze feature blocks and the haze-removed feature blocks into one-dimensional haze feature sequences and one-dimensional haze-removed feature sequences respectively; embedding learnable position information in the one-dimensional haze feature sequences and the one-dimensional haze-removed feature sequences. 3.The cross-modal interaction conversion method for haze remote sensing image target detection of claim 1, wherein, In the sequential query step and the reverse sequential query step: according to the number h of attention heads in the attention module, the one-dimensional haze feature sequence and the one-dimensional haze-removed feature sequence are dimensionally split to calculate a feature sequence of 1 / h dimensions corresponding to each attention head; each attention head performs a query-based calculation on the corresponding key and value to obtain a feature sequence of 1 / h dimensions, and then splices the feature sequence of 1 / h dimensions to obtain the original one-dimensional haze feature sequence and the one-dimensional haze-removed feature sequence.

4. The cross-modal interaction conversion method for haze remote sensing image target detection according to claim 3, characterized in that, The sequential query step adopts the following formula: MCA h-d = softmax(Q h K d T / d k 1 / 2 )V d , where MCA h-d denotes a function that computes the corresponding key and value based on the query, softmax denotes a normalized exponential function, Q h denotes a one-dimensional haze feature sequence as a query parameter, K d denotes a one-dimensional dehaze feature sequence as a key parameter, T denotes a matrix transpose, V d denotes a one-dimensional dehaze feature sequence as a value parameter, d k denotes the dimension of the feature sequence corresponding to each attention head.

5. The cross-modal interaction conversion method for haze remote sensing image target detection according to claim 4, characterized in that, The reverse sequential query step adopts the same formula structure as the sequential query step, wherein the query parameter is replaced by the one-dimensional haze-removed feature sequence, and the key parameter and the value parameter are replaced by the one-dimensional haze feature sequence.

6. The cross-modal interaction conversion method for haze remote sensing image target detection according to claim 5, characterized in that, The feedforward network of the cross-modal interaction conversion model adopts the following formula: FFN(x) = ReLU(xW1 + b1)W2 + b2, Wherein, FFN(x) represents a feedforward network, x represents an input feature sequence, ReLU represents an activation function, W1 and W2 represent a first full connection weight matrix and a second full connection weight matrix respectively, and b1 and b2 represent a first bias term and a second bias term respectively.

7. The cross-modal interaction conversion method for haze remote sensing image target detection according to claim 5, characterized in that, The first feature sequence and the second feature sequence are reshaped into two-dimensional feature maps, and are input into a cross-modal interaction conversion model for fusion and feature extraction. After the first feature sequence or the second feature sequence is reshaped into a two-dimensional feature map, during the process of outputting the two-dimensional feature map by the cross-modal interaction conversion model, nearest neighbor interpolation is used during the training period, and bilinear interpolation is used during the inference period. The two-dimensional feature map is superimposed with the corresponding haze feature map block or the dehazing feature map block to obtain an optimized two-dimensional feature map; and the optimized two-dimensional feature map is input into the cross-modal interaction conversion model.

8. A cross-modal interactive conversion device for haze remote sensing image target detection, characterized in that, It comprises: a dehazing unit, a feature extraction unit, a cross-modal interaction unit, a reshaping unit and a fusion unit, wherein: The dehazing unit is configured to obtain a haze remote sensing image, and perform dehazing processing on the haze remote sensing image to obtain a corresponding dehazing remote sensing image. The feature extraction unit is configured to perform feature extraction on the haze remote sensing image and the dehazing remote sensing image respectively to obtain corresponding haze feature map blocks and dehazing feature map blocks, and convert the haze feature map blocks and the dehazing feature map blocks into one-dimensional haze feature sequences and one-dimensional dehazing feature sequences respectively. The cross-modal interaction unit is configured to use an attention module of a cross-modal interaction conversion model established by a Transformer neural network to perform a sequential query step and a reverse sequential query step, wherein: in the sequential query step, the one-dimensional haze feature sequence is taken as a query, the one-dimensional dehazing feature sequence is taken as a key and a value, and a corresponding one-dimensional dehazing feature sequence is calculated based on the one-dimensional haze feature sequence query; in the reverse sequential query step, the one-dimensional dehazing feature is taken as a query, the one-dimensional haze feature is taken as a key and a value, and a corresponding one-dimensional haze feature sequence is calculated based on the one-dimensional dehazing feature sequence query. The reshaping unit is configured to superimpose the one-dimensional haze feature sequence and the corresponding one-dimensional dehazing feature sequence calculated in the sequential query step, input a feedforward network of the cross-modal interaction conversion model, and output a first feature sequence through a full connection layer; and superimpose the one-dimensional dehazing feature sequence and the corresponding one-dimensional haze feature sequence calculated in the reverse sequential query step, input the feedforward network of the cross-modal interaction conversion model, and output a second feature sequence through the full connection layer. The fusion unit is configured to reshape the first feature sequence and the second feature sequence into two-dimensional feature maps respectively, and input them into the cross-modal interaction conversion model for fusion and feature extraction, and use the extracted feature map for haze remote sensing image target detection.

Images