A method for road segmentation based on low-light-SAR cross-modal integrated feature fusion

Abstract

This invention belongs to the field of image processing technology and discloses a method for road segmentation based on low-light-SAR cross-modal integrated feature fusion. The method includes: acquiring low-light images and SAR images; extracting features from the low-light and SAR images using low-light feature extraction branches and SAR feature extraction branches respectively, with the low-light image requiring edge feature guidance structure for feature extraction; obtaining low-light coded features and SAR coded features through CBS convolutional blocks; fusing the low-light coded features and SAR coded features using a cross-modal shared feature extraction model (SFEM), and inputting them into a segmentation decoder and a fusion decoder respectively. The segmentation decoder decodes the features to obtain road segmentation results, while the fusion decoder reconstructs the shared features and outputs a fused image, thus constructing a low-light-SAR model. This invention, through a network constraint model, simultaneously possesses the ability to model spatial and semantic information, improving the accuracy of the lighting road segmentation network when generating high-quality fused images.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for road segmentation based on low-light-SAR cross-modal integrated feature fusion. Background Technology

[0002] Currently, multi-source remote sensing data used for road segmentation methods include low-light images and Synthetic Aperture Radar (SAR) images. These two types of images have different characteristics and advantages. Low-light images refer to images where low-light detectors passively capture light signals at night or under low-light conditions and convert these signals into visible images. They can display nighttime urban lighting conditions and are therefore widely used in road extraction, urban planning, and other fields. However, low-light images are highly dependent on lighting and weather conditions and are easily affected by haze, resulting in low signal-to-noise ratios. In contrast, SAR images are generated using active microwave imaging technology. This technology can penetrate clouds, smoke, and some vegetation, providing high-precision topographic information and has all-weather, all-time operation capabilities, unaffected by lighting and weather conditions. However, the grayscale or pseudo-color display format of SAR images differs significantly from human visual habits and is susceptible to noise such as speckle, making image interpretation more difficult.

[0003] Current research on low-light and SAR remote sensing image fusion methods and applications is relatively limited. Existing image fusion techniques tend to generate fused images that meet human visual perception and high evaluation metrics, but neglect the contribution of information from the fused image to road segmentation tasks, leading to excessive independence between image fusion and road segmentation. A few methods attempt to associate image fusion tasks with advanced visual tasks such as object detection and segmentation, using these tasks to guide the image fusion process. However, these methods cannot complete the image fusion and advanced visual tasks in an end-to-end manner, and achieving a performance balance between the two during training is difficult.

[0004] There is a lack of patented technologies for cross-modal fusion of low signal-to-noise ratio (SNR) low-light data and synthetic aperture radar (SAR) data. Existing patents focus excessively on improving the visual quality metrics of the fused image (such as sharpness and contrast), neglecting to optimize features for subsequent advanced visual tasks. This results in image fusion tasks being too independent of advanced visual tasks. Although Transformer encoders effectively improve the quality of fused images, the optimization goal of this method is limited to improving the visual effect or local feature complementarity of the fused image. It fails to establish a correlation constraint with advanced visual tasks such as semantic segmentation and object detection. As a result, the generated fused image may retain redundant features irrelevant to the task or lose key discriminative semantic information, making it difficult to meet the requirements of advanced visual tasks for high-level abstract semantic features. Summary of the Invention

[0005] To address the shortcomings of existing technologies in the background section, this invention proposes a method for road segmentation based on low-light-SAR cross-modal integrated feature fusion.

[0006] The technical solution adopted by this invention to solve this problem is: an agricultural hyperspectral image classification network based on a spatial-spectral sensing network. This invention firstly includes the following steps:

[0007] Step S1: Acquire low-light images and SAR images, and preprocess the low-light images;

[0008] Step S2: The low-light image and SAR image are processed separately through a dual-branch structure: the low-light detail features and SAR detail features are extracted using the Conv2D convolutional layer, and the edge feature information of the road is enhanced by the Sobel operator and 1×1 convolution in the low-light image.

[0009] Step S3: Based on low-light detail features, SAR detail features and edge feature information, after obtaining low-light coding features and SAR coding features with the help of CBS convolutional blocks, the features are shared through the cross-modal shared feature extraction module SFEM.

[0010] Step S4: Obtain the segmentation result and fused image through the segmentation feature decoder and fusion feature decoder, and construct the low-light-SAR cross-modal integrated feature fusion and road segmentation method model, namely the low-light-SAR model;

[0011] Step S5: Train the overall loss function of the low-light SAR model;

[0012] Step S6: Evaluate the performance of the low-light SAR model.

[0013] Preferably, in step S1, the specific process of acquiring the low-light image and the SAR image, and preprocessing the low-light image is as follows:

[0014] (1) The process of acquiring low-light images and SAR images is as follows:

[0015] Obtain panchromatic low-light images and SAR images with a resolution of 10 meters from remote sensing images;

[0016] (3) The process of preprocessing low-light images is as follows:

[0017] Threshold segmentation is performed on the low-light image, with the segmentation threshold set to (5, 255), and a binary image is output. Then, the binary image is processed using a dilation method with a kernel of size (5, 5) and a dilation iteration count of 3. Finally, a distance transform method is used to process the disconnected roads with large intervals, and the processed low-light image is output.

[0018] Preferably, in step S2, the specific process of the dual-branch structure processing the low-light image and the SAR image is as follows:

[0019] Low-light images and SAR images are used as input data. The low-light images are input into the low-light feature extraction branch, and the SAR images are input into the SAR feature extraction branch. A 2D convolutional layer Conv2D is used to extract the detailed features of the low-light images and SAR images, and the number of channels of the low-light images and SAR images is modified to obtain low-light detail features and SAR detail features. At the same time, the low-light images are input into the edge feature guidance structure, and the edge feature information of the road is extracted by the Sobel operator. Then, a 1×1 convolutional layer Conv is used to adjust the extracted edge feature information.

[0020] Preferably, in step S3, the specific process of obtaining micro-light coding features and SAR coding features with the help of CBS convolutional blocks, and sharing features through the cross-modal shared feature extraction module SFEM includes:

[0021] Step S3.1: For the low-light detail features, the low-light feature extraction branch uses CBS convolutional blocks for feature encoding to obtain low-light encoded features. The specific process is as follows: First, use convolution to extract features from the low-light detail features, then perform batch normalization (BN) processing, and finally use the SiLU activation function to activate the feature information to obtain low-light feature information. Then, concatenate the low-light feature information with the corresponding edge feature information in the channel dimension to obtain the low-light encoded features.

[0022] For SAR detail features, the SAR feature extraction branch uses CBS convolutional blocks to encode SAR detail features. The specific process is as follows: first, SAR detail features are extracted using convolution, then batch normalization (BN) processing is performed, and finally the SiLU activation function is used to activate the feature information, thereby obtaining SAR encoded features.

[0023] Step S3.2: In the cross-modal shared feature extraction module SFEM, the low-light coded features and SAR coded features are input into the SFEM module. First, deformable convolution DeConv is used to perform irregular extraction on the input low-light coded features and SAR coded features, adaptively adjusting the local spatial correspondence of the low-light coded features and SAR coded features to obtain low-light features and SAR features. Then, the low-light features and SAR features are separated by channel and spatial dimensions, and global information interaction between the low-light features and SAR features is realized in both channel and spatial dimensions. Finally, the low-light features and SAR features output in the channel and spatial dimensions are aggregated to obtain cross-modal long-range dependency features, which are the shared feature identifiers. The shared feature identifiers are then concatenated with the low-light coded features and SAR coded features in the channel dimension to obtain the concatenated features.

[0024] Step S3.3: Input the spliced ​​features into different CBS convolutional blocks in the next layer to obtain the low-light coding features and SAR coding features of the next layer. Then, input them into the new SFEM module and repeat the processing flow of step S3.2. After performing multi-layer processing, shared features are achieved.

[0025] Preferably, in step S4, the specific process of obtaining the segmentation result and fused image through the segmentation feature decoder and fusion feature decoder, and constructing the low-light-SAR cross-modal integrated feature fusion and road segmentation method model is as follows:

[0026] After processing based on CBS convolutional blocks and the cross-modal shared feature extraction module SFEM, the feature extraction results from the low-light feature extraction branch and the SAR feature extraction branch are concatenated along the channel dimension to obtain fused features. Then, 1×1 convolution is used to reduce the dimensionality of the fused features. After that, the dimensionality-reduced fused features are input into the segmentation feature decoder and the fusion feature decoder in the feature decoding stage to obtain the segmentation results and the fused image.

[0027] The feature decoding stage includes segmentation feature decoding and fusion feature decoding. The segmentation feature decoder consists of four segmentation feature upsampling modules (SDM). The SDM modules are used to aggregate cross-modal semantic shared features and fusion features at the same level, and upsample the dimensionality-reduced fusion features to obtain the segmentation result. The fusion feature decoder consists of four fusion feature upsampling modules (FDM). The FDM modules are used to aggregate cross-modal semantic shared features and fusion features at the same level. Through residual connections, pixel-level features are avoided from being lost after multi-layer convolution extraction, resulting in aggregated features. The aggregated features are then upsampled to reconstruct the fused image, thus constructing a low-light-SAR cross-modal integrated feature fusion and road segmentation method model, namely the low-light-SAR model.

[0028] Preferably, in step S5, the process of training the overall loss function of the low-light SAR model is as follows:

[0029] The overall loss function is designed, which consists of two parts: fusion loss and semantic-driven loss. The expression of the overall loss function is shown in Equation 1:

[0030] L total =L fu +αL seg_dri (1)

[0031] Among them, L total L represents the total loss of the proposed method. fu L represents the fusion loss. seg_dri α represents semantic-driven loss, and α represents the dynamic weighting factor.

[0032] The fusion loss consists of pixel intensity loss and texture loss, and the expression for the fusion loss is shown in Equation 2:

[0033] L fu =L int +λL texture (2)

[0034] Among them, L int L represents pixel intensity loss. texture λ represents the texture loss, and λ represents the weighting factor.

[0035] The expression for pixel intensity loss is shown in Equation 3:

[0036]

[0037] Where H,W represent the image size, I f I represents the fused image reconstructed by the fusion network. w ,I s Let ||·||1 represent a pair of low-light images and SAR images, respectively, where ||·||1 represents L1 regularization and max(·) represents the calculation of the maximum value of the element.

[0038] The texture loss expression is shown in Equation 4:

[0039]

[0040] Where ▽ represents the Sobel gradient operator;

[0041] A semantic-driven strategy based on dynamic factors is designed. By designing dynamically changing weighting factors, the co-optimization of image fusion and illuminated road segmentation tasks is ensured. The weight of the semantic-driven loss in the total loss is adjusted by a dynamic weighting factor α, so that the algorithm tends to rely on the fusion loss to update network parameters in the initial training phase. As training progresses, the dynamic weighting factor gradually increases, and the parameters of the illuminated road segmentation part are effectively updated accordingly, gradually injecting semantic feature information into the fusion features. The definition of α is shown in Equation 5.

[0042]

[0043] Where t∈[0,1] represents the ratio between the number of iterations in the current training process and the total number of iterations, η is the hyperparameter of the semantic-driven loss, and e represents the natural constant, which is a mathematical constant.

[0044] Preferably, in step S6, the performance evaluation process of the low-light SAR model is as follows:

[0045] The Mean Intersection over Union (MIoU) is used as the segmentation evaluation metric to calculate the degree of overlap between the predicted segment and the true segment label, as shown in Equation 6:

[0046]

[0047] Where N represents the number of segmentation categories, TP i FP represents the number of pixels correctly predicted for the i-th category. i FN represents the number of pixels incorrectly predicted for the i-th category. i This represents the number of pixels that actually belong to the i-th category but were not correctly predicted.

[0048] Compared with existing technologies, this invention proposes a road segmentation method based on low-light-SAR cross-modal integrated feature fusion. This invention has the following beneficial effects:

[0049] (1) An integrated cross-modal feature sharing architecture enables collaborative optimization of image fusion and road segmentation.

[0050] This invention couples the low-light and SAR image fusion task and the illuminated road segmentation task into the same framework, and designs a cross-modal shared feature extraction module (SFEM) to enable the two tasks to share cross-modal features, prompting the fusion network to generate feature representations that meet both visual quality requirements and semantic segmentation needs. Through joint optimization, the fusion network can enhance semantic features related to road segmentation while maintaining high-quality images and avoiding the generation of irrelevant and redundant information, thereby improving the overall performance of the task.

[0051] (2) Edge feature-guided structures enhance road edge perception in low-light environments

[0052] This invention introduces an edge feature guidance structure into the low-light image feature extraction branch to explicitly enhance the ability to extract edge information. This structure can highlight the contrast and structural features of road edges during the fusion process, thereby improving the recognition ability of the segmentation network. In particular, it can maintain robust road extraction performance even in low signal-to-noise ratio scenarios, reducing edge blurring and breakage problems.

[0053] (3) Based on the semantic-driven strategy of dynamic factors, adaptive weight balancing among tasks is achieved.

[0054] This invention proposes a semantic-driven strategy based on dynamic factors, which adaptively adjusts the weights of semantic segmentation loss and fusion loss. In the early stages of training, this strategy focuses on optimizing the fusion task to ensure the robustness of the fused features. As training progresses, the influence of the semantic-driven loss is gradually increased, allowing the fused features to continuously adapt to the needs of the road segmentation task. This dynamic adjustment mechanism effectively avoids the interference of semantic tasks on early feature learning, while ensuring that the final model has the ability to balance fusion quality and segmentation accuracy, thereby improving the generalization and practicality of the method in different application scenarios.

[0055] (4) Pioneering cross-modal fusion technology of low light and SAR

[0056] This method proposes a dedicated framework for cross-modal fusion of low-light and SAR data for the first time. Through network design, it combines the detailed texture of low-light images with the strong penetrating features of SAR, thus solving the technical deficiency of multimodal data fusion in low-light environments. Attached Figure Description

[0057] Figure 1 This is a flowchart of the method of the present invention;

[0058] Figure 2 This is a detailed network structure diagram of the method of the present invention;

[0059] Figure 3 This is a flowchart of the preprocessing process for low-light images according to the present invention;

[0060] Figure 4 This is a detailed structural diagram of the cross-modal shared feature extraction module SFEM of the present invention;

[0061] Figure 5 The diagram shows the structure of the segmented upsampling module (SDM) and the fusion upsampling module (FDM) of the present invention, where (a) is the structure of the segmented upsampling module (SDM) and (b) is the structure of the fusion upsampling module (FDM). Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be further described clearly and completely below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0063] To make the inventive objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings: In order to better understand the above-mentioned objectives, features, and advantages of this invention, the advantages of this invention will be further illustrated below by comparing the embodiments with the accompanying drawings and specific implementation methods.

[0064] This invention proposes a method for road segmentation based on low-light-SAR cross-modal integrated feature fusion. The flowchart of the method is as follows: Figure 1 The detailed network structure diagram of the method of the present invention is shown below. Figure 2 As shown, a pair of registered low-light images and SAR images are first input into the network. Next, the network extracts features from them using low-light feature extraction and SAR feature extraction branches, respectively. Specifically, the low-light image also needs to undergo feature extraction via an edge feature-guided structure, and then it is processed through a CBS convolutional block to obtain low-light coded features and SAR coded features. Then, the cross-modal shared feature extraction module (SFEM) is used to fuse these features, and the results are output to the segmentation decoder and fusion decoder, respectively. Finally, the segmentation decoder decodes the cross-modal shared features to obtain road segmentation results, while the fusion decoder reconstructs the cross-modal shared features and outputs the fused image.

[0065] The process of this invention specifically includes the following steps:

[0066] Step S1: Acquire low-light images and SAR images, and preprocess the low-light images;

[0067] Specifically, in step S1, the process of acquiring the low-light image and the SAR image, and preprocessing the low-light image, is as follows:

[0068] (1) The process of acquiring low-light images and SAR images is as follows:

[0069] Obtain panchromatic low-light images and SAR images with a resolution of 10 meters from remote sensing images;

[0070] (2) The process of preprocessing low-light images is as follows:

[0071] The preprocessing flowchart for low-light images is as follows: Figure 3As shown, due to the obstruction of roads by trees, clouds, and fog at night, the low-light data detected by satellite suffers from road disconnection. Morphological and distance transform methods are used to preprocess the low-light images. Specifically, to balance the interference of uneven illumination in the low-light images on subsequent road segmentation, threshold segmentation is performed on the low-light images, with the segmentation threshold set to (5, 255), and a binary image is output. Then, to connect the small-interval disconnected roads, a dilation method is used to process the binary image, employing a kernel of size (5, 5) and three dilation iterations. Finally, a distance transform method is used to process the larger-interval disconnected roads, and the processed low-light image is output. This effectively connects the disconnected roads in the low-light image, reducing the impact of trees, clouds, and fog on the low-light image quality. Furthermore, the processed low-light image maintains consistent road pixel values, avoiding the impact of uneven illumination on subsequent algorithms.

[0072] Step S2: The low-light image and SAR image are processed separately using a dual-branch structure: low-light detail features and SAR detail features are extracted using the Conv2D convolutional layer, and the edge feature information of the road is enhanced by the Sobel operator and 1×1 convolution in the low-light image.

[0073] Specifically, in step S2, the specific process of the dual-branch structure processing the low-light image and the SAR image is as follows:

[0074] Low-light images and SAR images are used as input data. The low-light images are input into the low-light feature extraction branch, and the SAR images are input into the SAR feature extraction branch. The input low-light images and SAR images need to be paired, that is, they correspond to each other geographically. A 2D convolutional layer Conv2D is used to extract texture, edge and other detailed features from the low-light images and SAR images, and the number of channels in the low-light images and SAR images is modified to obtain low-light detail features and SAR detail features, thereby enhancing the feature expression of the images in the channel dimension. At the same time, the low-light images are input into the edge feature guidance structure, and the edge feature information of the road is extracted by the Sobel operator. Then, a 1×1 convolutional layer Conv is used to adjust the extracted edge feature information.

[0075] Step S3: Based on low-light detail features, SAR detail features and edge feature information, after obtaining low-light coding features and SAR coding features with the help of CBS convolutional blocks, the features are shared through the cross-modal shared feature extraction module SFEM.

[0076] Specifically, in step S3, the process of obtaining micro-light coding features and SAR coding features using CBS convolutional blocks and sharing features through the cross-modal shared feature extraction module SFEM includes:

[0077] Step S3.1: For the low-light detail features, the low-light feature extraction branch uses CBS convolutional blocks (CBS1) for feature encoding to obtain low-light encoded features. The specific process is as follows: First, the low-light detail features are extracted using convolution. Then, batch normalization (BN) is performed to make the features have a similar distribution between different batches. Finally, the SiLU activation function is used to activate the feature information. The SiLU activation function introduces nonlinear factors to enhance the expressive power and obtain the low-light feature information. The low-light feature information is then concatenated with the corresponding edge feature information in the channel dimension to obtain the low-light encoded features.

[0078] For SAR detail features, the SAR feature extraction branch uses CBS convolutional blocks (CBS5) to encode SAR detail features. The specific process is as follows: first, SAR detail features are extracted using convolution, then batch normalization (BN) processing is performed, and finally, the SiLU activation function is used to activate the feature information, thereby obtaining SAR coded features.

[0079] Step S3.2: Design a cross-modal shared feature extraction module (SFEM) to extract shared feature identifiers from low-light images and SAR images, and gradually inject the shared feature identifiers into the low-light feature extraction branch and the SAR feature extraction branch to realize the interaction of feature information between low-light images and SAR images;

[0080] In the cross-modal shared feature extraction module SFEM1, micro-light coded features and SAR coded features are input into the SFEM1 module. A detailed structural diagram of the cross-modal shared feature extraction module SFEM is shown below. Figure 4 As shown, "DeConv" represents deformable convolution, Q, K, and V represent query vector, key vector, and value vector, respectively, and "Softmax" represents the Softmax function, which is used to calculate the correlation weights of the low-light coding features and SAR coding features.

[0081] To avoid the interference of the subsequent shared feature identification extraction caused by the shift in position and shape of the same feature in two images, the cross-modal shared feature extraction module SFEM first uses deformable convolution DeConv to perform irregular extraction on the input low-light coded features and SAR coded features, adaptively adjusting the local spatial correspondence of the low-light coded features and SAR coded features to obtain low-light features and SAR features. Then, the low-light features and SAR features are separated by channel and spatial dimensions, and global information interaction between the low-light features and SAR features is realized in both channel and spatial dimensions. Finally, the low-light features and SAR features output in the channel and spatial dimensions are aggregated to obtain cross-modal long-range dependency features, which are the shared feature identifiers. The shared feature identifiers are then concatenated with the low-light coded features and SAR coded features in the channel dimension to obtain the concatenated features.

[0082] Step S3.3: Input the spliced ​​features into different CBS convolutional blocks (CBS2 and CBS6) in the next layer to obtain the low-light coding features and SAR coding features of the next layer. Then, input them into the new SFEM module and repeat the processing flow of step S3.2. After four layers of processing, the shared features are realized.

[0083] In the four-layer processing flow, the four CBS convolutional blocks used in the low-light feature extraction branch are defined as CBS1, CBS2, CBS3 and CBS4 respectively. Similarly, the four CBS convolutional blocks used in the SAR feature extraction branch are defined as CBS5, CBS6, CBS7 and CBS8 respectively.

[0084] Because the CBS convolutional block continuously processes the input low-light detail features and SAR detail features and inputs the features into the SFEM module, a multi-level feature information interaction channel from shallow features to deep semantic features of low-light images and SAR images is established, which processes cross-modal interaction information of different fine granularities respectively. Based on this information interaction channel, the model can gradually combine the shallow details (such as edges and textures) of low-light images and SAR images with deep semantic information (such as object categories and structural features), thereby constructing a robust feature map that can be applied to segmentation and fusion tasks.

[0085] Step S4: Obtain the segmentation result and fused image through the segmentation feature decoder and fusion feature decoder, and construct a low-light-SAR cross-modal integrated feature fusion and road segmentation method model;

[0086] Specifically, in step S4, the process of obtaining the segmentation result and fused image through the segmentation feature decoder and fusion feature decoder, and constructing the low-light-SAR cross-modal integrated feature fusion and road segmentation method model is as follows:

[0087] After processing by CBS convolutional blocks and the cross-modal shared feature extraction module SFEM, the feature extraction results from the low-light feature extraction branch and the SAR feature extraction branch are concatenated in the channel dimension to obtain fused features. Then, 1×1 convolution is used to reduce the dimensionality of the fused features. After that, the dimensionality-reduced fused features are input into the segmentation feature decoder and the fusion feature decoder in the feature decoding stage to obtain the final segmentation results and fused image.

[0088] The structure diagram of the segmented upsampling module (SDM) is as follows: Figure 5 As shown in (a), “BN” represents the batch normalization layer, “ReLU” represents the corrected linear unit, and “Upsample” represents the bilinear interpolation upsampling layer; the FDM structure diagram of the fusion upsampling module is shown below. Figure 5 As shown in (b), "+" indicates residual connection;

[0089] The feature decoding stage includes segmentation feature decoding and fusion feature decoding. The segmentation feature decoder consists of four segmentation upsampling modules (SDM). The SDM modules are used to aggregate cross-modal semantic shared features and fusion features at the same level, and upsample the dimensionality-reduced fusion features to obtain the segmentation result. The fusion feature decoder consists of four fusion feature upsampling modules (FDM). The FDM modules are used to aggregate cross-modal semantic shared features and fusion features at the same level, and through residual connections, avoid the loss of pixel-level features after multi-layer convolution extraction to obtain aggregated features. Then, the aggregated features are upsampled to reconstruct the fused image, thus constructing a low-light-SAR cross-modal integrated feature fusion and road segmentation method model, namely the low-light-SAR model.

[0090] Step S5: Train the overall loss function of the low-light SAR model;

[0091] Specifically, in step S5, the process of training the overall loss function of the low-light SAR model is as follows:

[0092] To generate high-quality fused images and enhance the semantic information in the fused features to improve the road segmentation accuracy of the model, an overall loss function is designed. This overall loss function consists of two parts: fusion loss and semantic-driven loss. The expression of the overall loss function is shown in Equation 1.

[0093] L total =L fu +αL seg_dri (1)

[0094] Among them, L total L represents the total loss of the proposed method. fu L represents the fusion loss. seg_dri α represents semantic-driven loss, and α represents the dynamic weighting factor.

[0095] The fusion loss optimizes the pixel intensity and texture similarity between the fused image and the source image to guide the fusion network in extracting complementary feature information from the low-light image and the SAR image. To achieve this goal, the fusion loss consists of pixel intensity loss and texture loss, and the expression for the fusion loss is shown in Equation 2.

[0096] L fu =L int +λL texture (2)

[0097] Among them, L int L represents pixel intensity loss. texture λ represents the texture loss, and λ represents the weighting factor.

[0098] Pixel intensity loss is used to evaluate the difference in pixel intensity between the fused image and the source image. The expression for pixel intensity loss is shown in Equation 3:

[0099]

[0100] Where H,W represent the image size, I f I represents the fused image reconstructed by the fusion network. w ,I s Let ||·||1 represent a pair of low-light images and SAR images, respectively, where ||·||1 represents L1 regularization and max(·) represents the calculation of the maximum value of the element.

[0101] Texture loss is used to constrain the fusion network to extract complementary texture features from low-light images and SAR images. The texture loss expression is shown in Equation 4:

[0102]

[0103] in, This represents the Sobel gradient operator;

[0104] A semantic-driven strategy based on dynamic factors was designed. By designing dynamically changing weighting factors, the co-optimization of image fusion and illuminated road segmentation tasks is ensured. The weight of the semantic-driven loss in the total loss is adjusted by a dynamic weighting factor α, so that the algorithm tends to rely on the fusion loss to update network parameters in the initial training phase. As training progresses, the dynamic weighting factor gradually increases, and the parameters of the illuminated road segmentation part are effectively updated accordingly, gradually injecting semantic feature information into the fusion features. The definition of α is shown in Equation 5.

[0105]

[0106] Where t∈[0,1] represents the ratio between the number of iterations in the current training process and the total number of iterations, η is the hyperparameter of the semantic-driven loss, and e represents the natural constant, which is a mathematical constant.

[0107] Step S6, performance evaluation of the low-light SAR model;

[0108] Specifically, in step S6, the performance evaluation process of the low-light SAR model is as follows:

[0109] Mean Intersection over Union (MIoU) is used as the segmentation evaluation metric to calculate the degree of overlap between the predicted segment and the true segment label, as shown in Equation 6:

[0110]

[0111] Where N represents the number of segmentation categories, TP i FP represents the number of pixels correctly predicted for the i-th category. i FN represents the number of pixels incorrectly predicted for the i-th category. i This represents the number of pixels that actually belong to the i-th category but were not correctly predicted.

[0112] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0113] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for road segmentation based on low-light-SAR cross-modal integrated feature fusion, characterized in that, include: Step S1: Acquire low-light images and SAR images, and preprocess the low-light images; Step S2: The low-light image and SAR image are processed separately through a dual-branch structure: the low-light detail features and SAR detail features are extracted using the Conv2D convolutional layer, and the edge feature information of the road is enhanced by the Sobel operator and 1×1 convolution in the low-light image. Step S3: Based on low-light detail features, SAR detail features and edge feature information, after obtaining low-light coding features and SAR coding features with the help of CBS convolutional blocks, the features are shared through the cross-modal shared feature extraction module SFEM. Step S4: Obtain the segmentation result and fused image through the segmentation feature decoder and fusion feature decoder, and construct the low-light-SAR cross-modal integrated feature fusion and road segmentation method model, namely the low-light-SAR model; The specific process of constructing a low-light-SAR cross-modal integrated feature fusion and road segmentation method model by obtaining segmentation results and fused images through segmentation feature decoders and fusion feature decoders is as follows: After processing based on CBS convolutional blocks and the cross-modal shared feature extraction module SFEM, the feature extraction results from the micro-light feature extraction branch and the SAR feature extraction branch are spliced ​​together in the channel dimension to obtain fused features, and 1×1 convolution is used to reduce the dimensionality of the fused features. Then, the dimensionality-reduced fused features are input into the segmentation feature decoder and the fusion feature decoder in the feature decoding stage, respectively, to obtain the segmentation result and the fused image; The feature decoding stage includes segmentation feature decoding and fusion feature decoding. The segmentation feature decoder consists of four segmentation feature upsampling modules (SDM). The SDM module is used to aggregate cross-modal semantic shared features and fusion features at the same level, and upsample the dimensionality-reduced fusion features to obtain the segmentation result. The fusion feature decoder contains four fusion feature upsampling modules (FDM). The FDM modules are used to aggregate cross-modal semantic shared features and fusion features at the same level. Through residual connections, aggregated features are obtained. Then, the aggregated features are upsampled to reconstruct the fused image, thus constructing a low-light-SAR cross-modal integrated feature fusion and road segmentation method model, namely the low-light-SAR model. Step S5: Train the overall loss function of the low-light SAR model; Step S6: Evaluate the performance of the low-light SAR model.

2. The method for road segmentation based on low-light-SAR cross-modal integrated feature fusion as described in claim 1, characterized in that, In step S1, the specific process of acquiring low-light images and SAR images, and preprocessing the low-light images is as follows: (1) The process of acquiring low-light images and SAR images is as follows: Obtain panchromatic low-light images and SAR images with a resolution of 10 meters from remote sensing images; (2) The preprocessing process for low-light images is as follows: Threshold segmentation is performed on the low-light image, with the segmentation threshold set to (5, 255), and a binary image is output. Then, the binary image is processed using a dilation method with a kernel of size (5, 5) and a dilation iteration count of 3. Finally, a distance transform method is used to process the disconnected roads with large intervals, and the processed low-light image is output.

3. The method for road segmentation based on low-light-SAR cross-modal integrated feature fusion as described in claim 1, characterized in that, In step S2, the specific process by which the dual-branch structure processes the low-light image and the SAR image is as follows: Low-light images and SAR images are used as input data. The low-light images are input into the low-light feature extraction branch, and the SAR images are input into the SAR feature extraction branch. A 2D convolutional layer Conv2D is used to extract the detailed features of the low-light images and SAR images, and the number of channels of the low-light images and SAR images is modified to obtain low-light detail features and SAR detail features. At the same time, the low-light images are input into the edge feature guidance structure, and the edge feature information of the road is extracted by the Sobel operator. Then, a 1×1 convolutional layer Conv is used to adjust the extracted edge feature information.

4. The method for road segmentation based on low-light-SAR cross-modal integrated feature fusion as described in claim 1, characterized in that, In step S3, the specific process of obtaining micro-light coding features and SAR coding features with the help of CBS convolutional blocks, and sharing features through the cross-modal shared feature extraction module SFEM includes: Step S3.1: For the low-light detail features, the low-light feature extraction branch uses CBS convolutional blocks for feature encoding to obtain low-light encoded features. The specific process is as follows: First, use convolution to extract features from the low-light detail features, then perform batch normalization (BN) processing, and finally use the SiLU activation function to activate the feature information to obtain low-light feature information. Then, concatenate the low-light feature information with the corresponding edge feature information in the channel dimension to obtain the low-light encoded features. For SAR detail features, the SAR feature extraction branch uses CBS convolutional blocks to encode SAR detail features to obtain SAR coded features. The specific process is as follows: first, SAR detail features are extracted using convolution, then batch normalization (BN) processing is performed, and finally the SiLU activation function is used to activate the feature information to obtain SAR coded features. Step S3.2: In the cross-modal shared feature extraction module SFEM, the low-light coded features and SAR coded features are input into the SFEM module. First, deformable convolution DeConv is used to perform irregular extraction on the input low-light coded features and SAR coded features, adaptively adjusting the local spatial correspondence of the low-light coded features and SAR coded features to obtain low-light features and SAR features. Then, the low-light features and SAR features are separated by channel and spatial dimensions, and global information interaction between the low-light features and SAR features is realized in both channel and spatial dimensions. Finally, the low-light features and SAR features output in the channel and spatial dimensions are aggregated to obtain cross-modal long-range dependency features, which are the shared feature identifiers. The shared feature identifiers are then concatenated with the low-light coded features and SAR coded features in the channel dimension to obtain the concatenated features. Step S3.3: Input the spliced ​​features into different CBS convolutional blocks in the next layer to obtain the low-light coding features and SAR coding features of the next layer. Then, input them into the new SFEM module and repeat the processing flow of step S3.

2. After performing multi-layer processing, shared features are achieved.

5. The method for road segmentation based on low-light-SAR cross-modal integrated feature fusion as described in claim 1, characterized in that, In step S5, the process of training the overall loss function of the low-light SAR model is as follows: The overall loss function is designed, which consists of two parts: fusion loss and semantic-driven loss. The expression of the overall loss function is shown in Equation 1: （1） in, This represents the total loss of the proposed method. Indicates fusion loss, This represents semantically driven loss. Indicates dynamic weighting factor; The fusion loss consists of pixel intensity loss and texture loss, and the expression for the fusion loss is shown in Equation 2: (2) in, Indicates pixel intensity loss. Indicates texture loss. Indicates the weighting factor; The expression for pixel intensity loss is shown in Equation 3: （3） in, Indicates image size, The fused image represents the reconstruction from the fused network. These represent a pair of low-light images and SAR images, respectively. Indicates L1 regularization, This indicates the calculation of the maximum value of an element; The texture loss expression is shown in Equation 4: （4） in, This represents the Sobel gradient operator; We design a semantically driven strategy based on dynamic factors. By designing dynamically changing weighting factors, we ensure the synergistic optimization of image fusion and illumination road segmentation tasks. The weights of the semantic-driven loss in the total loss are adjusted so that the algorithm tends to rely on the fusion loss to update the network parameters in the initial training phase. As training progresses, the dynamic weight factor gradually increases, and the parameters of the illuminated road segmentation part are effectively updated accordingly, gradually injecting semantic feature information into the fusion features. The definition is shown in Formula 5: （5） in, This represents the ratio between the current number of iterations in the training process and the total number of iterations. is a hyperparameter for the semantically driven loss magnitude, where e represents the natural constant, which is a mathematical constant.

6. The method for road segmentation based on low-light-SAR cross-modal integrated feature fusion as described in claim 1, characterized in that, In step S6, the performance evaluation process of the low-light SAR model is as follows: The Mean Intersection over Union (MIoU) is used as the segmentation evaluation metric to calculate the degree of overlap between the predicted segment and the true segment label, as shown in Equation 6: （6） in, Indicates the number of segmentation categories. Indicates the first The number of pixels correctly predicted for each category Indicates the first The number of pixels that were mispredicted for each category Indicates that it truly belongs to the first The number of pixels in each category that were not correctly predicted.

Images