A method for segmenting degraded scene images based on feature reconstruction and fusion

Abstract

The application discloses a kind of based on feature reconstruction and fusion's degenerative scene image segmentation method, and the degenerative scene image data under real complex environment is obtained by camera intelligent equipment, and the degenerative scene image data is preprocessed, and the preprocessed degenerative image data is obtained;Through the channel attention fusion module and the spatial attention fusion module in the TDFF dynamic feature fusion module, the shallow reconstruction features extracted by APIR anti-perception image reconstruction subnetwork and the deep features extracted by semantic segmentation backbone network are fused to obtain the final enhanced feature map after dynamic collaborative repair;The final enhanced feature map is input into the classification head network for feature decoding and up-sampling calculation, and the final enhanced feature map is restored to the resolution of original scene image data, and finally a high-precision scene semantic segmentation mask is generated.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a method for segmenting degraded scene images based on feature reconstruction and fusion. Background Technology

[0002] In today's era of rapid technological advancement, edge devices such as drones and smart cars are widely deployed in numerous fields, including transportation and urban management, becoming crucial means of acquiring image data. Street scene image segmentation, as a highly significant application, plays a vital role in promoting intelligent transportation. However, edge devices face diverse acquisition scenarios in actual operation. In complex multi-target scenes, when images suffer from the loss of detailed features due to factors such as rain, snow, and fog, existing models are prone to confusion and make incorrect judgments, resulting in low segmentation accuracy. Although existing street scene segmentation algorithms, such as DeepLabv3+, perform excellently in conventional scenarios, feature fusion under adverse weather conditions is prone to problems such as dimensionality mismatch and insufficient extraction of multi-scale contextual information, making it difficult to meet the high-precision segmentation requirements of outdoor vision systems such as autonomous driving and intelligent monitoring. Summary of the Invention

[0003] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides a degraded scene image segmentation method based on feature reconstruction and fusion. The method performs a two-level dynamic fusion operation on the features extracted by the APIR adversarial perception image reconstruction sub-network and the features extracted by the semantic segmentation backbone network, thereby realizing dynamic compensation of multi-scale degraded features and finally generating a high-precision scene semantic segmentation mask.

[0004] Technical Solution: To achieve the above objectives, this invention provides a degraded scene image segmentation method based on feature reconstruction and fusion. The method acquires degraded scene image data from a real, complex environment using a camera-based intelligent device, and preprocesses the degraded scene image data to obtain preprocessed degraded image data. The preprocessed degraded image data is then input into the APIR adversarial perception image reconstruction subnetwork for feature extraction and reconstruction, resulting in two shallow reconstruction features. Finally, the preprocessed degraded image data is input into the semantic segmentation backbone network for deep semantic feature extraction, yielding deep feature F. r .

[0005] The two shallow reconstruction features and the deep feature F r Both are input into the TDFF dynamic feature fusion module, where the two shallow reconstruction features are concatenated and fused to obtain the intermediate feature F. f The intermediate features F are processed by the channel attention fusion module in the TDFF dynamic feature fusion module. f and deep features F r Perform fusion analysis to obtain the channel attention weights W. chThe intermediate features F are processed through the spatial attention fusion module in the TDFF dynamic feature fusion module. f and deep features F r Perform fusion analysis to obtain spatial attention weights W sp ; the intermediate feature F f and deep features F r The fused features are obtained by adding each element one by one, and then the fused features are sequentially added to the channel attention weights W. ch Spatial attention weight W sp Element-wise multiplication yields the final enhanced feature map F after dynamic collaborative repair. out The final enhanced feature map F out The input is fed into the classification head network for feature decoding and upsampling calculation, resulting in the final enhanced feature map F. out The resolution of the original scene image data is restored, and a high-precision scene semantic segmentation mask is finally generated.

[0006] The dual-stream feature extraction structure, consisting of the APIR adversarial perception image reconstruction subnetwork and the semantic segmentation backbone network, combined with the TDFF dynamic feature fusion module, constructs a degraded scene image segmentation model based on feature reconstruction and fusion. During the training of this model, a composite total loss function L, constructed from adversarial perception mapping loss, detail feature recovery loss, and semantic segmentation classification loss, is used. total Gradient co-constraints are applied to the model parameters, and the weight coefficients of the adversarial perception mapping loss term and the detail feature recovery loss term are dynamically adjusted based on the training stage during training. When calculating the adversarial perception mapping loss term, the detail feature recovery loss term, and the semantic segmentation and classification loss, the final enhanced feature map F output by the TDFF dynamic feature fusion module is introduced. out Perform joint gradient constraints.

[0007] Furthermore, in the channel attention fusion module, the two shallow reconstruction features F1 and F2 are concatenated and fused to obtain a concatenated feature. The concatenated feature is then subjected to a Cov2d convolution operation to obtain an intermediate feature F. f ; the intermediate feature F f and deep features F r The fused features are obtained by element-wise addition and fusion. The fused features are then subjected to AvgPool average pooling, Cov2d convolution, and Sigmoid activation to obtain the channel attention weights W. ch The calculation process is as follows:

[0008]

[0009] In the formula, σ is the Sigmoid activation function. This represents an element-wise addition operation.

[0010] Furthermore, in the spatial attention fusion module, the intermediate feature F f and deep features F r After performing Cov2d convolution operations on each feature, element-wise summation is performed to obtain spatial fusion features. Then, the spatial fusion features are activated using the Sigmoid activation function to obtain the spatial attention weights W. sp The calculation process is shown below;

[0011]

[0012] In the formula, σ is the Sigmoid activation function. This represents an element-wise addition operation.

[0013] Furthermore, the intermediate feature F f and deep features F r The fused feature is obtained by element-wise addition. The fused feature is then multiplied element-wise with the channel attention weights Wch, followed by a Cov2d convolution operation to obtain F. ch Feature map; F ch Feature map and spatial attention weight W sp Element-wise multiplication yields the final enhanced feature map Fout after dynamic collaborative repair; the calculation process is shown below:

[0014]

[0015]

[0016] In the formula, This represents an element-wise multiplication operation. This represents an element-wise addition operation.

[0017] Furthermore, the APIR adversarial perception image reconstruction subnetwork includes an input module, a first feature enhancement guidance module, a second feature enhancement guidance module, a third feature enhancement guidance module, a feature concatenation module, a residual module, and an output module. Preprocessed degraded image data is input into the input module for convolution to obtain an initial feature map. The initial feature map is then processed through a convolutional layer and input into the first feature enhancement guidance module for processing, outputting a first feature map. The first feature map is then processed by the second feature enhancement guidance module for processing, outputting a second feature map. The second feature map is then processed by the third feature enhancement guidance module for processing, outputting a third feature map. The initial feature map, the first feature map, the second feature map, and the third feature map are concatenated and fused in the feature concatenation module to obtain an aggregated feature map. This aggregated feature map is then input into the residual module for residual concatenation to obtain residual aggregated features. The residual aggregated features are then processed through two convolutional operations and input into the output module, which outputs two shallow reconstruction features.

[0018] Furthermore, the feature enhancement guidance module includes a first convolutional function layer, a second convolutional function layer, a third convolutional function layer, a channel attention module, and a spatial attention module. The input feature map of the feature enhancement guidance module is input to the first convolutional function layer, and the output of the first convolutional function layer and the input feature map are concatenated and fused to obtain a first fused feature. The first fused feature is input to the second convolutional function layer, and the output of the second convolutional function layer, the first fused feature, and the input feature map are concatenated and fused to obtain a second fused feature. The second fused feature is input to the third convolutional function layer, and the output of the third convolutional function layer, the first fused feature, and the input feature map are concatenated and fused to obtain a third fused feature. The third fused feature is input to a convolutional layer for convolutional operation to obtain a convolutional fused feature. The convolutional fused feature is input to the channel attention module to obtain channel weights, and the channel weights are multiplied element-wise with the convolutional fused feature to obtain a fourth fused feature. The fourth fused feature is input to the spatial attention module to obtain spatial weights, and the spatial weights are multiplied element-wise with the fourth fused feature to obtain a fifth fused feature. The fifth fused feature is the output of the feature enhancement guidance module.

[0019] Furthermore, the first, second, and third convolutional function layers have the same structure, each including a cascaded convolutional layer and a ReLU layer.

[0020] Furthermore, the semantic segmentation backbone network includes a ResNet feature extraction module, a global average pooling layer, a first multi-scale pooling layer, a second multi-scale pooling layer, and a third multi-scale pooling layer. Input image data is fed into the ResNet feature extraction module to extract features, obtaining an initial semantic feature map. The initial semantic feature map is then distributed and fed into the global average pooling layer, the first multi-scale pooling layer, the second multi-scale pooling layer, and the third multi-scale pooling layer for pooling operations. The outputs of the global average pooling layer, the first multi-scale pooling layer, the second multi-scale pooling layer, and the third multi-scale pooling layer are sequentially convolved and upsampled to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map. The initial semantic feature map, the first semantic feature map, the second semantic feature map, the third semantic feature map, and the fourth semantic feature map are concatenated and fused to obtain a semantic fusion feature map. The semantic fusion feature map is then sequentially upsampled and convolved to obtain a final semantic feature map. Finally, the final semantic feature map is segmented into a feature sequence to obtain deep features.

[0021] Beneficial Effects: This invention provides a degraded scene image segmentation method based on feature reconstruction and fusion, significantly improving segmentation accuracy in degraded scenes. In tests on the Cityscapes dataset for fog, rain, and snow degraded scenes, the model's average intersection-union ratio (mIoU) reached 60.67%, 61.92%, and 62.86%, respectively, representing a maximum improvement of 3.82% compared to the DeepLabv3+ model. It enhances the recognition ability for small-scale and dynamic targets by combining the texture detail reconstruction of the APIR network with the high-dimensional semantics of the PSPNet network, greatly improving the model's ability to characterize microscopic targets in adverse weather conditions. In foggy scenes, it also effectively reduces easily scattered noise. The IoU for the covered railing category was improved to 34.00%; in rainy scenes, the IoU for the truck category reached 63.95%; the robustness of the model in complex environments was improved, and the TDFF dynamic feature fusion module effectively solved the cross-modal dimension mismatch problem, realized the dynamic compensation of multi-scale degradation features, and successfully separated and suppressed the interference of raindrop trajectory and snow noise on the subject; the problem of low semantic segmentation accuracy caused by the degradation and loss of high-frequency detail features in street scene images under severe weather conditions such as fog, rain, and snow was solved; the reliable semantic understanding ability and scene adaptive generalization ability of outdoor vision systems such as autonomous driving and intelligent monitoring in complex degradation scene environments were improved. Attached Figure Description

[0022] Figure 1 This is a flowchart of a degraded scene image segmentation method based on feature reconstruction and fusion.

[0023] Figure 2 This is a network structure diagram of the TDFF dynamic feature fusion module.

[0024] Figure 3 The network structure diagram of the APIR adversarial perception image reconstruction subnetwork.

[0025] Figure 4 The network structure diagram for the feature enhancement guidance module.

[0026] Figure 5 This is a network structure diagram of the semantic segmentation backbone network. Detailed Implementation

[0027] The invention will now be further described with reference to the accompanying drawings.

[0028] like Figure 1 As shown, a degraded scene image segmentation method based on feature reconstruction and fusion is proposed. This method acquires degraded scene image data in real-world complex environments using a camera-based intelligent device. The degraded scene image data is preprocessed to obtain preprocessed degraded image data. This preprocessed degraded image data is then input into the APIR adversarial perception image reconstruction sub-network for feature extraction and reconstruction, resulting in two shallow reconstruction features. Finally, the preprocessed degraded image data is input into the semantic segmentation backbone network for deep semantic feature extraction, yielding deep feature F. r .

[0029] The two shallow reconstruction features and the deep feature F r Both are input into the TDFF dynamic feature fusion module, where the two shallow reconstruction features are concatenated and fused to obtain the intermediate feature F. f The intermediate features F are processed by the channel attention fusion module in the TDFF dynamic feature fusion module. f and deep features F r Perform fusion analysis to obtain the channel attention weights W. ch The intermediate features F are processed through the spatial attention fusion module in the TDFF dynamic feature fusion module. f and deep features F r Perform fusion analysis to obtain spatial attention weights W sp ; the intermediate feature F f and deep features F r The fused features are obtained by adding each element one by one, and then the fused features are sequentially added to the channel attention weights W. ch Spatial attention weight W sp Element-wise multiplication yields the final enhanced feature map F after dynamic collaborative repair. out The final enhanced feature map F out The input is fed into the classification head network for feature decoding and upsampling calculation, resulting in the final enhanced feature map F. out The resolution of the original scene image data is restored, and a high-precision scene semantic segmentation mask is finally generated.

[0030] The dual-stream feature extraction structure, consisting of the APIR adversarial perception image reconstruction subnetwork and the semantic segmentation backbone network, combined with the TDFF dynamic feature fusion module, constructs a degraded scene image segmentation model based on feature reconstruction and fusion. During the training of this model, a composite total loss function L, constructed from adversarial perception mapping loss, detail feature recovery loss, and semantic segmentation classification loss, is used. total Gradient co-constraints are applied to the model parameters, and the weight coefficients of the adversarial perception mapping loss term and the detail feature recovery loss term are dynamically adjusted based on the training stage during training. When calculating the adversarial perception mapping loss term, the detail feature recovery loss term, and the semantic segmentation and classification loss, the final enhanced feature map F output by the TDFF dynamic feature fusion module is introduced. out Perform joint gradient constraints.

[0031] The camera-based intelligent devices are edge-side intelligent devices, including drones, surveillance probes, and intelligent vehicles, etc., acquiring degraded scene image data in real, complex environments with varying and severe weather conditions such as fog, rain, and snow. Preprocessing operations are performed on the degraded scene image data, adjusting its size to a uniform resolution required by the model, and performing color conversion, frame extraction, distortion correction, and cropping to adapt to the input specifications of the deep learning neural network. The APIR adversarial perception image reconstruction sub-network filters dense noise and blur interference caused by degraded scenes in the image, performs dense extraction and reconstruction mapping of high-frequency details, and reconstructs lost high-frequency detail information in the image. The semantic segmentation backbone network is a PSPNet neural network, which uses a pyramid pooling module to extract deep global contextual semantic features at different scales. The TDFF dynamic feature fusion module effectively solves the dimensionality mismatch problem between two shallow reconstructed features and deep features.

[0032] like Figure 2 As shown, in the channel attention fusion module, two shallow reconstruction features F1 and F2 are concatenated and fused to obtain a concatenated feature. The concatenated feature is then subjected to a Cov2d convolution operation to obtain an intermediate feature F. f ; the intermediate feature F f and deep features F r The fused features are obtained by element-wise addition and fusion. The fused features are then subjected to AvgPool average pooling, Cov2d convolution, and Sigmoid activation to obtain the channel attention weights W. ch The calculation process is as follows:

[0033]

[0034] In the formula, σ is the Sigmoid activation function. This represents an element-wise addition operation.

[0035] In the spatial attention fusion module, the intermediate feature F f and deep features F r After performing Cov2d convolution operations on each feature, element-wise summation is performed to obtain spatial fusion features. Then, the spatial fusion features are activated using the Sigmoid activation function to obtain the spatial attention weights W. sp The calculation process is shown below;

[0036]

[0037] In the formula, σ is the Sigmoid activation function. This represents an element-wise addition operation.

[0038] intermediate feature F f and deep features F r The fused feature is obtained by element-wise addition. The fused feature is then multiplied element-wise with the channel attention weights Wch, followed by a Cov2d convolution operation to obtain F. ch Feature map; F ch Feature map and spatial attention weight W sp Element-wise multiplication yields the final enhanced feature map Fout after dynamic collaborative repair; the calculation process is shown below:

[0039]

[0040]

[0041] In the formula, This represents an element-wise multiplication operation. This represents an element-wise addition operation.

[0042] like Figure 3As shown, the APIR adversarial perception image reconstruction subnetwork includes an input module, a first feature enhancement guidance module, a second feature enhancement guidance module, a third feature enhancement guidance module, a feature concatenation module, a residual module, and an output module. Preprocessed degraded image data is input into the input module for convolution to obtain an initial feature map. The initial feature map is then processed through a convolutional layer and input into the first feature enhancement guidance module for processing, outputting a first feature map. The first feature map is then processed by the second feature enhancement guidance module for processing, outputting a second feature map. The second feature map is then processed by the third feature enhancement guidance module for processing, outputting a third feature map. The initial feature map, the first feature map, the second feature map, and the third feature map are concatenated and fused in the feature concatenation module to obtain an aggregated feature map. This aggregated feature map is then input into the residual module for residual concatenation to obtain residual aggregated features. The residual aggregated features are then processed through two convolutional operations and input into the output module, which outputs two shallow reconstruction features.

[0043] like Figure 4 As shown, the first, second, and third feature enhancement guidance modules are all FEGM feature enhancement guidance modules. Each FEGM feature enhancement guidance module includes a first convolutional function layer, a second convolutional function layer, a third convolutional function layer, a channel attention module, and a spatial attention module. The input feature map of the FEGM feature enhancement guidance module is input to the first convolutional function layer, and the output of the first convolutional function layer and the input feature map are concatenated and fused to obtain a first fused feature. The first fused feature is then input to the second convolutional function layer, and the output of the second convolutional function layer, the first fused feature, and the input feature map are concatenated and fused to obtain... The second fusion feature is input into the third convolutional function layer. The output of the third convolutional function layer, the first fusion feature, and the input feature map are concatenated and fused to obtain the third fusion feature. The third fusion feature is input into the convolutional layer for convolution operation to obtain the convolutional fusion feature. The convolutional fusion feature is input into the channel attention module to obtain the channel weights. The channel weights are multiplied element-wise with the convolutional fusion feature to obtain the fourth fusion feature. The fourth fusion feature is input into the spatial attention module to obtain the spatial weights. The spatial weights are multiplied element-wise with the fourth fusion feature to obtain the fifth fusion feature. The fifth fusion feature is used as the output of the FEGM feature enhancement guidance module.

[0044] The first, second, and third convolutional function layers have the same structure, each including a cascaded convolutional layer and a ReLU layer.

[0045] The APR adversarial image reconstruction subnetwork employs an implicit mapping mechanism based on the generative adversarial principle. Through the constraint of the adversarial loss function, the network is forced to map noisy shallow features into a clear, non-degenerate feature space in the hidden layer space, effectively avoiding secondary interference from pixel-level artifacts generated by front-end de-degeneration on back-end semantic segmentation. Furthermore, the APR adversarial image reconstruction subnetwork integrates the FEGM feature enhancement guidance module as a basic feature extraction building block. It utilizes a residual dense connection mechanism to enhance the flow of high-frequency features of small targets, significantly improving the model's ability to perceive and reconstruct small targets that are easily lost in complex cluttered backgrounds.

[0046] like Figure 5 As shown, the semantic segmentation backbone network includes a ResNet feature extraction module, a global average pooling layer, a first multi-scale pooling layer, a second multi-scale pooling layer, and a third multi-scale pooling layer. Input image data is fed into the ResNet feature extraction module to extract features, resulting in an initial semantic feature map. This initial semantic feature map is then fed into the global average pooling layer, the first multi-scale pooling layer, the second multi-scale pooling layer, and the third multi-scale pooling layer for pooling operations. The outputs of these layers are sequentially convolved and upsampled to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map. These four semantic feature maps are then concatenated and fused to obtain a semantic fusion feature map. This semantic fusion feature map is then sequentially upsampled and convolved to obtain a final semantic feature map. Finally, the final semantic feature map is segmented into a feature sequence to obtain deep features.

[0047] The semantic segmentation backbone network mainly consists of a ResNet feature extraction module composed of a ResNet basic feature extraction backbone network and a PPM pyramid pooling module composed of a global average pooling layer, a first multi-scale pooling layer, a second multi-scale pooling layer, and a third multi-scale pooling layer. After the preprocessed image is input into the ResNet feature extraction module to extract the initial semantic feature map, the initial semantic feature map enters the PPM pyramid pooling module. The initial semantic feature map is processed by spatial pyramid pooling through the global average pooling layer and multiple pooling layers of different scales to capture prior information of local and global environments at different scales. Subsequently, the semantic feature maps at each scale are concatenated, upsampled, and convolved to extract semantic features containing deep global contextual information, and output a deep feature sequence F. rUnlike the conventional PSPNet network, which operates independently and directly outputs the final image segmentation result, this invention improves its architecture by reconstructing it into a deep semantic extraction branch within a two-stream feature extraction structure; its output deep feature F r Instead of directly performing classification and decoding output, the feature is passed as input to the TDFF dynamic feature fusion module. Through adaptive allocation of attention weights, it performs cross-layer dynamic fusion and alignment with the high-frequency detail features extracted by the reconstruction branch of the APIR adversarial perception image reconstruction sub-network. After the TDFF dynamic feature fusion module performs cross-layer dynamic fusion and alignment of the outputs of the reconstruction branch and the deep semantic extraction branch of the APIR adversarial perception image reconstruction sub-network, the final enhanced feature map is obtained. This effectively makes up for the shortcomings of conventional PSPNet in dealing with degraded scenes such as fog, rain, and snow, which are prone to missed detection and misclassification of small targets due to the lack of a low-level feature recovery mechanism.

[0048] The upsampling calculation is a bilinear interpolation upsampling calculation; the final enhanced feature map F out After mapping the class channels through convolutional layers, feature decoding is performed using bilinear interpolation upsampling to compress the final enhanced feature map F. out The spatial size is progressively enlarged until it is restored to the resolution of the original scene image data, resulting in enhanced degraded scene image data. After decoding by the classification head network, the enhanced degraded scene image data is a feature map containing multiple channels. Each channel corresponds to the probability distribution of the category to which each pixel belongs in the degraded scene image data. The pixel categories include roads, sidewalks, trucks, and pedestrians, etc. Subsequently, by taking the maximum value operation Argmax, the category index with the highest probability score of each pixel is extracted and transformed into discrete category labels. Finally, a high-precision scene semantic segmentation mask is generated to complete the physical region segmentation of the image.

[0049] The dual-stream feature extraction structure consisting of the APIR adversarial perception image reconstruction subnetwork and the semantic segmentation backbone network, together with the TDFF dynamic feature fusion module, constructs a degraded scene image segmentation model based on feature reconstruction and fusion. The Cityscapes dataset and its fog, rain, and snow degradation variant datasets generated based on the atmospheric scattering physics model, as well as the BDD100K cross-domain dataset, are used and divided into training, validation, and test sets according to the proportions.

[0050] The training set was input into a degraded scene image segmentation model based on feature reconstruction and fusion. The SGD optimizer was used to train the model, and the learning rate was dynamically adjusted using a Poly learning rate decay strategy. The training hyperparameters were set as follows: initial learning rate of 0.001, batch size of 8, weight decay of 0.0001 to prevent overfitting, and total training epochs of 300. Model performance was evaluated on the validation set after each training epoch or periodically, using pixel accuracy (PA) and mean intersection-over-union (mIoU) as the core evaluation metrics. The weights with the highest mIoU on the validation set were saved as the final model weights.

[0051] To achieve end-to-end collaborative optimization of feature reconstruction and semantic segmentation tasks in a two-stream feature extraction structure, a degraded scene image segmentation model based on feature reconstruction and fusion adopts a multi-task weighted joint optimization strategy. Within a unified backpropagation framework, the model constructs a composite total loss function L that includes an adversarial perception mapping loss term, a detail feature recovery loss term, and a semantic segmentation classification loss term. total Gradient-coordinated constraints are applied to the parameters across the entire link. The composite total loss function L... total The calculation process is as follows:

[0052]

[0053] In the formula, L adv To overcome the adversarial perceptual mapping loss, it is used to constrain the adversarial game process between the generator and discriminator in the APIR adversarial perceptual image reconstruction subnet, realizing the mapping of degenerate features to the non-degenerate implicit space; L rec The detail feature recovery loss is used to constrain the fidelity of the first-level features to the original high-frequency details, forcing the network to enhance local fine-grained targets while denoising; L seg λ1 represents the semantic segmentation classification loss, used to constrain the consistency between the pixel-level classification predictions of the network's final output and the ground truth labeled mask; λ1, λ2, and λ3 are the dynamic weight coefficients of the adversarial perception mapping loss, detail feature recovery loss, and semantic segmentation classification loss, respectively.

[0054] To fully leverage the role of the TDFF dynamic feature fusion module in cross-modal dimensional alignment and break down the task barriers between feature reconstruction and semantic segmentation, this invention innovatively introduces the final enhanced feature map F output by the TDFF module when calculating the aforementioned loss function. out Joint gradient constraints are applied; the specific calculation formulas for each loss term are as follows:

[0055] The adversarial perception mapping loss L advImplicit spatial alignment constraints; extending the scope of adversarial games to the feature fusion layer, with the discriminator's input being the final enhanced feature map F output by the TDFF dynamic feature fusion module. out The loss function of Least Squares Generative Adversarial Network (LSGAN) is used for calculation, and the calculation process is as follows:

[0056]

[0057] In the formula, M is the number of samples in the batch; It is a feature-level discriminator network used to determine whether the final enhanced feature map of the input comes from the reconstruction and fusion of a clear scene image or a degraded scene image; This is the final enhanced feature map output by the TDFF module for the j-th degraded scene image sample. The adversarial perception mapping loss L... adv The forced APIR adversarial perception image reconstruction subnetwork and the TDFF dynamic feature fusion module are jointly optimized to make the final enhanced feature map difficult for the discriminator to distinguish in the data manifold distribution.

[0058] The loss of detail feature recovery L rec Based on knowledge transfer-based auxiliary constraints, and with knowledge transfer as the core methodology, paired clear, non-degradable images are simultaneously input into the network during the training phase to extract their corresponding ideal fusion features. As target domain knowledge, the detailed feature recovery loss consists of two parts: shallow reconstruction constraints and deep knowledge transfer constraints. The calculation formula is shown below:

[0059]

[0060] In the formula, and These are shallow reconstruction features output by the APIR adversarial perception image reconstruction subnetwork, representing degraded scene images and clear scene images, respectively. and These are the final enhanced feature maps output by the TDFF dynamic feature fusion module after the degraded scene image and the clear scene image are processed. Represented as L1 norm, The mean squared error is represented by α and β, which are balance coefficients. The loss L is recovered through detailed features. rec The deep feature distribution of clear scene images is transferred to the features of degraded scene images through knowledge transfer, and the forced orientation guides the final enhanced feature map F. out It approximates the ideal state of non-degeneracy in the feature space.

[0061] The semantic segmentation classification loss L seg The main task anchoring constraint; as the core optimization objective of the entire segmentation model, it directly depends on the final enhanced feature map F. outThe final enhanced feature map F out After decoding by the classification head network, the weighted multi-class cross-entropy loss function is used for calculation, as shown below:

[0062]

[0063] In the formula, N is the total number of pixels in the image; H and W are the height and width of the image; and C is the total number of semantic categories. This is the true category label for pixel i; it is 1 if it belongs to category c, and 0 otherwise. The classification head network is based on the final enhanced feature map F out Predict the probability that pixel i belongs to category c; This is the category weight coefficient, used to alleviate the data imbalance problem of long-tail categories.

[0064] To achieve smooth transition and accurate allocation of gradients across multiple tasks, the design concept of the CDIE cross-domain feature information extraction and evaluation framework is introduced into the joint optimization strategy. A two-stage progressive weight scheduling mechanism based on feature state feedback is proposed, which breaks away from the traditional open-loop decay strategy that relies solely on the training cycle. The two-stage progressive weight scheduling mechanism based on feature state feedback includes a static weight determination method based on main task anchoring and a dynamic adaptive weight calculation method based on feature evolution closed loop. The specific implementation is as follows:

[0065] The first stage is the training warm-up period, which occurs in the initial stage of model training, such as the first 30% of the total training iterations. Because the network is in a randomly initialized state, the final enhanced feature map F output by the TDFF dynamic feature fusion module... out Since the algorithm has not yet converged and contains a lot of chaotic noise, the static weight determination method based on the main task anchor is selected.

[0066] The static weight determination method based on main task anchoring sets the weight λ3=1.0 of the semantic segmentation classification loss as the absolute anchor point for global optimization, forcing the network to prioritize the establishment of basic semantic feature boundaries; the auxiliary task weights λ1 and λ2 are assigned relatively small fixed static values, where λ1 is 0.1 and λ2 is 0.2, only providing basic knowledge transfer warm-up constraints; the purpose of using static weights in the static weight determination method based on main task anchoring is to prevent excessive adversarial and reverse gradient reconstruction from destroying the formation of the initial semantic manifold in the early stage of training.

[0067] The second stage is the feature fine-tuning period. When the number of training iterations exceeds the preset warm-up threshold, that is, after the warm-up period, the model automatically switches to the dynamic adaptive weight calculation method based on feature evolution closed loop. Innovatively, this stage uses the final enhanced feature map output in real time by the TDFF dynamic feature fusion module during training as the closed loop feedback variable.

[0068] First, we define the feature alignment error ΔF, which is used to quantize the final enhanced feature map output by the TDFF dynamic feature fusion module after the current degraded scene image passes through it in real time. The final enhanced feature map output by the TDFF dynamic feature fusion module after the sharp scene image is processed. The difference is calculated using the following formula:

[0069]

[0070] In the formula, The mean squared error is represented by H and W, which are the height and width of the image, respectively, and C is the total number of channels in the feature map.

[0071] Based on the real-time evaluated feature alignment error ΔF, λ1 and λ2 are dynamically reconstructed and calculated. To ensure the generalization ability and convergence stability of the dynamic adaptive weight calculation formula in complex degradation scenarios, a hyperparameter determination mechanism based on ablation experiments and validation set feedback is established for the hyperparameters introduced in the formula. Among these, the dynamic weights of the adversarial perception mapping loss are... The calculation is as follows:

[0072]

[0073] In the formula, Basic adjustment coefficient, This is the scaling factor. The basic adjustment coefficient... scaling factor The hyperparameters were determined using a mechanism based on feedback from ablation experiments and validation sets; among which the baseline adjustment coefficient was used. The value of was determined through mesh search and ablation experiments on the Cityscapes and BDD100K validation sets; specifically, the base adjustment coefficient was set while freezing other parameters. The candidate set is By evaluating the average intersection-union ratio (mIoU) of the segmentation results under different values ​​and the feature distribution divergence of adversarial generation, the optimal empirical value that prevents the feature reconstruction gradient from inversely covering the semantic classification gradient is finally selected. In the preferred embodiment, The value is set to 0.1, and it is aligned with the static weights of the first stage to ensure the smoothness of the optimization. The scaling factor is... The functional form was established through structural ablation experiments. Experimental data show that, compared with the constant-fixed method and the linear scaling method, the scaling factor... The feature representation is set to be related to the network structure, i.e. At this time, the model has higher adaptive sensitivity to small deviations in the high-dimensional feature distribution, which can effectively prevent feature tearing caused by excessive adversarial training in the later stages; where C is the final enhanced feature map F. out The total number of channels.

[0074] The dynamic weights of the detail feature recovery loss The calculation is as follows:

[0075]

[0076] In the formula, Basic adjustment coefficient, An adaptive normalization parameter for controlling the steepness of the curve. The basic adjustment coefficient. Adaptive normalization parameters for controlling the steepness of the curve The hyperparameters are determined using a mechanism based on feedback from ablation experiments and validation sets; similarly, the baseline adjustment coefficients are... The value of is obtained through multi-task balanced ablation experiments. Tests have shown that when When the optimal value is 0.2, it maintains consistency with the static weights of the first stage, and the model achieves maximum gain in the segmentation IoU of distant small-scale targets. The adaptive normalization parameter controls the steepness of the curve. Through dynamic baseline ablation analysis, a method for setting... ,in The global mean of feature alignment error on the validation set is used at the end of the training warm-up period. Experiments show that by introducing an adaptive normalization parameter to control the kurtosis of the curve, the exponential decay component can be immune to the influence of the magnitude of the absolute value of the initial error of weather degradation of different orders of magnitude, such as heavy rain and light rain; thus enabling λ2 to robustly decay towards... Approaching the theoretical upper limit forces the network to accurately depict the edges of microscopic details.

[0077] By alternating between dynamic and static modes under the above two-stage mechanism, combined with the hyperparameter determination mechanism based on ablation experiments and validation set feedback, the problem of blind manual parameter tuning in multi-task loss function optimization is completely eliminated. This not only unifies the gradient flow direction in the backpropagation process at the theoretical level and eliminates gradient conflicts at the feature fusion point of the dual-flow feature extraction structure, but also ensures the robustness and generalization ability of the model under various severe weather scenarios by using detailed experimental feedback and closed-loop control.

[0078] The above description is merely a preferred embodiment of the present invention. Those skilled in the art can make several modifications and optimizations based on the above disclosure without departing from the basic principles described above. These modifications and optimizations should be considered within the scope of protection as understood by the present invention.

Claims

1. A degraded scene image segmentation method based on feature reconstruction and fusion, characterized in that: The image data of degraded scenes in real complex environments is acquired by a camera-based intelligent device, and the degraded scene image data is preprocessed to obtain preprocessed degraded image data. The preprocessed degraded image data is then input into the APIR adversarial perception image reconstruction subnetwork for feature extraction and reconstruction to obtain two shallow reconstruction features. The preprocessed degraded image data is input into the semantic segmentation backbone network for deep semantic feature extraction, resulting in deep feature F. r ; The two shallow reconstruction features and the deep feature F r Both are input into the TDFF dynamic feature fusion module, where the two shallow reconstruction features are concatenated and fused to obtain the intermediate feature F. f The intermediate features F are processed by the channel attention fusion module in the TDFF dynamic feature fusion module. f and deep features F r Perform fusion analysis to obtain the channel attention weights W. ch The intermediate features F are processed through the spatial attention fusion module in the TDFF dynamic feature fusion module. f and deep features F r Perform fusion analysis to obtain spatial attention weights W sp ; the intermediate feature F f and deep features F r The fused features are obtained by adding each element one by one, and then the fused features are sequentially added to the channel attention weights W. ch Spatial attention weight W sp Element-wise multiplication yields the final enhanced feature map F after dynamic collaborative repair. out The final enhanced feature map F out The input is fed into the classification head network for feature decoding and upsampling calculation, resulting in the final enhanced feature map F. out The resolution of the original scene image data is restored, and a high-precision scene semantic segmentation mask is finally generated. The dual-stream feature extraction structure, consisting of the APIR adversarial perception image reconstruction subnetwork and the semantic segmentation backbone network, combined with the TDFF dynamic feature fusion module, constructs a degraded scene image segmentation model based on feature reconstruction and fusion. During the training of this model, a composite total loss function L, constructed using adversarial perception mapping loss, detail feature recovery loss, and semantic segmentation classification loss, is employed. total Gradient co-constraints are applied to the model parameters, and the weight coefficients of the adversarial perception mapping loss term and the detail feature recovery loss term are dynamically adjusted based on the training stage during training. When calculating the adversarial perception mapping loss term, the detail feature recovery loss term, and the semantic segmentation and classification loss, the final enhanced feature map F output by the TDFF dynamic feature fusion module is introduced. out Perform joint gradient constraints.

2. The degradation scene image segmentation method based on feature reconstruction and fusion according to claim 1, characterized in that: In the channel attention fusion module, two shallow reconstruction features F1 and F2 are concatenated and fused to obtain a concatenated feature. The concatenated feature is then subjected to a Cov2d convolution operation to obtain an intermediate feature F. f ; the intermediate feature F f and deep features F r The fused features are obtained by element-wise addition and fusion. The fused features are then subjected to AvgPool average pooling, Cov2d convolution, and Sigmoid activation to obtain the channel attention weights W. ch The calculation process is as follows: In the formula, σ is the Sigmoid activation function. This represents an element-wise addition operation.

3. The degradation scene image segmentation method based on feature reconstruction and fusion according to claim 2, characterized in that: In the spatial attention fusion module, the intermediate feature F f and deep features F r After performing Cov2d convolution operations on each feature, element-wise summation is performed to obtain spatial fusion features. Then, the spatial fusion features are activated using the Sigmoid activation function to obtain the spatial attention weights W. sp The calculation process is shown below; In the formula, σ is the Sigmoid activation function. This represents an element-wise addition operation.

4. The degradation scene image segmentation method based on feature reconstruction and fusion according to claim 3, characterized in that: intermediate feature F f and deep features F r The fused feature is obtained by element-wise addition. The fused feature is then multiplied element-wise with the channel attention weights Wch, followed by a Cov2d convolution operation to obtain F. ch Feature map; F ch Feature map and spatial attention weight W sp Element-wise multiplication yields the final enhanced feature map Fout after dynamic collaborative repair; the calculation process is shown below: In the formula, This represents an element-wise multiplication operation. This represents an element-wise addition operation.

5. The degradation scene image segmentation method based on feature reconstruction and fusion according to claim 1, characterized in that: The APIR adversarial perception image reconstruction subnetwork includes an input module, a first feature enhancement guidance module, a second feature enhancement guidance module, a third feature enhancement guidance module, a feature concatenation module, a residual module, and an output module. Preprocessed degraded image data is input into the input module for convolution to obtain an initial feature map. The initial feature map is then processed through a convolutional layer and input into the first feature enhancement guidance module for processing, outputting a first feature map. The first feature map is then processed by the second feature enhancement guidance module for processing, outputting a second feature map. The second feature map is then processed by the third feature enhancement guidance module for processing, outputting a third feature map. The initial feature map, the first feature map, the second feature map, and the third feature map are concatenated and fused in the feature concatenation module to obtain an aggregated feature map. This aggregated feature map is then input into the residual module for residual concatenation to obtain residual aggregated features. The residual aggregated features are then processed through two convolutional operations and input into the output module, which outputs two shallow reconstruction features.

6. The degradation scene image segmentation method based on feature reconstruction and fusion according to claim 5, characterized in that: The feature enhancement guidance module includes a first convolutional function layer, a second convolutional function layer, a third convolutional function layer, a channel attention module, and a spatial attention module; The input feature map of the feature enhancement guidance module is input to the first convolutional function layer. The output of the first convolutional function layer and the input feature map are concatenated and fused to obtain the first fused feature. The first fused feature is input to the second convolutional function layer. The output of the second convolutional function layer, the first fused feature, and the input feature map are concatenated and fused to obtain the second fused feature. The second fused feature is input to the third convolutional function layer. The output of the third convolutional function layer, the first fused feature, and the input feature map are concatenated and fused to obtain the third fused feature. The third fused feature is input to a convolutional layer for convolution operation to obtain a convolutional fused feature. The convolutional fused feature is input to the channel attention module to obtain channel weights. The channel weights are multiplied element-wise with the convolutional fused feature to obtain the fourth fused feature. The fourth fused feature is input to the spatial attention module to obtain spatial weights. The spatial weights are multiplied element-wise with the fourth fused feature to obtain the fifth fused feature. The fifth fusion feature is the output of the feature enhancement guidance module.

7. The degradation scene image segmentation method based on feature reconstruction and fusion according to claim 6, characterized in that: The first, second, and third convolutional function layers have the same structure, each including a cascaded convolutional layer and a ReLU layer.

8. The degradation scene image segmentation method based on feature reconstruction and fusion according to claim 1, characterized in that: The semantic segmentation backbone network includes a ResNet feature extraction module, a global average pooling layer, a first multi-scale pooling layer, a second multi-scale pooling layer, and a third multi-scale pooling layer. Input image data is fed into the ResNet feature extraction module to extract features, resulting in an initial semantic feature map. This initial semantic feature map is then fed into the global average pooling layer, the first multi-scale pooling layer, the second multi-scale pooling layer, and the third multi-scale pooling layer for pooling operations. The outputs of these layers are sequentially convolved and upsampled to obtain a first semantic feature map, a second semantic feature map, a third semantic feature map, and a fourth semantic feature map. These four semantic feature maps are then concatenated and fused to obtain a semantic fusion feature map. This semantic fusion feature map is then sequentially upsampled and convolved to obtain a final semantic feature map. Finally, the final semantic feature map is segmented into a feature sequence to obtain deep features.

Images