Unmanned aerial vehicle semantic perception domain generalization method and system based on probability diffusion modeling, and storage medium

Abstract

The application provides a kind of unmanned aerial vehicle semantic perception domain generalization method and system based on probability diffusion modeling, first to the source scene data set is carried out luminosity enhancement to generate pseudo target scene image, then input semantic segmentation backbone network to obtain cross-domain feature map, output mean and variance by latent prior extractor, generate latent domain prior by reparameterization and add Gaussian noise by forward diffusion;Extract the feature of unlabeled new scene image, estimate the latent domain prior of new scene by reverse denoising processing of diffusion prior estimator, adjust the feature map of semantic segmentation backbone network;Output semantic segmentation result, combine task loss, semantic consistency loss, prior constraint loss to optimize model parameters.The application can quickly adapt to different light, weather, terrain scene under zero labeling condition, enhance the generalization and robustness of unmanned aerial vehicle semantic perception, architecture native compatible, can be seamlessly embedded in existing airborne perception module, greatly reduce the deployment and adaptation overhead of multiple scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and UAV perception technology, and particularly relates to a method, system and storage medium for generalizing the semantic perception domain of UAVs based on probability diffusion modeling. Background Technology

[0002] In the fields of computer vision and UAV autonomous perception technology, UAV semantic perception, as a core technology for achieving environmental understanding, target recognition, and safe operation, widely supports high-end applications such as power line inspection, geographic surveying, emergency rescue, and smart city surveying. This technology relies on semantic segmentation algorithms to achieve refined identification of key targets such as roads, buildings, vegetation, and obstacles, directly determining the reliability and intelligence level of UAV operations. As UAV operational scenarios rapidly expand to all-weather, all-terrain, and complex environments, cross-scenario generalization capability has become a key indicator for measuring the performance of semantic perception systems. However, existing technologies face significant technical bottlenecks in practical implementation.

[0003] From the perspective of current industry applications, the operating environment of drones is highly dynamic and unpredictable. Factors such as light intensity, weather changes, terrain undulations, and perspective differences can all cause significant domain shift problems. That is, a semantic segmentation model trained in a fixed scene will experience a sharp drop in segmentation accuracy once transferred to unfamiliar environments such as rainy days, foggy days, nighttime, or mountainous areas. Target misses and false detections will occur frequently, and environmental adaptability will be severely insufficient, making it difficult to meet the requirements of unmanned systems for continuous and stable operation in real and complex scenarios.

[0004] Analysis of existing technologies reveals two main categories of mainstream domain adaptation and generalization methods. The first category comprises data augmentation methods based on manually defined rules. These methods expand training samples through pre-set transformation strategies. However, they heavily rely on manual design, failing to accurately depict the domain shift patterns of real-world scenes and easily introducing a large amount of invalid interference, resulting in limited generalization improvement. The second category consists of domain alignment methods based on feature normalization. These methods reduce domain differences by forcibly unifying the feature distribution across different scenes. While this can alleviate the shift problem to some extent, it destroys the unique semantic details of the scene, leading to a degradation in the recognition of small, edge, and occluded targets. This deficiency is particularly pronounced in high-resolution, multi-scale, and highly variable-viewpoint scenarios such as drone aerial photography. More importantly, current mainstream methods heavily rely on labeled data from new scenes to support model fine-tuning or domain alignment learning. However, actual drone operations often involve remote mountainous areas, high-risk disaster sites, and areas without signal coverage—scenes that are difficult to label manually. Data acquisition is extremely costly and time-consuming, directly hindering the large-scale application of existing technologies in real-world engineering scenarios.

[0005] In recent years, some studies have attempted to introduce multi-scenario hybrid training or lightweight domain adaptation strategies, but most of these only optimize for single dimensions such as lighting and weather, failing to address complex domain shifts caused by multiple coupled factors. Other methods achieve multi-scenario adaptation through cloud training, but suffer from high computational overhead, insufficient real-time performance, and high deployment costs, making them difficult to adapt to the computing power limitations of airborne embedded platforms. Existing technologies generally cannot achieve efficient, accurate, and lightweight domain generalization in unlabeled new scenarios, becoming a core obstacle restricting UAV intelligent sensing technology from achieving autonomy, all-weather, and all-domain operations.

[0006] To address the aforementioned issues, this invention proposes a method, system, and storage medium for generalizing the semantic perception domain of unmanned aerial vehicles (UAVs) based on probability diffusion modeling. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a UAV semantic perception domain generalization method, system, and storage medium based on probabilistic diffusion modeling. Without requiring any new scene annotation data, it achieves cross-scene adaptive perception through latent domain offset learning and diffusion prior estimation. This allows for rapid adaptation to diverse and complex operating environments such as varying lighting conditions, severe weather, and complex terrain, significantly improving the accuracy, stability, and generalization ability of UAV semantic segmentation. Furthermore, this method is lightweight, easily integrated, and highly compatible, allowing direct embedding into existing mainstream semantic segmentation networks and UAV perception systems without requiring reconstruction of the original system architecture. This significantly reduces actual deployment costs and engineering implementation difficulties, providing more autonomous, reliable, and efficient theoretical support and technical implementation solutions for UAVs in all-weather, all-scenario operations such as power line inspection, geographic surveying, emergency rescue, and field exploration.

[0008] The present invention achieves the above-mentioned technical objectives by adopting the following technical solution:

[0009] A generalization method for the semantic perception domain of unmanned aerial vehicles (UAVs) based on probability diffusion modeling includes the following process:

[0010] Step 1: Collect source scene images and preprocess them to form a source scene dataset. Perform photometric enhancement on the images in the source scene dataset to generate pseudo target scene images. Finally, associate the verified pseudo target scene images with the corresponding source scene dataset.

[0011] Step 2: Input the preprocessed source scene image and pseudo target scene image into the semantic segmentation backbone network for processing to obtain cross-domain feature maps. Then, output the mean and variance parameters that characterize the domain offset between the source scene and the pseudo target scene through the latent prior extractor. Finally, generate the latent domain prior through reparameterization.

[0012] Step 3: Gaussian noise is added to the latent domain prior through forward diffusion using a diffusion prior estimator to extract features of the unlabeled new scene image collected in real time by the UAV. Inverse denoising is performed using the diffusion prior estimator to estimate the latent domain prior of the new scene. At the same time, the latent domain prior of the new scene is mapped to feature modulation parameters through the domain compensation module. The feature map of the semantic segmentation backbone network is adjusted to obtain an optimized feature map adapted to the new scene.

[0013] Step 4: Output the final semantic segmentation result based on the optimized feature map adapted to the new scene, and optimize the parameters of the semantic segmentation backbone network model using a joint training framework of task loss, semantic consistency loss, and prior constraint loss.

[0014] Furthermore, the specific process of step 1 is as follows:

[0015] First, images of the drone operation scene at different times and angles are acquired using image acquisition equipment. The target category in each image is semantically labeled, and the training set and validation set are divided proportionally to form the source scene dataset. Next, photometric enhancement is performed on each image in the source scene dataset to generate pseudo-target scene images that are semantically consistent with the source scene but have different scene styles. The pseudo-target scene images are then quality-verified. Finally, the verified pseudo-target scene images are associated with the source scene dataset to form paired data of source scene images and pseudo-target scene images, which are used for subsequent latent domain prior extraction.

[0016] Furthermore, the specific process of step 2 is as follows:

[0017] First, determine the type of semantic segmentation backbone network, and then preprocess the source scene image and the pseudo target scene image.

[0018] Next, the preprocessed source scene image and pseudo target scene image are input into the semantic segmentation backbone network. Features are extracted through the encoder part of the network to generate source scene feature map and pseudo target scene feature map. Then, the source scene feature map and pseudo target scene feature map are concatenated according to the channel dimension to form a cross-domain feature map.

[0019] Then, the core structure of the latent prior extractor is constructed, which consists of multiple residual blocks. The cross-domain feature map is input into the residual block sequence and processed through the residual connection mechanism. The output of the residual block sequence is processed through two projection layers. The first projection layer maps the cross-domain features to the feature space of the preset dimension. The activation function is a linear rectified function. The second projection layer outputs the mean and variance of the latent prior.

[0020] Then, the latent prior extractor generates the latent domain prior based on the mean and variance through reparameterization: a noise vector is randomly sampled from the standard normal distribution, the noise vector is multiplied element-wise with the variance, and then added element-wise with the mean, finally generating the latent domain prior that conforms to the standard normal distribution.

[0021] Finally, the validity of the generated latent domain priors is verified.

[0022] Furthermore, the specific process of step 3 is as follows:

[0023] First, determine the core parameters and initial state of the diffusion prior estimator, set the total number of diffusion steps and the noise variance sequence, take the latent domain prior generated by the latent prior extractor as the initial input of the forward diffusion, then execute the forward diffusion process to add Gaussian noise, and finally obtain the fully noisy latent state.

[0024] Then, features from unlabeled new scene images captured in real time by the drone are extracted, preprocessed, and input into the semantic segmentation backbone network encoder to extract multi-scale feature maps of the new scene images. ;

[0025] Then, a reverse denoising process is performed using a diffusion prior estimator to estimate the latent domain prior of the new scene;

[0026] Simultaneously, the domain compensation module is activated to adjust the feature map: the prior of the new scene's latent domain is input into the projection layer of the domain compensation module. This projection layer consists of a fully connected layer and an activation function, mapping the prior of the new scene's latent domain into two sets of feature modulation parameters, namely scale parameters. With offset parameter Used to adjust the weights of each channel in the feature map. This is used to correct the pixel value distribution of the feature map; the scale parameter is fused with the multi-scale feature map of the new scene image through element-wise multiplication, and then the offset parameter is incorporated through element-wise addition to adjust the feature map. The adjustment formula is as follows: ,in, This represents the optimized feature map adapted to the new scenario;

[0027] Finally, the effectiveness of the latent domain prior for the new scene and the optimized feature map adapted to the new scene is verified.

[0028] Furthermore, the specific process of performing the forward diffusion process to add Gaussian noise is as follows:

[0029] According to the preset diffusion steps, iterative noise addition operations are performed on the initial latent domain prior. The state update formula for each diffusion step is: ,in, , They represent the first Step, First The potential state after step diffusion, Is with the first The coefficients corresponding to the step noise variance , For the first The variance of the Gaussian noise added during the diffusion process. For the first The Gaussian noise vector of the step; and These are the fusion weights for the latent state and the noise vector, respectively. The fusion weights are determined by the cumulative variance. Auxiliary determination, among which, Indicates the preceding step The product of these factors is used to control the intensity of noise addition at each step, ensuring that the noise level of the prior in the latent domain gradually increases with the number of diffusion steps, ultimately yielding a fully noisy latent state. ,in, Indicates the first The coefficient corresponding to the step diffusion;

[0030] The specific process for estimating the prior potential of the new scenario is as follows:

[0031] The fully noisy latent state and the multi-scale feature map of the new scene image are jointly input into the denoising network of the diffusion prior estimator. The state update formula for each step of the denoising network is as follows: ,in, Indicates the first The latent state after noise reduction. The denoising network predicts the first... Step noise, The parameters are for the denoising network. Based on the current noise level and the characteristics of the new scene, the denoising network predicts and removes some noise from the latent state, updating the latent state. This process is repeated until all diffusion steps of denoising are completed, ultimately obtaining the prior latent domain of the new scene. .

[0032] Furthermore, the specific process of step 4 is as follows:

[0033] First, the optimized feature map adapted to the new scene is input into the decoder part of the semantic segmentation backbone network to generate a probability distribution map for each target category. Then, the argmax operation is performed on the probability distribution map to obtain the target category label corresponding to each pixel, forming the final semantic segmentation result.

[0034] Next, we construct the task loss function, the semantic consistency loss function, and the prior constraint loss function;

[0035] Finally, a joint training framework was constructed and model parameters were optimized: the task loss, semantic consistency loss, and prior constraint loss were weighted and summed according to preset weights to obtain the total loss function; the weights of the total loss function were set to balance the contribution of each loss, with the task loss weight set to 1.0, the semantic consistency loss weight set to 0.5, and the prior constraint loss weight set to 0.3, ensuring that the model prioritizes semantic segmentation accuracy while also considering domain generalization ability and prior estimation accuracy; the Adam optimizer was used to optimize the total loss function, and after each round of training, the average intersection-union ratio (OCR) of the model was calculated using the validation set of the source scene dataset, and the model parameters with the highest OCR were saved as the optimal model.

[0036] Furthermore, the semantic consistency loss function is constructed as follows: The source scene image and its corresponding pseudo-target scene image are selected as comparison samples, and both are input into the model during training to obtain the source scene semantic prediction map and the pseudo-target scene semantic prediction map; the semantic intersection-union ratio (CIU) of the two prediction maps is calculated. : ,in, Target region of the source scene prediction map The corresponding target region is predicted in the pseudo-target scene map, and the structural similarity index is calculated simultaneously. The semantic intersection-union ratio and the mean of the structural similarity index of the two predicted graphs are used as the semantic consistency score. : Then semantic consistency loss for: .

[0037] Furthermore, the method for constructing the prior constraint loss function is as follows:

[0038] Obtain the latent domain prior of the source scene and pseudo-target scene generated by the latent prior extractor. And the new scene latent domain prior generated by the diffusion prior estimator Calculate the L2 norm between the two: ;in, For prior constraint loss, It is the L2 norm. and The coordinates of the two sets of potential priors in the feature space are respectively. The value at the specified position quantifies the Euclidean distance between the two sets of latent domain priors in the feature space. By minimizing the prior constraint loss, it ensures that the new scene latent domain priors generated by the diffusion prior estimator are consistent with the domain offset patterns learned by the latent prior extractor.

[0039] A system for implementing the above-mentioned UAV semantic perception domain generalization method based on probability diffusion modeling includes:

[0040] The source scene image processing module receives source scene images acquired by the image acquisition device and performs semantic annotation, generates a source scene dataset, enhances the photometric properties of the source scene images, generates pseudo target scene images that are semantically consistent with the source scene but have different scene styles, performs verification, and associates the verified pseudo target scene images with the corresponding source scene dataset.

[0041] The latent domain prior generation module receives the data processed by the source scene image processing module, and obtains the parameters that characterize the domain offset pattern between the source scene and the pseudo-target scene based on the latent prior extractor: mean and variance, and generates the latent domain prior.

[0042] The new scene latent domain prior estimation module, based on the diffusion prior estimator, performs forward diffusion on the latent domain prior and adds Gaussian noise to extract features of the unlabeled new scene image collected in real time by the UAV. It then performs reverse denoising processing through the diffusion prior estimator to estimate the new scene latent domain prior. At the same time, the domain compensation module maps the new scene latent domain prior to feature modulation parameters and adjusts them to obtain an optimized feature map adapted to the new scene.

[0043] The joint training framework optimization module outputs the final semantic segmentation result based on the optimized feature map adapted to the new scenario. It uses task loss, semantic consistency loss, and prior constraint loss to optimize the parameters of the semantic segmentation backbone network model through a joint training framework.

[0044] A computer storage medium storing a computer program that, when run on a processor, executes the steps of the above-described generalization method for the semantic perception domain of unmanned aerial vehicles based on probability diffusion modeling.

[0045] The present invention has the following beneficial effects:

[0046] This invention achieves autonomous learning of domain offset patterns based on probability diffusion modeling. It can accurately adapt to complex scenes such as lighting, weather, and terrain without requiring new scene annotation data. It fundamentally solves the key problems of traditional methods, such as reliance on annotation, weak generalization ability, and low perception accuracy in complex scenes, and significantly reduces the cost of data collection, annotation, and model training.

[0047] This invention adopts a plug-and-play modular design, which can be seamlessly integrated into existing mainstream semantic segmentation networks without the need to reconstruct or modify the original UAV perception system framework, significantly reducing the difficulty of technical transformation and engineering deployment costs, and has strong engineering practicality and promotion value.

[0048] This invention optimizes the diffusion calculation process and the multi-constraint joint loss function, effectively improving the semantic segmentation accuracy and stability in unknown new scenarios while meeting the real-time requirements of the airborne platform. This provides more accurate, reliable and stable decision-making basis for UAVs in all-weather operations such as inspection, surveying and mapping, and emergency rescue.

[0049] In summary, this invention breaks through the dependence of traditional methods on labeled data. It autonomously learns the latent domain shift patterns between scenes through a probability diffusion mechanism, achieving efficient domain adaptation without any new scene labeling information. It is also lightweight, embeddable, and easily compatible, and can be seamlessly integrated into existing mainstream semantic segmentation architectures without significantly increasing computational overhead. It can simultaneously meet the engineering requirements of high precision, high real-time performance, and high compatibility, effectively solving a series of key pain points of existing technologies in complex UAV operation scenarios, such as poor adaptability, low accuracy, dependence on labeling, and difficult deployment. Attached Figure Description

[0050] Figure 1 The flowchart shows the generalization method for the semantic perception domain of UAVs based on probability diffusion modeling. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] The UAV semantic perception domain generalization method based on probability diffusion modeling described in this invention is as follows: Figure 1 As shown, the specific process includes the following:

[0053] Step 1: Acquire source scene images and perform semantic annotation to form a source scene dataset. Perform photometric enhancement on the source scene images to generate pseudo-target scene images that are semantically consistent with the source scene but have different scene styles. Verify these pseudo-target scene images and finally associate the verified pseudo-target scene images with the corresponding source scene dataset. The specific process is as follows:

[0054] First, the source scene was defined as a typical UAV operation scenario. Images of this scenario were collected at different times and angles using image acquisition equipment. Semantic annotations were performed on the target categories in each image, including core UAV semantic perception targets such as roads, buildings, vegetation, and obstacles. The annotated images were then divided into a training set and a validation set according to a predetermined ratio, forming the source scene dataset. The training set is used for subsequent model training, while the validation set is used for performance evaluation during training, ensuring that the model learns effective semantic information from the source scene.

[0055] Next, photometric enhancement is performed on each image in the source scene dataset. Enhancement methods include simulating rainy day fog effects, nighttime low-light dimness effects, foggy day image blurring effects, and snowy day partial occlusion effects. To address the cross-domain perception needs of UAVs, an innovative domain style perturbation generation strategy is designed. During photometric enhancement, pixel-level semantic alignment constraints ensure that the enhanced pseudo-target scene image is semantically identical to the original source scene image; that is, the position and contour of each target category in the pseudo-target scene image remain the same as in the source scene image, differing only in scene style. To achieve strict semantic preservation, this invention introduces a semantic structure preservation loss constraint: ,in, This is a semantic mask for the pseudo-target scene image. The true semantic mask of the source scene image. , These are the image height and width, respectively. A semantic structure preservation loss is employed. By minimizing this loss, we ensure that pseudo-target scene images do not undergo semantic shifts when style changes occur, providing high-quality pairing data for subsequent cross-domain modeling.

[0056] Then, the generated pseudo-target scene images are quality checked, with metrics including semantic consistency and style realism. Semantic consistency is calculated by determining the semantic intersection-union ratio (CIU) between the pseudo-target scene image and the source scene image. To verify, the calculation formula is as follows: ,in, For the target area in the pseudo-target scene, The target region of the source scene is used to ensure that the intersection-union ratio is not lower than a preset threshold. Style authenticity is verified through statistical distribution of scene features. This invention innovatively introduces a style similarity index. : ,in, , These represent the mean and variance of the brightness and texture features of the pseudo-target scene image, respectively. , This metric represents the statistical characteristics of the real target scene. It ensures that the pseudo-target image conforms to the visual characteristics of the real scene, thus avoiding style distortion or semantic misalignment.

[0057] Finally, the verified pseudo-target scene images are associated with the source scene dataset to form paired data of "source scene image - pseudo-target scene image", which is used for subsequent latent domain prior extraction. This provides data support for the model to learn the domain offset rules under different scenarios and provides dedicated data support for the probability diffusion domain generalization framework. This enables the model to automatically learn domain offset features in unlabeled new scenarios, significantly improving the semantic perception stability of UAVs across weather, lighting, and terrain.

[0058] Step 2: Input the preprocessed source scene image and pseudo-target scene image into the semantic segmentation backbone network for processing to obtain cross-domain feature maps. Then, the latent prior extractor outputs two parameters, the mean and variance, that characterize the domain shift between the source scene and the pseudo-target scene. Finally, latent domain priors are generated through reparameterization. The specific process is as follows:

[0059] First, the type of semantic segmentation backbone network is determined, and network structures suitable for UAV semantic perception tasks are selected, including DeepLabV3Plus and Mask2Former. DeepLabV3Plus uses a residual network 50 as the basic feature extraction network, while Mask2Former uses a SwinTransformer-Tiny network. Both networks can effectively capture multi-scale semantic features of images and are suitable for UAV application scenarios with different accuracy and computing power requirements. Then, the source scene image and the pseudo-target scene image are preprocessed, adjusting them to a preset size that matches the input requirements of the backbone network. Simultaneously, the image pixel values ​​are normalized, standardizing the pixel range to between 0 and 1 to eliminate the interference of pixel value differences on feature extraction. The normalization calculation formula is: ,in, This represents the normalized pixel value. Represents the original pixel value. This represents the minimum number of pixels in the image. This represents the maximum pixel value of the image.

[0060] Next, the preprocessed source scene image and pseudo-target scene image are input into the semantic segmentation backbone network, and features are extracted by the encoder part of the network. For the source scene image, after convolution and pooling operations by the backbone network encoder, a source scene feature map is generated, which contains semantic information and spatial structure information of the target category in the source scene. For the pseudo-target scene image, a pseudo-target scene feature map is generated using the same network process, and the pseudo-target scene feature map retains the same features as the source scene. Figure 1The source scene feature map retains the original semantic structure but incorporates stylistic features of the pseudo-target scene, such as the water mist texture features of a rainy scene and the low brightness features of a night scene. Then, the source scene feature map and the pseudo-target scene feature map are concatenated along the channel dimension to form a cross-domain feature map, providing a data foundation for subsequent modeling of cross-domain relationships. , Represents cross-domain feature maps, Represents the source scene feature map. Represents the feature map of the pseudo-target scene. This indicates a channel splicing operation.

[0061] Then, the core structure of the latent prior extractor is constructed, which consists of multiple layers of residual blocks. Cross-domain feature maps are input into the residual block sequence and processed through a residual connection mechanism. To mitigate the vanishing gradient problem during deep network training and ensure the network can effectively learn the correlations between cross-domain features, among other things... This represents the output characteristics of the residual block. This represents the input features of the residual block. This represents a convolution operation. The output of the residual block sequence is processed through two projection layers. The first projection layer maps the cross-domain features to a feature space of a preset dimension, and the activation function is a linear rectified function. Enhance the nonlinear expressive power of features, where, Indicates linear rectified output. The first projection layer represents the input features; the second projection layer outputs two key parameters, namely the mean and variance of the prior in the latent domain. These two parameters statistically characterize the domain shift pattern between the source scene and the pseudo-target scene. For example, the mean reflects the overall shift trend of cross-domain features, and the variance reflects the degree of dispersion of cross-domain features.

[0062] Subsequently, the latent prior extractor generates a latent domain prior based on the generated mean and variance using a reparameterization technique. Specifically, a noise vector is randomly sampled from a standard normal distribution. , ,in, Represent a standard normal distribution; this noise vector With variance Perform element-wise multiplication, then multiply by the mean. Element-wise addition is performed to generate a latent prior that conforms to a standard normal distribution. : ,in, This represents element-wise multiplication. This process ensures that the latent domain prior can capture the domain offset patterns between the source and pseudo-target scenes, and also introduces a certain generalization ability through random noise, thus avoiding overfitting during subsequent model training.

[0063] Finally, the effectiveness of the generated latent domain priors is validated. Validation metrics include domain offset capturing capability and semantic consistency preservation capability; the domain offset capturing capability is evaluated by calculating the correlation between the latent domain priors and cross-domain feature maps, using the following formula: ,in, Represents the correlation coefficient. This represents the covariance between the latent domain prior and the cross-domain feature map. This represents the variance of the prior in the latent domain. Represents the variance of the cross-domain feature map. The function used to calculate the square root must have a correlation higher than a preset threshold to ensure that the latent domain prior accurately reflects cross-domain feature differences. Semantic consistency is maintained by inputting the latent domain prior into the decoder of the semantic segmentation backbone network to generate a semantic prediction map. The intersection-union ratio (IU / U) of this prediction map and the semantic annotations of the source scene image is calculated. The IU / U must not be lower than a preset threshold to ensure that the latent domain prior records domain offset patterns without destroying the semantic structure of the image. The validated latent domain prior is used in subsequent domain compensation and diffusion prior estimation modules, providing crucial prior information for domain generalization in UAV semantic perception.

[0064] Step 3: Gaussian noise is added to the latent domain prior using a diffusion prior estimator to extract features from unlabeled new scene images acquired in real-time by the UAV. Inverse denoising is then performed using the diffusion prior estimator to estimate the new scene's latent domain prior. Simultaneously, the domain compensation module maps the new scene's latent domain prior to feature modulation parameters, adjusting the semantic segmentation backbone network feature map. Finally, the effectiveness of the new scene's latent domain prior and the optimized feature map adapted to the new scene is verified. The specific process is as follows:

[0065] First, the core parameters and initial state of the diffusion prior estimator are determined. The total number of diffusion steps is set, balancing estimation accuracy and computational efficiency; typically, four steps are chosen. A noise variance sequence is set, linearly increasing from an initial value of 0.1 to a final value of 0.99, simulating the gradual transition of the latent domain prior from clear to fuzzy. The latent domain prior generated by the latent prior extractor is used as the initial input for forward diffusion. This initial latent domain prior records the domain shift patterns between the source and pseudo-target scenes, providing a foundation for subsequent new scene estimation.

[0066] Next, a forward diffusion process is performed to add Gaussian noise. Following a preset number of diffusion steps, iterative noise addition operations are performed on the initial latent domain prior. The state update formula for each diffusion step is: ,in, , They represent the first Step, First The potential state after step diffusion, Is with the first The coefficients corresponding to the step noise variance , For the first The variance of the Gaussian noise added during the diffusion process. For the first The Gaussian noise vector of the step, The vectors follow a standard normal distribution with a mean of 0 and a covariance matrix equal to the identity matrix. and These are the fusion weights for the latent state and the noise vector, respectively. The fusion weights are determined by the cumulative variance. Auxiliary determination, among which, Indicates the preceding step The product of these factors, i.e., the cumulative variance, is used to control the intensity of noise addition at each step, ensuring that the noise level of the prior in the latent domain gradually increases with the number of diffusion steps, ultimately yielding a fully noisy latent state. ;in, Indicates the first The coefficient corresponding to the step diffusion satisfies , Represents the first step in the diffusion process. This process involves several steps. It simulates the change from known domain offset patterns to unknown domain offset patterns, laying the foundation for subsequent reverse denoising to adapt to new scenarios.

[0067] Then, features of unlabeled new scene images acquired in real time by the drone are extracted. The new scene images are adjusted to match the size of the input to the semantic segmentation backbone network, and after pixel value normalization, they are input into the backbone network encoder. Through operations such as convolution, pooling, and attention mechanisms, multi-scale feature maps of the new scene images are extracted. The feature map contains style and semantic structure information of the new scene, such as the water mist features of the rain scene and the vegetation distribution features of the rural scene. As conditional information for the diffusion prior estimator to reverse denoising, it guides the denoising process to move closer to the domain shift pattern of the new scene.

[0068] Next, a reverse denoising process is performed to estimate the prior of the new scene's latent domain. The fully noisy latent state and the multi-scale feature map of the new scene image are jointly input into the denoising network of the diffusion prior estimator. This denoising network consists of multiple layers of convolutions and residual connections, which can capture the correlation between noise and new scene features. The state update formula for each denoising step is: ,in, Indicates the first The latent state after noise reduction. The denoising network predicts the first... Step noise, These are the parameters for the denoising network. The denoising network predicts and removes some noise from the latent state based on the current noise level and the characteristics of the new scene, updating the latent state; this process is repeated until all diffusion steps of denoising are completed, ultimately obtaining the latent domain prior corresponding to the new scene. The latent domain prior accurately reflects the domain offset patterns between the new scene and the source scene, such as the brightness offset in a night scene and the contrast offset in a foggy scene, without relying on the semantic annotation of the new scene.

[0069] Simultaneously, the domain compensation module is activated to adjust the feature maps. The estimated latent domain priors for the new scene are then used. The projection layer of the input domain compensation module, which consists of a fully connected layer and an activation function, incorporates the prior of the latent domain of the new scene. The mapping is divided into two sets of feature modulation parameters, namely the scale parameter. With offset parameter Used to adjust the weights of each channel in the feature map. This is used to correct the pixel value distribution of the feature map. The scale parameter is fused with the new scene feature map output by the backbone network through element-wise multiplication, and then the offset parameter is incorporated through element-wise addition for feature map adjustment. The adjustment formula is as follows: After adjusting the feature map, an optimized feature map adapted to the new scene is obtained. .

[0070] Finally, the priors of the latent domain for the new scenario. Optimized feature maps adapted to new scenarios Perform validity verification. New scenario latent domain prior. The verification is performed by calculating its multi-scale feature map with the new scene image. The correlation is achieved, and the correlation must be higher than a preset threshold to ensure that prior information can accurately guide feature adjustment. The optimized feature map adapted to the new scene is validated by inputting it into the semantic segmentation backbone network decoder to generate a preliminary semantic prediction map. The structural similarity index of the prediction map is calculated, and the index must not be lower than a preset threshold to ensure that the feature map retains complete semantic structural information while adapting to the new scene. The validated new scene latent domain prior and the optimized feature map adapted to the new scene are used to generate the final semantic segmentation result, providing support for accurate semantic perception of UAVs in unlabeled new scenes.

[0071] Step 4: Based on the optimized feature maps adapted to the new scene, output the final semantic segmentation result. Optimize the parameters of the semantic segmentation backbone network model using a joint training framework combining task loss, semantic consistency loss, and prior constraint loss. The specific process is as follows:

[0072] First, generate the final semantic segmentation result. Then, use the optimized feature map adapted to the new scene. The decoder part of the input semantic segmentation backbone network gradually restores the spatial resolution of the image through upsampling operations based on the multi-scale semantic information of the feature maps. For the DeepLabV3Plus architecture, the decoder uses a dilated spatial pyramid pooling module to perform multi-scale feature fusion on the optimized feature maps, and then adjusts the number of feature channels to the number of target categories through 1×1 convolution. For the Mask2Former architecture, the decoder performs instance-level semantic modeling on the optimized feature maps through a mask attention mechanism to generate a probability distribution map for each target category. Finally, an argmax operation is performed on the probability distribution map to obtain the target category label corresponding to each pixel, forming the final semantic segmentation result. This result can accurately match the domain characteristics of new scenes, such as accurately distinguishing roads from waterlogged areas in rainy scenes and clearly identifying building and vegetation outlines in nighttime scenes.

[0073] Next, the task loss function is constructed. An appropriate loss calculation method is selected based on the type of semantic segmentation backbone network: if the semantic segmentation backbone network uses DeepLabV3Plus, the task loss uses weighted cross-entropy loss, assigning higher weights to categories with fewer samples in the semantic segmentation results to avoid decreased recognition accuracy of niche categories due to class imbalance; if the semantic segmentation backbone network uses Mask2Former, the task loss uses focus loss, applying low weights to easily classified samples and high weights to difficult-to-classify samples to improve the model's ability to recognize targets with blurred boundaries or occlusions. The task loss is calculated based on the semantic annotations of the source scene dataset. The probability distribution map output by the backbone network is compared with the semantic annotations to quantify the difference between the semantic segmentation results and the true labels, providing gradient signals for the model to learn basic semantic recognition capabilities.

[0074] Then, a semantic consistency loss function is designed. The source scene image and its corresponding pseudo-target scene image are selected as comparison samples, and both are input into the trained model to obtain the source scene semantic prediction map. semantic prediction graph of pseudo-target scene ; Calculate the semantic intersection-union ratio of the two prediction graphs. : ,in, Target region of the source scene prediction map The corresponding target region is predicted in the pseudo-target scene map, and the structural similarity index is calculated simultaneously. The mean of these two metrics is used as the semantic consistency score. : Semantic consistency loss for: This loss function ensures that the semantic prediction results of the source scene and the pseudo-target scene remain consistent, ensuring that the model does not compromise its basic ability to identify target categories while learning the offset rules of the new scene domain, and avoiding semantic misalignment problems such as misjudging roads as buildings in pseudo-target scenes.

[0075] Next, the prior constraint loss function is defined. The latent domain prior of the source scene and pseudo-target scene generated by the latent prior extractor is obtained. And the new scene latent domain prior generated by the diffusion prior estimator Calculate the L2 norm between the two. The norm formula is: ;in, For prior constraint loss, It is the L2 norm. and The coordinates of the two sets of potential priors in the feature space are respectively. The value at the specified position quantifies the Euclidean distance between the two sets of latent domain priors in the feature space. By minimizing the prior constraint loss, it ensures that the latent domain priors of the new scene generated by the diffusion prior estimator are consistent with the domain offset rules learned by the latent prior extractor, avoiding estimation errors of the latent domain priors of the new scene that deviate from a reasonable range, and improving the generalization stability of the model for different new scenes.

[0076] Finally, a joint training framework is constructed and model parameters are optimized. The task loss, semantic consistency loss, and prior constraint loss are weighted and summed according to preset weights to obtain the total loss function. The weights of the total loss function need to balance the contribution of each loss; typically, the task loss weight is set to 1.0, the semantic consistency loss weight to 0.5, and the prior constraint loss weight to 0.3, ensuring that the model prioritizes semantic segmentation accuracy while also considering domain generalization ability and prior estimation accuracy. The Adam optimizer is used to optimize the total loss function. After each training round, the average intersection-union ratio (AUC) of the model is calculated using the validation set of the source scene dataset, and the model parameters with the highest AUC are saved as the optimal model. During training, a gradient pruning strategy is adopted, setting an upper limit on the gradient norm to prevent gradient explosion and ensure stable model convergence. Through this joint training framework, the model can simultaneously master semantic recognition and domain adaptation capabilities, outputting accurate semantic segmentation results in unlabeled new scenes, meeting the semantic perception needs of scenarios such as UAV inspection, surveying, and rescue.

[0077] The present invention also provides a system for implementing the above-mentioned UAV semantic perception domain generalization method based on probability diffusion modeling, comprising:

[0078] The source scene image processing module receives source scene images acquired by the image acquisition device and performs semantic annotation, generates a source scene dataset, enhances the photometric properties of the source scene images, generates pseudo target scene images that are semantically consistent with the source scene but have different scene styles, performs verification, and associates the verified pseudo target scene images with the corresponding source scene dataset.

[0079] The latent domain prior generation module receives the data processed by the source scene image processing module, and based on the latent prior extractor, obtains the parameters that characterize the domain offset pattern between the source scene and the pseudo-target scene: mean and variance, and generates the latent domain prior.

[0080] The new scene latent domain prior estimation module, based on the diffusion prior estimator, performs forward diffusion on the latent domain prior and adds Gaussian noise to extract the features of the unlabeled new scene image collected by the UAV in real time. It performs reverse denoising processing through the diffusion prior estimator to estimate the new scene latent domain prior. At the same time, the domain compensation module maps the new scene latent domain prior to feature modulation parameters and adjusts the feature map of the semantic segmentation backbone network.

[0081] The joint training framework optimization module outputs the final semantic segmentation result based on the optimized feature map. It uses task loss, semantic consistency loss, and prior constraint loss to optimize the parameters of the semantic segmentation backbone network model.

[0082] The present invention also provides a computer storage medium storing a computer program that, when run on a processor, executes the steps of the above-described generalization method for the semantic perception domain of unmanned aerial vehicles based on probability diffusion modeling.

[0083] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A generalization method for the semantic perception domain of unmanned aerial vehicles based on probability diffusion modeling, characterized in that, The process includes the following: Step 1: Collect source scene images and preprocess them to form a source scene dataset. Perform photometric enhancement on the images in the source scene dataset to generate pseudo target scene images. Finally, associate the verified pseudo target scene images with the corresponding source scene dataset. Step 2: Input the preprocessed source scene image and pseudo target scene image into the semantic segmentation backbone network for processing to obtain cross-domain feature maps. Then, output the mean and variance parameters that characterize the domain offset between the source scene and the pseudo target scene through the latent prior extractor. Finally, generate the latent domain prior through reparameterization. Step 3: Gaussian noise is added to the latent domain prior through forward diffusion using a diffusion prior estimator to extract features of the unlabeled new scene image collected in real time by the UAV. Inverse denoising is performed using the diffusion prior estimator to estimate the latent domain prior of the new scene. At the same time, the latent domain prior of the new scene is mapped to feature modulation parameters through the domain compensation module. The feature map of the semantic segmentation backbone network is adjusted to obtain an optimized feature map adapted to the new scene. Step 4: Output the final semantic segmentation result based on the optimized feature map adapted to the new scene, and optimize the parameters of the semantic segmentation backbone network model using a joint training framework of task loss, semantic consistency loss, and prior constraint loss.

2. The UAV semantic perception domain generalization method based on probability diffusion modeling according to claim 1, characterized in that, The specific process of step 1 is as follows: First, images of the drone operation scene at different times and angles are acquired using image acquisition equipment. The target category in each image is semantically labeled, and the training set and validation set are divided proportionally to form the source scene dataset. Next, photometric enhancement is performed on each image in the source scene dataset to generate pseudo-target scene images that are semantically consistent with the source scene but have different scene styles. The pseudo-target scene images are then quality-verified. Finally, the verified pseudo-target scene images are associated with the source scene dataset to form paired data of source scene images and pseudo-target scene images, which are used for subsequent latent domain prior extraction.

3. The UAV semantic perception domain generalization method based on probability diffusion modeling according to claim 1, characterized in that, The specific process of step 2 is as follows: First, determine the type of semantic segmentation backbone network, and then preprocess the source scene image and the pseudo target scene image. Next, the preprocessed source scene image and pseudo target scene image are input into the semantic segmentation backbone network. Features are extracted through the encoder part of the network to generate source scene feature map and pseudo target scene feature map. Then, the source scene feature map and pseudo target scene feature map are concatenated according to the channel dimension to form a cross-domain feature map. Then, the core structure of the latent prior extractor is constructed, which consists of multiple residual blocks. The cross-domain feature map is input into the residual block sequence and processed through the residual connection mechanism. The output of the residual block sequence is processed through two projection layers. The first projection layer maps the cross-domain features to the feature space of the preset dimension. The activation function is a linear rectified function. The second projection layer outputs the mean and variance of the latent prior. Then, the latent prior extractor generates the latent domain prior based on the mean and variance through reparameterization: a noise vector is randomly sampled from the standard normal distribution, the noise vector is multiplied element-wise with the variance, and then added element-wise with the mean, finally generating the latent domain prior that conforms to the standard normal distribution. Finally, the validity of the generated latent domain priors is verified.

4. The UAV semantic perception domain generalization method based on probability diffusion modeling according to claim 1, characterized in that, The specific process of step 3 is as follows: First, determine the core parameters and initial state of the diffusion prior estimator, set the total number of diffusion steps and the noise variance sequence, take the latent domain prior generated by the latent prior extractor as the initial input of the forward diffusion, then execute the forward diffusion process to add Gaussian noise, and finally obtain the fully noisy latent state. Then, features from unlabeled new scene images captured in real time by the drone are extracted, preprocessed, and input into the semantic segmentation backbone network encoder to extract multi-scale feature maps of the new scene images. ; Then, a reverse denoising process is performed using a diffusion prior estimator to estimate the latent domain prior of the new scene; Simultaneously, the domain compensation module is activated to adjust the feature map: the prior of the new scene's latent domain is input into the projection layer of the domain compensation module. This projection layer consists of a fully connected layer and an activation function, mapping the prior of the new scene's latent domain into two sets of feature modulation parameters, namely scale parameters. With offset parameter Used to adjust the weights of each channel in the feature map. This is used to correct the pixel value distribution of the feature map; the scale parameter is fused with the multi-scale feature map of the new scene image through element-wise multiplication, and then the offset parameter is incorporated through element-wise addition to adjust the feature map. The adjustment formula is as follows: ,in, This represents the optimized feature map adapted to the new scenario; Finally, the effectiveness of the latent domain prior for the new scene and the optimized feature map adapted to the new scene is verified.

5. The UAV semantic perception domain generalization method based on probability diffusion modeling according to claim 4, characterized in that, The specific process of performing the forward diffusion process to add Gaussian noise is as follows: According to the preset diffusion steps, iterative noise addition operations are performed on the initial latent domain prior. The state update formula for each diffusion step is: ,in, , They represent the first Step, First The potential state after step diffusion, Is with the first The coefficients corresponding to the step noise variance , For the first The variance of the Gaussian noise added during the diffusion process. For the first The Gaussian noise vector of the step; and These are the fusion weights for the latent state and the noise vector, respectively. The fusion weights are determined by the cumulative variance. Auxiliary determination, among which, Indicates the preceding step The product of these factors is used to control the intensity of noise addition at each step, ensuring that the noise level of the prior in the latent domain gradually increases with the number of diffusion steps, ultimately yielding a fully noisy latent state. ,in, Indicates the first The coefficient corresponding to the step diffusion; The specific process for estimating the prior potential of the new scenario is as follows: The fully noisy latent state and the multi-scale feature map of the new scene image are jointly input into the denoising network of the diffusion prior estimator. The state update formula for each step of the denoising network is as follows: ,in, Indicates the first The latent state after noise reduction. The denoising network predicts the first... Step noise, The parameters are for the denoising network. Based on the current noise level and the characteristics of the new scene, the denoising network predicts and removes some noise from the latent state, updating the latent state. This process is repeated until all diffusion steps of denoising are completed, ultimately obtaining the prior latent domain of the new scene. .

6. The UAV semantic perception domain generalization method based on probability diffusion modeling according to claim 1, characterized in that, The specific process of step 4 is as follows: First, the optimized feature map adapted to the new scene is input into the decoder part of the semantic segmentation backbone network to generate a probability distribution map for each target category. Then, the argmax operation is performed on the probability distribution map to obtain the target category label corresponding to each pixel, forming the final semantic segmentation result. Next, we construct the task loss function, the semantic consistency loss function, and the prior constraint loss function; Finally, a joint training framework was constructed and model parameters were optimized: the task loss, semantic consistency loss, and prior constraint loss were weighted and summed according to preset weights to obtain the total loss function; the weights of the total loss function were set to balance the contribution of each loss, with the task loss weight set to 1.0, the semantic consistency loss weight set to 0.5, and the prior constraint loss weight set to 0.3, ensuring that the model prioritizes semantic segmentation accuracy while also considering domain generalization ability and prior estimation accuracy; the Adam optimizer was used to optimize the total loss function, and after each round of training, the average intersection-union ratio (OCR) of the model was calculated using the validation set of the source scene dataset, and the model parameters with the highest OCR were saved as the optimal model.

7. The UAV semantic perception domain generalization method based on probability diffusion modeling according to claim 6, characterized in that, The semantic consistency loss function is constructed as follows: Source scene images and corresponding pseudo-target scene images are selected as comparison samples. Both are input into the trained model to obtain source scene semantic prediction maps and pseudo-target scene semantic prediction maps. The semantic intersection-union ratio (CIU) of the two prediction maps is then calculated. : ,in, Target region of the source scene prediction map The corresponding target region is predicted in the pseudo-target scene map, and the structural similarity index is calculated simultaneously. The semantic intersection-union ratio and the mean of the structural similarity index of the two predicted graphs are used as the semantic consistency score. : Then semantic consistency loss for: .

8. The UAV semantic perception domain generalization method based on probability diffusion modeling according to claim 6, characterized in that, The method for constructing the prior constraint loss function is as follows: Obtain the latent domain prior of the source scene and pseudo-target scene generated by the latent prior extractor. and the new scene latent domain prior generated by the diffusion prior estimator Calculate the L2 norm between the two: ;in, For prior constraint loss, It is the L2 norm. and The coordinates of the two sets of potential priors in the feature space are respectively. The value at the specified position quantifies the Euclidean distance between the two sets of latent domain priors in the feature space. By minimizing the prior constraint loss, it ensures that the new scene latent domain priors generated by the diffusion prior estimator are consistent with the domain offset patterns learned by the latent prior extractor.

9. A system for implementing the UAV semantic perception domain generalization method based on probability diffusion modeling as described in claim 1, characterized in that, include: The source scene image processing module receives source scene images acquired by the image acquisition device and performs semantic annotation, generates a source scene dataset, enhances the photometric properties of the source scene images, generates pseudo target scene images that are semantically consistent with the source scene but have different scene styles, performs verification, and associates the verified pseudo target scene images with the corresponding source scene dataset. The latent domain prior generation module receives the data processed by the source scene image processing module, and obtains the parameters that characterize the domain offset pattern between the source scene and the pseudo-target scene based on the latent prior extractor: mean and variance, and generates the latent domain prior. The new scene latent domain prior estimation module, based on the diffusion prior estimator, performs forward diffusion on the latent domain prior and adds Gaussian noise to extract features of the unlabeled new scene image collected in real time by the UAV. It then performs reverse denoising processing through the diffusion prior estimator to estimate the new scene latent domain prior. At the same time, the domain compensation module maps the new scene latent domain prior to feature modulation parameters and adjusts them to obtain an optimized feature map adapted to the new scene. The joint training framework optimization module outputs the final semantic segmentation result based on the optimized feature map adapted to the new scenario. It uses task loss, semantic consistency loss, and prior constraint loss to optimize the parameters of the semantic segmentation backbone network model through a joint training framework.

10. A computer storage medium, characterized in that, A computer program is stored on a computer storage medium. When the program runs on a processor, it executes the steps of the UAV semantic perception domain generalization method based on probability diffusion modeling as described in claim 1.

Images