Adaptive hybrid prototype learning method for real-time semantic segmentation

Abstract

The invention discloses a self-adaptive hybrid prototype learning method for real-time semantic segmentation, and belongs to the technical field of prototype learning. The method aims to solve the problem that a static prototype is difficult to adapt to dynamic change of a scene. The method comprises the following steps: screening high-confidence pixel features for each category based on an initial segmentation prediction map, and weighting to generate a dynamic prototype of each category for a current input image; performing adaptive weighted fusion on the dynamic prototype and a learnable global category prototype through learnable parameters to obtain a category guidance prototype; calculating a cosine similarity graph of the feature graph and the category guidance prototype; and finally, by introducing learnable scale parameters, carrying out residual connection on the similarity graph and the initial segmentation prediction graph, and outputting a final segmentation result. The method can be combined with various real-time semantic segmentation algorithms, and on the premise that only a small amount of calculation amount and parameter amount are increased, the adaptive capacity and segmentation precision of the model to scene changes are improved.

Description

Technical Field

[0001] This invention relates to the field of prototype learning technology, and more specifically to an adaptive hybrid prototype learning method for real-time semantic segmentation. Background Technology

[0002] Real-time semantic segmentation, a core task of computer vision, aims to accurately classify each pixel in an image while meeting real-time processing requirements. It is a key technology for achieving environmental perception and scene understanding. This technology plays an irreplaceable role in fields such as intelligent driving, nighttime surveillance, military reconnaissance, and industrial inspection, especially in applications relying on infrared imaging.

[0003] Currently, deep convolutional neural networks are the mainstream approach in this field. However, most existing real-time semantic segmentation algorithms rely on discriminative classifiers to independently predict pixels, which makes it difficult to fully model the feature distribution structure and semantic similarity relationships within the same semantic category. Therefore, when faced with complex scenarios, large intra-class appearance differences, or interference, problems such as category confusion, blurred boundaries, and degraded generalization performance can easily arise.

[0004] In recent years, prototype learning has been introduced into the field of real-time semantic segmentation. This method transforms the traditional pixel classification problem into a distance metric problem of finding the nearest neighbor prototype in the feature space by learning a representative feature prototype for each semantic category. This metric-based paradigm can better model the semantic similarity within categories, providing a new technical path to improve the robustness of semantic segmentation. However, existing prototype learning-based methods mostly rely on static prototypes obtained by aggregating from the entire training set. Static prototypes are essentially global averages of category features, making it difficult to capture and adapt to dynamic appearance changes of the same category in different specific scenarios. For example, under changing lighting conditions, viewing angles, distance scales, or partial occlusion, the visual features of the target will change significantly. Due to its inherent rigidity, the representativeness and discriminative power of static prototypes will be greatly weakened in these scenarios, leading to a decline in segmentation performance.

[0005] In particular, the aforementioned limitations are further amplified in infrared imaging applications such as nighttime surveillance and military reconnaissance: First, there is low contrast and complex background: Infrared images generally suffer from low contrast between the target and the background and a lack of texture information. The target edges are blurred and easily confused with background clutter. Static prototypes are difficult to accurately represent the category information under such weak features.

[0006] Second, there is active interference: In military applications, targets often have heat source interference. These interferences may have local similarities with real target components in the feature space, which can easily cause model misactivation and introduce a lot of noise.

[0007] Third, the target scale and attitude change drastically: the same type of target component (such as the nose of an aircraft or a propeller) varies in scale and shape at different distances and viewing angles. As a fixed vector, the static prototype cannot adapt to such dynamic changes, resulting in poor segmentation of small-scale, non-standard attitude components.

[0008] Fourth, there is the constraint of real-time requirements: application scenarios require algorithms to complete processing within limited computing resources and strict time delays. This limits the use of overly complex or time-consuming prototype optimization and matching mechanisms, necessitating a balance between model efficiency and adaptability.

[0009] In summary, existing real-time semantic segmentation methods, especially those based on static prototypes, exhibit significant performance bottlenecks when dealing with dynamic changes in real-world scenes, particularly challenges in infrared image segmentation such as low contrast, active interference, and varying scale and pose. Therefore, there is an urgent need in this field for a novel method that can overcome the limitations of static prototypes under strict real-time constraints, achieving adaptive optimization of prototype representation, thereby improving the quality and discriminative ability of prototype representation. Summary of the Invention

[0010] The purpose of this invention is to provide an adaptive hybrid prototype learning method for real-time semantic segmentation, addressing the problem of insufficient representational ability of static prototypes in dynamic scenes. This method adaptively fuses general prior knowledge of categories with instance-specific information of the current image to construct a hybrid prototype capable of flexibly guiding pixel classification. Furthermore, similarity metrics based on this hybrid prototype are efficiently and stably integrated into the initial segmentation prediction. Ultimately, this achieves adaptive online optimization of prototype representation with almost no increase in computation or parameters, significantly enhancing the algorithm's ability to perceive and adapt to dynamic changes in the same category under different appearances, scales, viewpoints, and occlusion conditions. Simultaneously, it effectively improves the model's robustness and accuracy in complex infrared scenes with low contrast and active interference.

[0011] To achieve the above objectives, the technical solution adopted by this invention is: an adaptive hybrid prototype learning method for real-time semantic segmentation, comprising the following steps: S1. Extract features from the input image to obtain a feature map, and generate an initial segmentation prediction map from the feature map using a semantic segmentation head; S2. The feature map is refined by the feature refinement module to obtain a refined feature map. The feature refinement module is composed of a convolutional layer, a batch normalization layer and an activation function layer connected in series. S3. Based on the initial segmentation prediction map, select the top segments with the highest prediction confidence for each semantic category. The pixel locations and their normalized confidence values; S4, based on the obtained previous Each pixel location and its corresponding confidence value are used to extract the feature vector from the refined feature map. For each semantic category, the feature vector is calculated using the previous pixel location and its corresponding confidence value. The confidence value corresponding to each pixel is a weight for that category. The feature vectors are weighted and summed to generate dynamic prototypes for each category. These dynamic prototypes represent the feature distribution centers of the corresponding category in the current image. S5. Introduce a set of learnable global category prototypes; S6. Through a learnable fusion coefficient The dynamic prototype is adaptively and weightedly fused with the learnable global category prototype to generate the final category guidance prototype. S7. Calculate the cosine similarity between the features at each pixel position in the feature map described in step S1 and the class prototypes in the class guidance prototype, and use the similarity to measure the distance between classes to generate a similarity map; S8. Multiply the similarity map by a learnable scale parameter. Then, a residual connection is performed with the initial segmentation prediction map to obtain the final segmentation result.

[0012] Furthermore, in step S4, for the category Its dynamic prototype is calculated using the following formula: In the formula, Indicate category The dynamic prototype Indicate category The former A set of pixel indices Indicates index The pixel at that location belongs to the category The confidence level value, Indicates index Pixel in refined feature map The corresponding feature vector in the vector has a dimension of ; The dynamic prototypes corresponding to all categories are combined in order of category to form a dynamic prototype set.

[0013] Furthermore, in step S5, the learnable global category prototype is a parameter matrix that is randomly initialized and iteratively optimized through the backpropagation algorithm during model training. Each row vector in the matrix represents a general representation of a category.

[0014] Furthermore, in step S6, the adaptive weighted fusion is achieved through the following formula: In the formula, Indicates a category-guided prototype. Represents the learnable fusion coefficient. , Represents a learnable global category prototype. This represents a dynamic prototype.

[0015] Furthermore, in step S7, the formula for calculating cosine similarity is: In the formula, Represents the vector dot product. Representing vectors Norm, Indicates the location The similarity between the feature vector of a pixel and the prototype of each category; the higher the similarity value, the closer the pixel is to the original location. The higher the probability of belonging to the corresponding category.

[0016] Furthermore, in step S8, the process of fusing the initial segmentation prediction map and the similarity map is as follows: In the formula, This represents the final segmentation result image. This represents the initial segmentation prediction map. Represents a similarity graph. This is a learnable scale parameter.

[0017] Furthermore, the learnable fusion coefficients and learnable scale parameters All models are iteratively optimized during the model training process using the backpropagation algorithm.

[0018] Further, in step S2, the convolutional layer is a 1×1 convolutional layer, and the activation function layer is... Activation function layer.

[0019] Furthermore, in step S3, the top predictions with the highest confidence levels are selected. The pixel locations specifically include: for each category, selecting the pixel with the highest prediction confidence from the initial segmentation prediction map. Each pixel records its index position in the feature map and its corresponding path length. The confidence score after function normalization.

[0020] Furthermore, the method is integrated as a general module with the real-time semantic segmentation algorithm, and the integration position is after the semantic segmentation head.

[0021] According to the above technical solution, the beneficial effects of the present invention are: 1. This invention significantly improves the model's adaptability to dynamic scene changes and segmentation accuracy. By generating a dynamic prototype reflecting the current image features for each input image in real time, and adaptively weighting the dynamic prototype with a global category prototype using learnable fusion coefficients, the model can adaptively balance global general semantic features with the specific features of the current image instance. This mechanism allows the model to flexibly adjust its discrimination criteria when faced with appearance changes of the same category under different lighting, viewpoints, scales, or occlusions, thereby achieving stable optimization of segmentation results and effectively improving pixel-level classification accuracy.

[0022] 2. This invention achieves performance enhancement with almost no sacrifice in real-time performance, demonstrating outstanding engineering practical value. The entire adaptive hybrid prototype learning module adopts a lightweight design, rapidly completing prototype computation and fusion during forward propagation, and efficiently correcting initial predictions through residual connections. As shown in the embodiments, when combined with real-time networks such as SeaFormer, the number of model parameters hardly increases, the computational cost (FLOPs) only slightly increases, and the inference speed (FPS) remains at a high real-time level. This indicates that the module can be integrated as a plug-and-play component into various real-time semantic segmentation algorithms, achieving significant performance improvements with minimal computational overhead and parameter increments. This allows the advantages of prototype learning technology to be leveraged on resource-constrained mobile or edge computing platforms, while simultaneously meeting the stringent requirements of real-time applications for both low latency and high accuracy.

[0023] 3. This invention effectively enhances the robustness of the model in complex scenes, especially in suppressing noise and blurred boundary interference. By selecting high-confidence pixel features to generate a dynamic prototype, this mechanism filters out blurred edge pixels and background interference at the source, making the prototype representation purer. Furthermore, residual optimization of the initial segmentation prediction map using a similarity map can enhance the response in correctly predicted regions and suppress activation in mispredicted or incorrect regions. Figure 5 As shown in the heatmap comparison, after introducing AHPL, the model's response to the target region is more focused and accurate, and the feature activation region is closely attached to the target structure, effectively suppressing background interference. This is especially important in complex scenes such as low-contrast infrared images and those with thermal interference, directly improving the visual quality and robustness of the segmentation results.

[0024] 4. This invention provides a stable and efficient model optimization path. By introducing a learnable scale parameter to control the intensity of residual fusion, it effectively avoids gradient instability caused by scale mismatch between the similarity map and the initial segmentation prediction map, thereby ensuring smooth and efficient convergence during training (e.g., Figure 6As shown in the loss curve, the final convergence loss is lower. Meanwhile, the convergence trend of the fusion coefficients during training (e.g.) Figure 7 As shown, its value gradually adjusts and stabilizes at an equilibrium point from the initial stage of training, which focuses more on the dynamic prototype, thus verifying the effectiveness and reliability of the model design.

[0025] 5. This invention possesses good versatility and scalability. The adaptive hybrid prototype learning method is decoupled from the specific backbone network structure, and its design relies only on general feature maps and initial segmentation prediction maps. Therefore, this method can not only be applied to the specific networks in the embodiments but also be easily transferred to other real-time semantic segmentation algorithms. This demonstrates the universal value of this method as a basic performance enhancement module, providing an efficient and universal technical path for solving the adaptation problem of models in diverse scenarios. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the structure of the adaptive hybrid prototype learning method; Figure 2 This is a structural diagram of a semantic segmentation algorithm based on adaptive hybrid prototype learning; Figure 3 This is an example of a portion of the infrared dataset samples in an embodiment of the present invention; Figure 4 This is a comparison chart of segmentation results using different algorithms in this embodiment of the invention; Figure 5 This is a comparative heatmap used in embodiments of the present invention to verify the effectiveness of the adaptive hybrid prototype learning method; Figure 6 This is a visualization of the loss curve in an embodiment of the present invention; Figure 7 This is a visualization of the hybrid prototype fusion coefficient in an embodiment of the present invention; Figure 8 This is a visualization of the learnable scale parameters in an embodiment of the present invention. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0028] It should be noted that, unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0029] The specific structure of the Adaptive Hybrid Prototype Learning (AHPL) method of this invention is as follows: Figure 1 As shown, the AHPL method includes the following steps: (1) Initial segmentation prediction First, the feature maps output by the backbone network are processed using a conventional semantic segmentation head. Mapping is performed to obtain the initial class segmentation prediction map. ,in For the number of categories, Indicates the height of the feature map. This represents the width of the feature map. The initial segmentation prediction map contains confidence information about each pixel belonging to different categories. This preliminary classification result provides a coarse-grained semantic prior for guiding and optimizing subsequent steps.

[0030] (2) Feature Refinement Feature map of input segmentation head The input feature refinement module has a structure consisting of one A convolutional layer, a batch normalization (BN) layer, and a The activation functions are connected in series. Convolutional layers are responsible for extracting local spatial features, while batch normalization (BN) layers are used to stabilize the feature distribution. The activation function further introduces nonlinearity to suppress negative responses and improve the model's expressive power. Refined feature maps. The calculation process is shown in formula (1): (1) In the formula, express Convolutional layer Indicates the batch normalization layer. express Activation function.

[0031] After processing by this module, the output is a feature map that is similar to the input feature map. Refined feature maps of the same dimension This provides a high-quality feature base for subsequent prototype generation.

[0032] (3) High-confidence sampling of the initial segmentation prediction map Traditional prototype learning methods typically construct category prototypes by globally averaging pixel features across the entire region. However, in the initial prediction, object edges often contain a large number of blurry pixels, and may also be mixed with noise pixels that have been misclassified. If global averaging is performed indiscriminately, the prototype is easily affected by background noise and abnormal regions, thereby reducing its discriminative power and affecting segmentation accuracy.

[0033] To address this issue, this method selects the highest confidence level in each category prediction graph. Each pixel is used as a representative sample for that category. Specifically, for each category... ( ), from the initial segmentation prediction map generated in step (1) Perform high-confidence pixel sampling. Specifically, for For each of the categories, in the initial probability graph Iterate through all pixels and select the one with the highest prediction confidence for each category. Each pixel. Simultaneously, this is recorded. The index position of each pixel in the feature map and their corresponding processes The confidence score after function normalization.

[0034] Filtered by this method Each pixel is considered as the most representative pixel-level sample of the corresponding category in the current image, which can effectively avoid the interference of blurry pixels and noisy pixels on subsequent prototype construction.

[0035] (4) Dynamic prototype generation This step aims to generate a dynamic prototype for the current input image for each category. The specific process is as follows: Based on the results obtained in step (3), the top [number] of each category The pixel index is derived from the refined feature map output in step (2). In the process, the feature vector corresponding to each pixel index is extracted. Then, using the feature vector corresponding to these pixels... Confidence level As a weight, for each category before We perform a weighted summation of the corresponding feature vectors of each pixel to obtain a dynamic prototype that represents the spatial feature distribution center of each category in the current image. .

[0036] This dynamic prototype is adaptive because it is entirely determined by the content of the current input image, precisely matching the specific appearance of that category within the current image. The category... Dynamic prototype The calculation formula is shown in formula (2): (2) In the formula, Indicate category The former A set of pixel indices Indicates index The pixel at that location belongs to the category The confidence level value, Indicates index Pixel in refined feature map The corresponding feature vector in the vector has a dimension of ; .

[0037] Dynamic Prototype It consists of dynamic prototypes corresponding to all categories. It is formed by combining categories in the feature space, where This represents the total number of categories in the dataset. For feature dimensions.

[0038] This step generates a unique prototype representation for each input image and each category. This adaptability enables the algorithm to effectively handle objects with different scales, viewpoints, and appearances, overcoming the limitations of traditional static prototypes in adapting to dynamic scene changes in complex scenarios.

[0039] (5) Learnable global category prototype generation Introduce a set of learnable global category prototypes Each row vector in this matrix represents a general semantic representation of a category. First, the parameter matrix is ​​initialized. During forward propagation, this matrix is ​​directly used as a global prototype for each category in the computation. Subsequently, the parameter matrix is ​​iteratively updated using the backpropagation algorithm, enabling it to gradually learn the general representation of each category in the feature space.

[0040] (6) Prototype fusion The final category guidance prototype used to guide pixel classification This is achieved through learnable global category prototypes. and the dynamic prototype generated in step (4) The results are obtained through dynamic weighted fusion. The learnable global category prototype extracts general feature representations of categories from the entire training dataset, forming stable prior knowledge; while the dynamic prototype generates specific feature representations for the current input image, possessing context adaptability.

[0041] Using only dynamic prototypes may lead to overfitting due to the influence of single image features, while using only learnable global category prototypes may fail to adapt to the details of specific scenes. Therefore, this method designs a fusion mechanism through a learnable parameter. The learnable global category prototype and the dynamic prototype are adaptively weighted and fused to obtain the final category-guided prototype. The fusion process is shown in formula (3): (3) This fusion strategy achieves complementarity between global general knowledge and local scene adaptation, enabling the algorithm to achieve the best balance between "generality" and "specificity". This allows for fine-tuning of the details of specific images while maintaining an understanding of general features, thereby improving the adaptability of semantic segmentation algorithms in complex and diverse scenarios.

[0042] (7) Cosine similarity calculation After obtaining the final category guidance prototype, it needs to be compared with the feature map of the input segmentation head. Matching is performed to complete pixel-level prediction. Specifically, for feature maps... Each pixel position By calculating the feature vector at that location With the final category guide prototype The cosine similarity between them is used to obtain the semantic similarity score with each category.

[0043] This method utilizes the similarity calculation between pixel features and category prototypes to generate a refined similarity map. This allows us to calculate the similarity score for each pixel belonging to different categories. The calculation process for cosine similarity is shown in formula (4): (4) In the formula, Represents the vector dot product. Representing vectors Norm, Indicates the location The similarity between the feature vector of a pixel and the prototype of each category. A higher similarity value indicates that the pixel is closer to the target location. The higher the probability of belonging to the corresponding category.

[0044] (8) Residual connection fusion output Directly replacing the initial segmentation prediction map with a prototype-guided similarity map might result in the loss of valuable information from the initial prediction, especially in boundary or small object regions. To avoid this information loss, the similarity map is fused with the initial segmentation prediction map using a residual approach, optimizing the initial prediction map. In semantically consistent regions, the similarity map enhances the response of the initial prediction; in semantically inconsistent regions, it suppresses erroneous activation responses, thus refining the initial segmentation prediction map.

[0045] Directly using the similarity map for residual connection may lead to gradient instability, especially when the value range of the similarity map differs significantly from the initial prediction. Therefore, this method multiplies the similarity map obtained in step (7) by a scale adaptive parameter. Then, a residual connection is performed with the initial segmentation prediction map. The scale-adaptive parameter scales the similarity map to keep the value of the residual connection within a reasonable range, thereby helping the gradient flow to be smoother, accelerating the algorithm convergence and improving training stability.

[0046] The fusion process is shown in formula (5): (5) This residual fusion mechanism further improves the accuracy of prediction results while effectively preserving the original details.

[0047] Adaptive Hybrid Prototype Learning (AHPL) can be used as a general module to combine with mainstream semantic segmentation algorithms. To better illustrate the technical effects of this invention, the following detailed description uses the integration of the method of this invention into the SeaFormer algorithm and its application to infrared image semantic segmentation as a preferred embodiment.

[0048] SeaFormer is a real-time semantic segmentation network based on a lightweight Transformer, employing a dual-branch structure: a context branch and a spatial branch. The context branch achieves efficient local / global semantic modeling by alternately stacking MobileNetV2 blocks and axial attention-based SeaFormer layers; the spatial branch preserves high-resolution features to capture fine boundary information. Information exchange between the two branches is achieved through a feature fusion module. The SeaFormer algorithm is designed for resource-constrained scenarios, making it suitable for application on mobile devices and edge computing platforms, and capable of achieving high-precision semantic segmentation tasks with low latency.

[0049] like Figure 2 As shown, this invention integrates an adaptive hybrid prototype learning method into the SeaFormer real-time semantic segmentation algorithm, constructing the AHPL_SeaFormer algorithm. Specifically, after the prototype learning method is connected to the SeaFormer segmentation head in a concatenated manner, it uses the preliminary segmentation prediction map output by the segmentation head to generate a more discriminative hybrid guidance prototype, and then optimizes the final segmentation result through prototype matching, residual fusion and other steps.

[0050] SeaFormer itself provides high-quality feature representations with a lightweight and efficient attention mechanism, while the AHPL method further endows the algorithm with the ability to dynamically adapt to scene changes. This adaptive hybrid prototype learning method can significantly improve segmentation quality with only a small increase in computation and parameter count.

[0051] Experimental verification 1. Experimental Environment and Parameters Experimental operating system: Windows 10; Hardware configuration: CPU model: Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz, RAM: 32GB. GPU model: NVIDIA 3090, VRAM: 24GB. All algorithms are built and run on the PyTorch framework, using Python 3.8 as the programming language, and CUDA 11.3.1 and CUDNN 8.2.1 are used to accelerate the GPU.

[0052] Core parameter settings for the algorithm training phase: During the training phase, the image resolution was uniformly set to 640×640, the batch size was set to 8, the Adam optimizer was used, the learning rate was 0.004, and the weight decay was 0.0001. A total of 100 training epochs were conducted, with the learning rate gradually decreasing linearly. The number of sampling points used in the experiment... It is 128.

[0053] 2. Dataset The experimental data for this invention uses the publicly available infrared dataset LSOTB-TIR as the basic data source. 3984 images were selected from this dataset, and artificial interference was added to them, such as... Figure 3 As shown. The first row represents the original LSOTB-TIR data, the second row represents the image after adding noise, and the third row represents the corresponding annotation results.

[0054] The added interference follows the following rules: (1) The gray distribution satisfies the Gaussian model, that is, the gray value is the largest at the center and gradually decreases outward; (2) The interference release process goes through a process of change from small to large and then from large to small, and gradually moves away from the target; (3) The shape is modeled by the equation of an irregular circle.

[0055] Labelme software was used to generate JSON files containing pixel category and location information, and the targets were divided into four categories: nose (red), tail (green), propeller (yellow), and interference (blue). The dataset was finally divided into training, validation, and test sets in an 8:1:1 ratio.

[0056] 3. Data Analysis (1) Evaluation indicators Standard evaluation metrics in the semantic segmentation field are used: Mean Intersection over Union (MIoU) and Mean Pixel Accuracy (MPA). The efficiency of the algorithm is also evaluated using the number of parameters, computational complexity (FLOPs), and inference speed (FPS).

[0057] Mean Intersection over Union (MIoU) MIoU is the core accuracy metric for semantic segmentation. First, the interaction ratio (IoU) of each category is calculated, and then the average of the IoU results for all categories is calculated to obtain the final average interaction ratio. The calculation process of MIoU is shown in Equation (6): (6) Mean Pixel Accuracy (MPA) MPA is an indicator that measures the overall pixel classification accuracy of the algorithm. First, the pixel accuracy of each category is calculated separately, and then the pixel accuracy of all categories is averaged. The calculation process of MPA is shown in Equation (7): (7) eq \o\ac(○,3) Frame rate (Frames per Second, FPS) FPS represents the number of image frames that the algorithm can process per second, and is used to evaluate the inference speed of the algorithm. The calculation process of FPS is shown in Equation (8): (8) (2) Ablation analysis To verify the effectiveness of the adaptive hybrid prototype learning method in real-time semantic segmentation algorithms, this invention uses the SeaFormer real-time semantic segmentation algorithm as the basic framework for ablation experiments. The experimental results are shown in Table 1.

[0058] Table 1 Ablation Analysis To verify the effectiveness of the Adaptive Hybrid Prototype Learning (AHPL) method, experiments were conducted by incorporating AHPL into the real-time semantic segmentation algorithm SeaFormer. Compared to the basic SeaFormer algorithm, MIoU improved by 0.72 percentage points, MPA improved by 1.27 percentage points, FLOPs increased by 0.02G, Params remained almost unchanged, and FPS decreased by 4.45 frames / s. This demonstrates that the AHPL method can improve the segmentation performance of real-time semantic segmentation algorithms with only a small amount of overhead.

[0059] (3) Comparative experiment of different real-time semantic segmentation algorithms The AHPL_SeaFormer algorithm, constructed by integrating the method of this invention with the SeaFormer network, was compared with eight mainstream real-time semantic segmentation algorithms: SegNet, ERFNet, ESPNet, LEDNet, CGNet, BiSeNetv2, DDRNet, and SeaFormer. The segmentation accuracy and running efficiency of each algorithm were comprehensively evaluated. The results of the above comparative experiments are shown in Table 2.

[0060] Table 2 Segmentation results of different algorithms It can be seen that the real-time semantic segmentation network AHPL_SeaFormer based on adaptive hybrid prototype learning achieves an MIoU of 89.94%, an MPA of 94.35%, and the best average pixel accuracy and average intersection-union ratio in the infrared aircraft segmentation task; it also achieves the best results in terms of parameter quantity and computational cost compared to classic algorithms such as DDRNet.

[0061] The literature information corresponding to each comparison algorithm in Table 2 is as follows: SegNet: A Deep Convolutional Encoder-Decoder Architecture for ImageSegmentation; ERFNet: Efficient Residual Factorized ConvNet for Real-Time SemanticSegmentation; ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation; LEDNet: A Lightweight Encoder-Decoder Network for Real-Time SemanticSegmentation; CGNet: A Light-weight Context Guided Network for SemanticSegmentation; BiSeNet V2: Bilateral Network with Guided Aggregation for Real-timeSemantic Segmentation; DDRNet: Deep Dual-resolution Networks for Real-time and AccurateSemantic Segmentation of Road Scenes; SeaFormer: Squeeze-enhanced Axial Transformer for Mobile SemanticSegmentation.

[0062] Visualization results of different real-time semantic segmentation algorithms are as follows Figure 4 As shown in the figure, the segmentation comparison results of key components of the infrared aircraft sample are presented, including the original image, ground truth (GT), SegNet, ERFNet, ESPNet, LEDNet, CGNet, BiSeNetv2, DDRNet, SeaFormer, and AHPL_SeaFormer algorithm.

[0063] Rows 1, 4, 5, and 6 show a comparison of the propeller component segmentation. The results show that while all algorithms can roughly identify the propeller structure, except for AHPL_SeaFormer, the others generally suffer from structural breaks, blurred edges, or misclassified blades. AHPL_SeaFormer accurately recovers the number and geometry of the propeller blades with high detail completeness, almost perfectly matching the ground truth algorithm.

[0064] As observed from the second row of samples, AHPL_SeaFormer can still accurately locate and completely segment the nose region even in infrared images with complex backgrounds and low contrast. Compared to other algorithms, AHPL_SeaFormer not only closely resembles ground truth in terms of morphological structure but also clearly distinguishes the nose from surrounding components, effectively avoiding edge confusion. While other algorithms can detect the general area, they exhibit blurred outlines.

[0065] In lines 3 and 4, AHPL_SeaFormer reconstructs the tail outline relatively completely with a continuous structure. In contrast, other algorithms are unstable in the tail region, exhibiting partial structural loss or edge breakage.

[0066] A comprehensive observation of all samples reveals that AHPL_SeaFormer maintains high-quality segmentation performance under various flight attitudes and occlusion conditions. This indicates that the introduction of the AHPL method effectively improves the algorithm's generalization ability and robustness, enabling it to effectively cope with the challenges of complex flight conditions and local occlusion.

[0067] (4) Validation of the effectiveness of the adaptive hybrid prototype learning method like Figure 5 As shown, this set of heatmaps illustrates the comparison between the adaptive hybrid prototype learning method before (first row) and after (second row). It is clearly observed that before the method was introduced, the algorithm's response to the target region was weaker. After introducing the adaptive hybrid prototype learning method, the algorithm's response to the target region became more concentrated, the feature activation regions closely adhered to the target structure, and background interference was effectively suppressed.

[0068] (5) Visualization of loss curves Figure 6 The figure shows the training loss curves of the algorithm before and after introducing the adaptive hybrid prototype learning method. The horizontal axis represents the number of iterations, and the vertical axis represents the loss value.

[0069] Overall, both algorithms exhibit a typical training pattern of "rapid descent - slow descent - convergence," indicating that they effectively learn and reduce prediction errors during training. As the number of iterations increases, the rate of loss descent slows for both algorithms. However, the AHPL_SeaFormer algorithm consistently maintains a lower loss value than the SeaFormer algorithm, and the gap between them gradually widens. In the later stages of training, the loss curves of both algorithms tend to plateau, indicating that their performance has approached convergence. The difference in loss between the two algorithms demonstrates that by introducing the AHPL method, the algorithm can learn more refined and robust feature representations, thereby achieving lower prediction errors and higher segmentation accuracy on the training dataset.

[0070] (6) Hybrid prototype fusion coefficient Visualization like Figure 7 As shown in the figure, this figure illustrates the fusion coefficients between the dynamic prototype and the learnable global category prototype in step (6). The trend of change during training. The horizontal axis represents the number of training iterations, and the vertical axis represents the weight values ​​of the fusion coefficients.

[0071] The weights remain at a high level (approximately 0.57) in the early stages of training (about the first 20k iterations), at which point the algorithm mainly relies on dynamic prototypes to quickly adapt to the feature distribution. As training progresses, the weights gradually decrease and stabilize (approximately 0.51), indicating that the algorithm gradually increases its reliance on global category prototypes.

[0072] (7) Learnable scale parameters Visualization like Figure 8 As shown in the figure, this diagram illustrates the learnable scale parameters of step (8). The trend of change during training. The horizontal axis represents the number of training iterations, and the vertical axis represents the weight values ​​of the learnable scale parameters.

[0073] The figure shows the trend of this parameter as the number of iterations increases. It can be seen that the weight gradually increases at the beginning and shows a steady growth trend. Eventually, the parameter gradually converges during training, and the weight value stabilizes, with the final result approaching 0.51. This indicates that the algorithm has learned an optimal fusion strategy, that is, while preserving the initial prediction details, appropriately incorporating the similarity map to correct the segmentation results, proving the effectiveness of the method.

[0074] In summary, this invention enhances the adaptability of prototype learning to scene changes by constructing an adaptive hybrid prototype learning method. First, based on the confidence values ​​at each position in the initial segmentation prediction image, the method selects the most discriminative feature samples from the input features, effectively suppressing the interference of noise samples on prototype quality. Next, the selected feature samples are weighted and summed with the corresponding confidence values ​​in the initial segmentation prediction image to generate a dynamic prototype specific to the current image. This dynamic prototype represents the feature distribution center of each category in the current image. Second, the method employs learnable fusion coefficients to adaptively weight and fuse the global category prototype and the dynamic prototype, achieving a balance between global general features and instance-specific features. This allows the generated category-guided prototype to maintain category discriminativeness while flexibly adapting to different appearance changes. Furthermore, to avoid gradient instability, this method introduces a learnable scale parameter. The method scales the similarity map generated by the prototype guided by the final category and then performs a residual connection with the feature map to improve training stability and model convergence. This method can be combined with various real-time semantic segmentation algorithms to enhance their adaptability to different scenes and pose variations.

[0075] Finally, it should be noted that any parts of this invention not described in detail are prior art. Those skilled in the art will understand that the above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. An adaptive hybrid prototype learning method for real-time semantic segmentation, characterized in that, Includes the following steps: S1. Extract features from the input image to obtain a feature map, and generate an initial segmentation prediction map from the feature map using a semantic segmentation head; S2. The feature map is refined by the feature refinement module to obtain a refined feature map. The feature refinement module is composed of a convolutional layer, a batch normalization layer and an activation function layer connected in series. S3. Based on the initial segmentation prediction map, select the top segments with the highest prediction confidence for each semantic category. The pixel locations and their normalized confidence values; S4, based on the obtained previous Each pixel location and its corresponding confidence value are used to extract the feature vector from the refined feature map. For each semantic category, the feature vector is calculated using the previous pixel location and its corresponding confidence value. The confidence value corresponding to each pixel is a weight for that category. The feature vectors are weighted and summed to generate dynamic prototypes for each category. These dynamic prototypes represent the feature distribution centers of the corresponding category in the current image. S5. Introduce a set of learnable global category prototypes; S6. Through a learnable fusion coefficient The dynamic prototype is adaptively and weightedly fused with the learnable global category prototype to generate the final category guidance prototype. S7. Calculate the cosine similarity between the features at each pixel position in the feature map described in step S1 and the class prototypes in the class guidance prototype, and use the similarity to measure the distance between classes to generate a similarity map; S8. Multiply the similarity map by a learnable scale parameter. Then, a residual connection is performed with the initial segmentation prediction map to obtain the final segmentation result.

2. The adaptive hybrid prototype learning method for real-time semantic segmentation according to claim 1, characterized in that, In step S4, for the category Its dynamic prototype is calculated using the following formula: In the formula, Indicates category The dynamic prototype Indicates category The former A set of pixel indices Indicates index The pixel at that location belongs to the category The confidence level value, Indicates index Pixel in refined feature map The corresponding feature vector in the vector has a dimension of . ; ; All dynamic prototypes corresponding to each category are combined in order of category to form a dynamic prototype set.

3. The adaptive hybrid prototype learning method for real-time semantic segmentation according to claim 1, characterized in that, In step S5, the learnable global category prototype is a parameter matrix that is randomly initialized and iteratively optimized through the backpropagation algorithm during model training. Each row vector in the matrix represents a general representation of a category.

4. The adaptive hybrid prototype learning method for real-time semantic segmentation according to claim 2, characterized in that, In step S6, the adaptive weighted fusion is achieved through the following formula: In the formula, Indicates a category-guided prototype. Represents the learnable fusion coefficient. , Represents a learnable global category prototype. This represents a dynamic prototype.

5. The adaptive hybrid prototype learning method for real-time semantic segmentation according to claim 1, characterized in that, In step S7, the formula for calculating cosine similarity is: In the formula, Represents the vector dot product. Representing vectors Norm, Indicates the location The similarity between the feature vector of the pixel and the prototype of each category; The higher the similarity value, the closer the pixel is to the target location. The higher the probability of belonging to the corresponding category.

6. The adaptive hybrid prototype learning method for real-time semantic segmentation according to claim 5, characterized in that, In step S8, the process of fusing the initial segmentation prediction map and the similarity map is as follows: In the formula, This represents the final segmentation result image. This represents the initial segmentation prediction map. Represents a similarity graph. This is a learnable scale parameter.

7. The adaptive hybrid prototype learning method for real-time semantic segmentation according to claim 1, characterized in that, The learnable fusion coefficients and learnable scale parameters All models are iteratively optimized during the model training process using the backpropagation algorithm.

8. The adaptive hybrid prototype learning method for real-time semantic segmentation according to claim 1, characterized in that, In step S2, the convolutional layer is a 1×1 convolutional layer, and the activation function layer is... Activation function layer.

9. The adaptive hybrid prototype learning method for real-time semantic segmentation according to claim 1, characterized in that, In step S3, the top prediction confidence scores are selected. The pixel locations specifically include: for each category, selecting the pixel with the highest prediction confidence from the initial segmentation prediction map. Each pixel records its index position in the feature map and its corresponding path length. The confidence score after function normalization.

10. The adaptive hybrid prototype learning method for real-time semantic segmentation according to claim 1, characterized in that, The method is integrated as a general module with the real-time semantic segmentation algorithm, and the integration position is after the semantic segmentation head.

Images