Panoramic segmentation method and device in complex scene, and storage medium

Abstract

The invention provides a panoramic segmentation method and device in a complex scene and a storage medium, and the method comprises the steps: inputting an initial image into a panoramic segmentation network, and obtaining a panoramic segmentation result of the initial image; the panoramic segmentation network comprises: a backbone network; the feature coding layer comprises a DLK module and a DFF module which are cascaded, and the DLK module and the DFF module cooperate with each other to generate enhanced features E1 '-E4'; the edge guiding layer comprises an edge guiding fusion module and a feature pyramid, the EGF module fuses the C1 feature of the backbone network with the enhanced feature E4'to generate an edge enhanced feature F, and then performs channel splicing with E1 '-E3' to construct a four-level feature pyramid F1-F4; the dual-path decoding layer comprises a learnable query and a pixel updating path; the dual-path feature uses a cross attention mechanism to realize interaction, and a semantic category and an instance mask required by panoramic segmentation are output; the method can meet the adaptability in a complex scene and improve the segmentation quality.

Description

Technical Field

[0001] The present invention relates to the field of image segmentation technology, and in particular to a panoramic segmentation method in complex scenes based on a panoramic segmentation network based on feature fusion and edge guidance. Background Art

[0002] Although existing panoptic segmentation methods have achieved remarkable results, they still have the following problems in complex scenes:

[0003] 1. Static issues with multi-scale feature fusion strategies: Existing methods, such as Panoptic FPN, use a fixed-weight feature fusion mechanism, which makes it difficult to dynamically adjust the feature extraction strategy based on scene complexity. This results in insufficient adaptability to small objects and scale changes. This static fusion mechanism restricts the model's adaptive performance in different scenarios.

[0004] 2. Accuracy limitations of edge feature modeling: Existing decoders do not adequately retain high-frequency edge information, resulting in blurred segmentation boundaries for complex objects (such as power lines and streetlights) and detailed areas (such as vegetation and textures). Edge features are a key factor in segmentation accuracy, and their expressiveness directly affects the visual quality of the segmentation results.

[0005] 3. Imperfect collaborative optimization mechanism for global semantics and local details: When dealing with small-scale but semantically important targets, even the current optimal method, Mask2Former, has an accuracy rate of only 70.2% for traffic sign categories on the Cityscapes dataset, reflecting the obvious shortcomings of existing methods in fine target segmentation.

[0006] Therefore, there is an urgent need for a panoramic segmentation method in complex scenes that can meet the adaptability in complex scenes and improve the segmentation quality. Summary of the Invention

[0007] The purpose of the present invention is to provide a panoramic segmentation method for complex scenes, aiming to solve the technical problems that traditional panoramic segmentation methods are difficult to adapt to complex scenes and have low segmentation accuracy.

[0008] To achieve the above objectives, in a first aspect, the present invention provides a panoptic segmentation method for complex scenes, the steps of which include:

[0009] Inputting the initial image into a pre-built panoramic segmentation network to obtain a panoramic segmentation result of the initial image;

[0010] The panoptic segmentation network includes:

[0011] The backbone network is used to extract the initial features C1, C2, C3 and C4 from the initial image;

[0012] The feature encoding layer includes a cascaded dynamic large kernel (DLK) module and a dynamic feature fusion (DFF) module. The DLK module implements multi-scale feature extraction through cascaded deformable convolution kernels and a two-dimensional attention mechanism. The DFF module achieves adaptive aggregation of cross-level features through geometric alignment and dynamic weight generation. The DLK and DFF modules assist in generating enhanced features E1′-E4′ from the initial features extracted by the backbone network.

[0013] Edge-guided layer: includes edge-guided fusion module (EGF) and feature pyramid. The EGF module fuses the C1 feature of the backbone network with the deep feature E4′ of the encoder to generate edge-enhanced feature F through the edge gradient perception mechanism. The edge-enhanced feature F is downsampled at multiple levels and then channel-joined with E1′-E3′ to construct a four-level feature pyramid F1-F4.

[0014] Dual-path decoding layer: A parallel interactive feature pyramid is processed, including learnable query and pixel update paths. The query path uses the decoder to update query features, and the pixel path uses a convolutional network to extract detail information. A dual-path update strategy is used to update pixel features and query features, and the target category and segmentation mask are predicted in each decoder layer to achieve panoramic segmentation of the image.

[0015] As a further improvement to the above scheme, the DLK module implements multi-scale feature extraction through a deformable convolution kernel cascade attention selection mechanism. The specific steps are as follows:

[0016] First, cascade deformable convolution and use depthwise separable convolution to build a dual-path structure, which are configured as follows:

[0017]

[0018] Secondly, based on dual-dimensional dynamic selection, a channel-space collaborative attention mechanism is proposed. The calculation formula is as follows:

[0019] A s =σ(Conv 7×7 ([MaxPool(F);AvgPool(F)]))

[0020] A c =σ(MLP(GAP(F))+MLP(GMP(F)))

[0021]

[0022] Among them A s and A cRepresent the spatial and channel attention maps respectively, and ⊙ are element-wise multiplication and channel multiplication respectively, σ represents the activation function;

[0023] Finally, residual learning introduces cross-layer connections to enhance gradient propagation. The calculation formula is as follows:

[0024] F out =F in +Conv 1×1 (F′).

[0025] As a further improvement to the above solution, the DFF module implements adaptive aggregation of cross-level features through geometric alignment and dynamic weight generation. The following steps are performed:

[0026] First, geometric alignment is performed, and the high-level features E high Perform deformable upsampling and compare it with the low-level features E low Perform spatial alignment, the calculation formula is as follows:

[0027] E up =DeformConv(E high ,Δp)

[0028] Where Δp is the output of the offset prediction network;

[0029] Next is dynamic weight generation. Fusion weights are generated through global context modeling. The calculation formula is as follows:

[0030]

[0031] in Represents feature concatenation, Conv 1×1 It is a channel compression operation, and the weight matrix W of feature fusion is generated through Softmax normalization;

[0032] Finally, multimodal fusion uses a gating mechanism to integrate features. The calculation formula is as follows:

[0033] E fusion =W·E high +(1-W)·E low

[0034] Among them, E fusion is the final fusion result, W is the weight factor, E high is a high-level feature, E low It is a low-level feature.

[0035] As a further improvement to the above solution, the steps for constructing a four-level feature pyramid are as follows:

[0036] First, the edge features are encoded and the shallow features C1∈R of the main network are converted into h / 4×w / 4×256 With the dynamic multi-scale encoder deep feature E4′∈R h / 32×w / 32×256 Perform feature fusion and construct multi-scale edge representation based on the fused features, as shown below:

[0037]

[0038] Up8 is an 8x upsampling operation. is feature concatenation, Γ(·) is the improved channel attention mechanism, E′ edge Represents the edge features obtained by improving the channel attention mechanism;

[0039] Then, edge supervised learning is used to introduce auxiliary supervisory signals to enhance edge perception. The real edge map is generated by improving the Canny operator, and its calculation formula is as follows:

[0040]

[0041] Among them L edge is the loss function of edge supervision, Up4(E′edge) represents a 4-fold upsampling operation on these edge features to make their size match the real edge map, G edge It is a real edge map generated from the original image by the improved Canny operator, and the generated real edge map is used as the supervision signal; BCE is the binary cross entropy loss function, whose core function is to quantitatively measure the difference between the predicted edge and the real edge. Indicates that the average loss is calculated for N samples or pixels;

[0042] Multi-scale features are then injected to build an edge feature pyramid for cross-level guidance:

[0043]

[0044] in, Indicates that the edge features are downsampled, and the downsampling ratio is 2 k-1 Decide, Represents a series of edge features of different scales generated, forming an edge feature pyramid;

[0045] Finally, the features of each scale are fused with the backbone features through a gating mechanism:

[0046]

[0047] Among them, E′ k is the feature extracted by the backbone network at layer k, is the edge feature of the corresponding level (from the edge feature pyramid), Conv 3×3 (E' k ) performs 3×3 convolution on the backbone features, σ(...) is the sigmoid activation function, mapping the value to between 0 and 1. Represents element-wise multiplication, which is used here to implement the gating mechanism. Represents the feature connection operation, Conv1×1(...) is the final 1×1 convolution, which is used to fuse the connected features.

[0048] As a further improvement to the above solution, the dual-path update strategy includes:

[0049] A pixel feature update unit, configured to jointly process pixel features and query features through a cross-attention mechanism and fuse edge feature tensors during the attention weight calculation stage;

[0050] The query feature update unit is configured to iteratively optimize the query features using the edge-constrained Mask Attention Mechanism, specifically including:

[0051] Generate a dynamic region mask based on the instance mask predicted by the previous layer, perform matrix dot multiplication on it and the edge feature to generate boundary-enhanced spatial attention weights;

[0052] Multiply the spatial attention weight by the self-attention score of the query feature element-wise;

[0053] In the self-attention and feedforward network calculations, edge features are gated and fused with query features, and the update intensity of edge information on the query vector is controlled by learnable parameters;

[0054] The multi-resolution decoding unit utilizes the updated query features and pixel features for decoding and is configured to generate a segmentation mask through progressive resolution recovery.

[0055] As a further improvement to the above scheme, the calculation formula of the improved channel attention mechanism is as follows:

[0056] w c =σ(Conv 1×1 (GAP(E edge )))

[0057]

[0058] where w c is the weight coefficient of the channel feature, E edge The original feature.

[0059] As a further improvement to the above scheme, the overall loss function of the panoptic segmentation network is shown as follows:

[0060] L=L E +L pre

[0061] Among them L E Edge supervision loss formula L in the edge guidance module edge , L pre The loss is generated for the dual-path decoder prediction and is calculated as follows:

[0062]

[0063] Where D represents the number of layers in the Transformer decoder, i = 0 represents the prediction loss of the instance-guided activation query before it enters the Transformer decoder, and denote the binary cross entropy loss and dice loss of the segmentation mask, respectively, and is the cross entropy loss for object classification with a “no object” weight of 0.1. ce ,λ dice and λ cls is a hyperparameter that balances the three losses.

[0064] In a second aspect, the present invention also provides a panoramic segmentation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the panoramic segmentation method in a complex scene as described in the first aspect are implemented.

[0065] In a third aspect, the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a panoramic segmentation method in a complex scene as described in the first aspect.

[0066] Since the present invention adopts the above technical solution, the beneficial effects of this application are:

[0067] The present invention provides a panoramic segmentation method for complex scenes. Through the collaborative design of a cascaded dynamic large kernel module (DLK) and a dynamic feature fusion module (DFF), a spatial-channel dual-dimensional adaptive selection mechanism is constructed. The DLK module uses a cascaded deformable convolution kernel to replace the traditional large kernel convolution, which significantly reduces the computational complexity while expanding the effective receptive field. The convolution kernel offset is dynamically adjusted to achieve adaptive feature capture of the target size. The DFF module introduces a dynamic weight generation algorithm under geometric alignment constraints to solve the spatial misalignment problem during cross-level feature fusion, thereby reducing the multi-scale feature reconstruction error. The combination of the two improves the small target feature retention rate, achieving a small target recognition accuracy improvement of up to 6.8% on the Cityscapes dataset.

[0068] The Edge-Guided Fusion module (EGF) innovatively integrates shallow features C1 from the backbone network with deep semantic features E4' in the gradient domain. It constructs an edge gradient-aware branch using learnable operators to generate edge-enhanced features F with explicit geometric constraints. This module employs a cross-branch attention mechanism to align and compensate edge features with semantic representations. Compared to traditional methods, this module achieves improved Boundary Intersection over Union (BoundaryIoU) performance, effectively alleviating issues like blurred and broken object edges in complex scenes. It particularly improves the segmentation accuracy of high-frequency details such as building outlines and pedestrian limb edges.

[0069] The dual-path decoding layer captures global semantic associations and local detail features through parallel processing of a learnable query path and a pixel update path. A cross-attention mechanism establishes feature mappings between the two paths, enabling local feature correction guided by global context. This improves instance mask contour overlap and reduces pixel-level classification ambiguity. It also enhances instance differentiation while maintaining high-resolution detail, improving instance ID maintenance accuracy in densely occluded scenes.

[0070] The present invention achieves technological innovation in the entire process of feature extraction, fusion, and decoding: with a slight increase in the computational complexity of the backbone network, the panoramic quality (PQ) index has been improved on both the Cityscapes and COCO datasets; for small targets with a scale less than 64×64 pixels, the recognition recall rate can be improved; experiments have shown that this method has significant technical advantages in complex scenarios such as autonomous driving and remote sensing image analysis, and can effectively solve real-world application problems such as multi-target overlap, illumination changes, and edge blur. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0072] Figure 1 This is a schematic diagram of the network architecture of a panoramic segmentation network PSM-FFEG disclosed in the present invention;

[0073] Figure 2 Schematic diagram of the dynamic large convolution kernel disclosed in the present invention;

[0074] Figure 3 It is a schematic diagram of dynamic feature fusion disclosed in the present invention;

[0075] Figure 4 is a schematic diagram of the edge guide mold layer disclosed in the present invention;

[0076] Figure 5 This is the visualization result of the panoramic segmentation network disclosed in this invention on the Cityscapes dataset.

[0077] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0078] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0079] It should be noted that all directional indications (such as up, down, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0080] In addition, the terms "first," "second," and so on, used in this disclosure are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include at least one of these features.

[0081] Moreover, the technical solutions between the various embodiments of the present invention may be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0082] Example 1

[0083] See also Figure 1 The present invention provides a panoptic segmentation method for complex scenes, the steps of which include:

[0084] Inputting the initial image into a pre-built panoramic segmentation network to obtain a panoramic segmentation result of the initial image;

[0085] The panoptic segmentation network includes:

[0086] The backbone network is used to extract the initial features C1, C2, C3 and C4 from the initial image. Specifically, in this embodiment, ResNet50 is used as the basic feature extractor, and the multi-level features C1, C2, C3, and C4 output by stage1-stage4 are retained, with corresponding resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively. The C1 feature will be used as the high-resolution input of the edge guidance module.

[0087] The feature encoding layer performs feature optimization through the cascaded dynamic large kernel module (DLK) and dynamic feature fusion module (DFF);

[0088] The DLK module uses a cascade structure of two levels of deformable convolution kernels (5×5→7×7), with each convolution kernel followed by a channel-spatial dual-dimensional attention submodule. In the channel dimension, the global correlation of each feature channel is calculated to generate a channel attention weight matrix; in the spatial dimension, the spatial attention mask is generated using deformable convolution offsets, and feature recalibration is achieved through matrix dot multiplication. The DFF module achieves adaptive aggregation of cross-level features through geometric alignment and dynamic weight generation. The DLK and DFF modules assist in generating enhanced features E1′-E4′ from the initial features extracted by the backbone network.

[0089] Edge-guided layer: includes edge-guided fusion module (EGF) and feature pyramid. Specifically, the high-resolution feature C1 (containing rich edge details) output by the backbone network and the deep semantic feature E4' output by the encoding layer are input into the edge-guided fusion module (EGF):

[0090] The improved Canny operator is used to calculate the gradient amplitude of the C1 feature to generate the edge energy map;

[0091] The spatial correspondence between the E4' feature and the edge energy map is established through the cross-layer attention mechanism to generate an edge-aware attention weight matrix. The C1 and E4' features are weighted and fused using the matrix to output the edge-enhanced feature F. The edge-enhanced feature F is then subjected to 3×3 convolution and bilinear downsampling to generate a four-level feature pyramid F1-F4 with a resolution coverage of 1 / 4-1 / 32.

[0092] Dual-path decoding layer: includes a parallel query path (global semantic path) and a pixel path (local detail path);

[0093] The query path uses a learnable positional encoding vector as the initial query and iteratively updates the query features through a 6-layer Transformer decoder. Each decoder layer contains a multi-head cross-attention mechanism to calculate the global correlation between the query vector and the F4 features.

[0094] The pixel path uses cascaded 3×3 depth-wise separable convolutions to perform multi-level feature refinement on the F1-F3 features, with each level of convolution followed by a dynamic upsampling operation;

[0095] A dual-path update strategy is used to update pixel features and query features, and target categories and segmentation masks are predicted in each decoder layer to achieve panoramic segmentation of the image.

[0096] The present invention adopts a dynamic multi-scale feature fusion mechanism, which realizes the adaptive capture and fusion of contextual information through the cascade of deformable convolution kernels and the channel-space dual-dimensional attention mechanism. This mechanism breaks through the limitations of traditional fixed-weight fusion and can adaptively adjust the contribution weights of features of different scales according to the complexity of the input image, significantly improving the segmentation performance of small and multi-scale objects.

[0097] The edge feature guidance module uses a bidirectional enhancement path that combines pixel semantic features with edge gradient information. Through explicit supervision of edge information, the model is guided to focus on the fine-grained representation of object boundaries. This not only enhances the model's perception of high-frequency details but also establishes an effective bridge between edge features and semantic representation.

[0098] The dual-path decoding layer uses a cross-attention mechanism to achieve coordinated optimization of global semantic understanding and local detail features. The global path is responsible for capturing long-range dependencies and scene semantic understanding, while the local path focuses on preserving high-resolution features and fine boundary information. The two paths achieve complementary enhancement through an interactive attention mechanism.

[0099] The present invention achieves technological innovation in the entire process of feature extraction, fusion, and decoding. With a slight increase in the computational complexity of the backbone network, the panoramic quality (PQ) index has been improved on both the Cityscapes and COCO datasets. The recognition recall rate is also improved for small targets with a scale of less than 64×64 pixels. Experiments have shown that this method has significant technical advantages in complex scenarios such as autonomous driving and remote sensing image analysis, and can effectively solve real-world application problems such as multi-target overlap, illumination changes, and edge blur.

[0100] As a preferred embodiment, see Figure 2 , the DLK module realizes multi-scale feature extraction through the deformable convolution kernel cascade attention selection mechanism. The specific steps are as follows:

[0101] Construct a deformable convolution kernel cascade structure:

[0102] Preferably, a dual-path structure is constructed using depthwise separable convolution, which is configured as follows:

[0103]

[0104] The above configuration achieves the cumulative effect of receptive field, which is equivalent to obtaining a 23×23 large core effect. The calculation formula is as follows:

[0105] R i =R i-1 +(k i -1)×j i

[0106] where R i-1 is the effective receptive field of the i-1th layer, k i is the convolution kernel size of layer i, j i is the jump distance (stride or dilation rate) of the i-th layer. Since the first layer uses a 5×5 convolution kernel with dilation = 1 and the initial R0 = 1, then R1 = 1 + (5-1) × 1 = 5. The second layer uses a 7×7 convolution kernel with dilation = 3, the effective kernel size = 7×3 = 21 (because dilation = 3), and R2 = 5 + (19-1) × 1 = 23. Cascading these two convolution layers can achieve an effective receptive field size similar to 23×23. This cascading strategy avoids the huge computational overhead of directly using a larger convolution kernel.

[0107] Secondly, based on dual-dimensional dynamic selection, a channel-space collaborative attention mechanism is proposed;

[0108] Spatial attention branch:

[0109] Parallel maximum pooling (MaxPool) and average pooling (AvgPool) operations are used to perform feature fusion through a 7×7 convolutional layer. The activation function σ is preferably Sigmoid. The specific calculation formula is as follows:

[0110] A s =σ(Conv 7×7 ([MaxPool(F);AvgPool(F)]))

[0111] Channel attention branch:

[0112] Global average pooling (GAP) and global maximum pooling (GMP) are applied, and a dual-path multi-layer perceptron (MLP) is used for channel interaction. The specific calculation formula is as follows:

[0113] A c =σ(MLP(GAP(F))+MLP(GMP(F)))

[0114] Feature Recalibration: Execution Operation,

[0115] Among them A s and A c Represent the spatial and channel attention maps respectively, and ⊙ are element-wise multiplication and channel multiplication respectively, σ represents the activation function;

[0116] Residual connection optimization:

[0117] Introducing cross-layer residual connection structure:

[0118] Feature dimension matching through 1×1 convolution;

[0119] Perform feature superposition F out =F in +Conv 1×1 (F′);

[0120] Preferably, the initial weight of the residual branch is set to 0 to ensure the stability of the network in the initial stage.

[0121] Through the progressive cascading design of deformable convolution kernels, an equivalent large kernel receptive field of 23×23 is achieved while maintaining the actual kernel sizes of 5×5 and 7×7. Compared to directly using 23×23 conventional convolutions, this reduces computational complexity while enhancing adaptability to irregular features through dynamic adjustment of the expansion rate. An innovative dual-dimensional attention mechanism is employed: the spatial attention branch employs a dual-mode pooling strategy, with maximum pooling capturing salient features and average pooling preserving background information. A 7×7 convolution achieves local context awareness. The channel attention branch utilizes GAP / GMP dual-path aggregation, coupled with the nonlinear mapping of the MLP, to effectively identify complex relationships between channels. The synergistic feature recalibration enhances the response strength of important features.

[0122] As a preferred embodiment, see Figure 3 The steps of the DFF module to achieve adaptive aggregation of cross-level features through geometric alignment and dynamic weight generation are as follows:

[0123] Geometric alignment processing:

[0124] Preferably, deformable convolution is used to achieve spatial alignment of cross-level features, specifically including:

[0125] a) Construct an offset prediction network consisting of two layers of 3×3 convolution kernels, with LeakyReLU as the preferred activation function and 2N output channels (N is the number of deformable convolution sampling points);

[0126] b) For high-level features E high Perform a deformable upsampling operation, the calculation formula is:

[0127] E up =DeformConv(E high ,Δp)

[0128] where Δp∈R^ (h×w×2N) is the output of the offset prediction network, h and w are the feature map height and width respectively;

[0129] c) Compensation with low-level features E is achieved through bilinear interpolation low Pixel-level alignment, the alignment error is controlled within 0.5 pixels;

[0130] Dynamic weight generation mechanism:

[0131] Implementing global context-aware fusion weight generation includes:

[0132] a) Feature splicing operation: The upsampled high-level features E up With low-level features E low Splicing along the channel dimension, that is, E up ⊕E low ∈R^(h×w×(C_high+C_low))

[0133] b) Channel compression processing: 1×1 convolution is used to compress the number of splicing feature channels to 1, that is, Conv1×1 (E up ⊕E low )∈R^ (h×w×1)

[0134] c) Weight normalization: Generate a spatially adaptive weight matrix through the Softmax function. The calculation formula is:

[0135] W=Softmax(Conv 1×1 (E up ⊕E low ))

[0136] Where ⊕ represents feature concatenation, Conv 1×1 For channel compression operation, through Softmax normalization, the weight matrix of feature fusion W∈[0,1]^ (h×w) , the sum of the weight values of different spatial positions is 1;

[0137] Gated Fusion Execution:

[0138] The specific process of implementing multimodal feature fusion includes:

[0139] a) Dynamically allocate the contribution of high- and low-level features according to the weight matrix. The weighted fusion calculation formula is:

[0140]

[0141] Among them, E fusion is the final fusion result, W is the weight factor, E high is a high-level feature, E low It is a low-level feature; Represents the element-wise multiplication corresponding to the spatial position;

[0142] b) Gradient protection mechanism: Gradient truncation is implemented on the weight matrix W to prevent gradient saturation during back propagation;

[0143] c) Residual connection optimization scheme: After fusion, feature E fusion 10% of the original low-level features are superimposed to enhance feature diversity;

[0144] By adopting deformable convolution upsampling combined with an offset prediction network, the alignment error is effectively reduced compared to the traditional bilinear interpolation method. The dynamically adjusted sampling point position effectively compensates for the positional deviation between high-level feature semantic information and low-level feature detail information, and the pixel alignment accuracy in the target edge area is also effectively improved.

[0145] As a preferred embodiment, see Figure 4 , the steps to construct a four-level feature pyramid are as follows:

[0146] Cross-level edge feature encoding:

[0147] a) Dynamic feature fusion

[0148] Extract the shallow features of the backbone network C1∈R h / 4×w / 4×256 With the dynamic multi-scale encoder deep feature E4′∈R h / 32×w / 32×256 ;

[0149] Perform 8x bilinear upsampling (Up8) on E4′ to align its spatial resolution with C1;

[0150] Concatenate features along the channel dimension: Concat(Up8(E4′),C1)∈R^ (h / 4×w / 4×512)

[0151] b) Improved channel attention mechanism Γ(·)

[0152] Use 1×1 depth-wise separable convolution to process the splicing features and calculate the channel attention weight:

[0153] w c =σ(Conv 1×1 (GAP(E edge )))

[0154]

[0155] Where wc is the weight coefficient of channel feature, E edge is the original feature, GAP is the global average pooling

[0156] Output features:

[0157] E′ edge Represents the edge features obtained by improving the channel attention mechanism;

[0158] Supervised Learning Enhancement at the Edge:

[0159] a) Improve the Canny operator to generate a real edge map

[0160] Adaptive Gaussian kernel size: σ = 0.3 × min(h,w) / 256

[0161] Dynamic dual threshold calculation: high threshold T_high = 0.2 × max(Gradient), low threshold T_low = 0.5 × T_high;

[0162] Generate a binary edge graph G edge∈{0,1}^(h×w)

[0163] b) Assisted supervision training

[0164] For E′ edge Perform 4x upsampling (Up4) to match G_edge resolution

[0165] The marginal loss function is as follows:

[0166]

[0167] Among them L edge is the loss function of edge supervision, Up4(E′edge) represents a 4-fold upsampling operation on these edge features to make their size match the real edge map, G edge It is a true edge map generated from the original image by the improved Canny operator, and the generated true edge map is used as the supervision signal; BCE is the binary cross entropy loss function, and its core function is to quantitatively measure the difference between the predicted edge and the true edge. Indicates that the average loss is calculated for N samples or pixels;

[0168] Multi-scale features are then injected to build an edge feature pyramid for cross-level guidance:

[0169]

[0170] in, Indicates that the edge features are downsampled, and the downsampling ratio is 2 k-1 Decide, Represents a series of edge features of different scales generated, forming an edge feature pyramid;

[0171] Finally, the features of each scale are fused with the backbone features through a gating mechanism:

[0172]

[0173] Among them, E′ k is the feature extracted by the backbone network at layer k, E k edge is the edge feature of the corresponding level (from the edge feature pyramid), Conv 3×3 (E' k ) performs 3×3 convolution on the backbone features, σ(...) is the sigmoid activation function, mapping the value to between 0 and 1. Represents element-wise multiplication, which is used here to implement the gating mechanism. Represents the feature connection operation, Conv1×1(...) is the final 1×1 convolution, which is used to fuse the connected features;

[0174] The improved channel attention mechanism enhances the response strength of edge features; the high-quality supervision signal generated by the dynamic Canny operator reduces the edge positioning error to 1.2 pixels; the multi-scale edge pyramid also effectively improves the mIoU indicator in the target contour area; the cross-layer feature calibration mechanism can improve the recall rate of small target detection, reduce the jagged rate of image segmentation edges, and reduce key point positioning errors.

[0175] As a preferred embodiment, the dual-path update strategy includes:

[0176] A pixel feature update unit is configured to jointly process pixel features and query features through a cross-attention mechanism and fuse edge feature tensors during the attention weight calculation phase. Specifically, this includes: performing feature enhancement on edge features and original pixel features through differentiable concatenation or channel-wise weighted addition;

[0177] The enhanced features are used to generate a key-value pair matrix, which is then cross-attended with the query features, so that the pixel features can simultaneously aggregate global semantic information and edge detail features.

[0178] The query feature update unit is configured to iteratively optimize the query features using the edge-constrained Mask Attention Mechanism, specifically including:

[0179] Generate a dynamic region mask based on the instance mask predicted by the previous layer, perform matrix dot multiplication on it and the edge feature to generate boundary-enhanced spatial attention weights;

[0180] Multiplying the spatial attention weight by the self-attention score of the query feature element-wise to suppress noise interference in non-target areas;

[0181] In the self-attention and feedforward network calculations, edge features are gated and fused with query features, and the update intensity of edge information on the query vector is controlled by learnable parameters;

[0182] A multi-resolution decoding unit is configured to generate a segmentation mask by progressive resolution restoration, specifically comprising:

[0183] The updated query features are upsampled step by step through the transposed convolution operation and concatenated with the deconvolution output of the corresponding resolution in the pixel path in the channel dimension.

[0184] At each resolution stage, the edge features are spatially aligned with the spliced dual-path features, and the boundary-sensitive feature responses are extracted through a 3×3 convolution kernel;

[0185] At the final resolution layer, a 1×1 convolution kernel is used to output a semantic category probability map and an instance mask, where the boundary area of the instance mask is enhanced by edge features.

[0186] By integrating edge features into the cross-attention mechanism of pixel feature updating, the global semantic feature aggregation process can synchronously perceive the gradient changes of object boundaries, effectively solving the problem of blurred or jagged mask boundaries caused by ignoring edge details in traditional methods, and significantly improving the segmentation accuracy of object contours in complex scenes; the edge-constrained Mask attention mechanism is introduced in the query feature update stage, and spatial attention weights are generated by the dot product operation of dynamic region masks and edge features to suppress noise interference in non-target areas, enhance the positioning sensitivity of object boundaries in self-attention calculations, and thus improve the instance segmentation stability in challenging scenarios such as occlusion and deformation; a progressive resolution recovery strategy is adopted in the decoding process, and the semantic context of the query feature and the spatial details of the pixel path are feature aligned and channel-stitched at different resolution levels, combined with the convolution operation guided by edge features to achieve multi-scale collaborative optimization of semantic information and geometric structure, and generate high-fidelity panoramic segmentation results.

[0187] As a preferred embodiment, in the training of deep learning networks, the loss function can guide and supervise the network model, directly affecting the model's convergence speed, generalization ability, and final performance. The loss function of this panoramic segmentation network consists of two parts, one for edge-guided supervision and the other for dual-path decoder prediction generation. The former will learn feature map edge information, and the latter will continuously update the generated prediction. The overall loss function of the panoramic segmentation network is shown in the following formula:

[0188] L=L E +L pre

[0189] Among them L E Edge supervision loss formula L in the edge guidance module edge , L pre The loss is generated for the dual-path decoder prediction and is calculated as follows:

[0190]

[0191] Where D represents the number of layers in the Transformer decoder, i = 0 represents the prediction loss of the instance-guided activation query before it enters the Transformer decoder, and denote the binary cross entropy loss and dice loss of the segmentation mask, respectively, and is the cross entropy loss for object classification with a “no object” weight of 0.1. ce ,λ dice and λ clsis a hyperparameter that balances the three losses.

[0192] In order to further illustrate the effectiveness of the panoramic segmentation method provided by the present invention in complex scenes, under the same experimental environment and configuration, the method PSM-FFEG provided by the present invention is compared with various panoramic segmentation methods with Res50 as the backbone network, and the common and representative indicators PQ, PQ th and PQ st . The panoramic segmentation results on the Cityscapes and MSCOCO datasets are shown in Tables 1 and 2 respectively; the results show that the panoramic segmentation method based on feature fusion and edge guidance proposed in the present invention outperforms all other methods under the same Res50 backbone network configuration. Specifically, Res50 is selected as the backbone network of the entire model, the parameter batch size is set to 16, the learning rate is 0.0001, the number of training iterations is 90,000, and the AdamW optimizer is used. The experiment is carried out on a single GPU. The GPU model used is NVIDIA Tesla A100, the video memory is 40GB, the deep learning framework is Pytorch 2.1.0, the CUDA version is 12.1, and the loss function adopts L=L E +L p , loss weight λ ce ,λ dice and λ cls Set to 2.0, 5.0 and 5.0 respectively;

[0193] On the Cityscapes dataset, the method PSM-FFEG provided by the present invention performs well in PQ, PQ th and PQ st The indicators reached 64.7%, 59.4% and 69.1% respectively, which are 2.6, 4.6 and 1.8 percentage points higher than the baseline model Mask2Former. th and PQ st They are 54.3%, 61.4%, and 44.3% respectively, which are 2.4, 3.7, and 1.3 percentage points higher than the baseline model Mask2Former.

[0194] Table 1 Comparison of panoramic segmentation results on the Cityscapesval dataset

[0195]

[0196] Table 2 Comparison of panoramic segmentation results on COCO dataset

[0197]

[0198] In order to more intuitively demonstrate the panoramic segmentation effect of the proposed method, four outdoor complex scenes are randomly selected from the Cityscapes panoramic segmentation dataset to generate panoramic segmentation, which is presented through intuitive and effective visualization and compared with the panoramic segmentation results generated by the benchmark model Mask2Former method. The results are shown in the figure. Figure 5 As shown in the figure (from left to right: input image, real annotated image, Mask2Former generated image, and image generated by this method). The key areas are marked with red boxes. As can be seen from the figure, the method proposed in the present invention is more accurate in the segmentation of edge details and target boundaries, especially in the segmentation of small targets and occluded targets. This method has a significant improvement over Mask2Former. These results further verify the effectiveness of the method proposed in the present invention in feature fusion and edge guidance, and can better adapt to panoramic segmentation tasks in complex scenes.

[0199] Example 2

[0200] The present invention further provides a panoptic segmentation device, comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute some or all of the steps in embodiment 1;

[0201] The processor may include one or more processing units, for example, an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0202] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on instruction operation codes and timing signals to complete the control of instruction fetching and execution.

[0203] The processor may also include a memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or is reusing. If the processor needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0204] Example 3

[0205] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed, the device where the storage medium is located is controlled to execute part or all of the steps in Example 1.

[0206] The computer-readable storage medium may include a high-speed RAM memory, and may also include a non-volatile memory (nonvolatile memory), such as at least one disk storage. It is understood that the storage medium can be a random access memory (RAM), a magnetic disk, a hard disk, a solid state drive (SSD), or a non-volatile memory, etc., which can store program code, and other machine-readable media.

[0207] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods or storage media. Therefore, embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. All equivalent structural transformations made based on the contents of the present invention's description and drawings, or directly or indirectly applied to other related technical fields, are within the scope of patent protection of the present invention.

Claims

1. A panoptic segmentation method for complex scenes, characterized in that: The steps include: Inputting the initial image into a pre-built panoramic segmentation network to obtain a panoramic segmentation result of the initial image; The panoptic segmentation network includes: The backbone network is used to extract the initial features C1, C2, C3 and C4 from the initial image; The feature encoding layer includes a cascaded DLK module and a DFF module. The DLK module implements multi-scale feature extraction through cascaded deformable convolution kernels and a two-dimensional attention mechanism. The DFF module implements adaptive aggregation of cross-level features through geometric alignment and dynamic weight generation. The DLK and DFF modules assist in generating enhanced features E1′-E4′ from the initial features extracted by the backbone network. Edge-guided layer: It includes an edge-guided fusion module and a feature pyramid. The EGF module fuses the C1 feature of the backbone network with the deep feature E4′ of the encoder to generate edge-enhanced features F through the edge gradient perception mechanism. The edge-enhanced features F are downsampled at multiple levels and then channel-joined with E1′-E3′ to construct a four-level feature pyramid F1-F4. Dual-path decoding layer: A parallel interactive feature pyramid is processed, including learnable query and pixel update paths. The query path uses the decoder to update query features, and the pixel path uses a convolutional network to extract detail information. A dual-path update strategy is used to update pixel features and query features, and the target category and segmentation mask are predicted in each decoder layer to achieve panoramic segmentation of the image.

2. The method for panoptic segmentation in complex scenes according to claim 1, characterized in that: The DLK module implements multi-scale feature extraction through a deformable convolution kernel cascade attention selection mechanism. The specific steps are as follows: First, cascade deformable convolution and use depthwise separable convolution to build a dual-path structure, which are configured as follows: Secondly, based on dual-dimensional dynamic selection, a channel-space collaborative attention mechanism is proposed. The calculation formula is as follows: A s =σ(Conv 7×7 ([MaxPool(F);AvgPool(F)])) A c nσ(MLP(GAP(F))+MLP(GMP(F))) Among them A s and A c Represent the spatial and channel attention maps respectively, and ⊙ are element-wise multiplication and channel multiplication respectively, σ represents the activation function; Finally, residual learning is used to introduce cross-layer connections to enhance gradient propagation. The calculation formula is as follows: F out =F in +Conv 1×1 (F′)。 3. The method for panoptic segmentation in complex scenes according to claim 1 or 2, characterized in that: The steps of the DFF module to achieve adaptive aggregation of cross-level features through geometric alignment and dynamic weight generation are as follows: First, geometric alignment is performed, and the high-level features E high Perform deformable upsampling and compare it with the low-level features E low Perform spatial alignment, the calculation formula is as follows: E up =DeformConv(E high ,Δp) Where Δp is the output of the offset prediction network; Then, dynamic weight generation is performed and fusion weight is generated through global context modeling. The calculation formula is as follows: in Represents feature concatenation, Conv 1×1 It is a channel compression operation, and the weight matrix W of feature fusion is generated through Softmax normalization; Finally, multimodal fusion uses a gating mechanism to integrate features. The calculation formula is as follows: HAVE BEEN fusion =W·E high +(1-W)·E low Among them, E fusion is the final fusion result, W is the weight factor, E high is a high-level feature, E low It is a low-level feature.

4. The method for panoptic segmentation in complex scenes according to claim 1 or 2, characterized in that: The steps to construct a four-level feature pyramid are as follows: First, the edge features are encoded, and the shallow features C1 of the backbone network are fused with the deep features E4′ of the dynamic multi-scale encoder. The multi-scale edge representation is constructed based on the fused features, as shown below: Up8 is an 8x upsampling operation. is feature concatenation, Γ(·) is the improved channel attention mechanism, E′ edge Represents the edge features obtained by improving the channel attention mechanism; Then, edge supervised learning is used to introduce auxiliary supervisory signals to enhance edge perception. The real edge map is generated by improving the Canny operator, and its calculation formula is as follows: Among them L edge is the loss function of edge supervision, Up4(E′edge) represents a 4-fold upsampling operation on these edge features, G edge It is a true edge map generated from the original image by the improved Canny operator, and the generated true edge map is used as the supervision signal; BCE is the binary cross entropy loss function; Multi-scale features are then injected to build an edge feature pyramid for cross-level guidance: in, Indicates downsampling of the obtained edge features. Represents a series of edge features of different scales generated, forming an edge feature pyramid; Finally, the features of each scale are fused with the backbone features through a gating mechanism: Among them, E′ k is the feature extracted by the backbone network at layer k, E k edge is the edge feature of the corresponding level, Conv 3×3 (E' k ) Perform 3×3 convolution on the backbone features. Represents element-wise multiplication, used to implement the gating mechanism, Represents a feature connection operation.

5. The method for panoptic segmentation in complex scenes according to claim 4, characterized in that: The calculation formula of the improved channel attention mechanism is as follows: w c =σ(Conv 1×1 (GAP(E edge ))) where w c is the weight coefficient of the channel feature, E edge The original feature.

6. The method for panoptic segmentation in complex scenes according to claim 1 or 2, characterized in that: The dual-path update strategy includes: A pixel feature update unit, configured to jointly process pixel features and query features through a cross-attention mechanism and fuse edge feature tensors during the attention weight calculation stage; The query feature update unit is configured to iteratively optimize the query features using the edge-constrained Mask Attention Mechanism, specifically including: Generate a dynamic region mask based on the instance mask predicted by the previous layer, perform matrix dot multiplication on it and the edge feature to generate boundary-enhanced spatial attention weights; Multiply the spatial attention weight by the self-attention score of the query feature element-wise; In the self-attention and feedforward network calculations, edge features are gated and fused with query features, and the update intensity of edge information on the query vector is controlled by learnable parameters; The multi-resolution decoding unit utilizes the updated query features and pixel features for decoding and is configured to generate a segmentation mask through progressive resolution recovery.

7. The method for panoptic segmentation in complex scenes according to claim 4, characterized in that: The overall loss function of the panoptic segmentation network is shown as follows: L=L E +L pre Among them L E Edge supervision loss formula L in the edge guidance module edge , L pre Generate loss for dual-path decoder prediction.

8. The method for panoptic segmentation in complex scenes according to claim 7, characterized in that: L pre The calculation formula is as follows: Where D represents the number of layers in the Transformer decoder, i = 0 represents the prediction loss of the instance-guided activation query before it enters the Transformer decoder, and denote the binary cross entropy loss and dice loss of the segmentation mask, respectively, and is the cross entropy loss for object classification with a "no object" weight of 0.

1. ce ,λ dice and λ cls is a hyperparameter that balances the three losses.

9. A panoptic segmentation device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method implements the steps of a panoramic segmentation method in a complex scene as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the panoramic segmentation method in a complex scene according to any one of claims 1 to 8 are implemented.

Images