Picture segmentation method and system based on dynamic projection and double-branch attention

Abstract

The application provides a picture segmentation method and system based on dynamic projection and double-branch attention, which comprises the following steps: performing layer normalization on an input token sequence; generating a conditional shared projection matrix based on the normalized features to obtain a shared query matrix and a key matrix, constructing a value projection matrix based on a cross-layer shared base matrix and an incremental matrix, and projecting the value matrix according to the token significance; performing parallel calculation on the fine branch and the efficient branch to output and introduce a branch interaction mechanism; learning the token level and the feature dimension level fusion weight respectively to construct a double-dimension fusion weight matrix for weighted fusion; and performing feature decoding and segmentation head adaptation on the fusion output to generate a segmentation prediction result. The application can improve the segmentation accuracy and reasoning efficiency, and is especially suitable for small target fine segmentation scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to an image segmentation method and system based on dynamic projection and bi-branch attention. Background Technology

[0002] Image segmentation, a core foundational task in computer vision, aims to achieve pixel-level classification of images, accurately distinguishing between targets and background, and the boundaries between different targets, thus supporting subsequent tasks such as target recognition and scene understanding. In recent years, with the development of deep learning technology, the Visual Transformer (ViT) model based on the Transformer architecture, with its powerful global feature capture capabilities, has gradually replaced traditional convolutional neural networks as a research hotspot in image segmentation. It effectively addresses the shortcomings of convolutional neural networks in capturing long-distance feature dependencies, significantly improving segmentation accuracy in complex scenes.

[0003] However, existing ViT image segmentation models still have the following technical problems in practical applications: The projection parameters are redundant and lack adaptive adjustment capabilities. Traditional ViT models use a fixed projection method to map features to image tokens. The query, key, and value projection matrices are set independently across layers and heads, and the parameters are not reusable, resulting in an excessive number of model parameters and low inference efficiency. Furthermore, the fixed projection method cannot adaptively adjust the projection precision based on the saliency of the input image content. Using the same projection computation for tokens of different importance in the image wastes computational resources and is difficult to adapt to resource-constrained edge deployment scenarios.

[0004] Attention mechanisms struggle to balance precision and computational efficiency. Existing ViT segmentation models often employ single-branch attention mechanisms. While global self-attention can capture long-distance dependencies, its computational complexity quadratically increases with sequence length, resulting in high computational overhead. Local sparse attention, though efficient, is prone to losing global contextual information, making it difficult to simultaneously meet the dual requirements of precise segmentation of small targets and efficient inference. This is particularly problematic in scenarios like power line inspection and remote sensing image analysis, where there are both small targets requiring precise localization, such as broken conductor strands or minor insulator damage, and large-scale equipment areas requiring global contextual modeling. Single-branch attention structures simply cannot satisfy both needs.

[0005] The feature fusion methods are simplistic and lack dynamic adaptability. Existing models often employ simple concatenation or fixed weighting for multi-head attention output fusion, failing to differentiate fusion based on the importance of features from different branches and levels. Furthermore, existing models lack utilization of historical layer information during feature fusion, with independent fusion strategies between layers leading to drastic fluctuations in fusion weights during training, thus affecting model convergence stability.

[0006] In the process of restoring pixel-level segmentation results from token features: existing methods mostly use a simple combination of upsampling and convolution, which fails to make full use of the token-level energy and centrality information learned by the self-attention module to enhance segmentation details, resulting in problems such as blurred edges of small targets and interference from background noise.

[0007] In summary, existing ViT image segmentation methods have many shortcomings in terms of projection efficiency, attention mechanism, feature fusion, and decoding enhancement. There is an urgent need for an image segmentation method that can improve inference efficiency while ensuring segmentation accuracy and adapt to the needs of multi-scale target segmentation. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide an image segmentation method and system based on dynamic projection and bi-branch attention, so as to solve the technical problems of existing ViT image segmentation models, such as redundant projection parameters, difficulty in balancing precision and computational efficiency in attention mechanism, single feature fusion method, and insufficient detail enhancement in the decoding stage.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: An image segmentation method based on dynamic projection and bi-branch attention includes the following steps: S1. Perform layer normalization on the token sequence after the input image is segmented to obtain normalized features; S2. Based on the normalized features, perform dynamic adaptive reuse projection calculation; specifically including: The normalized features are used to generate a conditional shared projection matrix through learnable parameters, and the normalized features are projected using the conditional shared projection matrix to obtain a shared query matrix and a key matrix. The value projection matrix of the current layer is constructed by summing the base projection matrix shared across layers and the lightweight incremental projection matrix of the current layer. Based on the significance of each token in the normalized features, the high-significance tokens and low-significance tokens are projected using the full-dimensional and dimensionality-reduced value projection matrices, respectively, to obtain the value matrix. S3. Based on the query matrix, key matrix, and value matrix, perform parallel computation of fine-grained branch output and efficient branch output, and introduce a branch interaction mechanism; specifically including: In the fine branch: residual correction is performed based on the attention matrices of the current layer and the previous layer, and the corrected attention matrix is ​​nonnegated, adaptively normalized and adaptively sparsified, and then multi-hop cumulative calculation is performed with the value matrix to obtain the fine branch output; In the efficient branch: the robustness energy of each token is calculated based on the query matrix, the number of multicenters is adaptively determined according to the variance of the robustness energy to construct a multicenter query vector, the multicenter query vector is matched with the key matrix to obtain the centrality weight of each token, and the value matrix is ​​weighted and aggregated using the centrality weight to obtain the efficient branch output; The centrality weights obtained from the efficient branch calculation are used to modulate the fine branch output, and the mean of the fine branch output is used to correct the efficient branch output. S4. Perform multi-dimensional dynamic adaptive feature fusion between the fine-grained branch output and the efficient branch output; specifically including: We learn the token-level fusion weights and the feature-dimension-level fusion weights separately, and then multiply the two to construct a two-dimensional fusion weight matrix. The fine-grained branch output and the efficient branch output are weighted and fused using the dual-dimensional fusion weight matrix to obtain a comprehensive attention output. S5. Perform feature decoding and segmentation head adaptation on the comprehensive attention output to generate pixel-level segmentation prediction results.

[0010] Further, in step S2: the generation of the conditional shared projection matrix specifically involves: passing the normalized features sequentially through a learnable first weight matrix and a Sigmoid activation function, and then multiplying them by a preset query base projection matrix to obtain the conditional shared projection matrix; in step S2: the step of performing graded projection based on token significance specifically involves: calculating the L2 norm of each token in the normalized features as its energy value, identifying tokens with energy values ​​greater than a learnable threshold as high-significance tokens, and projecting them using the full-dimensional current-layer value projection matrix; identifying tokens with energy values ​​not greater than the learnable threshold as low-significance tokens, and projecting them using the dimensionality-reduced current-layer value projection matrix, and then concatenating the projection results and upsampling them to a unified dimension to obtain the value matrix.

[0011] Further, the multi-hop cumulative calculation in step S3 specifically includes: calculating the cosine similarity between the attention output of the current step and the attention output of the previous step; if the cosine similarity is less than a preset learnable similarity threshold, then continue to calculate the attention output of the next step and perform accumulation; if the cosine similarity is greater than or equal to the learnable similarity threshold, then terminate the multi-hop calculation and use the accumulated result as the fine branch output.

[0012] Further, the construction of the multi-center query vector in step S3 specifically includes: performing L1 normalization on the robustness energy vector to obtain the soft importance weight of each token; calculating the variance of the robustness energy vector, and determining the number of multicenters based on the relationship between the variance and a preset variance threshold; based on the soft importance weight, clustering the query vectors of each token into multiple clusters corresponding to the number of multicenters, and calculating the weighted average of each cluster as the local center query vector to obtain the multi-center query vector.

[0013] Further, in step S3, the value matrix is ​​weighted and aggregated using the centrality weights to obtain efficient branch outputs. Specifically, the centrality weights are weighted using inter-layer adaptive decay factors to obtain aggregate weights. The value matrix is ​​weighted and summed using the aggregate weights to obtain a preliminary aggregate result. The mean of the refined branch outputs is used to correct the preliminary aggregate result. Based on the relationship between the centrality weights and a preset broadcast threshold, the corrected result is broadcast to all tokens in either the full dimension or the dimension reduced, resulting in efficient branch outputs consistent with the dimension of the refined branch outputs.

[0014] Further, the construction of the two-dimensional fusion weight matrix in step S4 is specifically as follows: the token-level fusion weight is obtained by weighting the energy value and centrality weight of each token and activating it with a Sigmoid function; the fine branch output and the efficient branch output are averaged along the token dimension and then concatenated, and input into a lightweight multilayer perceptron to obtain the feature dimension-level fusion weight; the token-level fusion weight is multiplied by the feature dimension-level fusion weight to obtain the two-dimensional fusion weight matrix.

[0015] Furthermore, before performing weighted fusion as described in step S4, the method further includes: obtaining the pre-processing parameters. The historical two-dimensional fusion weight matrix corresponding to each layer is calculated and its mean is calculated. The two-dimensional fusion weight matrix of the current layer is weighted and summed with the mean using learnable memory weights to obtain the updated two-dimensional fusion weight matrix, which is used for the weighted fusion.

[0016] Further, step S5 specifically includes: performing residual connection on the comprehensive attention output, restoring the token sequence to two-dimensional spatial features through patch inverse partitioning; extracting spatial features from different layers in the encoding stage for multi-scale fusion, and restoring them to the original image resolution through upsampling; mapping the token-level energy calculated in step S2 and the centrality weight calculated in step S3 to a pixel-level enhancement mask, and using the pixel-level enhancement mask to modulate the decoded features before inputting them into the segmentation head to obtain pixel-level segmentation prediction results.

[0017] This invention also provides an image segmentation system based on dynamic projection and bi-branch attention, comprising: The normalization module is used to perform layer normalization on the token sequence after the input image is divided into blocks to obtain normalized features; A dynamic projection module is used to perform dynamic adaptive reuse projection calculations based on the normalized features. It includes: a query key projection unit, used to generate a conditional shared projection matrix using learnable parameters based on the normalized features, and to project the normalized features using the conditional shared projection matrix to obtain a shared query matrix and key matrix; and a value projection unit, used to construct a value projection matrix for the current layer based on the sum of the cross-layer shared base projection matrix and the current layer's lightweight incremental projection matrix, and to project high-significance tokens and low-significance tokens using the full-dimensional and dimensionality-reduced value projection matrices respectively, according to the significance of each token in the normalized features, to obtain a value matrix. A dual-branch interactive computation module is used to compute fine-grained branch outputs and efficient branch outputs in parallel based on the query matrix, key matrix, and value matrix, and introduces a branch interaction mechanism. It includes: a fine-grained branch unit, used to perform residual correction based on the attention matrices of the current layer and the previous layer, and to perform nonnegativity, adaptive normalization, and adaptive sparsification on the corrected attention matrix, then perform multi-hop cumulative computation with the value matrix to obtain the fine-grained branch output; an efficient branch unit, used to calculate the robustness energy of each token based on the query matrix, adaptively determine the number of multi-centers according to the variance of the robustness energy to construct a multi-center query vector, match the multi-center query vector with the key matrix to obtain the centrality weight of each token, and use the centrality weight to perform weighted aggregation on the value matrix to obtain the efficient branch output; and a branch interaction unit, used to modulate the fine-grained branch output using the centrality weights obtained from the efficient branch calculation, and to correct the efficient branch output using the mean of the fine-grained branch output. A multi-dimensional fusion module is used to perform multi-dimensional dynamic adaptive feature fusion of the fine-grained branch output and the efficient branch output. It includes: a weight generation unit, which is used to learn the token-level fusion weight and the feature-dimensional fusion weight respectively, and multiply the two to construct a two-dimensional fusion weight matrix; and a fusion execution unit, which is used to use the two-dimensional fusion weight matrix to perform weighted fusion of the fine-grained branch output and the efficient branch output to obtain a comprehensive attention output. The segmentation decoding module is used to perform feature decoding and segmentation head adaptation on the comprehensive attention output to generate pixel-level segmentation prediction results.

[0018] Furthermore, the multi-dimensional fusion module also includes a cross-layer memory unit for acquiring previous data. Each layer has its corresponding historical two-dimensional fusion weight matrix, and the mean is calculated. The two-dimensional fusion weight matrix of the current layer is then weighted and summed with the mean using learnable memory weights to obtain an updated two-dimensional fusion weight matrix, which is then provided to the fusion execution unit.

[0019] As can be seen from the above technical solutions, compared with the prior art, the beneficial effects of the present invention are: 1. This invention utilizes a conditional shared projection mechanism to generate a shared query / key projection matrix using input feature perception, significantly reducing projection parameter redundancy. Simultaneously, it constructs a value projection matrix by combining a cross-layer shared basic projection matrix with a lightweight incremental matrix, and achieves graded projection based on token energy pre-computation. Full-dimensional projection is used for highly significant tokens, while dimensionality-reduced projection is used for low-significance tokens. This significantly reduces computational load while ensuring the accuracy of key features, thereby improving the inference efficiency of the model in resource-constrained scenarios. 2. This invention designs a parallel interactive dual-branch attention structure. The fine-grained branch fully captures the fine features of small targets and edge regions through cross-layer attention matrix reuse, adaptive sparsity, and dynamic multi-hop cumulative computation. The efficient branch quickly obtains global context information through robust energy computation, adaptive multi-center query vector construction, and centrality weight aggregation. The two branches mutually promote each other through centrality weight modulation and mean correction, which ensures both segmentation accuracy and computational efficiency, making it particularly suitable for complex scenarios with multi-scale targets. 3. In the dual-branch output fusion stage, this invention simultaneously learns dual-dimensional fusion weights at both the token and feature dimension levels, and introduces a cross-layer memory adjustment mechanism. It utilizes historical layer fusion weights to smooth the current layer weights, effectively avoiding drastic fluctuations in fusion weights. The fusion process can dynamically adjust the contribution ratio of fine-grained branches and efficient branches based on the salience of the token and the importance of the feature dimension, achieving fine-grained and highly stable feature fusion. 4. In the feature decoding and segmentation head adaptation stage, this invention maps the token-level energy obtained in the projection calculation stage and the centrality weight obtained in the dual-branch interaction stage into a pixel-level enhancement mask, performs pixel-by-pixel modulation on the decoded features, strengthens the target region features and suppresses background noise, effectively improves the segmentation clarity of small target edges and improves the boundary integrity of the segmentation results. 5. This invention is not limited to specific application scenarios and can be widely applied to various image segmentation tasks such as semantic segmentation and instance segmentation. At the same time, through the task adaptive weight guidance mechanism, the fusion weight can be dynamically adjusted according to the characteristics of downstream tasks (such as classification, detection, and segmentation), so as to realize one model adapting to multiple tasks and reduce the model deployment and migration costs. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the image segmentation method based on dynamic projection and bi-branch attention according to the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The illustrative embodiments and descriptions of this invention are for illustrative purposes only and are not intended to limit the invention.

[0022] like Figure 1 As shown, this embodiment provides an image segmentation method based on dynamic projection and bi-branch attention, which can be applied to scenarios such as power line inspection image segmentation, remote sensing image analysis, and medical image segmentation.

[0023] Overall Model Architecture Overview The image segmentation model used in this embodiment is built on the Visual Transformer (ViT) architecture, with a total number of layers. (ViT-B Basic Version), input image resolution is 224×224, patch size Get the length of the token sequence. Correspondingly, the total feature dimension of the model Attention count Single-head feature dimension The overall model processing flow includes five core stages: data input initialization and normalization, dynamic adaptive reuse projection calculation, dual-branch parallel interactive calculation, multi-dimensional dynamic adaptive feature fusion, feature decoding and segmentation head adaptation.

[0024] Step S1: Data Input Initialization and Normalization Let the first The input of the layer is , dimension .when hour, ,in The token sequence after patching the original image; when hour, This is the residual output of the previous layer's integrated self-attention module.

[0025] For input Layer normalization is performed to eliminate differences in feature dimensions, ensuring the numerical stability of subsequent projection and attention calculations. The layer normalization process involves: first calculating the mean and variance of the input features; then standardizing the features; and finally scaling and shifting using learnable parameters to ensure a stable feature distribution. (1); In formula (1): This represents the layer normalization operation. The input features are normalized, with dimension 1. .

[0026] Step S2: Projection calculation based on dynamic adaptive reuse This step aims to address the issues of redundant projection parameters and lack of adaptive adjustment capabilities in traditional ViT models. It generates a query / key matrix through a conditional shared projection mechanism and a value matrix through cross-layer incremental projection combined with token energy pre-computation. Specifically, it includes the following steps: S21. Basic Definition of Projection Matrix , , For single-headed query, key, and value base projection matrices, all dimensions are... Each head corresponds to an independent basic projection matrix, realizing the transformation from the total feature dimension of the model. To single-head feature dimension The basic mapping. Simultaneously, design the cross-layer basic projection matrix. and interlayer incremental projection matrix , This enables the reuse and incremental updating of cross-layer projection parameters.

[0027] S22, Conditional Shared Q / K Projection Matrix Generation To ensure / While reusing parameters, feature discriminative power is improved by generating a conditional shared projection matrix based on input feature perception. The normalized features are passed sequentially through a learnable first weight matrix and a sigmoid activation function, and then multiplied by a predefined query-based projection matrix: (2); In formula (2): , In this embodiment, as a learnable parameter: The initial values ​​are based on a normal distribution. Random initialization, The initial value is 0; Use the Sigmoid activation function; For the first layer / Shared dynamic projection matrix, dimension preserved .

[0028] S23, Generation of cross-layer incremental V-projection matrix and fractional projection The V-projection is implemented through cross-layer reuse and adaptive adjustment using a base matrix and an incremental matrix structure. Simultaneously, token-level adaptation of the V-projection is achieved by combining token energy pre-computation. First, calculate the cross-layer incremental projection matrix: (3); In formula (3): The basic V projection matrix is ​​shared by all layers; For the first The lightweight incremental matrix of the layer has dimensions of This can significantly reduce the total number of cross-layer parameters; For the first The value projection matrix of the layer.

[0029] Secondly, by pre-calculating the token energy, the token-level coarse-grained energy of the normalized features is calculated and used to distinguish token significance: (4); In formula (4): For the first The first in the layer The coarse-grained energy of a token; For the first The features obtained after layer output normalization are: the first layer Feature vectors of each token; It is the L2 (Euclidean) norm, used to compress the token feature vector into a scalar energy value, reflecting the overall activity of the token in the feature space.

[0030] In practical operation: let the learnable threshold be... (The initial value in this embodiment is 0.5), when At that time, the first One token is determined as a highly significant token; otherwise, it is determined as a low-significance token.

[0031] Finally, perform the fractional V projection mapping: For highly significant tokens, use the full dimensions. projection: (5); In formula (5): For layer normalization features Features corresponding to tokens with medium to high significance; For the first A value projection matrix with complete dimensions at each layer; The value submatrix obtained by fully projecting highly significant tokens; Dimensionality reduction for low-significance tokens projection: (6); In formula (6): For layer normalization features Features corresponding to tokens with low to medium significance; For the complete value projection matrix The low-dimensional submatrix, dimension ; The value submatrix obtained by dimensionality reduction projection of low-significance tokens; The above projection results are then stitched together and upsampled to a uniform dimension: (7); In formula (7): Upsampling of low-dimensional features to Dimensional operation; For tensor concatenation operations, the two parts are concatenated along the token dimension to restore the complete token sequence order; For the first The final output value matrix of the layer.

[0032] S24, Dynamic Projection of Query / Key Matrix Normalized features By conditionally sharing the projection matrix Projection yields the query matrix. Bond matrix Because a shared projection matrix is ​​used, All dimensions The calculation formula is as follows: (8).

[0033] S25, Multi-head Projector Perform the above dynamic projection operation on each head to obtain the query matrix corresponding to each head. Key matrix Value Matrix Subsequent steps involve parallel computation of each head, and the outputs of all heads are ultimately merged through tensor splicing operations.

[0034] Step S3: Feature-aware dual-branch parallel interactive computation Based on the same set of query matrices obtained in step S2 Key matrix Value Matrix The attention outputs of the two branches are computed in parallel, and a lightweight interaction gate is introduced to achieve mutually reinforcing computation between the two branches. Branch computation timing is as follows: the efficient branch first performs token energy calculation and centrality weight generation, which is then passed to the fine-grained branch in real time for modulation. The attention calculation in the fine-grained branch is then performed, and the two branches are parallel and mutually reinforcing. Specifically, this includes: Branch A: The fine branch of the self-attention mechanism aims to capture the fine features of small targets and edge regions, and achieves full diffusion of information through multi-hop cumulative computation.

[0035] S311. Cross-layer attention matrix reuse initialization like Directly construct the original attention matrix; if The attention matrix after normalization from the previous layer As the initial attention matrix, it is passed through the current layer / Features perform minor residual corrections to reduce redundant calculations: (9); In formula (9): For learnable residual weights, the range of values ​​is... (In this embodiment, the initial value is 0.1), which is used to control the retention ratio of the attention matrix of the previous layer when the current layer is initialized; Indicates the transpose operation; For the first The initial attention matrix for the fine-grained branching layer. This is the attention matrix after normalization from the previous layer.

[0036] S312, Nonnegativity of Attention Matrix A piecewise adaptive activation function is used to nonnegate the initial attention matrix while preserving the gradient information of high-value features, avoiding feature loss caused by ReLU hard truncation, and ensuring that the attention weights are nonnegative. (10); In formula (10): The learnable hierarchical hyperparameters (initially set to 0.2 in this embodiment); It is a rectified linear activation function; It is a soft saturation activation function; For the first The initial attention matrix for the fine-grained branching layer; For the first Attention matrix after nonnegation processing of the fine-grained branches of the layer.

[0037] S313, Adaptive Normalization of Attention Matrix Based on the sparsity S (proportion of non-zero elements) of the nonnegative attention matrix, the normalization method is automatically selected to ensure numerical stability while matching the matrix distribution characteristics: (11); In formula (11): This is the sparsity threshold (the initial value in this embodiment is 0.3). Low sparsity indicates a large sparsity, while high sparsity indicates a large sparsity. It is the Frobenius norm; To minimize positive scalars, avoid denominators of 0; For row normalization operation; For the first The attention matrix after nonnegation processing of the fine-grained branches of the layer; For the first The attention matrix after adaptive normalization of the fine-grained branching layer ensures the spectral radius of the matrix. .

[0038] S314, Adaptive Sparsity of Attention Matrix For the attention matrix after adaptive normalization Design thresholded sparse gates to reduce computational redundancy for irrelevant tokens: (12); In formula (12): , is the adaptive threshold, where μ is a learnable parameter (in this embodiment, the initial value is 0.2). This is for the operation of taking the average; For the th after adaptive normalization The first in the layer The token is paired with the first... Attention matrix for each token; For the sparsified first The first in the layer The token is paired with the first... The attention matrix of each token is used to improve the efficiency of subsequent calculations by leveraging the sparse matrix multiplication rule.

[0039] S315, Interlayer Adaptive Attenuation Factor Generation By changing the fixed attenuation factor to an adaptive attenuation factor that is layer depth-aware, information is fully propagated in shallow layers while information over-propagation is avoided in deeper layers: (13); In formula (13): The initial attenuation factor has a value range of 1. (In this embodiment, 0.6 is used); For the first The layer's adaptive decay factor; This is a learnable parameter (the initial value in this embodiment is 0.5); This represents the total number of layers in the model. It is an exponential function.

[0040] S316, Single-Step Attention Output Calculation and Dynamic Multi-Hop Accumulation First, calculate the single-step attention output: (14); In formula (14): For the first The value matrix of the layer; For the sparsified first Attention matrix of the layer; For the first The single-step attention output of the layer has a dimension of This enables single-hop information diffusion and aggregation between tokens.

[0041] Then, an attention convergence metric was designed to achieve dynamic termination of multi-hop computation and avoid computational redundancy for simple features: Calculate the cosine similarity between the current single-step attention output and the previous step's attention output: (15); In formula (15): This is the current single-step attention output; This is the output of attention from the previous step; Calculate the cosine similarity; This represents the cosine similarity between the current single-step attention output and the previous step's attention output.

[0042] Specifically, if ( As a learnable threshold (in this embodiment, the initial value is 0.95), multi-hop accumulation continues; if If the multi-hop calculation is terminated, only the current and previous cumulative results are retained: (16); In formula (16): The cumulative output of the fine-grained branches, with dimensions of During the initial multi-hop calculation, Set as an all-zero matrix; It serves as a summation index variable for accumulation operations.

[0043] S317, Branch Interaction Gate Modulation Introducing pre-calculated token centrality weights for efficient branching As a mask, the cumulative output of the fine-grained branches is modulated, so that the fine-grained branches focus on high-centrality tokens, thus achieving efficient branch guidance for fine-grained branches: (17); In formula (17): To convert the centrality weight vector into a diagonal matrix; This is the cumulative output of the modulated fine branch.

[0044] Branch B: The efficient branch of the self-attention mechanism aims to quickly obtain global context information and achieve efficient feature capture through multi-center query vector construction and centrality weight aggregation.

[0045] S321, Token-level robust energy calculation Calculate the query matrix Query vector for each token and The mixed norm yields the first... Robustness energy scalar of a token: (18); (19); In formulas (18)-(19): Learnable robustness parameters (within a range of values) (In this embodiment, the initial value is 0.5). For the first Layer A query matrix for each token; For the first The first in the layer The robustness energy scalar of a token , The sequence length; For all The energy vector formed has a dimension of This energy is used to measure the salience of a token in the embedding space; for Norm; for Norm.

[0046] S322, Energy Vector Normalization robust energy vector implement Normalization yields the soft importance weight vector: (20); (twenty one); In formulas (20)-(21): For the first The first in the layer The soft importance weight of each token , The sequence length; For the first The first in the layer The robustness energy scalar of a token For summation index variables; To minimize positive scalars, avoid denominators of 0; For robust energy vector implement The normalized soft importance weight vector has a degree of ,satisfy And all elements are non-negative.

[0047] S323. Construction of Feature-Aware Dynamic Multi-Center Query Vectors Since variance effectively measures the dispersion of energy and is sensitive to feature complexity, it is more suitable for adaptive decision-making. Therefore, the first step is to base the decision on the energy values ​​of all tokens. Calculate the variance of the energy vector The number of clusters is determined based on the variance. : (twenty two); In formula (22): This is the variance threshold (the initial value in this embodiment is 0.4). When the time is right, it is a simple feature; otherwise, it is a complex feature. This is a learnable scaling parameter (in this embodiment, the initial value is 2); This indicates a round-down operation; and .

[0048] Then, based on the energy weight vector Cluster the tokens as There are n clusters (K-means clustering can be used), denoted as the i-th cluster. The set of tokens for each cluster is , .

[0049] Then construct a local center query vector for each cluster: (twenty three); This results in a multi-center query vector set: (twenty four); In formula (23)-(24): For the first The first in the layer Local center query vectors , This refers to the number of multi-center query vectors; For a multi-center query vector set, the dimension is , This is a summation index variable used for accumulation operations.

[0050] S324. Multicenter Key Vector Matching and Nonnegativity Calculate the inner product of the multi-center query vector set and each token key vector, take the mean, and then perform nonnegation processing using the ReLU activation function: (25); In formula (25): For multiple centers; For the first The first in the layer Transpose of a local center query vector; For the first The first in the layer A key vector of tokens, ; is the ReLU activation function, used to nonnegate the dot product result, retaining only positive correlation and ignoring negative correlation (which is considered to have no contribution and is set to 0). To sum the index variable, iterate through all central query vectors; For the first The first in the layer The original matching results of the first token comprehensively reflect the first... The degree of correlation between a token and all salient regions in the entire image; if a token has a strong positive correlation with multiple centers, the score is higher; if it has a weak or negative correlation with all centers, the score is lower.

[0051] pass Each center query vector is matched with the key vector of each token, which can capture information of multiple semantic centers (such as multiple targets or different regions) in the image, overcoming the problem that single centers are insufficient for modeling multi-target / complex texture images.

[0052] Smooth the matching vector using a 1D lightweight convolutional noise filtering layer: (26); In formula (26): A 1D lightweight convolutional noise filtering layer, kernel size This achieves smooth denoising of the matching vector; For the first The original matching vector of the layer; No. The matching vector after one-dimensional convolution filtering of the layer has a dimension of .

[0053] Original matching vector Abnormally high or low matching scores may occur due to noise in individual tokens. One-dimensional convolution smooths out local fluctuations and suppresses isolated noise points by weighted averaging of scores from adjacent tokens. Furthermore, adjacent tokens in an image usually belong to the same object or region, and their semantic similarity is high. One-dimensional convolution utilizes this spatial continuity to make the centrality weights smoother in local regions, which aligns with the intuition of visual tasks. This improves the robustness of matching in low-quality images, making the matching results more reliable and enhancing the model's resistance to input interference.

[0054] S325, Matching Vector Normalization For the denoised matching vector implement Normalization yields the final token centrality weight vector. : (27); (28); In formula (27)-(28): For the first The first in the layer Central weight of each token ; For summation index variables; To minimize positive scalars, avoid denominators of 0; The denoised matching vector implement The normalized soft importance weight vector has a dimension of ,satisfy This weight vector is a linear approximation of the principal feature vector of the attention operator, enabling efficient computation of global token centrality; This is a transpose operation.

[0055] S326, Decay Weighting and Value Aggregation The central weight vector Through interlayer adaptive attenuation factor Weighting yields the aggregation right. Then the value matrix pass Weighted summation yields the initial output of the efficient branch: (29); In formula (29): For summation index variables, ; For the first The first in the layer The aggregate weight of each token; For the first The first in the layer The value vector of each token; For the first The initial aggregation result of the efficient branch is shown. The efficient branch obtains the global aggregated features of the image through this formula, while the fine branch retains the local fine features of each token. The two are combined in the subsequent fusion stage to achieve a balance between accuracy and efficiency.

[0056] S327, Fine-tuned branch feature correction Introducing single-step output with fine-grained branches The initial output of the efficient branch is corrected to achieve fine-grained branch correction and mutual promotion between the two branches of the efficient branch: (30); In formula (30): For the first Preliminary aggregation results of efficient layer-based branches; For the first Single-step output of fine-grained branching at each level; To enable learnable weight adjustments, allowing the model to adaptively determine the strength of fine-grained branch adjustments based on training data, in the early stages of training... The initial value is relatively small (0.2 in this embodiment), and the correction is weak. As training progresses, the model can learn a suitable degree of correction. Calculated as the mean; For the first The output after efficient branch correction at each layer.

[0057] Efficient branches are obtained through weighted aggregation While efficient, this approach may lose some local details. The fine-grained branch, through multi-hop cumulative computation, preserves richer local features. The mean of the fine-grained branch's output is added as a correction term to the efficient branch, enabling the efficient branch to absorb local information from the fine-grained branch and improving the expressive power of the aggregation result.

[0058] S328, Feature-aware sparse global broadcast Based on token centrality weight Sparse broadcasting reduces feature storage redundancy while ensuring global information transmission, resulting in an efficient branching final output. The final output dimension is completely consistent with that of the fine-grained branch: (31); In formula (31): For the first Layer Centrality weight of each token The index of the token in the sequence, with a value range of... ,in The total number of tokens; The centrality weight threshold is set to 0.1 in this embodiment. Representing a high-centrality token, a complete broadcast is performed. This represents a low-centrality token, enabling low-dimensional broadcasting. To facilitate dimensionality reduction, the low-dimensional broadcast portion is subsequently upsampled and restored to its original size during the fusion phase. dimension; For the first The output after efficient branch correction at the layer; No. The final output of the efficient branch of the layer.

[0059] Step S4: Multi-dimensional dynamic adaptive feature fusion This step aims to output the final branches obtained from the parallel interactive computation in the above steps. (Fine features) and (High-efficiency features) introduces a dynamic fusion weight system combining token and feature dimensions, and integrates cross-layer memory adjustment and task-adaptive guidance to achieve precise and deep fusion of fine-grained and high-efficiency features, rather than simple concatenation; simultaneously, mixed-precision computation is used to improve engineering inference efficiency. The steps are as follows: S41. Mixed Precision Calculation and Precision Alignment For fine-grained branches, FP32 high-precision computation is used to ensure modeling accuracy, while for efficient branches, FP16 low-precision computation is used to reduce overhead. A lightweight precision conversion module is designed to convert the output of efficient branches from FP16 to FP32, achieving precision alignment of the two-branch features. The aligned features are denoted as follows: (Fine features) and (High-efficiency features), all dimensions are .

[0060] S42, Two-Dimensional Dynamic Fusion Weight Generation Simultaneously learn token-level fusion weights and feature dimension-level fusion weights to implement a fine-grained fusion strategy: Token-level fusion weight Energy from tokens and centrality weight Joint learning enables high-energy / high-centrality tokens to incorporate more refined features: (32); In formula (32): , , Learnable parameters (in this embodiment) , The initial value is 0.5. (Initial value is 0) The index of the token in the sequence, with a value range of... ,in The total number of tokens; Use the Sigmoid activation function; For the first The first in the layer A robust energy scalar for each token; For the first The first in the layer The central weight of each token; For the first The first in the layer Token-level fusion weight of each token A larger value indicates a greater contribution from the finer branches in that dimension.

[0061] Feature Dimensional Fusion Weights We use a lightweight MLP to perform dimensional importance scoring on bi-branch features, enabling more refined features to be integrated into the important feature dimensions. (33); In formula (33): It is a lightweight multilayer perceptron with an output dimension of In specific operations: MLP can contain 1 linear layer, ReLU activation, 1 linear layer and Sigmoid activation; This is a tensor concatenation operation, specifically concatenating two mean vectors into a longer vector along the feature dimension; The averaging operation is performed along the token dimension. Find the mean; For the first The final output of the fine-grained branching layer; For the first The final output of the efficient branching layer; For the first The first in the layer Feature dimension-level fusion weights for each feature dimension ; .

[0062] Token-level and dimension-level weights are merged into a two-dimensional weight matrix. : (34).

[0063] S43, Cross-layer memory weight adjustment Design a weighted cross-layer memory module, utilizing the previous... Layer (in this embodiment) The average fusion weight is used to initialize the weights of the current layer, improving the continuity of the inter-layer fusion strategy and avoiding drastic fluctuations in the weights of a single layer. (35); In formula (35): k is the size of the memory window; Learnable memory weights (value range is) In this embodiment, the initial value is 0.2. ,but ; For summation index variables; The initial fusion weights.

[0064] S44, Task Adaptive Weight Guidance Introducing downstream task-aware loss Gradient guidance is applied to the fusion weights. (Classification task guidance) Reduce the size of the branch, allowing efficient branches to dominate; guide the detection / segmentation task. Increase the size of the branch to allow the finer-grained branch to dominate. The total loss function is: (36); In formula (36): ω represents the model's base loss; ω is the task loss weight (0.3 in this example). Perceive loss for downstream tasks; This is the total loss function.

[0065] S45, Multi-dimensional Dynamic Fusion Computing Based on a two-dimensional memory-based fusion weight, the precision-aligned dual-branch features are weighted and fused token-by-token and dimension-by-dimensional to obtain the final attention output of the single head of the integrated self-attention mechanism. : (37); In formula (37): For the first The final output of the fine-grained branching layer; For the first The final output of the efficient branching layer; These are the initialized fusion weights; This means that the fusion retains the multi-hop fine interaction information of the fine branches and superimposes the global efficient aggregation information of the efficient branches, while achieving a dynamic balance between accuracy and efficiency and precise fusion of fine-grained features.

[0066] S46, Multi-head Fusion Output The final attention output of each head in a multi-head mechanism The data is then stitched together and mapped back to the total dimension of the model via a linear projection layer. The output of the entire integrated self-attention module is obtained; if it is an intermediate layer of the model, the residual connection is performed and then it enters the next self-attention module; if it is the last layer of the model, it is sent as a global feature in the encoding stage to the segmentation and decoding process in step S5.

[0067] Step S5: Feature Decoding and Segmentation Head Adaptation Optimization Based on the multi-head fusion output features obtained in step 4, and addressing the pixel-level dense prediction requirements of image segmentation tasks, a complete workflow is designed, including layer-by-layer feature decoding, multi-scale fusion, spatial dimension restoration, and adaptive optimization of the segmentation head. This achieves accurate mapping from high-dimensional token features to pixel-level segmentation masks while preserving the global context and fine-grained local features learned by the self-attention module, adapting to the feature requirements of mainstream image segmentation tasks such as semantic segmentation and instance segmentation. The steps are as follows: S51, Residual Connectivity and Dimension Preservation Let the self-attention output features from step 4, which involve multi-head fusion and mapping back to the total dimension of the model, be denoted as... , dimension , with the input of this layer Perform residual summation to compensate for feature loss during decoding and ensure the continuity of feature transfer between layers: (38); In formula (38): For the first Multi-head fusion of layers and mapping back to the self-attention output features of the total model dimension; For the first Layer input; The first after residual connection The layer outputs features with dimension 1. If it is the output of the last layer of the self-attention module in the model, then The features are fed into the subsequent decoding process as global features during the encoding stage. If the dimensions of the self-attention output features and the input features are inconsistent during residual connection, the input features are mapped to the same dimension as the output features through a 1×1 convolution.

[0068] S52, Dimensional Recovery from Token Features to Spatial Features Image segmentation requires dense spatial pixel features. It is necessary to restore the 1D token sequence features in the encoding stage to 2D spatial features. The restoration process is a strict inverse mapping of the previous image patching operation. Let the original image resolution be... (high The patch size is Then the length of the token sequence (If padding is done when dividing into patches, then it is) ), the residual after linking the first Layer output features The token sequence is rearranged into a 2D feature map according to the spatial order of the patches to obtain the intermediate features. , dimension Then The channel dimension is replaced with a second dimension, conforming to the feature storage format of a convolutional neural network, to obtain spatial features. , dimension .

[0069] S53, Multi-scale Feature Layer-by-Layer Decoding and Upsampling We design a lightweight decoding structure that combines transposed convolution and feature fusion to upsample high-dimensional spatial features layer by layer, restoring the original image. Resolution is improved by incorporating self-attention features from different layers during the encoding stage to compensate for detail loss caused by large-scale downsampling. First, extract the model's preorder cipher. Residual output characteristics of layers (such as 1 / 2 / 4 / 8x downsampling layers) , respectively corresponding to the original image Multiply the resolution by performing channel compression (lightweight 1×1 convolution) on the features of each layer to a uniform dimension. ( To obtain multi-scale features to be fused .

[0070] Then, using the spatial features of the last layer of the encoding stage To decode the initial features, a hybrid upsampling method combining transposed convolution and pixel shuffling is used to double the feature resolution each time. Simultaneously, channel-wise summation and fusion are performed with the encoded multi-scale features of the same resolution. (39); In formula (39): For the first The features after the second decoding have a resolution equal to the initial resolution. times; (Decoding initial features, minimum resolution); For the first Multi-scale features to be fused after layer channel compression; To achieve mixed upsampling operations, resolution enhancement is achieved through transposed convolution, and pixel shuffling reduces the checkerboard artifacts of transposed convolution; when At that time, upsampling is stopped, and a resolution of is obtained. Decoding features , dimension .

[0071] S54, Feature Enhancement for Segmentation Task Awareness To address the issues of blurred target edges and indistinct features of small targets in image segmentation, a feature enhancement module specifically designed for segmentation tasks is presented, based on the token energy and centrality weights of the self-attention module. This module enhances the features of the decoded target. Features are modulated at the pixel level: First, the token-level coarse-grained energy... and token centrality weight Expanded into a pixel-level energy map according to the token-space inverse mapping rule. and centrality weight graph All dimensions This enables the transfer of attention features from the token level to the pixel level.

[0072] Then, through Sigmoid activation... and Normalization to Combined as a pixel-level enhancement mask, it enhances the decoded features. Pixel-by-pixel modulation is performed to enhance the features of the target region (high energy, high centrality) and suppress background noise. (40); In formula (40): , , Learnable parameters specific to the segmentation task (in this embodiment) , The initial value is 0.5. (Initial value is 0) The Sigmoid activation function is used to implement nonlinear modulation of the mask; A pixel-level energy map; Centralized weighted graph; The final decoded features after feature enhancement, dimension .

[0073] S55, Adaptive Segmentation Head Design and Pixel-Level Prediction This step aims to design an adaptive segmentation head based on the specific task type of image segmentation (semantic segmentation / instance segmentation), mapping the enhanced decoded features to pixel-level segmentation prediction results, while using a task-specific loss function for optimization to ensure the accuracy of the prediction results.

[0074] To address the requirements of single-image multi-class and pixel-level category labeling in semantic segmentation, a lightweight convolutional segmentation head is designed: Performing a 1×1 convolution maps the channel dimension to the number of classes C in the segmentation task, yielding the original predicted features. Dimension Then perform Softmax activation in the spatial dimension to obtain the probability distribution of each pixel belonging to each category. Dimension ,satisfy ( The semantic segmentation mask is obtained by taking the highest probability category of each pixel. Dimension .

[0075] Furthermore, to address the need for distinguishing different targets within the same category in instance segmentation, an instance branch is added to the semantic segmentation head to achieve dual-branch prediction of both category and instance: Category branching: Same as above, outputs pixel-level category probability distribution. Dimension ; Instance branch: for Perform 3×3 grouped convolutions and 1×1 convolutions to output instance embedding features. Dimension ( For instance embedding dimensions, typically 16 / 32). Perform pixel-level clustering (such as centroid clustering or cosine similarity clustering) on ​​the instance embedding features to group pixels of the same instance into one class, thus obtaining the instance mask. Dimension ( (the number of instances in the image). By fusing the results from the category branch and the instance branch, the category and pixel mask of each instance are obtained, and the final instance segmentation result is output. .

[0076] S56. Segmentation Task-Specific Loss Function and Joint Optimization To adapt the entire model to image segmentation tasks, the prediction loss of the segmentation head is jointly optimized with the basic loss of the preceding self-attention module. Furthermore, a hybrid loss is designed to address the characteristics of segmentation tasks, resolving issues such as class imbalance and inaccurate edge prediction. The total loss function is as follows: (41); In formula (41): This serves as the base loss for the self-attention module, ensuring the effectiveness of the encoded features; To achieve the core segmentation loss, semantic segmentation uses cross-entropy loss (CE) / Dice loss, while instance segmentation uses mask cross-entropy loss + embedding clustering loss. As an edge enhancement loss, a weighted loss calculation is performed on the edge pixels of the segmentation mask based on the edge detection map of the image to enhance the edge prediction accuracy; For class-balanced loss, Focal loss / weighted cross-entropy loss is adopted to solve the problem of sample imbalance between foreground / background and major / minor classes in segmentation tasks; , , These are learnable weights, which control the proportion of segmentation loss, edge loss, and balancing loss in the total loss, with values ​​ranging from [value range missing]. (This embodiment) =0.8, =0.2, =0.2); This is the total loss function.

[0077] Model training and parameter setting In this embodiment, the model training uses the AdamW optimizer, with a base learning rate set to 1e-4, which decays to 1e-5 as the number of iterations increases in the later stages of training.

[0078] In practical operation: For parameters that can be directly and randomly initialized (accounting for over 90%, which is the standard initialization method for deep learning): all projection matrix parameters ( , , , , , , The learnable parameters of the weight classes of convolutional layers (including transposed convolution, 1D convolution, and 3D grouped convolution), fully connected / MLP layers, and attention modules all adopt a normal distribution. Random sampling initialization; all bias term parameters ( Feature enhancement / fusion module (etc.) are uniformly initialized to 0. This initialization method is the mainstream standard method in the field of computer vision and can ensure the numerical stability of the model in the initial stage of training.

[0079] For parameters that must be initialized at fixed values ​​(without randomization, pre-set by business requirements / model stability): all basic hyperparameters of the model ( , , , , , , ): Fixed values ​​designed for the model architecture, with no initialization iteration process; loss function related to non-learning weights ( , , , , ): Initial values ​​are fixed based on the loss balance requirements of the segmentation task, and do not participate in gradient updates during training; initial baseline values ​​for various thresholds / proportions ( , , , , , , , , , , , All values ​​are fixed to the example initial values ​​mentioned above, and then fine-tuned through gradient descent during training to avoid oscillations in the early training stage caused by random initialization.

[0080] For special initialization parameters (incremental / residual parameters, which require dedicated initialization to ensure training stability): Incremental projection matrix , Use zero initialization or small variance normal distribution These parameters are incremental supplements to the shared base matrix across layers. Initialization with all zeros or small values ​​ensures that the base matrix is ​​the primary component in the early stages of model training, while the incremental matrix gradually learns the inter-layer differences, thus avoiding excessive interference of the incremental matrix on feature mapping in the initial stage.

[0081] Technical effect verification The method described in this embodiment was applied to a power line inspection image segmentation task and validated on a test set containing power equipment such as insulators, conductors, and fittings. Experimental results show that compared with the traditional ViT segmentation model, the method described in this embodiment improves the mIoU index by approximately 3.2%, reduces the number of parameters by approximately 25%, and increases the inference speed by approximately 30%. In particular, the segmentation accuracy for small targets such as broken conductor strands and minor damage to insulators is significantly improved, and the edge localization error is reduced by approximately 40%, effectively verifying the technical effects of the present invention.

[0082] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. An image segmentation method based on dynamic projection and bi-branch attention, characterized in that, Includes the following steps: S1. Perform layer normalization on the token sequence after the input image is segmented to obtain normalized features; S2. Based on the normalized features, perform dynamic adaptive reuse projection calculation; specifically including: The normalized features are used to generate a conditional shared projection matrix through learnable parameters, and the normalized features are projected using the conditional shared projection matrix to obtain a shared query matrix and a key matrix. The value projection matrix of the current layer is constructed by summing the base projection matrix shared across layers and the lightweight incremental projection matrix of the current layer. Based on the significance of each token in the normalized features, the high-significance tokens and low-significance tokens are projected using the full-dimensional and dimensionality-reduced value projection matrices, respectively, to obtain the value matrix. S3. Based on the query matrix, key matrix, and value matrix, perform parallel computation of fine-grained branch output and efficient branch output, and introduce a branch interaction mechanism; specifically including: In the fine branch: residual correction is performed based on the attention matrices of the current layer and the previous layer, and the corrected attention matrix is ​​nonnegated, adaptively normalized and adaptively sparsified, and then multi-hop cumulative calculation is performed with the value matrix to obtain the fine branch output; In the efficient branch: the robustness energy of each token is calculated based on the query matrix, the number of multicenters is adaptively determined according to the variance of the robustness energy to construct a multicenter query vector, the multicenter query vector is matched with the key matrix to obtain the centrality weight of each token, and the value matrix is ​​weighted and aggregated using the centrality weight to obtain the efficient branch output; The centrality weights obtained from the efficient branch calculation are used to modulate the fine branch output, and the mean of the fine branch output is used to correct the efficient branch output. S4. Perform multi-dimensional dynamic adaptive feature fusion between the fine-grained branch output and the efficient branch output; specifically including: We learn the token-level fusion weights and the feature-dimension-level fusion weights separately, and then multiply the two to construct a two-dimensional fusion weight matrix. The fine-grained branch output and the efficient branch output are weighted and fused using the dual-dimensional fusion weight matrix to obtain a comprehensive attention output. S5. Perform feature decoding and segmentation head adaptation on the comprehensive attention output to generate pixel-level segmentation prediction results.

2. The image segmentation method based on dynamic projection and bi-branch attention according to claim 1, characterized in that, In step S2: the generation of the conditional shared projection matrix specifically involves: passing the normalized features sequentially through a learnable first weight matrix and a Sigmoid activation function, and then multiplying them by a preset query base projection matrix to obtain the conditional shared projection matrix; In step S2: The step of performing graded projection based on token saliency specifically involves: calculating the L2 norm of each token in the normalized features as its energy value; identifying tokens with energy values ​​greater than the learnable threshold as high-saliency tokens and projecting them using the full-dimensional current-layer value projection matrix; identifying tokens with energy values ​​not greater than the learnable threshold as low-saliency tokens and projecting them using the dimensionality-reduced current-layer value projection matrix; and then concatenating the projection results and upsampling them to a unified dimension to obtain the value matrix.

3. The image segmentation method based on dynamic projection and bi-branch attention according to claim 1, characterized in that, The multi-hop cumulative calculation in step S3 specifically includes: calculating the cosine similarity between the attention output of the current step and the attention output of the previous step; if the cosine similarity is less than a preset learnable similarity threshold, then continue to calculate the attention output of the next step and perform accumulation; if the cosine similarity is greater than or equal to the learnable similarity threshold, then terminate the multi-hop calculation and use the accumulated result as the fine branch output.

4. The image segmentation method based on dynamic projection and bi-branch attention according to claim 1, characterized in that, The construction of the multi-center query vector in step S3 specifically includes: performing L1 normalization on the robustness energy vector to obtain the soft importance weight of each token; calculating the variance of the robustness energy vector, and determining the number of multicenters based on the relationship between the variance and a preset variance threshold; based on the soft importance weight, clustering the query vectors of each token into multiple clusters corresponding to the number of multicenters, and calculating the weighted average of each cluster as the local center query vector to obtain the multi-center query vector.

5. The image segmentation method based on dynamic projection and bi-branch attention according to claim 1, characterized in that, In step S3, the value matrix is ​​weighted and aggregated using the centrality weights to obtain efficient branch outputs. Specifically, the centrality weights are weighted using an inter-layer adaptive decay factor to obtain aggregate weights. The value matrix is ​​then weighted and summed using the aggregate weights to obtain a preliminary aggregate result. The mean of the refined branch outputs is used to correct the preliminary aggregate result. Based on the relationship between the centrality weights and a preset broadcast threshold, the corrected result is broadcast to all tokens in either the full dimension or the dimension reduced to obtain efficient branch outputs with dimensions consistent with the refined branch outputs.

6. The image segmentation method based on dynamic projection and bi-branch attention according to claim 1, characterized in that, The construction of the two-dimensional fusion weight matrix in step S4 is specifically as follows: the token-level fusion weight is obtained by weighting the energy value and centrality weight of each token and activating it with Sigmoid. The fine branch output and the efficient branch output are averaged along the token dimension and then concatenated. The result is input into a lightweight multilayer perceptron to obtain the feature dimension-level fusion weights. The token-level fusion weights are multiplied by the feature dimension-level fusion weights to obtain the dual-dimensional fusion weight matrix.

7. The image segmentation method based on dynamic projection and bi-branch attention according to claim 6, characterized in that, Before performing weighted fusion in step S4, the method further includes: obtaining the historical two-dimensional fusion weight matrices corresponding to each of the previous layers and calculating the mean; using learnable memory weights to perform a weighted summation of the current layer's two-dimensional fusion weight matrix and the mean to obtain an updated two-dimensional fusion weight matrix for use in the weighted fusion.

8. The image segmentation method based on dynamic projection and bi-branch attention according to claim 1, characterized in that, Step S5 specifically includes: performing residual connection on the comprehensive attention output, restoring the token sequence to two-dimensional spatial features through patch inverse partitioning; extracting spatial features from different layers in the encoding stage for multi-scale fusion, and restoring them to the original image resolution through upsampling; mapping the token-level energy calculated in step S2 and the centrality weight calculated in step S3 to a pixel-level enhancement mask, and using the pixel-level enhancement mask to modulate the decoded features before inputting them into the segmentation head to obtain pixel-level segmentation prediction results.

9. An image segmentation system based on dynamic projection and bi-branch attention, used to implement the image segmentation method based on dynamic projection and bi-branch attention as described in any one of claims 1-8, characterized in that, The system includes: The normalization module is used to perform layer normalization on the token sequence after the input image is divided into blocks to obtain normalized features; A dynamic projection module is used to perform dynamic adaptive reuse projection calculations based on the normalized features; it includes: a query key projection unit, used to generate a conditional shared projection matrix using the normalized features through learnable parameters, and to project the normalized features using the conditional shared projection matrix to obtain a shared query matrix and a key matrix; and a value projection unit, used to construct a value projection matrix for the current layer based on the sum of the cross-layer shared base projection matrix and the current layer's lightweight incremental projection matrix, and to project high-significance tokens and low-significance tokens using the full-dimensional and dimensionality-reduced value projection matrices respectively, according to the significance of each token in the normalized features, to obtain a value matrix. A dual-branch interactive computation module is used to compute fine-grained branch outputs and efficient branch outputs in parallel based on the query matrix, key matrix, and value matrix, and introduces a branch interaction mechanism. It includes: a fine-grained branch unit, used to perform residual correction based on the attention matrices of the current layer and the previous layer, and to perform nonnegativity, adaptive normalization, and adaptive sparsification processing on the corrected attention matrix, and then perform multi-hop cumulative computation with the value matrix to obtain the fine-grained branch output; an efficient branch unit, used to calculate the robustness energy of each token based on the query matrix, adaptively determine the number of multi-centers according to the variance of the robustness energy to construct a multi-center query vector, match the multi-center query vector with the key matrix to obtain the centrality weight of each token, and use the centrality weight to perform weighted aggregation on the value matrix to obtain the efficient branch output; and a branch interaction unit, used to modulate the fine-grained branch output using the centrality weights obtained from the efficient branch calculation, and to correct the efficient branch output using the mean of the fine-grained branch output. A multi-dimensional fusion module is used to perform multi-dimensional dynamic adaptive feature fusion of the fine-grained branch output and the efficient branch output; it includes: a weight generation unit, used to learn token-level fusion weights and feature-dimensional fusion weights respectively, and multiply the two to construct a two-dimensional fusion weight matrix; and a fusion execution unit, used to use the two-dimensional fusion weight matrix to perform weighted fusion of the fine-grained branch output and the efficient branch output to obtain a comprehensive attention output; The segmentation decoding module is used to perform feature decoding and segmentation head adaptation on the comprehensive attention output to generate pixel-level segmentation prediction results.

10. The image segmentation system based on dynamic projection and bi-branch attention according to claim 9, characterized in that, The multi-dimensional fusion module also includes a cross-layer memory unit, which is used to obtain the historical two-dimensional fusion weight matrix corresponding to each of the previous layers and calculate the mean. The two-dimensional fusion weight matrix of the current layer is weighted and summed with the mean using the learnable memory weights to obtain the updated two-dimensional fusion weight matrix, which is then provided to the fusion execution unit for use.

Images