An image fine structure intelligent detection algorithm based on a double-branch encoder

Abstract

The application discloses an image fine structure intelligent detection algorithm based on a double-branch encoder, and comprises the following steps: first, constructing an image segmentation system comprising a feature prior branch based on a lightweight pre-training model, a fine structure extraction branch based on a structure perception visual state space module, a cross-modal fusion module and a multi-scale feature complementary mapping decoder; then, inputting an original image into the feature prior branch and the fine structure extraction branch of the image segmentation system respectively to perform feature extraction and preliminary fusion; then, performing second stage fusion on the preliminarily fused features through the cross-modal fusion module; and finally, inputting the features into the multi-scale feature complementary mapping decoder to obtain a binary segmentation graph. Through the design of the feature prior branch and the fine structure extraction branch, the generalization ability of the model in a few sample scene is enhanced, and long-distance dependence and complex irregular morphology of fine structure are efficiently captured at a lower calculation cost. A two-stage cross-modal fusion mechanism improves the representation ability of the model to fine structure features.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and digital image processing technology, and in particular to an intelligent algorithm for detecting fine structures in images based on a dual-branch encoder. Background Technology

[0002] In the field of deep learning-based image segmentation, the mainstream technical solutions are currently based on two main architectures: convolutional neural networks and visual Transformers.

[0003] Traditional convolutional neural networks rely on local convolutional kernel sliding window operations to extract features. Their inherent inductive bias enables them to efficiently capture local features. However, in order to obtain global information, convolutional neural networks must stack multiple layers of convolution or use large kernel convolution. This approach is inefficient and makes it difficult to directly establish dependencies between distant pixels. For fine structure segmentation tasks with narrow shapes, irregular structures, and large spans, it is difficult to maintain their continuity and integrity.

[0004] Visual Transformer uses a self-attention mechanism, which can calculate the association weight between any two image patches in a sequence, and has a powerful global modeling capability by nature; however, the computational complexity of its self-attention mechanism is proportional to the square of the length of the input sequence, resulting in huge computational and memory overhead when processing high-resolution images, making it difficult to deploy in real time on edge devices with limited computing resources.

[0005] In recent years, models such as EdgeSAM have emerged, which obtain a lightweight image encoder based on convolutional networks by performing knowledge distillation on the ViT-based SAM model, a fundamental visual model.

[0006] Although the EdgeSAM encoder can output rich general visual prior features, when it is directly used for specific tasks (such as fine structure segmentation), an effective mechanism still needs to be designed to combine its prior knowledge with task-specific features. Otherwise, when the scale of a specific dataset is small, the generalization ability and accuracy of the model are still insufficient. Therefore, this invention proposes an intelligent image fine structure detection algorithm based on a dual-branch encoder to solve the problems existing in the prior art. Summary of the Invention

[0007] To address the shortcomings of existing technologies, such as the weak global modeling capability of CNNs, the high computational cost of ViT, and the insufficient generalization ability and inadequate utilization of prior knowledge by visual foundation models in specific tasks, this invention aims to propose an intelligent image fine structure detection algorithm based on a dual-branch encoder. This algorithm fully utilizes the rich general visual prior features provided by the visual foundation model to enhance the model's generalization ability in scenarios with few samples. It constructs a fine structure extraction branch through a specially designed structure-aware visual state space module, efficiently capturing long-distance dependencies and complex irregular shapes of fine structures at a lower computational cost. Through a two-level cross-modal fusion mechanism, it effectively integrates general prior features with task-specific features, improving the model's ability to represent fine structure features. Simultaneously, the algorithm maintains high accuracy while having low computational complexity, making it suitable for practical application deployment.

[0008] To achieve the objectives of this invention, the invention is implemented through the following technical solution: an intelligent image fine structure detection algorithm based on a dual-branch encoder, comprising the following steps:

[0009] Step 1: Construction of the image segmentation system. The system includes a feature prior branch based on a lightweight pre-trained model, a fine structure extraction branch based on a structure-aware visual state space module, a cross-modal fusion module, and a multi-scale feature complementary mapping decoder.

[0010] Step 2: Feature extraction. The original image is input into the feature prior branch and the fine structure extraction branch of the image segmentation system, respectively. The feature prior branch outputs a feature extraction function containing F. s1 F s2 F s3 F s4 Hierarchical features at four scales and the semantically rich F in the last layer final Five different levels of features;

[0011] Step 3: Preliminary fusion. Then, the features of each level are extracted from the fine structure branch and input together with the features of the same level from the feature prior branch into the cross-modal fusion module for the first stage of fusion, resulting in the fused feature F. fusion1 F fusion2 F fusion3 and F fusion4 ;

[0012] Step 4: Secondary fusion, combining the initially fused features F fusion1 F fusion2 and F fusion3 The data is sequentially input into the corresponding layers of the fine structure extraction branch for further processing, and then the features F are processed. fusion1 F fusion2 Ffusion3 and F fusion4 The final semantic features F of the feature prior branches are respectively final The second stage of fusion is performed through the cross-modal fusion module to obtain the final encoder features F1, F2, F3 and F4;

[0013] Step 5: Decode and segment. Input the final encoder features F1, F2, F3 and F4 into the multi-scale feature complementary mapping decoder to obtain the final binary segmentation map.

[0014] A further improvement is that the training strategy for the lightweight pre-trained model in step one adopts a fine-tuning method that only updates the parameters of the normalization layer.

[0015] The further improvement lies in the following: In step two, the fine structure extraction branch first divides the input image into image patches and adds positional encoding, and then sends them to the SAVSS module for processing. The SAVSS module consists of a gated bottleneck convolutional layer, a two-dimensional selective scanning module with a structure-aware scanning strategy, a pixel attention-guided fusion module, and residual connections.

[0016] A further improvement lies in the following: the specific process of the gated bottleneck convolutional layer is as follows:

[0017] S1. For the input feature x, after bottleneck convolution, group normalization, and ReLU activation function, g1(x) and x1 are obtained, which are expressed by the following formula.

[0018] g1(x)=ReLU(GN(BottConv(x)))

[0019] x1=ReLU(GN(BottConv(g1(x))))

[0020] In the formula, BottConv represents bottleneck convolution, and GN represents group normalization;

[0021] S2 and g2(x) are obtained by another path through bottleneck convolution, group normalization, and ReLUU activation function, and are represented by g2(x) = ReLU(GN(BottConv(x))).

[0022] S3. Then, g2(x) and x1 are combined through the Hadamard product to obtain the gated feature map m(x) = g2(x) ⊙ x1;

[0023] S4. Next, the gated feature map m(x) is further refined to obtain x2=ReLU(GN(BottConv(m(x))));

[0024] S5. Finally, perform a residual concatenation between X2 and the original input x to obtain the final output y = x2 + x.

[0025] Further improvements are made in the following aspects: The scanning strategy includes horizontal alternating scanning, vertical alternating scanning, main diagonal scanning, and secondary diagonal scanning. Horizontal alternating scanning starts from the last row and alternates between scanning from left to right and from right to left; vertical alternating scanning starts from the first column and alternates between scanning from top to bottom and from bottom to top; main diagonal scanning is performed along the diagonal direction from the upper left to the lower right; and secondary diagonal scanning is performed along the diagonal direction from the upper right to the lower left.

[0026] A further improvement is that the pixel attention-guided fusion module is used to fuse the initial sequence features x1 and the scanned sequence features x2, and the fusion process is expressed by the following formula.

[0027] f1(x) =(Norm(BottConv(x1))

[0028] f2(x) = Norm(BottConv((x2)))

[0029] σ = Sigmoid(Norm(BottConv( f2(x)⊙f1(x))))

[0030] y = (1-σ) ⊙ f1(x) + σ ⊙ f2(x)

[0031] Where ⊙ represents the Hadamard product, Norm represents batch normalization, and BottConv represents the bottleneck convolution.

[0032] A further improvement is made in step four, where the cross-modal fusion module first uses a 1×1 point convolution and size scaling operation to convert the feature prior branch's features F before fusion. s Features F of the fine structure extraction branch m Alignment is performed in both channel and spatial dimensions, and the aligned features F s and F m The features are concatenated along the channel dimension, and then subjected to 1×1 point convolution, group normalization, and ReLU activation to obtain the intermediate feature F. mid =ReLU(GN(PConv(Concat(F s ,F m )))), for F mid Two independent 1×1 point convolutions and group normalization operations are performed respectively. Convolutional modulation is then used to generate two attention weight maps, A1=GN(PConv(Fmid)) and A2=GN(PConv(Fmid)). Finally, the features F are fused. fusion F is obtained by weighted summation and concatenation with residuals. fusion = A1⊙F s +A2⊙F m +F m .

[0033] A further improvement is made in step five, where the multi-scale feature complementarity mapping decoder consists of a linear layer, a deconvolutional upsampling layer, and a feature complementarity mapping module. Specifically, the input multi-scale features F1, F2, F3, and F4 are first adjusted for the number of channels using a linear layer, and then the deconvolutional upsampling layer is used to upsample each feature map to a uniform size, as expressed by the following formula. In the formula, UpSample represents a dynamic convolutional layer, and MLP represents a linear layer. The upsampled features are then concatenated along the channel dimension and fed into a feature complementarity mapping module. Through channel attention and spatial attention mechanisms, channel weights enhance the semantic guidance of the spatial representation, while spatial weights strengthen the spatial complementarity of the semantic representation. Finally, the convolutional layer outputs the final segmentation result image, represented as... .

[0034] The beneficial effects of this invention are as follows: By introducing a feature prior branch based on EdgeSAM, this invention enables the algorithm to have excellent performance and generalization ability, and can more accurately handle diverse and complex scenarios.

[0035] By extracting innovative structural-aware visual state space modules from the branches, the continuity and integrity of fine structures can be better maintained;

[0036] The two-level cross-modal fusion module can adaptively and efficiently fuse general prior features and task-specific features, avoiding information conflicts or loss caused by simple splicing or addition, giving full play to the synergistic advantages of the dual-branch architecture, and improving the quality and efficiency of feature fusion.

[0037] By combining the pixel attention-guided fusion module and the multi-scale feature complementary mapping decoder, the ability to perceive and reconstruct details such as fine structural edges can be enhanced, resulting in more accurate segmentation results and clearer boundaries. Attached Figure Description

[0038] Figure 1 This is a diagram of the overall architecture of the present invention.

[0039] Figure 2 This is an architectural diagram of the Structure-Aware Visual State Space Module (SAVSS) of the present invention.

[0040] Figure 3 This is a structural diagram of the gated bottleneck convolutional layer of the present invention. Detailed Implementation

[0041] To enhance understanding of the present invention, the present invention will be further described in detail below with reference to embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0042] according to Figure 1, Figure 2 and Figure 3 As shown, this embodiment provides an intelligent image fine structure detection algorithm based on a dual-branch encoder, including the following steps:

[0043] Step 1: Construction of the image segmentation system. This involves constructing an image segmentation system comprising a feature prior branch based on a lightweight pre-trained model, a fine structure extraction branch based on the Structure-Aware Visual State Space Module (SAVSS), a cross-modal fusion module, and a multi-scale feature complementarity mapping decoder. The overall architecture is shown in the appendix of the manual. Figure 1 As shown;

[0044] The training strategy of the lightweight pre-trained model adopts a fine-tuning method that only updates the parameters of the normalization layer to better adapt to the downstream fine structure segmentation task and efficiently utilize pre-trained knowledge. This method significantly reduces the number of training parameters and lowers the risk of overfitting.

[0045] Step 2: Feature extraction. The original image is input into the feature prior branch and the fine structure extraction branch of the image segmentation system, respectively. The feature prior branch outputs a feature extraction function containing F. s1 F s2 F s3 F s4 Hierarchical features at four scales and the semantically rich F in the last layer final Five different levels of features;

[0046] The fine structure extraction branch first segments the input image into image patches and adds positional encoding, then feeds them into the SAVSS module for processing. The SAVSS module consists of a gated bottleneck convolutional layer, a two-dimensional selective scanning module with a structure-aware scanning strategy, a pixel attention-guided fusion module, and residual connections, as shown in the attached manual. Figure 2 As shown;

[0047] The gated bottleneck convolutional layer structure is shown in the attached instruction manual. Figure 3 As shown, the specific process is as follows:

[0048] S1. For the input feature x, after bottleneck convolution (BottConv), group normalization (GN), and ReLU activation function, g1(x) and x1 are obtained, which are expressed by the following formula.

[0049] g1(x)=ReLU(GN(BottConv(x)))

[0050] x1=ReLU(GN(BottConv(g1(x))))

[0051] In the formula, BottConv represents bottleneck convolution, and GN represents group normalization;

[0052] S2 and g2(x) are obtained by another path through bottleneck convolution, group normalization, and ReLUU activation function, and are represented by g2(x) = ReLU(GN(BottConv(x))).

[0053] S3. Then, g2(x) and x1 are combined through the Hadamard product to obtain the gated feature map m(x) = g2(x) ⊙ x1;

[0054] S4. Next, the gated feature map m(x) is further refined to obtain x2=ReLU(GN(BottConv(m(x))));

[0055] S5. Finally, perform a residual concatenation between X2 and the original input x to obtain the final output y = x2 + x.

[0056] This layer is used for preliminary feature transformation and gating adjustment to enhance feature representation capabilities.

[0057] Two-dimensional selective scanning module: Since different scanning strategies affect the model's sensitivity to direction, in order to enable the two-dimensional selective scanning module to better perceive irregular fine structures, a structure-aware scanning strategy that is more in line with the extension characteristics of fine structures is designed, including horizontal alternating scanning, vertical alternating scanning, main diagonal scanning, and sub-diagonal scanning.

[0058] Horizontal alternating scan: Starting from the last row, scan alternately from left to right and from right to left;

[0059] Vertical alternating scan: Starting from the first column, scan alternately from top to bottom and from bottom to top;

[0060] Main diagonal scan: Scan along the diagonal direction from the top left to the bottom right;

[0061] Subdiagonal scan: Scan along the diagonal direction from the upper right to the lower left.

[0062] Each path unfolds the two-dimensional image features into a one-dimensional sequence along a specific direction, and then inputs these sequences into the core SSM equation for processing.

[0063] ① Calculation of Discretization Parameters

[0064] In the formula, G and H are learnable parameter matrices that control the dynamics of the hidden state and the influence of the input, respectively; Δ is a learnable time step parameter used to control the granularity of discretization. and These are the discretized state transition matrix and input influence matrix.

[0065] ② Hidden state update

[0066] Hidden state yk The update formula for time step k is:

[0067] In the formula y k -1 represents the hidden state of the previous time step, r k This is the input for the current time step.

[0068] ③ Output Calculation

[0069] The output c of the current time step k Determined by both the hidden state and the input, c k =E yk +T rk , where E and T are learnable output projection matrices.

[0070] After processing, the output sequences of the four paths are reconstructed into feature maps and summed to obtain the output of the module. This strategy enables the model to fully perceive the contextual information of fine structures in different directions.

[0071] Pixel attention-guided fusion module: used to fuse initial sequence features x1 (output of gated bottleneck convolutional layer) and scanned sequence features x2 (output of 2D selective scan module). The fusion process is represented by the following formula.

[0072] f1(x) =(Norm(BottConv(x1))

[0073] f2(x) = Norm(BottConv((x2)))

[0074] σ = Sigmoid(Norm(BottConv( f2(x)⊙f1(x))))

[0075] y = (1-σ) ⊙ f1(x) + σ ⊙ f2(x)

[0076] Where ⊙ represents the Hadamard product, Norm represents batch normalization, and BottConv represents the bottleneck convolution.

[0077] Step 3: Preliminary fusion. Then, the features of each level are extracted from the fine structure branch and input together with the features of the same level from the feature prior branch into the cross-modal fusion module for the first stage of fusion, resulting in the fused feature F. fusion1 F fusion2 F fusion3 and F fusion4 ;

[0078] Step 4: Secondary fusion, combining the initially fused features F fusion1 F fusion2 and F fusion3The data is sequentially input into the corresponding layers of the fine structure extraction branch for further processing, and then the features F are processed. fusion1 F fusion2 F fusion3 and F fusion4 The final semantic features Ffinal of the feature prior branch are fused in the second stage through the cross-modal fusion module to obtain the final encoder features F1, F2, F3 and F4;

[0079] In the cross-modal fusion module, since the features of the two branches differ in the number of channels and resolution, the cross-modal fusion module first uses 1×1 point convolution and size scaling operations to merge the features F of the prior branch. s Features F of the fine structure extraction branch m Alignment is performed in both channel and spatial dimensions, and the aligned features F s and F m The features are concatenated along the channel dimension, and then subjected to 1×1 point convolution, group normalization, and ReLU activation to obtain the intermediate feature F. mid =ReLU(GN(PConv(Concat(F s ,F m Two independent 1×1 point convolutions and group normalization operations are performed on Fmid, and two attention weight maps A1=GN(PConv(F) are generated using convolution modulation. mid )) and A2=GN(PConv(Fmid)), finally fusing features F fusion F is obtained by weighted summation and concatenation with residuals. fusion = A1⊙F s +A2⊙F m +F m .

[0080] Step 5: Decode and segment. Input the final encoder features F1, F2, F3 and F4 into the multi-scale feature complementary mapping decoder to obtain the final binary segmentation map.

[0081] The multi-scale feature complementarity mapping decoder consists of linear layers, deconvolutional upsampling layers, and a feature complementarity mapping module. Specifically, the input multi-scale features F1, F2, F3, and F4 are first adjusted for the number of channels using linear layers, and then upsampled to a uniform size using deconvolutional upsampling layers, as expressed by the following formula.

[0082] In the formula, UpSample represents a dynamic convolutional layer, and MLP represents a linear layer. The upsampled features are then concatenated along the channel dimension and fed into a feature complementarity mapping module. Through channel attention and spatial attention mechanisms, channel weights enhance the semantic guidance of the spatial representation, while spatial weights strengthen the spatial complementarity of the semantic representation. Finally, the convolutional layer outputs the final segmentation result image, represented as... .

[0083] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent image fine structure detection algorithm based on a dual-branch encoder, characterized in that, Includes the following steps: Step 1: Construction of the image segmentation system. The system includes a feature prior branch based on a lightweight pre-trained model, a fine structure extraction branch based on a structure-aware visual state space module, a cross-modal fusion module, and a multi-scale feature complementary mapping decoder. Step 2: Feature extraction. The original image is input into the feature prior branch and the fine structure extraction branch of the image segmentation system, respectively. The feature prior branch outputs a feature extraction function containing F. s1 F s2 F s3 F s4 Hierarchical features at four scales and the semantically rich F in the last layer final Five different levels of features; The fine structure extraction branch first segments the input image into image patches and adds position encoding, and then sends them to the SAVSS module for processing. The SAVSS module consists of a gated bottleneck convolutional layer, a two-dimensional selective scanning module with a structure-aware scanning strategy, a pixel attention-guided fusion module, and residual connections. Step 3: Preliminary fusion. Then, the features of each level are extracted from the fine structure branch and input together with the features of the same level from the feature prior branch into the cross-modal fusion module for the first stage of fusion, resulting in the fused feature F. fusion1 F fusion2 F fusion3 and F fusion4 ; Step 4: Secondary fusion, combining the initially fused features F fusion1 F fusion2 and F fusion3 The data is sequentially input into the corresponding layers of the fine structure extraction branch for further processing, and then the features F are processed. fusion1 F fusion2 F fusion3 and F fusion4 The final semantic features F of the feature prior branches are respectively final The second stage of fusion is performed through the cross-modal fusion module to obtain the final encoder features F1, F2, F3 and F4; Before fusion, the cross-modal fusion module first uses 1×1 point convolution and size scaling operations to convert the feature prior branch's features F. s Features F of the fine structure extraction branch m Alignment is performed in both channel and spatial dimensions, and the aligned features F s and F m The features are concatenated along the channel dimension, and then subjected to 1×1 point convolution, group normalization, and ReLU activation to obtain the intermediate feature F. mid =ReLU(GN(PConv(Concat(F s ,F m )))), for F mid Two independent 1×1 point convolutions and group normalization operations are performed respectively, and two attention weight maps A1=GN(PConv(F) are generated using convolution modulation. mid )) and A2=GN(PConv(F mid Finally, the feature F is fused. fusion F is obtained by weighted summation and concatenation with residuals. fusion = A1⊙F s +A2⊙F m +F m ; Step 5: Decode and segment. Input the final encoder features F1, F2, F3 and F4 into the multi-scale feature complementary mapping decoder to obtain the final binary segmentation map.

2. The image fine structure intelligent detection algorithm based on a dual-branch encoder according to claim 1, characterized in that: The training strategy for the lightweight pre-trained model in step one adopts a fine-tuning method that only updates the parameters of the normalization layer.

3. The image fine structure intelligent detection algorithm based on a dual-branch encoder according to claim 1, characterized in that: The specific process of the gated bottleneck convolutional layer is as follows: S1. For the input feature x, after bottleneck convolution, group normalization, and ReLU activation function, g1(x) and x1 are obtained, which are expressed by the following formula. g1(x)=ReLU(GN(BottConv(x))) x1=ReLU(GN(BottConv(g1(x)))) In the formula, BottConv represents bottleneck convolution, and GN represents group normalization; S2 and g2(x) are obtained by another path through bottleneck convolution, group normalization, and ReLUU activation function, and are represented by g2(x) = ReLU(GN(BottConv(x))). S3. Then, g2(x) and x1 are combined through the Hadamard product to obtain the gated feature map m(x) = g2(x) ⊙ x1; S4. Next, the gated feature map m(x) is further refined to obtain x2=ReLU(GN(BottConv(m(x)))); S5. Finally, perform a residual concatenation between X2 and the original input x to obtain the final output y = x2 + x.

4. The image fine structure intelligent detection algorithm based on a dual-branch encoder according to claim 1, characterized in that: The scanning strategy includes horizontal alternating scanning, vertical alternating scanning, main diagonal scanning, and secondary diagonal scanning. Horizontal alternating scanning starts from the last row and alternates between left to right and right to left. Vertical alternating scanning starts from the first column and alternates between top to bottom and bottom to top. Main diagonal scanning is performed along the diagonal direction from the upper left to the lower right. Secondary diagonal scanning is performed along the diagonal direction from the upper right to the lower left.

5. The image fine structure intelligent detection algorithm based on a dual-branch encoder according to claim 1, characterized in that: The pixel attention-guided fusion module is used to fuse the initial sequence feature x1 and the scanned sequence feature x2. The fusion process is expressed by the following formula. f1(x) =(Norm(BottConv(x1)) f2(x) = Norm(BottConv((x2))) σ = Sigmoid(Norm(BottConv( f2(x)⊙f1(x)))) y = (1-σ) ⊙ f1(x) + σ ⊙ f2(x) Where ⊙ represents the Hadamard product, Norm represents batch normalization, and BottConv represents the bottleneck convolution.

6. The image fine structure intelligent detection algorithm based on a dual-branch encoder according to claim 1, characterized in that: The multi-scale feature complementarity mapping decoder in step five consists of a linear layer, a deconvolutional upsampling layer, and a feature complementarity mapping module. Specifically, the input multi-scale features F1, F2, F3, and F4 are first adjusted for the number of channels using a linear layer, and then the deconvolutional upsampling layer is used to upsample each feature map to a uniform size, as expressed by the following formula. ； In the formula, UpSample represents a dynamic convolutional layer, and MLP represents a linear layer. The upsampled features are then concatenated along the channel dimension and fed into a feature complementarity mapping module. Through channel attention and spatial attention mechanisms, channel weights enhance the semantic guidance of the spatial representation, while spatial weights strengthen the spatial complementarity of the semantic representation. Finally, the convolutional layer outputs the final segmentation result image, represented as... .

Images