A small-scale target segmentation method based on hierarchical multi-scale feature fusion

Abstract

The application discloses a small-scale target segmentation method based on hierarchical multi-scale feature fusion, and belongs to the technical field of image processing and computer vision, and solves the problems of multi-scale fusion information redundancy and the like in the prior art. The method comprises the following steps: inputting an input image into an encoder; the encoder comprises n encoding modules which are sequentially cascaded; taking the encoding features of the deepest layer encoding module as the fusion features of the corresponding layer; starting from the fusion features of the deepest layer encoding module, the fusion features of the current layer encoding module and the encoding features of the previous layer encoding module are hierarchically fused in a top-down manner layer by layer to obtain the fusion features corresponding to each layer encoding module; inputting the fusion features of the deepest layer into a decoder; the decoder comprises n-1 decoding modules which are sequentially cascaded; the fusion features corresponding to each layer encoding module except the deepest layer are respectively taken as modulation signals and fed into the decoding modules of the corresponding layers of the decoder, and a segmentation probability map is generated by the decoder.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to a small-scale target segmentation method based on hierarchical multi-scale feature fusion. Background Technology

[0002] In recent years, with the rapid development of imaging technology and artificial intelligence, the widespread application of high-resolution real-time dynamic imaging equipment has provided a rich data foundation for accurate detection and recognition, while continuous breakthroughs in artificial intelligence technology have further promoted its deep integration in multiple fields. In intelligent visual analysis systems, image segmentation technology is a key link in improving recognition accuracy and intelligence. Accurate segmentation results can provide target location information in advance, assisting path planning and reducing operational risks; during real-time processing, dynamic segmentation can guide the system to adjust operations, avoiding target misjudgment or omission. Compared to large-scale targets, the segmentation task of small-scale targets is more challenging: the target size is small, feature information is scarce, and boundaries are blurred. In practical applications, small-scale targets are easily missed; therefore, achieving high-precision automatic segmentation of small-scale targets remains a pressing technical challenge.

[0003] The development of deep learning technology, especially convolutional neural networks and transformer networks, has significantly advanced small-scale target segmentation techniques. Compared to traditional segmentation methods, deep learning methods, with their powerful feature representation capabilities and end-to-end optimization mechanisms, can maintain high segmentation accuracy even under complex imaging conditions, and typically require no complex manual intervention. Meanwhile, continuous breakthroughs in various high-resolution imaging technologies have not only improved the detection sensitivity of small-scale targets but also provided high-quality labeled data for the training and testing of deep learning models, forming an important foundation for the development of deep learning.

[0004] Most current deep learning segmentation methods are based on encoder-decoder architectures, achieving segmentation through encoder feature extraction and decoder spatial probability map mapping, such as U-Net, FCN, Transformer models and their variants. Frequent downsampling operations in encoder-decoder models can lead to the loss of spatial information of small-scale targets, affecting model performance. Furthermore, existing methods improve the accuracy of small-scale target segmentation through multi-scale methods and attention mechanisms. Multi-scale methods extract features in parallel using multiple convolutional kernels of different sizes, improving the model's ability to perceive targets at different scales; however, this method may cause information redundancy, and the model's receptive field remains limited. Simultaneously, traditional multi-scale fusion strategies lack hierarchical modeling of the correlation between feature levels, making it difficult to effectively retain detailed localization information and contextual semantic information of small targets. This results in weak responses of small-scale targets in the fused feature map, susceptibility to interference from background or large targets, and low segmentation and detection accuracy. The introduction of attention mechanisms provides a new approach to selecting important information in small-scale target segmentation; however, since attention information is calculated from encoded features, the loss of encoded features can introduce bias into the attention process, and existing methods have not yet addressed the problem of correcting for biased encoded features.

[0005] In summary, the loss of spatial information of small-scale targets due to downsampling, the information redundancy and insufficient feature fusion brought about by multi-scale methods, and the attention calculation bias caused by the lack of encoded features all pose serious challenges to the stability and reliability of small-scale target segmentation results, i.e., the robustness of the algorithm. Summary of the Invention

[0006] Based on the above analysis, the embodiments of the present invention aim to provide a small-scale target segmentation method based on hierarchical multi-scale feature fusion, in order to solve the technical problems in the prior art such as information redundancy in multi-scale fusion, inability to effectively preserve the detailed localization and contextual semantic information of small targets, and attention calculation deviation caused by the loss of encoded features.

[0007] This invention proposes a small-scale target segmentation method based on hierarchical multi-scale feature fusion, the method comprising: The input image is fed into the encoder; wherein the encoder includes n sequentially cascaded encoding modules, and each encoding module outputs encoded features at a corresponding scale; The coding features of the deepest coding module are used as the fusion features of the corresponding layer. Starting from the fusion features of the deepest coding module, the fusion features of the current layer coding module are fused with the coding features of the previous layer coding module in a top-down manner to obtain the fusion features corresponding to each layer coding module. The deepest layer's fused features are input to the decoder; wherein, the decoder includes n-1 decoder modules cascaded in sequence; the fused features corresponding to each layer's encoding module, excluding the deepest layer, are fed as modulation signals into the corresponding layer's decoding module of the decoder, and the decoder generates a segmentation probability map.

[0008] Based on the above solution, the present invention also makes the following improvements: Furthermore, the step of performing layer-by-layer fusion of the fusion features of the current layer encoding module with the encoding features of the previous layer encoding module in a top-down manner includes: Upsample and adjust the channels of the fused features of the current layer coding module to align its spatial size and number of channels with the coding features of the previous layer coding module; The fused features of the current layer coding module and the coding features of the previous layer coding module after alignment are respectively segmented into blocks to obtain the corresponding first feature block set and second feature block set; Calculate the correlation between each feature block in the second feature block set and each feature block in the first feature block set to obtain a correlation matrix; A confidence map is generated based on the correlation matrix. The confidence map is used to characterize the probability that each spatial location in the coding features of the previous layer coding module belongs to the segmentation target. The confidence map is used to perform weighted fusion of the fusion features of the aligned current layer coding module and the coding features of the previous layer coding module to obtain the fusion features corresponding to the current layer coding module.

[0009] Furthermore, the step of segmenting the fused features of the aligned current layer coding module and the coding features of the previous layer coding module into blocks includes: With the first Each encoding module serves as the current layer encoding module. Using a core size of , interval is The sliding window expansion operation divides the aligned fused features of the current layer coding module and the coding features of the previous layer coding module into N groups of size . Feature blocks; in, Where N is the number of feature channels and N is the number of feature blocks.

[0010] Furthermore, after the slicing operation, the following steps are also included: Using a linear transformation with shared weights, the first feature block set and the second feature block set are mapped to a low-dimensional embedding space, respectively, to obtain the compressed first and second embedded feature block sets. Each feature vector in the first set of embedded feature blocks and the second set of embedded feature blocks is normalized using the L2 norm to obtain a normalized first set of feature blocks and a normalized second set of feature blocks.

[0011] Further, the step of calculating the correlation between each feature block in the second feature block set and each feature block in the first feature block set to obtain a correlation matrix includes: Perform a dot product operation between each eigenvector of the normalized second feature block set and each eigenvector of the normalized first feature block set to obtain an N×N correlation matrix. The element in the x-th row and y-th column of the correlation matrix represents the cosine similarity between the x-th feature block in the second feature block set and the y-th feature block in the first feature block set.

[0012] Furthermore, the step of generating a confidence map based on the correlation matrix includes: The correlation matrix is ​​normalized within each row to obtain a confidence vector, where each element represents the confidence that the corresponding feature block in the second feature block set belongs to the segmentation target. The confidence vector is reshaped into a two-dimensional confidence matrix; The two-dimensional confidence matrix is ​​upsampled so that its spatial size is consistent with the spatial size of the coding features of the previous layer coding module, thus obtaining the confidence map.

[0013] Furthermore, the step of using the confidence map to perform weighted fusion of the fused features of the aligned current layer coding module and the coding features of the previous layer coding module includes: The confidence value at each position in the confidence graph is used as the fusion weight of the corresponding position of the coding feature of the previous layer coding module, and 1 minus the confidence value is used as the fusion weight of the corresponding position of the fusion feature of the current layer coding module after alignment. We perform a weighted summation based on position to obtain the fusion feature corresponding to the current layer encoding module.

[0014] Furthermore, the method also includes: Select at least one intermediate layer's fusion feature from the fusion features corresponding to each layer, input it into the auxiliary detection branch, and generate the target detection result; A joint loss function is calculated based on the segmentation probability map and the target detection result. The joint loss function includes a segmentation loss term and a detection loss term. The joint loss function is used to train a segmentation model consisting of an encoder connected to a decoder.

[0015] Furthermore, the step of selecting at least one intermediate layer's fusion feature from the fusion features corresponding to each layer includes: The fusion features with spatial dimensions between 1 / 4 and 1 / 16 of the original input image size are selected as the input to the auxiliary detection branch.

[0016] Furthermore, the auxiliary detection branch includes a classification sub-branch and a regression sub-branch; wherein, The classification sub-branch is used to predict the probability that the anchor frame at each spatial location belongs to each category; The regression sub-branch is used to predict the positional offset of each anchor frame.

[0017] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: The small-scale target segmentation method based on hierarchical multi-scale feature fusion proposed in this invention has the following beneficial effects: (1) Effectively enhance the spatial feature representation of small-scale targets This invention employs a top-down, hierarchical, scale-wise fusion strategy, progressively injecting deep semantic information into shallow, high-resolution features, thereby avoiding the loss of small-scale target spatial information due to multiple downsampling. Simultaneously, by leveraging the confidence map calculated based on block correlation, it adaptively highlights the regions containing small targets, significantly improving their response intensity in the fused feature map.

[0018] (2) Reduce feature redundancy and expand the effective receptive field Unlike traditional multi-scale parallel convolution, which is prone to information redundancy and limited receptive field, this invention utilizes the extensive contextual information of high-level features to perform correlation matching with low-level local blocks. Without significantly increasing parameters, it imbues low-level features with semantic guidance, thereby balancing local details with global discriminative capabilities.

[0019] (3) Mitigating the impact of coding feature bias on attention mechanism Existing attention mechanisms rely directly on encoded features. When these features are biased due to operations such as downsampling, the accuracy of the attention weights decreases. This invention uses normalized correlation between high- and low-level feature blocks to calculate the fusion confidence, which is not directly affected by biases in a single encoded feature and exhibits strong robustness. Even when the quality of the encoded features is not ideal, the fusion weights can still be effectively calibrated.

[0020] (4) Improve the segmentation accuracy and robustness of the decoder This invention replaces the original encoded features with layered fusion features to modulate each level of the decoder, enabling the decoder to obtain enhanced features containing prior confidence in the small target space at each stage. Experiments show that this method can effectively reduce the false negative and false positive rates for small-scale medical targets (such as nodules, microtumors, and microvessels), demonstrating superior performance and stability compared to existing methods in small target segmentation tasks.

[0021] (5) Strong versatility and scalability The hierarchical multi-scale feature fusion module proposed in this invention is a lightweight design that can be plugged and played into various existing encoder-decoder network structures (such as U-Net, TransUNet, SwinUnet, etc.) without modifying the original network architecture, making it easy to deploy quickly on existing systems and improve performance.

[0022] In summary, the small-scale target segmentation scheme based on hierarchical multi-scale feature fusion proposed in this invention can effectively solve several problems existing in the prior art: loss of spatial information of small-scale targets due to encoder-decoder downsampling; lack of inter-level correlation modeling in existing multi-scale fusion schemes, which easily leads to defects such as information redundancy and limited receptive field; and the inability of attention mechanism to correct coding feature bias, which ultimately results in low response intensity of small-scale targets in the fused feature map, susceptibility to interference, low segmentation accuracy, and poor robustness.

[0023] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0024] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 A flowchart of a small-scale target segmentation method based on hierarchical multi-scale feature fusion provided in an embodiment of the present invention; Figure 2 A general design diagram of the hierarchical multi-scale feature fusion mechanism provided in the embodiments of the present invention; Figure 3 This is a schematic diagram illustrating the specific implementation of the hierarchical multi-scale feature fusion mechanism provided in an embodiment of the present invention. Detailed Implementation

[0025] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0026] A specific embodiment of the present invention discloses a small-scale target segmentation method based on hierarchical multi-scale feature fusion. This method fuses encoded features scale-wise from low to high scale, and then uses the fused encoded features to modulate the output of the decoder. The flowchart of this method is shown below. Figure 1 As shown in the figure, the overall design diagram of the hierarchical multi-scale feature fusion mechanism is as follows: Figure 2 As shown, the specific implementation method is explained below.

[0027] Step S1: Input the input image to the encoder; wherein the encoder includes n sequentially cascaded encoding modules, and each encoding module outputs encoded features at a corresponding scale.

[0028] In this embodiment, the segmentation model thus includes an encoder and a decoder. The encoder consists of n sequentially cascaded encoding modules. Input image The input of the first encoding module is used as the input terminal, and the output of the previous encoding module is the input of the next encoding module. Therefore, the first... Encoding modules The output is Represented as: (1) Step S2: Use the encoding features of the deepest encoding module as the fusion features of the corresponding layer; starting from the fusion features of the deepest encoding module, perform layer-by-layer fusion of the fusion features of the current layer encoding module with the encoding features of the previous layer encoding module in a top-down manner to obtain the fusion features corresponding to each layer encoding module.

[0029] Preferably, in this embodiment, the step of performing layer-by-layer fusion of the fusion features of the current layer encoding module and the encoding features of the previous layer encoding module in a top-down manner includes the following steps.

[0030] Step S21: Upsample and adjust the channels of the fusion features of the current layer coding module to align its spatial size and number of channels with the coding features of the previous layer coding module.

[0031] Step S22: Perform block segmentation operations on the fused features of the current layer coding module and the coding features of the previous layer coding module after alignment, respectively, to obtain the corresponding first feature block set and second feature block set.

[0032] The step of segmenting the fused features of the aligned current layer coding module and the coding features of the previous layer coding module into blocks includes: With the first Each encoding module is used as the current layer encoding module. Using a core size of , interval is The sliding window expansion operation divides the aligned fused features of the current layer coding module and the coding features of the previous layer coding module into N groups of size . Feature blocks; in, Where N is the number of feature channels and N is the number of feature blocks.

[0033] Preferably, after the slicing operation, the method further includes: using a linear transformation with shared weights to map the first feature block set and the second feature block set to a low-dimensional embedding space, respectively, to obtain a compressed first embedded feature block set and a second embedded feature block set; and performing L2 norm normalization on each feature vector in the first embedded feature block set and the second embedded feature block set to obtain a normalized first feature block set and a normalized second feature block set.

[0034] Step S23: Calculate the correlation between each feature block in the second feature block set and each feature block in the first feature block set to obtain the correlation matrix.

[0035] Preferably, the step of calculating the correlation between each feature block in the second feature block set and each feature block in the first feature block set to obtain a correlation matrix includes: Perform a dot product operation between each eigenvector of the normalized second feature block set and each eigenvector of the normalized first feature block set to obtain an N×N correlation matrix. The element in the x-th row and y-th column of the correlation matrix represents the cosine similarity between the x-th feature block in the second feature block set and the y-th feature block in the first feature block set.

[0036] Step S24: Generate a confidence map based on the correlation matrix. The confidence map is used to characterize the probability that each spatial location in the coding features of the previous layer coding module belongs to the segmentation target.

[0037] Preferably, generating a confidence map based on the correlation matrix includes: normalizing each row of the correlation matrix to obtain a confidence vector, where each element in the confidence vector represents the confidence that the corresponding feature block in the second feature block set belongs to the segmentation target; reshaping the confidence vector into a two-dimensional confidence matrix; and upsampling the two-dimensional confidence matrix so that its spatial size is consistent with the spatial size of the encoded features of the previous layer encoding module to obtain the confidence map.

[0038] Step S25: Use the confidence map to perform weighted fusion of the fusion features of the aligned current layer coding module and the coding features of the previous layer coding module to obtain the fusion features corresponding to the current layer coding module.

[0039] Preferably, the step of using the confidence map to perform weighted fusion of the fusion features of the aligned current layer coding module and the coding features of the previous layer coding module includes: using the confidence value of each position in the confidence map as the fusion weight of the corresponding position of the coding feature of the previous layer coding module, subtracting the confidence value from 1 as the fusion weight of the corresponding position of the fusion feature of the aligned current layer coding module; and performing weighted summation according to position to obtain the fusion feature corresponding to the current layer coding module.

[0040] In the specific implementation process, after obtaining the output of all encoding modules, from arrive Reverse scale-wise fusion: First, and By merging, we obtain Then and By merging, we obtain And so on. The above calculation process can be expressed by the following formula: (2) (3) in, For feature fusion module, for and The size of the cut pieces.

[0041] A schematic diagram illustrating the specific implementation of the hierarchical multi-scale feature fusion mechanism is shown below. Figure 3 As shown, the specific method of feature fusion corresponds to steps S21 to S25 above, and is described in detail below.

[0042] (1) To Using bilinear upsampling, it is spatially similar to... Align. Then use a The convolution operation makes its channel size equal to... Alignment. After the above operations, and The same size is denoted as Then use a core size of , interval is The sliding window expansion operation achieves feature segmentation, and this process will... and Transform into Size characteristics, among which The number of feature blocks, at this time and They are respectively denoted as and .

[0043] (2) Linear operations using shared weights and To achieve block embedding, and Mapped to size Features, here , This is the channel compression factor.

[0044] (4) (5) in For shared weights.

[0045] (3) To determine and The similarity of each feature block in the shallow spatial attention. For each image patch, calculate the features and depth features. The similarity of features of each spatial attention image patch. When the similarity is high, it can be considered that... This image patch contains foreground objects, otherwise it is noise or a false object. This process can be implemented using the following calculation. First, a normalization operation is applied to... and Normalize each d-dimensional feature vector in the dataset to obtain the normalized feature vector. and .

[0046] (6) (7) yes Norm.

[0047] (4) Calculate using vector dot product and The correlation is obtained by determining the correlation matrix. Since the norm of the normalized eigenvectors is 1, the dot product of the eigenvectors is equivalent to the cosine similarity: (8) To improve computational efficiency, the above process can be transformed into matrix multiplication, as follows: (9) (5) Since the area of ​​small-scale targets is smaller than the area of ​​the entire image, the cosine similarity between the feature vectors of small-scale targets and most other feature vectors should be low, while the feature similarity between background features should be high. Therefore, for each feature vector, the total similarity with all other feature vectors can be calculated, and the foreground and background can be distinguished based on this similarity: normalize each row of S to obtain the confidence vector. This confidence level is used to describe The probability that a patch in the image contains the segmented target: (10) (6) To undergo deformation and recombination, from Transform into Then Each position in the file is generated through a copy operation. This step can be achieved using the nearest neighbor upsampling operation to obtain a matrix of size . The confidence matrix.

[0048] (11) (12) (7) Finally, use confidence scores to test the confidence level. and The final fusion feature is obtained by performing a weighted summation. : (13) Step S3: Input the deepest layer's fused features into the decoder; wherein the decoder includes n-1 decoder modules cascaded in sequence; feed the fused features corresponding to each layer's encoding module (excluding the deepest layer) as modulation signals into the corresponding layer's decoding module of the decoder, and the decoder generates a segmentation probability map.

[0049] In the specific implementation process, this embodiment obtains the hierarchical fusion feature set. Then, it is fed as the modulation signal into the corresponding levels of the decoder. Specifically: for the decoder's... Layer (with encoder) (corresponding to layer space size), the original encoded features are no longer used. Instead, use the fused version. . Having undergone the aforementioned layer-by-layer fusion process, it simultaneously incorporates global semantic discrimination information from the deep layer and high-resolution spatial details from the shallow layer, and through confidence... Adaptive enhancement emphasizes the features of the region containing the small target. The decoder receives... Afterwards, further operations such as upsampling, convolution, and interaction with features from adjacent decoder layers can be performed to finally generate a segmentation probability map.

[0050] Furthermore, the method may further include: selecting at least one intermediate layer's fusion feature from the fusion features corresponding to each layer, inputting it into an auxiliary detection branch, and generating a target detection result; calculating a joint loss function based on the segmentation probability map and the target detection result, the joint loss function including a segmentation loss term and a detection loss term; and training a segmentation model composed of an encoder connected to a decoder using the joint loss function. Preferably, selecting at least one intermediate layer's fusion feature from the fusion features corresponding to each layer includes: selecting fusion features with a spatial size between 1 / 4 and 1 / 16 of the original input image size as input to the auxiliary detection branch. Preferably, the auxiliary detection branch includes a classification sub-branch and a regression sub-branch; wherein the classification sub-branch is used to predict the probability that the anchor box at each spatial location belongs to each category; and the regression sub-branch is used to predict the positional offset of each anchor box.

[0051] Specifically, the fusion features obtained based on the above method This embodiment also includes an auxiliary learning branch (i.e., an auxiliary detection branch). Based on the response scale of small-scale targets on the feature map, several levels suitable for detection are selected from the fused feature set. With the largest size, it retains the richest spatial details, making it suitable for detecting extremely small targets; As the number of fusion layers increases, the resolution decreases while the semantics are enhanced, making it suitable for detecting slightly larger small targets. The selection strategy is as follows: select fusion features with a spatial size between 1 / 4 and 1 / 16 of the original image as the detection layer, for example... , and Since segmentation is essentially pixel-level classification, the proportion of positive sample pixels is extremely low for small-scale targets. This results in extremely sparse gradient signals from small target regions during training, making it difficult for the model to gain sufficient "attention" to strengthen the feature representation of these tiny regions. The detection branch, however, provides a completely different type of supervision signal: target-level localization and bounding box regression. The detection head needs to predict the precise coordinate offset of the bounding box, which forces feature fusion. Encoding clear, sharp edge responses and accurate center point responses at the location of small targets is crucial. This stringent requirement for spatial accuracy precisely compensates for the weakness of segmentation tasks in being insensitive to blurred boundaries and localization offsets. For target detection tasks, even if the target is only a few dozen pixels long, a complete localization box is required. This forces features to learn the complete outline of the small target and its contrast relationship with the surrounding background, rather than just scattered activation regions. This information is directly passed to the segmentation decoder through a shared fusion feature layer, which helps to segment more complete small target regions.

[0052] A shared-weight detection head is attached to each selected fusion feature layer. This detection head contains two branches: a classification branch (input is the fusion feature) and a classification branch. After passing through several convolutional modules, and then through Convolution output One channel, among which The output represents the number of categories. The output is for each spatial location. Each anchor box has a probability of belonging to a category. Regression branch: Has the same structure as the classification branch, running parallel to it, and outputs... One channel, representing The offset of each anchor frame.

[0053] The output of the segmentation model for object detection is calculated. With the output of the segment decoder Then, the joint loss function for object detection and segmentation can be calculated.

[0054] (14) in, Let the target detection loss function be used. For the segmentation loss function; Labels for detecting real targets; The loss function provides the true pixel-level segmentation labels. This function can be used for fully supervised learning as well as weakly supervised learning, where some samples only have location labels but no pixel-level segmentation labels. When samples have pixel-level segmentation labels... When, it can be converted into At this point, fully supervised learning is used. , When the sample only has location labels and no pixel-level segmentation labels, , 0, .

[0055] In summary, the functional modules corresponding to the method proposed in this invention can be plugged into existing encoder-decoder segmentation models as plug-and-play modules. Existing methods (such as U-Net, FCN, etc.) typically employ simple skip connections or unidirectional multi-scale parallel fusion, lacking hierarchical modeling of the correlation between feature levels. This application proposes starting from the deepest encoding layer and fusing layer by layer downwards with the previous encoding layer to obtain the fused multi-scale features. This top-down hierarchical progressive fusion method can gradually transfer high-level semantic information to shallow-level detailed features, effectively alleviating the problem of small-scale target spatial information loss caused by downsampling. In addition, existing multi-scale fusion methods (such as multi-convolutional kernel parallel) are prone to information redundancy, and the attention mechanism may fail due to encoding feature deviations. The feature fusion mechanism designed in this invention can dynamically emphasize the feature contribution of the region where the small target is located, suppress irrelevant background or large target interference, and correct the possible deviations of low-level encoding features through the participation of high-level features.

[0056] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0057] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A small-scale target segmentation method based on hierarchical multi-scale feature fusion, characterized in that, The method includes: The input image is fed into the encoder; wherein the encoder includes n sequentially cascaded encoding modules, and each encoding module outputs encoded features at a corresponding scale; The coding features of the deepest coding module are used as the fusion features of the corresponding layer. Starting from the fusion features of the deepest coding module, the fusion features of the current layer coding module are fused with the coding features of the previous layer coding module in a top-down manner to obtain the fusion features corresponding to each layer coding module. The deepest layer's fused features are input to the decoder; wherein, the decoder includes n-1 decoder modules cascaded in sequence; the fused features corresponding to each layer's encoding module, excluding the deepest layer, are fed as modulation signals into the corresponding layer's decoding module of the decoder, and the decoder generates a segmentation probability map.

2. The small-scale target segmentation method based on hierarchical multi-scale feature fusion according to claim 1, characterized in that, The method of performing layer-by-layer fusion of the fusion features of the current layer encoding module with the encoding features of the previous layer encoding module in a top-down manner includes: Upsample and adjust the channels of the fused features of the current layer coding module to align its spatial size and number of channels with the coding features of the previous layer coding module; The fused features of the current layer coding module and the coding features of the previous layer coding module after alignment are respectively segmented into blocks to obtain the corresponding first feature block set and second feature block set; Calculate the correlation between each feature block in the second feature block set and each feature block in the first feature block set to obtain a correlation matrix; A confidence map is generated based on the correlation matrix. The confidence map is used to characterize the probability that each spatial location in the coding features of the previous layer coding module belongs to the segmentation target. The confidence map is used to perform weighted fusion of the fusion features of the aligned current layer coding module and the coding features of the previous layer coding module to obtain the fusion features corresponding to the current layer coding module.

3. The small-scale target segmentation method based on hierarchical multi-scale feature fusion according to claim 2, characterized in that, The step of segmenting the fused features of the aligned current layer coding module and the coding features of the previous layer coding module into blocks includes: With the first Each encoding module is used as the current layer encoding module. Using a core size of , interval is The sliding window expansion operation divides the aligned fused features of the current layer coding module and the coding features of the previous layer coding module into N groups of size . Feature blocks; in, Where N is the number of feature channels and N is the number of feature blocks.

4. The small-scale target segmentation method based on hierarchical multi-scale feature fusion according to claim 3, characterized in that, Following the slicing operation, the following also applies: Using a linear transformation with shared weights, the first feature block set and the second feature block set are mapped to a low-dimensional embedding space, respectively, to obtain the compressed first and second embedded feature block sets. Each feature vector in the first set of embedded feature blocks and the second set of embedded feature blocks is normalized using the L2 norm to obtain a normalized first set of feature blocks and a normalized second set of feature blocks.

5. The small-scale target segmentation method based on hierarchical multi-scale feature fusion according to claim 4, characterized in that, The step of calculating the correlation between each feature block in the second feature block set and each feature block in the first feature block set to obtain a correlation matrix includes: Perform a dot product operation between each eigenvector of the normalized second feature block set and each eigenvector of the normalized first feature block set to obtain an N×N correlation matrix. The element in the x-th row and y-th column of the correlation matrix represents the cosine similarity between the x-th feature block in the second feature block set and the y-th feature block in the first feature block set.

6. The small-scale target segmentation method based on hierarchical multi-scale feature fusion according to claim 5, characterized in that, The generation of the confidence map based on the correlation matrix includes: The correlation matrix is ​​normalized within each row to obtain a confidence vector, where each element in the confidence vector represents the confidence that the corresponding feature block in the second feature block set belongs to the segmentation target. The confidence vector is reshaped into a two-dimensional confidence matrix; The two-dimensional confidence matrix is ​​upsampled so that its spatial size is consistent with the spatial size of the coding features of the previous layer coding module, thus obtaining the confidence map.

7. The small-scale target segmentation method based on hierarchical multi-scale feature fusion according to claim 6, characterized in that, The step of using the confidence map to perform weighted fusion of the fused features of the aligned current layer coding module and the coding features of the previous layer coding module includes: The confidence value at each position in the confidence graph is used as the fusion weight of the corresponding position of the coding feature of the previous layer coding module, and 1 minus the confidence value is used as the fusion weight of the corresponding position of the fusion feature of the current layer coding module after alignment. The fusion feature corresponding to the current layer encoding module is obtained by performing a weighted summation based on position.

8. The small-scale target segmentation method based on hierarchical multi-scale feature fusion according to claim 7, characterized in that, The method further includes: Select at least one intermediate layer's fusion feature from the fusion features corresponding to each layer, input it into the auxiliary detection branch, and generate the target detection result; A joint loss function is calculated based on the segmentation probability map and the target detection result. The joint loss function includes a segmentation loss term and a detection loss term. The joint loss function is used to train a segmentation model consisting of an encoder connected to a decoder.

9. The small-scale target segmentation method based on hierarchical multi-scale feature fusion according to claim 8, characterized in that, The step of selecting at least one intermediate layer's fusion feature from the fusion features corresponding to each layer includes: The fusion features with spatial dimensions between 1 / 4 and 1 / 16 of the original input image size are selected as the input to the auxiliary detection branch.

10. The small-scale target segmentation method based on hierarchical multi-scale feature fusion according to claim 9, characterized in that, The auxiliary detection branch includes a classification sub-branch and a regression sub-branch; wherein... The classification sub-branch is used to predict the probability that the anchor frame at each spatial location belongs to each category; The regression sub-branch is used to predict the positional offset of each anchor frame.

Images