Ultrasonic image few-sample target detection method and system based on Mangbar model

Abstract

The invention discloses an ultrasonic image few-sample target detection method and system based on a Mangbar model, and the method comprises the following steps: extracting the image features of an ultrasonic image, and generating a suggestion box; performing position coding on the picture features, obtaining new picture features by using a global information enhancement module, and generating suggestion box features in combination with the generated suggestion box; obtaining new suggestion box features by using a category relationship learning module; generating a suggestion box classification score and a coordinate prediction value; parameter updating is carried out, and if the training round reaches the maximum number of iterations, the process is ended; otherwise, repeating the steps. A Mamba model is used for obtaining global information, and the Mamba model shows deeper semantic understanding and performance enhancement potential in context learning by means of the capabilities of context modeling and long-range dependency relationship capturing.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method and system for detecting a small number of targets in an ultrasound image based on a Mamba model. Background Art

[0002] Ultrasound is a commonly used medical imaging technology that uses ultrasound to obtain images of the internal structure of the human body. Ultrasound images have the advantages of being non-invasive, low-cost, and real-time, so they are widely used in clinical practice to detect and diagnose a variety of diseases, such as tumors, cardiovascular diseases, gynecological diseases, etc. Compared with other medical images, ultrasound imaging has more artifacts and noise, low contrast between tissues leading to blurred boundaries, highly subjective diagnosis, and high dependence on experienced doctors. In order to overcome these problems, many computer-aided diagnosis systems have been developed to assist doctors in improving the efficiency and accuracy of diagnosis. With the continuous increase in the amount of medical imaging data and the limited medical resources, automated auxiliary diagnosis technology has become increasingly important. Ultrasound image target detection can identify and locate regions of interest, provide assistance for subsequent medical diagnosis, treatment, and segmentation tasks, and is widely used in tumor detection and prenatal diagnosis.

[0003] In recent years, with the rapid development of deep learning technology, especially the emergence of convolutional neural networks (CNN), deep learning has achieved great success in the field of medical image analysis. Compared with traditional image analysis methods, deep learning-based methods have more outstanding capabilities in extracting invariant features, and do not require manual feature design, and have excellent performance and speed. In the field of object detection, various CNN-based network architectures have been proposed and improved, such as Faster R-CNN, YOLO, SSD, etc. These methods have achieved significant improvements in detection speed and accuracy, making deep learning-based object detection methods gradually become the mainstream method of medical image object detection. However, deep learning-based object detection methods still have shortcomings: the accuracy and precision of deep learning models depend largely on a large number of training sets. The scarcity of labeled data is a problem faced by these methods. This problem may not cause major problems in many medical scenarios, but in some cases, a large amount of data cannot be obtained due to ethical and privacy regulations, and for some rare cases such as rare diseases, it is difficult to obtain available samples, and deep learning models ultimately cannot provide satisfactory detection results.

[0004] Few-shot learning aims to improve the accuracy and generalization ability of the model in identifying new categories by training on rich basic category data and a small amount of new category data, providing a way to solve the problem of scarcity of medical image training data for deep learning models. Few-shot learning is widely used in the research of medical image analysis, such as tumor segmentation, disease classification, and image registration. A large number of studies on few-shot learning of medical images have shown that in most scenarios, few-shot learning can overcome the scarcity of medical image training data. Although there are a large number of studies on few-shot learning of medical images and few-shot object detection on natural images, there is still a lack of research on few-shot object detection in medical images. Therefore, in order to solve the problem of scarce ultrasound image data and assist doctors in improving the efficiency and accuracy of diagnosis, a few-shot object detection method for ultrasound images is needed. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for detecting small-sample targets in ultrasound images based on the Mamba model in order to address the above-mentioned problems.

[0006] The technical solution of the present invention is as follows:

[0007] A method for detecting a small number of targets in an ultrasound image based on a Mamba model comprises the following steps:

[0008] Extract the image features of the ultrasound image and generate suggestion boxes;

[0009] Encode the image features by position, use the global information enhancement module to obtain new image features, and combine them with the generated suggestion boxes to generate suggestion box features;

[0010] Use the category relationship learning module to obtain new suggestion box features;

[0011] Generate classification scores and coordinate prediction values ​​for suggestion boxes;

[0012] Update the parameters. If the training round reaches the maximum number of iterations, the training ends; otherwise, repeat the above steps.

[0013] Furthermore, the global information enhancement module includes a Mamba layer and a feature fusion operation, and the Mamba layer is used to capture the global information in the image.

[0014] Furthermore, the specific operation steps of the global information enhancement module include:

[0015] Reshape the acquired image features of dimensions (B, C, H, W) into dimensions (B, C, N), where B, C, H, W represent batch size, number of channels, height, and width respectively, N is the number of tokens and N = H × W;

[0016] The reshaped data enters the normalization layer for layer normalization;

[0017] The normalized data enters two branches respectively. The data of one branch passes through the linear layer, 1D convolution layer, SSM layer, and SiLU layer successively. The data of the other branch passes through the linear layer and SiLU layer successively. Finally, the data obtained from the two branches are multiplied and the features are output;

[0018] Reshape the output features and restore their dimensions to (B, C, H, W);

[0019] The output features are reduced in dimension using a 1×1 convolution operation and then concatenated with the initially acquired features in the second dimension. The concatenated features are again subjected to a 1×1 convolution operation for dimensionality reduction to obtain feature outputs of the same dimension as the input features.

[0020] Furthermore, the category relationship learning module includes an external attention operation, which reshapes the input features into (B, C, N) and then performs the external attention operation.

[0021] Furthermore, the category relationship learning module includes the following steps:

[0022] Single external attention layer: The input features are multiplied by the external matrix and then normalized. The obtained data is multiplied by the external attention matrix K to obtain the initial attention weight. After softmax and normalization operations, it is multiplied by the external attention matrix V to obtain the final attention weight. The attention weight is added to the input features at the element level and output.

[0023] Multiple outer attention layers: The output of the previous outer attention layer is used as the input of the next outer attention layer, and the operations in the outer attention layer are repeated.

[0024] The features output by the last outer attention layer are reshaped into (B, C, H, W) and then output.

[0025] Furthermore, the position encoding of the image features specifically includes the following steps:

[0026] The positions are encoded using the following formula:

[0027] C x [i,j]=i / W;

[0028] C y [i,j]=j / H;

[0029] Where C, W, and H are the number of channels, width, and height of the input features respectively;

[0030] Feature fusion: The two position (x and y) encodings are concatenated with the input features, and a 1×1 convolution operation is used to reduce the dimension of the concatenated features to obtain features with the same dimension as the input features.

[0031] The present invention also includes a small sample target detection system for ultrasonic images based on the Mamba model, comprising:

[0032] Backbone network: extracts image features of ultrasound images;

[0033] Position encoding module: performs position encoding on the image features extracted by the backbone network;

[0034] Global information enhancement module: captures the global information in the features output by the position encoding module;

[0035] Feature alignment and pooling module: Generates proposal box features based on the proposal boxes output by the region proposal network and the features delivered by the global information enhancement module;

[0036] Category relationship learning module: performs external attention operations on the features output by the feature alignment and pooling modules;

[0037] Detection head: Outputs the suggestion box category score and suggestion box position based on the features output by the category relationship learning module.

[0038] Furthermore, the global information enhancement module includes a Mamba layer and a feature fusion operation, and the Mamba layer is used to capture the global information in the image.

[0039] Furthermore, the category relationship learning module includes an external attention operation, which reshapes the input features into (B, C, N) and then performs the external attention operation.

[0040] Furthermore, the position encoding module includes position encoding and feature fusion operations.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. A method and system for detecting few-shot targets in ultrasound images based on the Mamba model. The Mamba model is used to obtain global information. The Mamba model shows the potential for deeper semantic understanding and performance enhancement in context learning by virtue of its ability to model context and capture long-range dependencies. The Mamba model can more efficiently obtain global information in ultrasound images, enhance the model's understanding of the overall context, reduce the interference of a large amount of noise and artifacts in ultrasound images, and improve detection performance and robustness.

[0043] 2. A few-shot target detection method and system for ultrasound images based on the Mamba model uses external attention to learn the semantic and spatial relationships between categories. By encoding the position information of the suggestion box into the feature information, the use of external attention can efficiently learn the semantic and spatial relationships between anatomical structure categories at a low computational cost, helping the model to use these relationships to better detect and locate targets, thereby improving the detection performance and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flowchart of a few-sample target detection method for ultrasound images based on the Mamba model.

[0045] Figure 2 This is a framework diagram of a few-sample target detection system for ultrasound images based on the Mamba model. DETAILED DESCRIPTION

[0046] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0047] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0048] The technical problems that this application intends to solve are:

[0049] Limitations in obtaining global information: Due to low contrast, the boundaries of targets in ultrasound images are often not obvious, and local information is difficult to effectively distinguish the difference between tissues or lesions and the surrounding environment. In addition, ultrasound images are easily disturbed by noise, artifacts, etc., resulting in blurred and unclear local information. Therefore, relying solely on local features may not accurately detect the target. Global information can provide the model with the overall context of the ultrasound image, enabling the model to make inferences based on the information of the entire image rather than relying solely on local features. In addition, some features in ultrasound images may be distributed in different areas of the image, and local features are difficult to capture these long-range dependencies, while global information can help the model understand these long-range relationships. Therefore, for target detection models, how to efficiently obtain global information, enhance the model's understanding of the overall context, and reduce the interference of a large amount of noise and artifacts in ultrasound images is the key to improving detection performance and robustness. However, traditional models are limited in their ability to obtain global information: CNN is limited to its local receptive field when obtaining global information; although Transformer is better at capturing long-range dependencies, it faces challenges in high computational complexity and data requirements.

[0050] Insufficient use of spatial relationships between categories: Due to the inherent characteristics of the human body structure, there are strong semantic and spatial correlations between anatomical structures in ultrasound images. The model can use these relationships to better detect and locate targets, especially in few-shot scenarios, when there are insufficient samples of certain categories, the model can infer the target of that category based on the positional relationship of other targets. In addition, in ultrasound images, targets of different categories (such as lesions and surrounding normal tissues) may look similar, leading to classification errors. Spatial relationships help the model to make inferences based on the location of the target in the image, thereby reducing confusion. Previous few-shot learning methods focused on learning semantic relationships when learning relationships between categories, while ignoring the learning of spatial relationships. On the other hand, previous few-shot learning methods mainly used graph neural networks or transformers to learn semantic relationships between categories, which has a high computational cost.

[0051] See also Figure 1-2 , a few-shot target detection method for ultrasound images based on the Mamba model, such as Figure 1 As shown, the following steps are included:

[0052] Extract the image features of the ultrasound image and generate suggestion boxes;

[0053] Encode the image features by position, use the global information enhancement module to obtain new image features, and combine them with the generated suggestion boxes to generate suggestion box features;

[0054] Use the category relationship learning module to obtain new suggestion box features;

[0055] Generate classification scores and coordinate prediction values ​​for suggestion boxes;

[0056] Update the parameters. If the training round reaches the maximum number of iterations, the training ends; otherwise, repeat the above steps.

[0057] The global information enhancement module includes the Mamba layer and feature fusion operations. The Mamba layer is used to capture the global information in the image. The specific operation steps of the global information enhancement module include:

[0058] Reshape the image features with dimensions (B, C, H, W) to (B, C, N), where B, C, H, W represent batch size, number of channels, height, and width, respectively, and N represents the number of tokens and N = H × W;

[0059] The reshaped data enters the normalization layer for layer normalization;

[0060] The normalized data enters two branches respectively. The data of one branch passes through the linear layer, 1D convolution layer, SSM (Structured State Space Model) layer, SiLU (Sigmoid Linear Unit) layer, and the data of the other branch passes through the linear layer and SiLU layer. Finally, the data obtained from the two branches are multiplied and the features are output;

[0061] Reshape the output features and restore their dimensions to (B, C, H, W);

[0062] The output features are reduced in dimension using a 1×1 convolution operation and then concatenated with the initially acquired features in the second dimension. The concatenated features are again subjected to a 1×1 convolution operation for dimensionality reduction to obtain feature outputs of the same dimension as the input features.

[0063] The category relationship learning module includes an external attention operation, which reshapes the input features into (B, C, N) and then performs the external attention operation. The category relationship learning module includes the following steps:

[0064] Single external attention layer: The input features are multiplied by the external matrix and then normalized. The obtained data is multiplied by the external attention matrix K to obtain the initial attention weight. After softmax and normalization operations, it is multiplied by the external attention matrix V to obtain the final attention weight. The attention weight is added to the input features at the element level and output.

[0065] Multiple outer attention layers: The output of the previous outer attention layer is used as the input of the next outer attention layer, and the operations in the outer attention layer are repeated.

[0066] The features output by the last outer attention layer are reshaped into (B, C, H, W) and then output.

[0067] Position encoding of image features specifically includes the following steps:

[0068] The positions are encoded using the following formula:

[0069] C x [i,j]=i / W;

[0070] C y [i,j]=j / H;

[0071] Where C, W, H are the number of channels (Channels), width (Width) and height (Height) of the input features respectively;

[0072] Feature fusion: The two position (x and y) encodings are concatenated with the input features, and a 1×1 convolution operation is used to reduce the dimension of the concatenated features to obtain features with the same dimension as the input features.

[0073] like Figure 2 As shown, the present application also includes a few-sample target detection system for ultrasound images based on the Mamba model, including:

[0074] Backbone network: extracts image features of ultrasound images;

[0075] Position encoding module: performs position encoding on the image features extracted by the backbone network;

[0076] Global information enhancement module: captures the global information in the features output by the position encoding module;

[0077] Feature alignment and pooling module: Generates proposal box features based on the proposal boxes output by the region proposal network and the features delivered by the global information enhancement module;

[0078] Category relationship learning module: performs external attention operations on the features output by the feature alignment and pooling modules;

[0079] Detection head: Outputs the suggestion box category score and suggestion box position based on the features output by the category relationship learning module.

[0080] The global information enhancement module includes a Mamba layer and feature fusion operations. The Mamba layer is used to capture the global information in the image.

[0081] The category relationship learning module includes an external attention operation, which reshapes the input features into (B, C, N) and then implements the external attention operation.

[0082] The position encoding module includes position encoding and feature fusion operations.

[0083] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A method for detecting target in ultrasound images with few samples based on the Mamba model, characterized in that: The following steps are involved: Extract the image features of the ultrasound image and generate suggestion boxes; Encode the image features by position, use the global information enhancement module to obtain new image features, and combine them with the generated suggestion boxes to generate suggestion box features; Use the category relationship learning module to obtain new suggestion box features; Generate classification scores and coordinate prediction values ​​for suggestion boxes; Update the parameters. If the training round reaches the maximum number of iterations, the training ends; otherwise, repeat the above steps.

2. The method for detecting target in ultrasound images with a small number of samples based on the Mamba model according to claim 1, characterized in that: The global information enhancement module includes a Mamba layer and a feature fusion operation, and the Mamba layer is used to capture the global information in the image.

3. The method for detecting target in ultrasound images with a small number of samples based on the Mamba model according to claim 2, characterized in that: The specific operation steps of the global information enhancement module include: Reshape the image features with dimensions (B, C, H, W) to (B, C, N), where B, C, H, W represent the batch size, number of channels, height, and width respectively, N is the number of tokens and N = H × W; The reshaped data enters the normalization layer for layer normalization; The normalized data enters two branches respectively. The data of one branch passes through the linear layer, 1D convolution layer, SSM layer, and SiLU layer successively. The data of the other branch passes through the linear layer and SiLU layer successively. Finally, the data obtained from the two branches are multiplied and the features are output; Reshape the output features and restore their dimensions to (B, C, H, W); The output features are reduced in dimension using a 1×1 convolution operation and then concatenated with the initially acquired features in the second dimension. The concatenated features are again subjected to a 1×1 convolution operation for dimensionality reduction to obtain feature outputs of the same dimension as the input features.

4. The method for detecting target in ultrasound images with a small number of samples based on the Mamba model according to claim 3, characterized in that: The category relationship learning module includes an external attention operation, which reshapes the input features into (B, C, N) and then performs the external attention operation.

5. The method for detecting target in ultrasound images with a small number of samples based on the Mamba model according to claim 4, characterized in that: The category relationship learning module includes the following steps: Single external attention layer: The input features are multiplied by the external matrix and then normalized. The obtained data is multiplied by the external attention matrix K to obtain the initial attention weight. After softmax and normalization operations, it is multiplied by the external attention matrix V to obtain the final attention weight. The attention weight and the input features are added element-wise and output; Multiple external attention layers: the output of the previous external attention layer is used as the input of the next external attention layer, and the operation in the external attention layer is repeated; The features output by the last outer attention layer are reshaped into (B, C, H, W) and then output.

6. The method for detecting target in ultrasound images with a small number of samples based on the Mamba model according to claim 1, characterized in that: The position encoding of the image features specifically includes the following steps: The positions are encoded using the following formula: C x [i,j]=i / W; C y [i,j]=j / H; Where C, W, and H are the number of channels, width, and height of the input features respectively; Feature fusion: The two position (x and y) encodings are concatenated with the input features, and a 1×1 convolution operation is used to reduce the dimension of the concatenated features to obtain features with the same dimension as the input features.

7. A few-sample target detection system for ultrasound images based on the Mamba model, characterized in that: include: Backbone network: extracts image features of ultrasound images; Position encoding module: performs position encoding on the image features extracted by the backbone network; Global information enhancement module: captures the global information in the features output by the position encoding module; Feature alignment and pooling module: Generates proposal box features based on the proposal boxes output by the region proposal network and the features delivered by the global information enhancement module; Category relationship learning module: performs external attention operations on the features output by the feature alignment and pooling modules; Detection head: Outputs the suggestion box category score and suggestion box position based on the features output by the category relationship learning module.

8. The ultrasound image few-sample target detection system based on the Mamba model according to claim 7, characterized in that: The global information enhancement module includes a Mamba layer and a feature fusion operation, and the Mamba layer is used to capture the global information in the image.

9. The ultrasound image few-sample target detection system based on the Mamba model according to claim 7, characterized in that: The category relationship learning module includes an external attention operation, which reshapes the input features into (B, C, N) and then performs the external attention operation.

10. The ultrasound image few-sample target detection system based on the Mamba model according to claim 7, characterized in that: The position encoding module includes position encoding and feature fusion operations.

Images