A medical image retrieval method and system

Abstract

This invention proposes a medical image retrieval method and system, relating to the field of medical image processing. It addresses the problem that existing technologies cannot deeply understand the intrinsic content of medical images, and struggle with issues such as high-resolution, multimodal data, and background noise interference, resulting in inaccurate and ineffective retrieval of corresponding medical images, and poor retrieval robustness and accuracy. The method includes: acquiring a medical image dataset; preprocessing the medical image dataset; using a trained feature extraction model to extract features and obtain image fusion features; constructing an image feature library based on the image fusion features; using the trained feature extraction model to extract features from the medical images to be retrieved, obtaining the fusion features of the medical images to be retrieved; and matching these features with the image feature library to retrieve the corresponding images. This invention significantly improves the accuracy and efficiency of medical image retrieval.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing, and in particular relates to a medical image retrieval method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Image retrieval is the process of searching for similar images based on their visual content, and it has significant applications in the field of medical imaging. In medicine, doctors often rely on examining patients' pathological images and related medical records to make diagnoses and develop treatment plans; medical images play a crucial role in patient diagnosis and treatment. Therefore, doctors frequently need to retrieve valuable images from various medical image databases to improve their clinical practice.

[0004] However, medical image retrieval differs from general image retrieval in the following ways:

[0005] (1) Strong domain specificity: Medical imaging contains multimodal data (such as CT, MRI, ultrasound), requiring algorithms to be optimized for specific anatomical regions and lesion characteristics;

[0006] (2) Data quality challenges: Medical images are often accompanied by noise, artifacts and uneven resolution, and effective preprocessing is required to improve image quality;

[0007] (3) High accuracy requirement: Medical image retrieval results directly affect clinical diagnosis, requiring the model to have higher robustness and accuracy.

[0008] Therefore, medical image retrieval not only needs to quickly find content similar to the target image from the database, but also needs to pay attention to subtle differences in lesions in order to assist doctors in diagnosis and decision-making.

[0009] Currently, the inventors have found that existing medical image retrieval technologies often fail to deeply understand the intrinsic content of images, making it difficult to distinguish between highly similar cases with different diagnoses due to minor differences. Although feature fusion-based descriptors have achieved significant success in the field of image retrieval, in medical image applications, they cannot accurately and effectively retrieve corresponding medical images due to problems such as high resolution, multimodal data, and background noise interference, resulting in poor robustness and accuracy of the retrieval algorithms. Summary of the Invention

[0010] To overcome the shortcomings of the prior art, this invention provides a medical image retrieval method and system. By constructing the Swin-AttnNet feature extraction model for medical image retrieval, it accurately and comprehensively extracts multi-scale local and global features of medical images, thereby significantly improving the accuracy and efficiency of medical image retrieval.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0012] Firstly, a medical image retrieval method is disclosed, including:

[0013] Obtain the medical image to be retrieved, wherein the medical image contains at least medical image data and tags;

[0014] The medical images to be retrieved are input into the trained feature extraction model, and the multi-scale local feature module is used to extract the multi-scale local features of the medical images to be retrieved.

[0015] Global features are extracted from the medical images to be retrieved using the global feature module;

[0016] The multi-scale local and global features of the medical image to be retrieved are fused together to obtain the fused features of the image to be retrieved.

[0017] In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are searched to obtain the corresponding image;

[0018] The process of constructing the image feature library of the feature extraction model is as follows: using the trained feature extraction model, features are extracted from the acquired image dataset, and the fused features of each extracted image are stored in the database to obtain the image feature library of the feature extraction model.

[0019] As a further technical solution, the feature extraction model includes a backbone network module, a multi-scale local feature extraction module, and a global feature extraction module;

[0020] During training, the training set is input into the backbone network module, and the first input features are obtained by downsampling by 2 times.

[0021] The first input feature is downsampled by a factor of 2 to obtain the second input feature;

[0022] The second input feature is downsampled by a factor of 6 to obtain the third input feature;

[0023] The second input feature is input into the multi-scale local feature extraction module for feature extraction to obtain multi-scale local features;

[0024] The third input feature is fed into the global feature extraction module for feature extraction to obtain the global feature.

[0025] Multi-scale local features are fused with global features to obtain the fused features of the fused image.

[0026] As a further technical solution, the acquired medical image dataset is preprocessed before the training process to obtain a preprocessed dataset. The specific process is as follows:

[0027] Traverse the image files in the medical image dataset and match each image file with its corresponding label.

[0028] For image files, construct an image dictionary;

[0029] Add complete image paths to the image dictionary;

[0030] Split the image attributes in the image dictionary into separate lists;

[0031] Based on the image attributes, the image labels are re-labeled as continuous values ​​to obtain the preprocessed dataset.

[0032] As a further technical solution, the second input feature is input into a multi-scale local feature extraction module for feature extraction to obtain multi-scale local features. The multi-scale local feature module includes an overlapping window partitioning module, a multi-head self-attention window module, a feedforward neural network, a window merging module, convolutional layers, layer normalization, and fully connected layers. The specific process is as follows:

[0033] The second input feature map is divided into small windows by the overlapping window partitioning module, and the first output feature map of each window is obtained.

[0034] The first output feature map is input into the window multi-head self-attention module for attention calculation, and the second output feature map of each window is obtained.

[0035] The second output feature map is input into the feedforward neural network for nonlinear transformation to obtain the third output feature map for each window;

[0036] The window merging module merges the third output feature maps of each window to obtain the fourth output feature map.

[0037] The number of channels in the fourth output feature map is adjusted through convolutional layers;

[0038] The fourth output feature map is standardized by batch standard layer to obtain standardized output features;

[0039] The standardized output features are input into a fully connected layer for mapping to obtain multi-scale local features.

[0040] As a further technical solution, the third input feature is fed into the global feature extraction module for feature extraction to obtain global features. The global feature extraction module includes convolutional layers, GeM pooling layers, and fully connected layers. The specific process is as follows:

[0041] The number of channels for the third input feature is adjusted using convolutional layers;

[0042] The third input feature is fed into the GeM pooling layer for pooling to obtain the fifth output feature map;

[0043] The fifth output feature map is mapped through a fully connected layer to obtain global features.

[0044] As a further technical solution, multi-scale local features and global features are fused to obtain the fused features of the fused image. Specifically, the multi-scale features and global features are concatenated in the last channel through a cross function to obtain the fused features of the image.

[0045] As a further technical solution, the fusion features of the image to be retrieved are searched in the image feature library of the feature extraction model to obtain the corresponding image. Specifically, the similarity search of the fusion features of the image to be retrieved is performed in the image feature library using the dot product method to obtain the retrieval results and output them.

[0046] Secondly, a medical image retrieval system is disclosed, comprising:

[0047] An image acquisition module is used to acquire medical images to be retrieved, wherein the medical images contain at least medical image data and tags;

[0048] The image retrieval module is used to input preprocessed medical images into a trained feature extraction model, and to extract multi-scale local features of the medical images to be retrieved using the multi-scale local feature module.

[0049] Global features are extracted from the medical images to be retrieved using the global feature module;

[0050] The multi-scale local and global features of the medical image to be retrieved are fused together to obtain the fused features of the image to be retrieved.

[0051] In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are searched to obtain the corresponding image;

[0052] The process of constructing the image feature library of the feature extraction model is as follows: using the trained feature extraction model, features are extracted from the acquired image dataset, and the fused features of each extracted image are stored in the database to obtain the image feature library of the feature extraction model.

[0053] A third aspect of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described medical image retrieval method.

[0054] A fourth aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above-described medical image retrieval method.

[0055] The above one or more technical solutions have the following beneficial effects:

[0056] In this embodiment, a novel model for medical image retrieval is constructed. This model uses an improved Swing Transformer as the backbone feature extraction network to process images in the dataset. An optimized multi-scale local feature extraction module, Self-ATT, efficiently extracts multi-scale local features. Then, the GeM (Generalized-Mean) pooling layer of the global feature extraction module generates global feature representations. Finally, a weighted feature fusion method further integrates multi-scale local and global feature information to generate more accurate image descriptors. The feature extraction model in this embodiment considers the challenges of high resolution, multimodal data, and background noise interference faced by medical image retrieval, significantly improving the accuracy and efficiency of image retrieval.

[0057] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0058] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0059] Figure 1 This is a schematic diagram of a medical image retrieval model architecture in this embodiment.

[0060] Figure 2 This is a schematic diagram of the multi-scale feature extraction module architecture in Embodiment 1.

[0061] Figure 3 This is a schematic diagram of the global feature extraction module architecture in this embodiment. Detailed Implementation

[0062] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0063] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0064] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0065] Example 1

[0066] This embodiment discloses a medical image retrieval method.

[0067] To more clearly illustrate this embodiment, the implementation process of the medical image retrieval method can be specifically described as follows:

[0068] A medical image retrieval method, comprising:

[0069] S1. Obtain the medical image to be retrieved, wherein the medical image contains at least medical image data and tags;

[0070] S2. Input the medical image to be retrieved into the trained feature extraction model, and use the multi-scale local feature module to extract the multi-scale local features of the medical image to be retrieved.

[0071] Global features are extracted from the medical images to be retrieved using the global feature module;

[0072] The multi-scale local and global features of the medical image to be retrieved are fused together to obtain the fused features of the image to be retrieved.

[0073] In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are searched to obtain the corresponding image;

[0074] The process of constructing the image feature library of the feature extraction model is as follows: using the trained feature extraction model, features are extracted from the acquired image dataset, and the fused features of each extracted image are stored in the database to obtain the image feature library of the feature extraction model.

[0075] like Figure 1 As shown, in step S1, the medical image to be retrieved is obtained, and the medical image contains at least medical image data and tags.

[0076] In this embodiment, a medical image of a breast tumor was obtained, containing relevant medical data and tags, with the tag being benign adenomas.

[0077] like Figure 1 As shown, in step S2, the medical image to be retrieved is input into the trained feature extraction model, and the multi-scale local feature module is used to extract the multi-scale local features of the medical image to be retrieved.

[0078] Global features are extracted from the medical images to be retrieved using the global feature module;

[0079] The multi-scale local and global features of the medical image to be retrieved are fused together to obtain the fused features of the image to be retrieved.

[0080] In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are searched to obtain the corresponding image.

[0081] Before inputting the medical images to be retrieved into the trained feature extraction model, an image dataset is first obtained, a feature extraction model is constructed, and the obtained image dataset is input into the constructed feature extraction model for training and testing. During the training and testing process, features are extracted, and an image feature library of the feature extraction model is constructed.

[0082] S2-1. Obtain a medical image dataset, which includes at least medical image data and labels.

[0083] The BreaKHis_v1 dataset was used as a medical image dataset, and image data and labels were obtained from the BreaKHis_v1 dataset.

[0084] S2-2. Preprocess the medical image dataset to obtain the preprocessed dataset; divide the preprocessed dataset into training set and test set.

[0085] In this embodiment, a preprocessing module is constructed to correspond images with labels, having the following structure [(image, label)]. The preprocessed dataset P is divided into a training set P1 and a test set P2 in a ratio of 9:1.

[0086] S2-2-1. Preprocess the medical image dataset to obtain the preprocessed dataset.

[0087] The medical image files and labels are preprocessed to obtain a preprocessed dataset. The specific process is as follows:

[0088] (1) Traverse the image files from the medical image dataset and match the image files with the labels one by one.

[0089] In this embodiment, the contents of the CSV file (landmark_ID / tag->image) are loaded, and the image files are traversed from the dataset directory using the os library and the tf.io.gfile module.

[0090] (2) For image files, construct an image dictionary;

[0091] In this embodiment, an image dictionary named images is constructed (key = image_id, value = {label, file_id}).

[0092] Specifically, 1) Create an empty dictionary named "images" to store information related to each image.

[0093] Then iterate through the data frame: iterate through each row of the data CSV file, extract the landmark_id (i.e., label) from the current row, assign it to the variable label, and extract the images column, which contains one or more image file IDs separated by spaces.

[0094] 2) Split the file IDs and update the dictionary. Perform the following operations for each file ID:

[0095] In the dictionary images, create a new nested dictionary using the file ID (file_id) as the key;

[0096] Then, the image label is stored in a nested dictionary with the key 'label'; at the same time, the file_id itself is also stored in the nested dictionary with the key 'file_id'.

[0097] (3) Add complete image paths to the image dictionary.

[0098] 1) Obtain all image paths.

[0099] The tf.io.gfile.glob() function reads all image file paths with the .jpg extension under the specified directory image_dir and stores these paths in the list image_paths.

[0100] Perform the following operations on each image path in the image_paths list:

[0101] Use os.path.basename() to get the name of the image file (including the extension);

[0102] The path is normalized using os.path.normpath(), and then the last four characters of the filename (i.e., the .jpg extension) are removed using [:-4], thus extracting the file ID (file_id).

[0103] 2) Check if file_id exists in the dictionary images.

[0104] If it exists, add a new key 'image_path' to the dictionary entry corresponding to that file_id, with the value being the full path to the image, image_path. The updated images dictionary will then contain a label, file ID, and the full path to the corresponding image file for each file ID entry, forming a dictionary format that facilitates searching.

[0105] (4) Split the image attributes in the image dictionary into separate lists.

[0106] 1) Create three empty lists to store the following attributes respectively.

[0107] image_paths: Stores the full path of the image, file_ids: Stores the file ID of the image, labels: Stores the label of the image.

[0108] 2) Use a for loop to iterate through each key-value pair in the dictionary images and extract the attributes of each image item by item.

[0109] Take 'image_path' from the dictionary value and add it to the image_paths list;

[0110] Retrieve 'file_id' from the dictionary value and add it to the file_ids list;

[0111] Take 'label' from the dictionary value and add it to the labels list.

[0112] The elements in these lists are arranged in the same order, and each element at each index position corresponds to one of the three attributes of the same image.

[0113] This step helps transform the data in the dictionary into a more manageable list format, thus facilitating subsequent batch processing or model input operations.

[0114] (5) Based on the image attributes, the image labels are re-labeled as continuous values ​​to obtain the preprocessed dataset.

[0115] 1) Extract the unique labels from the images and sort them to obtain an ordered list containing the unique labels.

[0116] Specifically, use set(labels) to extract all unique values ​​from the label list;

[0117] Use sorted() to sort these unique labels, ensuring they are arranged in ascending order, resulting in an ordered list of unique labels, unique_labels.

[0118] 2) Map the unique labels of the ordered list to consecutive integer values ​​to produce an integer-original label pair.

[0119] Create a mapping dictionary and use dictionary comprehensions to map each unique_label to a consecutive integer value starting from 0.

[0120] The `enumerate(unique_labels)` function generates a pair of (index, label) elements, where index is a consecutive integer and label is the original label.

[0121] 3) Generate a new tag dictionary based on the integer-original tag pairs.

[0122] The generated relabeling dictionary is used to convert the original labels into consecutive integers. For example, labels = [305, 101, 450, 305] will be remapped to new_labels = [1, 0, 2, 1].

[0123] This step is crucial for label preprocessing in classification tasks, especially when training deep learning models, as consecutive integer labels facilitate the correct calculation of loss functions (such as cross-entropy).

[0124] S2-2-2, Divide the preprocessed dataset into a training set and a test set.

[0125] In this embodiment, the preprocessed dataset P is divided into a training set P1 and a test set P2 in a 9:1 ratio.

[0126] S2-3. Construct a feature extraction model. Input the training set into the feature extraction model for training to obtain a trained feature extraction model.

[0127] The feature extraction model includes a backbone network module, a multi-scale local feature extraction module, and a global feature extraction module.

[0128] In this embodiment, a trainable framework for retrieving similar images from the Biskeh cancer database is obtained using deep learning technology. It consists of three parts: a backbone network (Swin-Transformer), a global feature extraction module (GFC), and a multi-scale local feature extraction module (Self-ATT) as the main structure.

[0129] The Swin Transformer is used as the backbone network for model feature extraction. Branches are made from the backbone network, and the pre-modeled model (without parameters) in the keras_swin_transformer library is used to model the Swin Transformer, and branches are made in the corresponding layers.

[0130] The backbone network consists of an image slicing layer, a linear embedding layer, a Swing Trasnformer module, and an image fusion layer.

[0131] During the training process, (1) the training set is input into the backbone network module and the first input feature is obtained by downsampling by 2 times.

[0132] In this embodiment, in the backbone network module, the training set images are divided into non-overlapping image patches through an image segmentation layer. Then, each non-overlapping image patch is reduced in dimensionality through a linear embedding layer. Next, a SwinTransformer Block convolution operation is performed, followed by a 2x downsampling through a SwinTransformer Block x2 module, and finally fused through a Patch Merging layer to obtain the first input features.

[0133] (2) The first input feature is downsampled by 2 times to obtain the second input feature.

[0134] In this embodiment, the first input feature Continue with the Swin Transformer Block convolution operation, then perform a 2x downsampling through the Swin Transformer Block x2 module, and finally fuse the data through the PatchMerging layer to obtain the second input feature.

[0135] (3) The second input feature is downsampled by 6 times to obtain the third input feature.

[0136] In this embodiment, the second input feature Continue with the Swin Transformer Block convolution operation, using 6 Swin Transformer Blocks, and perform a 6x downsampling to obtain the third input feature.

[0137] (4) Input the second input feature into the multi-scale local feature extraction module for feature extraction to obtain multi-scale local features.

[0138] like Figure 2 As shown, in this embodiment, the multi-scale local feature module includes a local feature extraction layer and a fully connected layer, specifically including an overlapping window partitioning module, a window multi-head self-attention module, a feedforward neural network, a window merging module, a convolutional layer, a layer normalization layer, and a fully connected layer.

[0139] The process of obtaining multi-scale local features is as follows:

[0140] 1) The second input feature map is divided into small windows by the overlapping window partitioning module, and the first output feature map of each window is obtained.

[0141] Specifically, in this embodiment, the second input feature The input image is divided into overlapping small windows by the Overlapped Window Split module, which facilitates feature extraction in local areas.

[0142] Then, using tf.image.extract_patches, small patches of shape [window_size, window_size] are extracted from the input image, with the stride controlling the overlapping area of ​​the window.

[0143] Finally, the first output feature map P of each window is obtained. l 1 The output shape is (batch_size, num_windows, num_windows, window_size^2*channels).

[0144] 2) Input the first output feature map into the multi-head self-attention module of the window to perform attention calculation, and obtain the second output feature map of each window.

[0145] In this embodiment, the first output feature map P l 1 Input the data into the multi-head self-attention module in the Win-MSA window for multi-head self-attention calculation.

[0146] Specifically, the self-attention mechanism is implemented using Keras' MultiHeadAttention layer to obtain the second output feature map P for each window. l 2 The output shape is

[0147] (batch_size,num_windows,window_size2×channels).

[0148] 3) Input the second output feature map into the feedforward neural network for nonlinear transformation to obtain the third output feature map for each window.

[0149] In this embodiment, the second output feature map P l 2 The input is fed into a Feedforward Network, which applies the feedforward network to each window to perform a nonlinear transformation on the window features.

[0150] Specifically, a two-layer fully connected network is used with ReLU activation function.

[0151] The first Dense layer receives the input shape:

[0152] (batch_size,num_windows,feature_dim),

[0153] The output shape is: (batch_size, num_windows, units);

[0154] The input to the second Dense layer comes from the previous layer, and the input shape is:

[0155] (batch_size,num_windows,units),

[0156] The output shape remains:

[0157] (batch_size,num_windows,units).

[0158] Finally, the third output feature map of each window is obtained. The feature dimension of each window is projected from feature_dim to a new feature dimension units, but the batch size and the number of windows remain unchanged.

[0159] 4) The third output feature map of each window is merged through the window merging module to obtain the fourth output feature map.

[0160] In this embodiment, in the Window Merge module, the third output feature map P of each window is... i 3 Merge.

[0161] Specifically, tf.reduce_mean is used to average the window dimensions, thus reducing the window features. Compressing into a unified representation yields the fourth output feature map. Output shape: (batch_size, units).

[0162] 5) Adjust the number of channels in the fourth output feature map through convolutional layers.

[0163] In this embodiment, the fourth output feature map is adjusted through a 1x1 convolutional layer. The number of channels is R, using Conv2D with a kernel size of (1,1).

[0164] 6) The fourth output feature map is standardized by batch standard layer to obtain the standardized output features.

[0165] In this embodiment, batch normalization and activation are performed using a BatchNormalization layer to normalize the fourth output feature map. Standardize it.

[0166] 7) Input the standardized output features into the fully connected layer for mapping to obtain multi-scale local features.

[0167] In this embodiment, the standardized output features are processed through a fully connected layer containing 256 neurons. The mapping is transformed into 256-bit multi-scale local features.

[0168] (5) Input the third input feature into the global feature extraction module to extract the feature and obtain the global feature.

[0169] like Figure 3 As shown, in this embodiment, the global feature extraction module includes a convolutional layer, a GeM pooling layer, and a fully connected layer. The global feature extraction module (GFC) uses GeM pooling for shallow feature representation, which can extract image features and retain more detailed information.

[0170] The third input feature is fed into the global feature extraction module for feature extraction to obtain the global features. The specific process is as follows:

[0171] 1) Adjust the number of channels of the third input feature through convolutional layers.

[0172] In this embodiment, the third input feature Pre-adjust the channels using a 1x1 convolutional layer.

[0173] 2) Input the third input feature into the GeM pooling layer for pooling to obtain the fifth output feature map.

[0174] In this embodiment, the fifth output feature map P is obtained through GeM pooling. l 5 .

[0175] 3) Map the fifth output feature map through a fully connected layer to obtain global features.

[0176] In this embodiment, the fifth output feature map P l 5 The fifth output feature map P is processed through a fully connected layer with 1024 neurons, followed by a connected layer with 128 neurons. l 5 The mapping is converted into a 128-bit feature descriptor, thus obtaining the global features.

[0177] The training dataset P1 is input into the designed multi-scale fusion feature extraction network for training. During training, ArcFace loss is used to train the overall network. The Adam optimizer is used to adaptively adjust the learning rate of each parameter based on the first and second moment estimates of the gradient. The trained model and parameters are saved in the corresponding .h5 database file for subsequent model loading.

[0178] S2-4. Using the trained feature extraction model, extract features from the training set to obtain the fusion features of the image;

[0179] The fusion features of the images are stored in a database to build an image feature library;

[0180] Using the trained feature extraction model, features are extracted from the test set to obtain the fused features of the test image;

[0181] The fusion features of the test image are matched with the image feature database to retrieve the corresponding image features.

[0182] S2-4-1. Using the trained feature extraction model, extract features from the training set to obtain the fused features of the image.

[0183] Multi-scale local features are fused with global features to obtain the fused features of the fused image.

[0184] In this embodiment, the tf.concat function is used to concatenate multi-scale local and global features according to the last channel.

[0185] Input shapes: (batch_size, 128), (batch_size, 256)

[0186] tf.concat([input1,input2],axis=-1): Concatenates the two input tensors along the last dimension (i.e. the feature dimension).

[0187] Output a fused feature that contains 128-bit global features and 256-bit local features.

[0188] S2-4-2. Store the fusion features of the images in the database to build an image feature library.

[0189] In this embodiment, the trained feature extraction network is used to extract fused features from the P1 dataset, and the resulting fused features are stored in a database.

[0190] S2-4-3. Using the trained feature extraction model, feature extraction is performed on the test set to obtain the fusion features of the test image. The fusion features of the test image are then matched with the image feature library to retrieve the corresponding image features.

[0191] In this embodiment, a pre-trained feature extraction network model is used to extract fused features from the test dataset P2, and a similarity search is performed in the P1 database using dot products. The most similar result is returned and output.

[0192] The test dataset P2 is input into the trained feature extraction model. Multi-scale local features are extracted by the model's multi-scale local feature extraction module, and global features are extracted by the global feature extraction module. The multi-scale local features and global features are fused to obtain the fused features. Finally, the dot product method is used to perform a similarity search on P2 in the feature map of image P1 of the feature extraction model, and the most similar result is returned and output, that is, the corresponding image is retrieved.

[0193] In this embodiment, Swin-AttnNet significantly improves the model's recognition performance across different categories by combining local and global features. In subclass classification tasks, Swin-AttnNet demonstrates significant advantages, improving average precision by approximately 10% and recall by approximately 30%. The results show that the proposed feature extraction model significantly improves the accuracy and efficiency of medical image retrieval. The feature extraction model in this embodiment can be widely applied to medical image retrieval to assist doctors in diagnosis and decision-making.

[0194] S2-5. Extract multi-scale local features from the medical images to be retrieved using the multi-scale local feature module.

[0195] In this embodiment, a trained feature extraction model was obtained through training and testing, and then applied to medical image retrieval.

[0196] Specifically, the medical images of breast tumors are input into the feature extraction model. The backbone network of the feature extraction model, Swin-Transformer, is used to perform two 2x convolution downsampling operations on the medical images of breast tumors to obtain the second input features. The second input features are then input into the multi-scale local feature extraction module to extract the multi-scale local features of the medical images of breast tumors.

[0197] S2-6. Use the global feature module to extract global features of the medical image to be retrieved.

[0198] In this embodiment, the second input feature is further downsampled by 6 times through convolution to obtain the third input feature. The third input feature is then input into the global feature extraction module to extract the global features of the breast tumor medical image.

[0199] S2-7. The multi-scale local and global features of the medical image to be retrieved are fused to obtain the fused features of the image to be retrieved.

[0200] In this embodiment, the multi-scale local features and global features of the breast tumor medical image are fused to obtain the fused features of the breast tumor medical image.

[0201] S2-8. In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are searched to obtain the corresponding image.

[0202] In this embodiment, images matching the fusion features of breast tumor images are retrieved from the image feature library of the feature extraction model.

[0203] In the field of medical imaging applications, the feature extraction model in this application addresses the challenges of high-resolution, multimodal data, and background noise interference in medical image retrieval. It can accurately and effectively retrieve the corresponding medical images, greatly improving the robustness and accuracy of the retrieval algorithm.

[0204] Example 2

[0205] The purpose of this embodiment is to provide a medical image retrieval system, including:

[0206] An image acquisition module is used to acquire medical images to be retrieved, wherein the medical images contain at least medical image data and tags;

[0207] The image retrieval module is used to input preprocessed medical images into a trained feature extraction model and use the multi-scale local feature module to extract multi-scale local features of the medical images to be retrieved.

[0208] Global features are extracted from the medical images to be retrieved using the global feature module;

[0209] The multi-scale local and global features of the medical image to be retrieved are fused together to obtain the fused features of the image to be retrieved.

[0210] In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are searched to obtain the corresponding image;

[0211] The process of constructing the image feature library of the feature extraction model is as follows: using the trained feature extraction model, features are extracted from the acquired image dataset, and the fused features of each extracted image are stored in the database to obtain the image feature library of the feature extraction model.

[0212] Based on the provision of a medical image retrieval system, the method steps in Embodiment 1 are implemented.

[0213] Example 3

[0214] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the aforementioned medical image retrieval method.

[0215] Example 4

[0216] The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the aforementioned medical image retrieval method.

[0217] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0218] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0219] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A medical image retrieval method, characterized in that, include: Obtain the medical image to be retrieved, wherein the medical image contains at least medical image data and tags; The medical images to be retrieved are input into the trained feature extraction model, and the multi-scale local feature module is used to extract the multi-scale local features of the medical images to be retrieved. Global features are extracted from the medical images to be retrieved using the global feature module; The multi-scale local and global features of the medical image to be retrieved are fused together to obtain the fused features of the image to be retrieved. In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are searched to obtain the corresponding image; The process of constructing the image feature library of the feature extraction model is as follows: using the trained feature extraction model, features are extracted from the acquired image dataset, and the fused features of each extracted image are stored in the database to obtain the image feature library of the feature extraction model. The feature extraction model includes a backbone network module, a multi-scale local feature extraction module, and a global feature extraction module; During training, the training set is input into the backbone network module, and the first input features are obtained by downsampling by 2 times. The first input feature is downsampled by a factor of 2 to obtain the second input feature; The second input feature is downsampled by a factor of 6 to obtain the third input feature; The second input feature is input into the multi-scale local feature extraction module for feature extraction to obtain multi-scale local features; The third input feature is fed into the global feature extraction module for feature extraction to obtain the global feature. Multi-scale local features are fused with global features to obtain the fused features of the fused image; The second input feature is input into the multi-scale local feature extraction module for feature extraction to obtain multi-scale local features. The multi-scale local feature extraction module includes an overlapping window partitioning module, a multi-head self-attention window module, a feedforward neural network, a window merging module, convolutional layers, layer normalization, and fully connected layers. The specific process is as follows: The second input feature map is divided into small windows by the overlapping window partitioning module, and the first output feature map of each window is obtained. The first output feature map is input into the window multi-head self-attention module for attention calculation, and the second output feature map of each window is obtained. The second output feature map is input into the feedforward neural network for nonlinear transformation to obtain the third output feature map for each window; The window merging module merges the third output feature maps of each window to obtain the fourth output feature map. The number of channels in the fourth output feature map is adjusted through convolutional layers; The fourth output feature map is standardized by batch standard layer to obtain standardized output features; The standardized output features are input into a fully connected layer for mapping to obtain multi-scale local features.

2. The medical image retrieval method as described in claim 1, characterized in that, Before the training process, the acquired medical image dataset is preprocessed to obtain a preprocessed dataset. The specific process is as follows: Traverse the image files in the medical image dataset and match each image file with its corresponding label. For image files, construct an image dictionary; Add complete image paths to the image dictionary; Split the image attributes in the image dictionary into separate lists; Based on the image attributes, the image labels are re-labeled as continuous values ​​to obtain the preprocessed dataset.

3. The medical image retrieval method as described in claim 1, characterized in that, The third input feature is fed into the global feature extraction module for feature extraction to obtain global features. The global feature extraction module includes convolutional layers, GeM pooling layers, and fully connected layers. The specific process is as follows: The number of channels for the third input feature is adjusted using convolutional layers; The third input feature is fed into the GeM pooling layer for pooling to obtain the fifth output feature map; The fifth output feature map is mapped through a fully connected layer to obtain global features.

4. The medical image retrieval method as described in claim 1, characterized in that, The fusion of multi-scale local features and global features is achieved by fusing them together to obtain the fused features of the image. Specifically, the multi-scale features and global features are concatenated in the last channel using a cross function to obtain the fused features of the image.

5. The medical image retrieval method as described in claim 1, characterized in that, In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are searched to obtain the corresponding image. Specifically, the similarity of the fusion features of the image to be retrieved is searched in the image feature library using the dot product method to obtain the retrieval results and output them.

6. A medical image retrieval system, characterized in that, include: An image acquisition module is used to acquire medical images to be retrieved, wherein the medical images contain at least medical image data and tags; The image retrieval module is used to input preprocessed medical images into a trained feature extraction model and use the multi-scale local feature module to extract multi-scale local features of the medical images to be retrieved. Global features are extracted from the medical images to be retrieved using the global feature module; The multi-scale local and global features of the medical image to be retrieved are fused together to obtain the fused features of the image to be retrieved. In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are searched to obtain the corresponding image; The process of constructing the image feature library of the feature extraction model is as follows: using the trained feature extraction model, features are extracted from the acquired image dataset, and the fused features of each extracted image are stored in the database to obtain the image feature library of the feature extraction model. The feature extraction model includes a backbone network module, a multi-scale local feature extraction module, and a global feature extraction module; During training, the training set is input into the backbone network module, and the first input features are obtained by downsampling by 2 times. The first input feature is downsampled by a factor of 2 to obtain the second input feature; The second input feature is downsampled by a factor of 6 to obtain the third input feature; The second input feature is input into the multi-scale local feature extraction module for feature extraction to obtain multi-scale local features; The third input feature is fed into the global feature extraction module for feature extraction to obtain the global feature. Multi-scale local features are fused with global features to obtain the fused features of the fused image; The second input feature is input into the multi-scale local feature extraction module for feature extraction to obtain multi-scale local features. The multi-scale local feature extraction module includes an overlapping window partitioning module, a multi-head self-attention window module, a feedforward neural network, a window merging module, convolutional layers, layer normalization, and fully connected layers. The specific process is as follows: The second input feature map is divided into small windows by the overlapping window partitioning module, and the first output feature map of each window is obtained. The first output feature map is input into the window multi-head self-attention module for attention calculation, and the second output feature map of each window is obtained. The second output feature map is input into the feedforward neural network for nonlinear transformation to obtain the third output feature map for each window; The window merging module merges the third output feature maps of each window to obtain the fourth output feature map. The number of channels in the fourth output feature map is adjusted through convolutional layers; The fourth output feature map is standardized by batch standard layer to obtain standardized output features; The standardized output features are input into a fully connected layer for mapping to obtain multi-scale local features.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it performs the steps of the method as described in any one of claims 1-5.

Images