Medical image retrieval method and system

Abstract

The invention provides a medical image retrieval method and system, relates to the field of medical image processing, and aims to solve the problems that the internal content of a medical image cannot be deeply understood in the prior art, and corresponding medical images cannot be accurately and effectively retrieved for the problems of high resolution, multi-modal data, background noise interference and the like faced by medical image retrieval. And the retrieval robustness and accuracy are poor. The method comprises the following steps: acquiring a medical image data set, and preprocessing the medical image data set; performing feature extraction by using the trained feature extraction model to obtain fusion features of the image; constructing an image feature library based on the fusion features of the images; performing feature extraction on the to-be-retrieved medical image by using the trained feature extraction model to obtain a fusion feature of the to-be-retrieved medical image; and the image feature library is matched with the image feature library, and a corresponding image is retrieved. According to the method, the accuracy and efficiency of medical image retrieval are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to a medical image retrieval method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Image retrieval is the process of searching for similar images based on the visual content of an image. It has important application value in the field of medical imaging. In the medical field, doctors often make diagnoses and develop treatment plans by examining patients' pathological images and related cases. Medical images play a very important role in the diagnosis and treatment of patients. Therefore, doctors often need to retrieve valuable images from different medical image databases to improve their clinical practice.

[0004] However, medical image retrieval is different from ordinary image retrieval and has the following unique features:

[0005] (1) Strong domain specificity: Medical images contain multimodal data (such as CT, MRI, and ultrasound), and algorithms need to be optimized for specific anatomical regions and lesion characteristics;

[0006] (2) Data quality challenges: Medical images are often accompanied by noise, artifacts, and uneven resolution, and image quality needs to be improved through effective preprocessing;

[0007] (3) High precision requirement: Medical image retrieval results directly affect clinical diagnosis, requiring the model to have higher robustness and accuracy.

[0008] Therefore, medical image retrieval not only needs to quickly find content similar to the target image from the database, but also needs to pay attention to the subtle differences in the lesions to assist doctors in diagnosis and decision-making.

[0009] At present, the inventors have found that existing medical image retrieval technologies often cannot deeply understand the intrinsic content of images, and it is difficult to distinguish cases that are highly similar but have different diagnostic results due to minor differences. Although descriptors based on feature fusion have achieved remarkable success in the field of image retrieval, in the application of medical images, due to the problems of high resolution, multimodal data and background noise interference faced by medical image retrieval, it is impossible to accurately and effectively retrieve the corresponding medical images, and the robustness and accuracy of the retrieval algorithm are poor. Summary of the invention

[0010] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a medical image retrieval method and system, which accurately and comprehensively extracts multi-scale local features and global features of medical images by constructing a feature extraction model Swin-AttnNet applied to medical image retrieval, thereby significantly improving the accuracy and efficiency of medical image retrieval.

[0011] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0012] In a first aspect, a medical image retrieval method is disclosed, comprising:

[0013] Acquire a medical image to be retrieved, wherein the medical image at least contains medical image data and a label;

[0014] The medical image to be retrieved is input into the trained feature extraction model, and the multi-scale local features of the medical image to be retrieved are extracted using the multi-scale local feature module;

[0015] The global feature module is used to extract the global features of the medical images to be retrieved;

[0016] The multi-scale local features and global features of the medical image to be retrieved are fused to obtain the fusion features of the image to be retrieved;

[0017] In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are retrieved to obtain the corresponding image;

[0018] Among them, the image feature library construction process of the feature extraction model is: using the trained feature extraction model to extract features from the acquired image data set, and storing the fusion features of each extracted image in the database to obtain the image feature library of the feature extraction model.

[0019] As a further technical solution, the feature extraction model includes a backbone network module, a multi-scale local feature extraction module and a global feature extraction module;

[0020] During the training process, the training set is input into the backbone network module and downsampled by 2 times to obtain the first input feature;

[0021] Downsample the first input feature by 2 times to obtain the second input feature;

[0022] Downsample the second input feature by 6 times to obtain the third input feature;

[0023] Inputting the second input feature into a multi-scale local feature extraction module for feature extraction to obtain a multi-scale local feature;

[0024] Inputting the third input feature into the global feature extraction module for feature extraction to obtain a global feature;

[0025] The multi-scale local features are fused with the global features to obtain the fusion features of the fused image.

[0026] As a further technical solution, before the training process, the acquired medical image data set is preprocessed to obtain a preprocessed data set. The specific process is as follows:

[0027] Traverse the image files from the medical image dataset and match the image files and labels one by one;

[0028] For image files, build an image dictionary;

[0029] Add the full image path to the image dictionary;

[0030] Split image attributes in the image dictionary into separate lists;

[0031] According to the image attributes, the image labels are re-labeled as continuous values ​​to obtain the preprocessed data set.

[0032] As a further technical solution, the second input feature is input into a multi-scale local feature extraction module for feature extraction to obtain a multi-scale local feature, wherein the multi-scale local feature module includes an overlapping window division module, a window multi-head self-attention module, a feedforward neural network, a window merging module, a convolution layer, a layer normalization and a fully connected layer. The specific process is:

[0033] Divide the second input feature map into small windows through the overlapping window division module, and obtain the first output feature map of each window;

[0034] Input the first output feature map into the window multi-head self-attention module for attention calculation to obtain the second output feature map of each window;

[0035] Input the second output feature map into the feedforward neural network for nonlinear transformation to obtain the third output feature map of each window;

[0036] Merging the third output feature map of each window through the window merging module to obtain a fourth output feature map;

[0037] Adjust the number of channels of the fourth output feature map through the convolution layer;

[0038] Standardize the fourth output feature map through a batch standard layer to obtain a standardized output feature;

[0039] The standardized output features are input into the fully connected layer for mapping to obtain multi-scale local features.

[0040] As a further technical solution, the third input feature is input into the global feature extraction module for feature extraction to obtain the global feature, wherein the global feature extraction module includes a convolutional layer, a GeM pooling layer and a fully connected layer, and the specific process is as follows:

[0041] Adjust the number of channels of the third input feature through the convolution layer;

[0042] Input the third input feature to the GeM pooling layer for pooling to obtain a fifth output feature map;

[0043] The fifth output feature map is mapped through a fully connected layer to obtain global features.

[0044] As a further technical solution, the multi-scale local features are fused with the global features to obtain the fusion features of the fused image. Specifically, the multi-scale features and the global features are spliced ​​in the last channel through a cross function to obtain the fusion features of the image.

[0045] As a further technical solution, in the image feature library of the feature extraction model, the fused features of the image to be retrieved are retrieved to obtain the corresponding image. Specifically, the dot product method is used to perform a similarity search on the fused features of the image to be retrieved in the image feature library to obtain and output the retrieval results.

[0046] In a second aspect, a medical image retrieval system is disclosed, comprising:

[0047] An image acquisition module, used to acquire a medical image to be retrieved, wherein the medical image at least contains medical image data and a label;

[0048] An image retrieval module is used to input the preprocessed medical images into the trained feature extraction model and use the multi-scale local feature module to extract the multi-scale local features of the medical images to be retrieved;

[0049] The global feature module is used to extract the global features of the medical images to be retrieved;

[0050] The multi-scale local features and global features of the medical image to be retrieved are fused to obtain the fusion features of the image to be retrieved;

[0051] In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are retrieved to obtain the corresponding image;

[0052] Among them, the image feature library construction process of the feature extraction model is: using the trained feature extraction model to extract features from the acquired image data set, and storing the fusion features of each extracted image in the database to obtain the image feature library of the feature extraction model.

[0053] The third aspect of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned medical image retrieval method when executing the program.

[0054] A fourth aspect of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above-mentioned medical image retrieval method.

[0055] One or more of the above technical solutions have the following beneficial effects:

[0056] In this embodiment, a new model for medical image retrieval is constructed. This model uses the improved Swin Transformer as the backbone feature extraction network to process the images in the data set; through the optimized multi-scale local feature extraction module Self-ATT, it can efficiently extract multi-scale local features; then use the GeM (Generalized-Mean) pooling layer of the global feature extraction module to generate global feature representation; finally, through the weighted feature fusion method, the multi-scale local features and global feature information are further integrated to generate a more accurate image descriptor. The feature extraction model in this embodiment takes into account the problems faced by medical image retrieval, such as high resolution, multi-modal data, and background noise interference, and significantly improves the accuracy and efficiency of image retrieval.

[0057] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0059] Figure 1 This is a schematic diagram of a medical image retrieval model architecture of the first embodiment;

[0060] Figure 2 This is a schematic diagram of the architecture of the multi-scale feature extraction module of the first embodiment;

[0061] Figure 3 Schematic diagram of the global feature extraction module architecture of the first embodiment. DETAILED DESCRIPTION

[0062] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0063] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0064] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0065] Embodiment 1

[0066] This embodiment discloses a medical image retrieval method.

[0067] In order to more clearly illustrate this embodiment, the implementation process of the medical image retrieval method can be specifically described as follows:

[0068] A medical image retrieval method, comprising:

[0069] S1. Obtain a medical image to be retrieved, wherein the medical image at least contains medical image data and a label;

[0070] S2, inputting the medical image to be retrieved into the trained feature extraction model, and using the multi-scale local feature module to extract the multi-scale local features of the medical image to be retrieved;

[0071] The global feature module is used to extract the global features of the medical images to be retrieved;

[0072] The multi-scale local features and global features of the medical image to be retrieved are fused to obtain the fusion features of the image to be retrieved;

[0073] In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are retrieved to obtain the corresponding image;

[0074] Among them, the image feature library construction process of the feature extraction model is: using the trained feature extraction model to extract features from the acquired image data set, and storing the fusion features of each extracted image in the database to obtain the image feature library of the feature extraction model.

[0075] like Figure 1 As shown, in step S1, a medical picture to be retrieved is obtained, and the medical picture at least contains medical image data and a label.

[0076] In this embodiment, a medical image of a breast tumor is obtained, and the image contains relevant medical data and a label, and the label is benign adenosis.

[0077] like Figure 1 As shown, in step S2, the medical image to be retrieved is input into the trained feature extraction model, and the multi-scale local features of the medical image to be retrieved are extracted using the multi-scale local feature module;

[0078] The global feature module is used to extract the global features of the medical images to be retrieved;

[0079] The multi-scale local features and global features of the medical image to be retrieved are fused to obtain the fusion features of the image to be retrieved;

[0080] In the image feature library of the feature extraction model, the fused features of the image to be retrieved are retrieved to obtain the corresponding image.

[0081] Before inputting the medical images to be retrieved into the trained feature extraction model, an image data set is first obtained, a feature extraction model is constructed, and the obtained image data set is input into the constructed feature extraction model for training and testing. During the training and testing process, features are extracted and an image feature library of the feature extraction model is constructed.

[0082] S2-1. Obtain a medical image dataset, where the medical image dataset at least includes medical image data and labels.

[0083] The BreaKHis_v1 dataset is used as a medical image dataset, and image data and labels are obtained from the BreaKHis_v1 dataset.

[0084] S2-2. Preprocess the medical image data set to obtain a preprocessed data set; divide the preprocessed data set into a training set and a test set.

[0085] In this embodiment, a preprocessing module is constructed to correspond images to labels, having the following structure [(image, label)], and the preprocessed data set P is divided into a training set P1 and a test set P2, with a ratio of 9:1.

[0086] S2-2-1. Preprocess the medical image dataset to obtain a preprocessed dataset.

[0087] Preprocess the medical image files and labels to obtain the preprocessed data set. The specific process is as follows:

[0088] (1) Traverse the image files from the medical image dataset and match the image files and labels one by one.

[0089] In this example, the CSV file content (landmark_ID / label->image) is loaded, and the image files are traversed from the dataset directory using the os library and the tf.io.gfile module.

[0090] (2) For image files, construct an image dictionary;

[0091] In this embodiment, an image dictionary images is constructed, (key=image_id, value={label, file_id}).

[0092] Specifically, 1) create an empty dictionary images to store the relevant information of each image,

[0093] Then traverse the data frame: iterate each row in the data CSV file, extract the landmark_id (i.e. label) from the current row, assign it to the variable label, and extract the images column, which contains one or more image file IDs separated by spaces.

[0094] 2) Split the file ID and update the dictionary, performing the following operations on each file ID:

[0095] In the dictionary images, create a new nested dictionary with the file ID (file_id) as the key;

[0096] Then the image label label is stored in the nested dictionary with the key 'label'; at the same time, the file_id itself is also stored in the nested dictionary with the key 'file_id'.

[0097] (3) Add the complete image path to the image dictionary.

[0098] 1) Get all image paths.

[0099] Use the tf.io.gfile.glob() function to read all image file paths with the extension .jpg under the specified directory image_dir and store these paths in the list image_paths.

[0100] For each image path in the image_paths list do the following:

[0101] Use os.path.basename() to get the name of the image file (including the extension);

[0102] Normalize the path using os.path.normpath(), and then use [:-4] to remove the last four characters of the file name (i.e., the .jpg extension) to extract the file ID (file_id).

[0103] 2) Check if file_id exists in the dictionary images.

[0104] If it exists, add a new key 'image_path' to the dictionary entry corresponding to the file_id, and its value is the full path image_path of the image. After the images dictionary is updated, each file ID item will contain the label, file ID, and the corresponding full path of the image file. It forms a dictionary format that is easy to query.

[0105] (4) Split the image attributes in the image dictionary into independent lists.

[0106] 1) Create three empty lists to store the following attributes.

[0107] image_paths: stores the full path of the image, file_ids: stores the file ID of the image, labels: stores the label of the image.

[0108] 2) Use a for loop to iterate over each key-value pair in the dictionary images and extract the attributes of each image one by one.

[0109] Take 'image_path' from the dictionary value and add it to the image_paths list;

[0110] Take 'file_id' from the dictionary value and add it to the file_ids list;

[0111] Take 'label' from the dictionary value and add it to the labels list.

[0112] The elements in these lists are arranged in the same order, and the element at each index position corresponds to the three attributes of the same image.

[0113] This step helps convert the data in the dictionary into a more easily processable list form, which is convenient for subsequent batch processing or model input operations.

[0114] (5) According to the image attributes, the image labels are re-labeled as continuous values ​​to obtain the preprocessed data set.

[0115] 1) Extract and sort the unique labels of the images to obtain an ordered list containing unique labels.

[0116] Specifically, use set(labels) to extract all unique values ​​in the label list;

[0117] Use sorted() to sort these unique labels and ensure that they are arranged in ascending order, resulting in an ordered list unique_labels containing unique labels.

[0118] 2) Map the unique labels of the ordered list to consecutive integer values, producing an integer-original label pair.

[0119] Create a mapping dictionary, using dictionary comprehension, to map each unique_label to a consecutive integer value starting from 0.

[0120] enumerate(unique_labels) produces an (index, label) pair, where index is a consecutive integer and label is the corresponding original label.

[0121] 3) Generate a new label dictionary based on the integer-original label pair.

[0122] The generated relabeling dictionary is used to convert the original labels into continuous integers. For example, labels = [305, 101, 450, 305] will be remapped to new_labels = [1, 0, 2, 1].

[0123] This step is very important for label preprocessing in classification tasks, especially when training deep learning models, where continuous integer labels are conducive to the correct calculation of loss functions such as cross entropy.

[0124] S2-2-2. Divide the preprocessed data set into a training set and a test set.

[0125] In this embodiment, the preprocessed data set P is divided into a training set P1 and a test set P2 in a ratio of 9:1.

[0126] S2-3. Construct a feature extraction model, input the training set into the feature extraction model for training, and obtain a trained feature extraction model.

[0127] The feature extraction model includes a backbone network module, a multi-scale local feature extraction module and a global feature extraction module.

[0128] In this embodiment, a trainable framework for retrieving similar images from the Biskeh cancer database through deep learning technology is composed of three parts: the backbone network Swin-Transformer, the global vital signs extraction module GFC, and the multi-scale local feature extraction module Self-ATT main structure.

[0129] Use Swin Trasnformer as the backbone network for model feature extraction, make branches from the backbone network, directly use the model (excluding parameters) that has been modeled in the keras_swin_transformer library to model Swin Trasnformer, and make branches in the corresponding levels.

[0130] The backbone network includes image segmentation layer, linear embedding layer, Swin Transformer module and image fusion layer.

[0131] During the training process, (1) the training set is input into the backbone network module and downsampled by 2 times to obtain the first input feature.

[0132] In this embodiment, in the backbone network module, the training set images are divided into non-overlapping image blocks through the image slicing layer, and then each non-overlapping image block is down-dimensioned through the linear embedding layer, followed by a Swin Transformer Block convolution operation, followed by a 2x downsampling through the Swin Transformer Block x2 module, and then fused through the Patch Merging layer to obtain the first input feature

[0133] (2) Downsample the first input feature by a factor of 2 to obtain the second input feature.

[0134] In this embodiment, the first input feature Continue the Swin Transformer Block convolution operation, downsample by 2 times through the Swin Transformer Block x2 module, and fuse through the PatchMerging layer to obtain the second input feature

[0135] (3) Downsample the second input feature by 6 times to obtain the third input feature.

[0136] In this embodiment, the second input feature Continue the Swin Transformer Block convolution operation, Swin Transformer Block x6, perform 6 times downsampling, and obtain the third input feature

[0137] (4) Inputting the second input feature into the multi-scale local feature extraction module for feature extraction to obtain a multi-scale local feature.

[0138] like Figure 2 As shown, in this embodiment, the multi-scale local feature module includes a local feature extraction layer and a fully connected layer, specifically including an overlapping window division module, a window multi-head self-attention module, a feedforward neural network, a window merging module, a convolutional layer, a layer normalization and a fully connected layer.

[0139] Get multi-scale local features. The specific process is as follows:

[0140] 1) The second input feature map is divided into small windows through the overlapping window division module, and the first output feature map of each window is obtained.

[0141] Specifically, in this embodiment, the second input feature Input to Overlapped Window Split module, which divides the input image into small overlapping windows to facilitate feature extraction in a local area;

[0142] Then, use tf.image.extract_patches to extract small patches of shape [window_size, window_size] from the input image, and control the overlapping area of ​​the window by stride.

[0143] Finally, the first output feature map P of each window is obtained l 1 , the output shape is (batch_size,num_windows,num_windows,window_size^2*channels).

[0144] 2) Input the first output feature map into the window multi-head self-attention module for attention calculation to obtain the second output feature map of each window.

[0145] In this embodiment, the first output feature map P l 1 Input into the Win-MSA window multi-head self-attention module for multi-head self-attention calculation.

[0146] Specifically, the MultiHeadAttention layer of Keras is used to implement the self-attention mechanism and obtain the second output feature map P of each window. l 2 , the output shape is

[0147] (batch_size,num_windows,window_size2×channels).

[0148] 3) Input the second output feature map into the feedforward neural network for nonlinear transformation to obtain the third output feature map of each window.

[0149] In this embodiment, the second output feature map P l 2 Input to the Feedforward Network feedforward neural network, apply the feedforward neural network in each window, and perform nonlinear transformation on the window features.

[0150] Specifically, a two-layer fully connected network is used, and the activation function is ReLU.

[0151] The first Dense layer receives an input shape:

[0152] (batch_size,num_windows,feature_dim),

[0153] The output shape is: (batch_size, num_windows, units);

[0154] The input of the second Dense layer comes from the previous layer, and the input shape is:

[0155] (batch_size,num_windows,units),

[0156] The output shape is still:

[0157] (batch_size,num_windows,units).

[0158] Finally, the third output feature map of each window is obtained Project the feature dimension of each window from feature_dim to a new feature dimension units, but keep the batch size and number of windows unchanged.

[0159] 4) The third output feature map of each window is merged through the window merging module to obtain a fourth output feature map.

[0160] In this embodiment, in the Window Merge window merging module, the third output feature map P of each window is i 3 to merge.

[0161] Specifically, use tf.reduce_mean to average the window dimensions and convert the window features Compressed into a unified representation, the fourth output feature map is obtained Output shape: (batch_size, units).

[0162] 5) Adjust the number of channels of the fourth output feature map through the convolution layer.

[0163] In this embodiment, the fourth output feature map is adjusted by a 1x1 convolution layer. The number of channels is , using Conv2D, the convolution kernel size is (1,1), and the number of output channels is R.

[0164] 6) The fourth output feature map is normalized by a batch standard layer to obtain a standardized output feature.

[0165] In this embodiment, batch normalization and activation are used to normalize the fourth output feature map using the BatchNormalization layer. To standardize.

[0166] 7) The standardized output features are input into the fully connected layer for mapping to obtain multi-scale local features.

[0167] In this embodiment, the standardized output features are transformed into Mapped into 256-bit multi-scale local features.

[0168] (5) Input the third input feature into the global feature extraction module for feature extraction to obtain the global feature.

[0169] like Figure 3 As shown, in this embodiment, the global feature extraction module includes a convolutional layer, a GeM pooling layer and a fully connected layer. The global feature extraction module (GFC) uses GeM pooling for shallow feature expression, which can extract image features and retain more detail information.

[0170] The third input feature is input into the global feature extraction module for feature extraction to obtain the global feature. The specific process is as follows:

[0171] 1) Adjust the number of channels of the third input feature through the convolution layer.

[0172] In this embodiment, the third input feature The channels are pre-adjusted through a 1x1 convolutional layer.

[0173] 2) Input the third input feature into the GeM pooling layer for pooling to obtain the fifth output feature map.

[0174] In this embodiment, the fifth output feature map P is obtained by GeM pooling. l 5 .

[0175] 3) Map the fifth output feature map through a fully connected layer to obtain global features.

[0176] In this embodiment, the fifth output feature map P l 5 The fifth output feature map P is passed through a fully connected layer containing 1024 neurons and then through a connection layer containing 128 neurons. l 5 Mapped to a 128-bit feature descriptor, the global feature is obtained.

[0177] The training dataset P1 is input into the designed multi-scale fusion feature extraction network for training. During the training process, ArcFace loss is used to train the entire network. The Adam optimizer is used to adaptively adjust the learning rate of each parameter according to the first-order moment estimation and second-order moment estimation of the gradient. The trained model and parameters are saved in the corresponding .h5 database file for subsequent model loading and use.

[0178] S2-4, using the trained feature extraction model, extract features from the training set to obtain the fusion features of the image;

[0179] Store the fused features of the image in the database to build an image feature library;

[0180] Using the trained feature extraction model, perform feature extraction on the test set to obtain the fusion features of the test image;

[0181] The fused signs of the test image are matched with the image feature library to retrieve the corresponding image features.

[0182] S2-4-1. Use the trained feature extraction model to extract features from the training set to obtain the fusion features of the image.

[0183] The multi-scale local features are fused with the global features to obtain the fusion features of the fused image.

[0184] In this embodiment, the tf.concat function is used to concatenate multi-scale local features and global features according to the last channel.

[0185] Input shape: (batch_size, 128), (batch_size, 256)

[0186] tf.concat([input1, input2], axis = -1): concatenates two input tensors along the last dimension (i.e., the feature dimension).

[0187] Output a fusion feature containing 128-bit global features and 256-bit local features

[0188] S2-4-2. Store the fused features of the image in a database to build an image feature library.

[0189] In this embodiment, the trained feature extraction network is used to extract fusion features from the P1 data set, and the obtained fusion features are stored in a database.

[0190] S2-4-3. Use the trained feature extraction model to extract features from the test set to obtain the fusion features of the test image, match the fusion features of the test image with the image feature library, and retrieve the corresponding image features.

[0191] In this embodiment, a pre-trained feature extraction network model is used to extract fusion features from the test data set P2, and a dot product method is used to perform a similarity search in the P1 database. The most similar result is returned and output.

[0192] The test data set P2 is input into the trained feature extraction model, and the multi-scale local features are extracted by the multi-scale local feature extraction module of the model. The global features are extracted by the global feature extraction module, and the multi-scale local features are fused with the global features to obtain the fused features. Finally, the dot product method is used in the P1 image feature map of the feature extraction model to perform a similarity search for P2, and the most similar result is returned and output, that is, the corresponding image is retrieved.

[0193] In this embodiment, Swin-AttnNet uses a combination of local features and global features to significantly improve the model's recognition performance for different categories. In the subclass classification task, Swin-AttnNet showed significant advantages, with an average precision of about 10% and a recall of about 30%. The results show that the proposed feature extraction model significantly improves the accuracy and efficiency of medical image retrieval. The feature extraction model in this embodiment can be widely used in medical image retrieval to assist doctors in diagnosis and decision-making.

[0194] S2-5. Use the multi-scale local feature module to extract the multi-scale local features of the medical image to be retrieved.

[0195] In this embodiment, a trained feature extraction model is obtained through training and testing, and is applied to medical image retrieval.

[0196] Specifically, the breast tumor medical image is input into the feature extraction model, and the backbone network Swin-Transformer of the feature extraction model is used to perform 2 times 2x convolution downsampling on the breast tumor medical image to obtain the second input feature, and the second input feature is input into the multi-scale local feature extraction module to extract the multi-scale local features of the breast tumor medical image.

[0197] S2-6. Use the global feature module to extract the global features of the medical image to be retrieved.

[0198] In this embodiment, the second input feature is further convolved and downsampled by 6 times to obtain a third input feature, and the third input feature is input into a global feature extraction module to extract the global features of the breast tumor medical image.

[0199] S2-7, the multi-scale local features and global features of the medical image to be retrieved are fused to obtain the fused features of the image to be retrieved.

[0200] In this embodiment, the multi-scale local features of the breast tumor medical image and the global features of the breast tumor medical image are fused to obtain the fused features of the breast tumor medical image.

[0201] S2-8. In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are retrieved to obtain the corresponding image.

[0202] In this embodiment, an image matching the fusion feature of the breast tumor image is retrieved from the image feature library of the feature extraction model.

[0203] In the application of medical imaging, the feature extraction model in this application can accurately and effectively retrieve the corresponding medical images to address the problems faced by medical image retrieval, such as high resolution, multimodal data, and background noise interference, greatly improving the robustness and accuracy of the retrieval algorithm.

[0204] Embodiment 2

[0205] The purpose of this embodiment is to provide a medical image retrieval system, comprising:

[0206] An image acquisition module, used to acquire a medical image to be retrieved, wherein the medical image at least contains medical image data and a label;

[0207] An image retrieval module is used to input the preprocessed medical images into the trained feature extraction model and use the multi-scale local feature module to extract the multi-scale local features of the medical images to be retrieved;

[0208] The global feature module is used to extract the global features of the medical images to be retrieved;

[0209] The multi-scale local features and global features of the medical image to be retrieved are fused to obtain the fusion features of the image to be retrieved;

[0210] In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are retrieved to obtain the corresponding image;

[0211] Among them, the image feature library construction process of the feature extraction model is: using the trained feature extraction model to extract features from the acquired image data set, and storing the fusion features of each extracted image in the database to obtain the image feature library of the feature extraction model.

[0212] Based on providing a medical image retrieval system, the method steps in embodiment 1 are implemented.

[0213] Embodiment 3

[0214] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned medical image retrieval method when executing the program.

[0215] Embodiment 4

[0216] The purpose of this embodiment is to provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the above-mentioned medical image retrieval method are performed.

[0217] The steps involved in the apparatus of the above embodiment correspond to the method embodiment 1, and the specific implementation method can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0218] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0219] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A medical image retrieval method, characterized in that: include: Acquire a medical image to be retrieved, wherein the medical image at least contains medical image data and a label; The medical image to be retrieved is input into the trained feature extraction model, and the multi-scale local features of the medical image to be retrieved are extracted using the multi-scale local feature module; The global feature module is used to extract the global features of the medical images to be retrieved; The multi-scale local features and global features of the medical image to be retrieved are fused to obtain the fusion features of the image to be retrieved; In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are retrieved to obtain the corresponding image; Among them, the image feature library construction process of the feature extraction model is: using the trained feature extraction model to extract features from the acquired image data set, and storing the fusion features of each extracted image in the database to obtain the image feature library of the feature extraction model.

2. A medical image retrieval method as claimed in claim 1, characterized in that: The feature extraction model includes a backbone network module, a multi-scale local feature extraction module and a global feature extraction module; During the training process, the training set is input into the backbone network module and downsampled by 2 times to obtain the first input feature; Downsample the first input feature by 2 times to obtain the second input feature; Downsample the second input feature by 6 times to obtain the third input feature; Inputting the second input feature into a multi-scale local feature extraction module for feature extraction to obtain a multi-scale local feature; Inputting the third input feature into the global feature extraction module for feature extraction to obtain a global feature; The multi-scale local features are fused with the global features to obtain the fusion features of the fused image.

3. A medical image retrieval method as claimed in claim 1, characterized in that: Before the training process, the acquired medical image dataset is preprocessed to obtain a preprocessed dataset. The specific process is as follows: Traverse the image files from the medical image dataset and match the image files and labels one by one; For image files, build an image dictionary; Add the full image path to the image dictionary; Split image attributes in the image dictionary into separate lists; According to the image attributes, the image labels are re-labeled as continuous values ​​to obtain the preprocessed data set.

4. A medical image retrieval method as claimed in claim 1, characterized in that: The second input feature is input into the multi-scale local feature extraction module for feature extraction to obtain multi-scale local features, wherein the multi-scale local feature module includes an overlapping window division module, a window multi-head self-attention module, a feedforward neural network, a window merging module, a convolution layer, a layer normalization and a fully connected layer. The specific process is as follows: Divide the second input feature map into small windows through the overlapping window division module, and obtain the first output feature map of each window; Input the first output feature map into the window multi-head self-attention module for attention calculation to obtain the second output feature map of each window; Input the second output feature map into the feedforward neural network for nonlinear transformation to obtain the third output feature map of each window; Merging the third output feature map of each window through the window merging module to obtain a fourth output feature map; Adjust the number of channels of the fourth output feature map through the convolution layer; Standardize the fourth output feature map through a batch standard layer to obtain a standardized output feature; The standardized output features are input into the fully connected layer for mapping to obtain multi-scale local features.

5. A medical image retrieval method as claimed in claim 1, characterized in that: The third input feature is input into the global feature extraction module for feature extraction to obtain the global feature, wherein the global feature extraction module includes a convolution layer, a GeM pooling layer and a fully connected layer. The specific process is as follows: Adjust the number of channels of the third input feature through the convolution layer; Input the third input feature to the GeM pooling layer for pooling to obtain a fifth output feature map; The fifth output feature map is mapped through a fully connected layer to obtain global features.

6. A medical image retrieval method as claimed in claim 1, characterized in that: The multi-scale local features are fused with the global features to obtain the fusion features of the fused image. Specifically, the multi-scale features and the global features are spliced ​​in the last channel through the cross function to obtain the fusion features of the image.

7. A medical image retrieval method as claimed in claim 1, characterized in that: In the image feature library of the feature extraction model, the fused features of the image to be retrieved are retrieved to obtain the corresponding image. Specifically, the dot product method is used to perform similarity search on the fused features of the image to be retrieved in the image feature library to obtain and output the retrieval results.

8. A medical image retrieval system, characterized in that: include: An image acquisition module, used to acquire a medical image to be retrieved, wherein the medical image at least contains medical image data and a label; An image retrieval module is used to input the preprocessed medical images into the trained feature extraction model and use the multi-scale local feature module to extract the multi-scale local features of the medical images to be retrieved; The global feature module is used to extract the global features of the medical images to be retrieved; The multi-scale local features and global features of the medical image to be retrieved are fused to obtain the fusion features of the image to be retrieved; In the image feature library of the feature extraction model, the fusion features of the image to be retrieved are retrieved to obtain the corresponding image; Among them, the image feature library construction process of the feature extraction model is: using the trained feature extraction model to extract features from the acquired image data set, and storing the fusion features of each extracted image in the database to obtain the image feature library of the feature extraction model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are performed.

Images