A spatial image retrieval system and retrieval method based on multi-source feature fusion

Abstract

The present invention discloses a spatial image retrieval system and retrieval method based on multi-source feature fusion, which specifically relates to the field of medical image processing and is used to solve the problems of efficient preprocessing and accurate retrieval of multimodal medical images. The system extracts multi-scale features of multimodal medical images through a multi-scale convolutional neural network to fully capture image details and global information. The spatial attention mechanism and the channel attention mechanism are used to perform weighted processing on the features, highlight important information and suppress irrelevant information, and enhance the distinguishability of the features. The weighted spatial and channel features are fused through the convolution layer to generate a comprehensive feature vector containing multimodal information. The multimodal density distribution index and the high-dimensional feature entropy index are analyzed by support vector machine regression, and the optimal retrieval index structure is dynamically selected to quickly and efficiently match the image features in the database, ensure the accuracy and reliability of the matching, and provide technical support for the rapid and accurate diagnosis and analysis of medical images.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and more specifically, to a spatial image retrieval system and a retrieval method based on multi-source feature fusion. Background Art

[0002] Spatial image retrieval with multi-source feature fusion refers to the technology of accurately matching and retrieving images in high-dimensional space by fusing feature information from multiple image modalities. This technology is particularly important in the field of medical image processing. Multimodal medical images (such as CT, MRI, PET, etc.) each contain different diagnostic information. By fusing these multi-source features, the diagnostic accuracy and comprehensiveness of the image can be improved. By extracting and fusing features from different modalities, constructing a unified feature representation, and retrieving in a high-dimensional feature space, efficient and accurate matching of the target image can be achieved.

[0003] When processing multimodal medical images, existing technologies have deficiencies in image enhancement mode selection and retrieval structure mode selection. First, most image enhancement methods mainly rely on a single indicator, such as global contrast or single texture complexity, which cannot fully reflect the multidimensional characteristics of the image, resulting in inaccurate enhancement mode selection. The lack of consideration of the balance between local image details and global structure makes it difficult to process images with uneven contrast or rich details. Secondly, the existing feature retrieval index structure selection relies on a fixed algorithm and fails to adaptively select the optimal retrieval structure according to the distribution and complexity of feature data, resulting in unsatisfactory retrieval performance and efficiency in different data scenarios. Therefore, an adaptive image enhancement method that comprehensively considers the local contrast and overall complexity of the image, as well as a dynamic retrieval structure selection method based on multidimensional feature fusion coefficients, are proposed, which can effectively improve the accuracy and efficiency of image preprocessing and retrieval.

[0004] In order to solve the above problems, a technical solution is now provided. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present invention provide a spatial image retrieval system and retrieval method based on multi-source feature fusion. By comprehensively considering the local contrast and overall complexity of the image, a method for adaptively selecting an image enhancement mode is proposed to ensure that the best preprocessing effect can be obtained in different types of images. First, a multi-scale convolutional neural network (MSCNN) is used to extract multi-scale features of the preprocessed multimodal medical image to fully capture the details and global information of the image, significantly improving the expressiveness and information content of the features. On this basis, the spatial attention mechanism and the channel attention mechanism are applied to weight the multi-scale features, and the irrelevant information is suppressed by accurately extracting and emphasizing important spatial positions and channel features, further enhancing the distinguishability of the features. Subsequently, the weighted spatial feature map and the channel feature map are fused through a 1x1 convolutional layer to ensure the richness and accuracy of the features. The weighted multimodal medical image features are fused through a feature fusion algorithm to generate a comprehensive feature vector containing multimodal information, significantly improving the information content and expressiveness of the image. Support vector machine regression is used to analyze the multimodal density distribution index and high-dimensional feature entropy index, and the optimal retrieval index structure is dynamically selected to ensure that the image features in the database can be matched quickly and efficiently in different data scenarios. The candidate features are quickly located through an efficient index structure, and combined with a similarity measurement method, the accuracy and reliability of feature matching are guaranteed. Overall, the present invention significantly improves the processing performance and efficiency of multimodal medical images through systematic feature extraction, weighting, fusion and matching methods, providing solid technical support for the rapid and accurate diagnosis and analysis of medical images, so as to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] S1, preprocessing the input multimodal medical images, analyzing the details and structural characteristics of the images, and deciding the image enhancement mode to ensure the image quality and subsequent processing quality;

[0008] S2, uses a multi-scale convolutional neural network to extract multi-scale features from the pre-processed multi-modal medical images to obtain information of different scales of the image to capture details and global information in the image;

[0009] S3, based on the extracted multi-scale features, applies the spatial attention mechanism and channel attention mechanism to weight the features and obtain weighted multimodal medical image features to highlight important features and suppress irrelevant information;

[0010] S4, fusing the weighted multimodal medical image features through a feature fusion algorithm to obtain a comprehensive feature vector containing multimodal information to enhance the information content and expression ability of the image;

[0011] S5, based on the data distribution and information content of the comprehensive feature vector, generates multi-dimensional feature fusion coefficients through support vector machine regression, selects the optimal index structure, quickly matches the database image features, and retrieves the image most similar to the input image;

[0012] S6, sorting the retrieved images according to their similarity, selecting several images with the highest similarity as the final retrieval results, and displaying them to the user.

[0013] In a preferred embodiment, step S1 includes the following contents:

[0014] S1-1, preprocessing the input multimodal medical image, including image denoising, enhancement and normalization operations;

[0015] S1-2, denoising the image using mean filtering to reduce the noise in the image;

[0016] S1-3, extracting local equalization information and global equalization information of the image, wherein the local equalization information includes a local contrast index, and the global equalization information includes a complexity sensitivity factor, constructing a contrast complexity coefficient through the local contrast index and the complexity sensitivity factor, and selecting and deciding whether to use histogram equalization or adaptive histogram equalization;

[0017] S1-4, normalize the image, scale the image pixel values ​​to a specific range, and standardize the image data.

[0018] In a preferred embodiment, the logic for obtaining the local contrast index is as follows:

[0019] The image is divided into multiple local regions, each of which is Pixels;

[0020] For each local area, calculate the mean ;

[0021] Calculate the local contrast index: ;

[0022] in, Represents a set of local regions in an image; is a constant used to prevent division by zero; Represents the total number of pixels in the image; Indicates that the image is at position The pixel value of

[0023] The logic for obtaining the complexity sensitive factor is as follows:

[0024] Use the Laplace operator to detect edges and get edge images : ;

[0025] in, Indicates that the edge image is at position The edge strength value of Indicates that the image is The second derivative in direction indicates the horizontal edge change; Indicates that the image is The second derivative in direction indicates vertical edge changes;

[0026] For each pixel, calculate the gray value probability distribution within its local window ;

[0027] Calculate the local entropy value for each pixel: ;

[0028] in, Indicates at location The local entropy value of Represents pixel value Probability distribution within the surrounding window;

[0029] Combine edge strength and local entropy to calculate complexity sensitivity factor : ;

[0030] in, It represents the complexity sensitivity factor, which comprehensively measures the edge strength and texture complexity of the image;

[0031] Combining the local contrast index and the complexity sensitivity factor, the contrast complexity coefficient is calculated using the complementary product method: ;

[0032] in, represents the contrast complexity coefficient;

[0033] , and is a tuning factor that ensures a balanced influence of the local contrast index and the complexity sensitivity factor;

[0034] Used to adjust the product weight between local contrast and complexity;

[0035] Used to smooth the sum of local contrast index and complexity sensitivity factor to prevent extreme value influence;

[0036] Used to balance the relative importance of the local contrast index and complexity sensitivity factor in the numerator and denominator;

[0037] The mode is selected according to the value of the contrast complexity coefficient: if the contrast complexity coefficient is greater than the contrast complexity threshold, the local contrast enhancement mode is selected; if the contrast complexity coefficient is less than or equal to the contrast complexity threshold, the adaptive histogram equalization mode is selected;

[0038] The image is processed accordingly according to the selected enhancement mode.

[0039] In a preferred embodiment, step S2 includes the following contents:

[0040] S2-1, build a multi-scale convolutional neural network model, including multiple convolutional layers, pooling layers, and activation function layers, to ensure that the network can capture the multi-scale features of the image;

[0041] S2-2, data augmentation is performed on the preprocessed multimodal medical images, including random cropping, rotation, flipping, and scaling operations;

[0042] S2-3, input the preprocessed multimodal medical image into the multi-scale convolutional neural network model to perform forward propagation calculation;

[0043] S2-4, during the training process, the back propagation algorithm and optimizer are used to optimize the network parameters and minimize the loss function;

[0044] S2-5, use the cross-validation method to evaluate the model performance, divide the data set into training set, validation set and test set, which are used for model training, validation and testing respectively;

[0045] S2-6, after the model training is completed, multi-scale feature extraction is performed on the preprocessed multimodal medical images, and the images are input layer by layer into the trained multi-scale convolutional neural network model to extract feature maps of different scales.

[0046] In a preferred embodiment, step S3 includes the following contents:

[0047] S3-1, applying spatial attention mechanism to weighted processing of the extracted multi-scale features;

[0048] Perform global average pooling on multi-scale features to obtain the average pooling map of the feature map; perform global maximum pooling on multi-scale features to obtain the maximum pooling map of the feature map; splice the average pooling map and the maximum pooling map along the channel dimension to form a new feature map; input the spliced ​​feature map into the 1x1 convolution layer, extract the spatial attention map through the convolution operation, and generate the spatial attention map; multiply the original multi-scale features and the spatial attention map element by element to generate a weighted spatial feature map;

[0049] S3-2, apply the channel attention mechanism to weighted processing of the extracted multi-scale features;

[0050] Perform global average pooling on the multi-scale features to obtain the average pooling value of each channel to form a first feature vector; perform global maximum pooling on the multi-scale features to obtain the maximum pooling value of each channel to form a second feature vector; input the first feature vector and the second feature vector into a shared multi-layer perceptron to generate a channel attention vector; multiply the original multi-scale features by the channel attention vector channel by channel to generate a weighted channel feature map;

[0051] S3-3, fusing the weighted spatial feature map and the channel feature map to generate the final weighted multimodal medical image feature;

[0052] S3-4, normalize the weighted multimodal medical image features.

[0053] In a preferred embodiment, step S4 includes the following contents:

[0054] S4-1, performing feature extraction and dimensionality reduction processing on the weighted spatial feature map and channel feature map respectively;

[0055] The weighted spatial feature map is extracted using a fully connected layer to convert the high-dimensional feature map into a reduced-dimensional spatial feature vector. The weighted channel feature map is processed in the same way and the high-dimensional channel feature map is converted into a reduced-dimensional channel feature vector through a fully connected layer.

[0056] S4-2, concatenate the reduced-dimensional spatial feature vector and the channel feature vector to form a new high-dimensional feature vector;

[0057] S4-3, applying a feature fusion algorithm to fuse the concatenated high-dimensional feature vectors;

[0058] The attention mechanism is used to weight the high-dimensional feature vector to generate an attention weight vector; the high-dimensional feature vector is multiplied element by element by the attention weight vector to obtain a weighted feature vector; the fully connected layer is used to fuse and reduce the dimension of the weighted feature vector to generate the final comprehensive feature vector;

[0059] S4-4, normalize the fused comprehensive feature vector.

[0060] In a preferred embodiment, step S5 includes the following contents:

[0061] S5-1, constructing a comprehensive feature vector data set according to the standardized comprehensive feature vector in step S4-4, extracting density distribution information and feature entropy information, wherein the density distribution information includes a multimodal density distribution index, and the feature entropy information includes a high-dimensional feature entropy index, and obtaining a multidimensional feature fusion coefficient by subjecting the multimodal density distribution index and the high-dimensional feature entropy index to a support vector machine regression method;

[0062] Select the inverted index, kd tree or LSH feature retrieval index structure through the multi-dimensional feature fusion coefficient;

[0063] Among them, the process of obtaining the multimodal density distribution index is as follows:

[0064] For each comprehensive feature vector, calculate the Euclidean distance between it and the adjacent comprehensive feature vectors;

[0065] Calculate the number of neighboring comprehensive feature vectors within the neighborhood threshold;

[0066] The local density is equal to the number of neighboring comprehensive feature vectors divided by the power of the dimension of the neighboring threshold;

[0067] The average density of all integrated eigenvectors is obtained by calculating the average of all local densities;

[0068] The standard deviation of the density distribution is obtained by calculating the average of the sum of squares of the differences between each local density and the mean density and then taking the square root;

[0069] The standard deviation of the density distribution is divided by the mean density to obtain the multimodal density distribution index;

[0070] Among them, the process of obtaining the high-dimensional feature entropy index is as follows:

[0071] For each feature dimension, calculate the probability distribution of the corresponding value;

[0072] The entropy value of the feature dimension is obtained by calculating the probability of each eigenvalue multiplied by its negative logarithm and then summing all eigenvalues;

[0073] By calculating the average of the entropy values ​​of all feature dimensions, the entropy values ​​of all feature dimensions are averaged to obtain the comprehensive entropy value;

[0074] Combining the entropy values ​​of all feature dimensions, we can get the high-dimensional feature entropy index;

[0075] Through support vector machine regression analysis, a nonlinear model is constructed by combining multimodal density distribution index and high-dimensional feature entropy index to generate multidimensional feature fusion coefficients;

[0076] According to the size of the multi-dimensional feature fusion coefficient, different feature retrieval index structures are used:

[0077] If the multi-dimensional feature fusion coefficient is less than or equal to the first threshold, select the inverted index;

[0078] If the multi-dimensional feature fusion coefficient is greater than the first threshold and less than or equal to the second threshold, select the kd tree;

[0079] If the multi-dimensional feature fusion coefficient is greater than the second threshold, LSH is selected;

[0080] S5-2, matching the comprehensive feature vector with the image features in the database;

[0081] The similarity between the comprehensive feature vector and the candidate feature vector is calculated one by one. By calculating the similarity, the similarity between the input image feature and each image feature in the database is evaluated.

[0082] In a preferred embodiment, step S6 includes the following contents:

[0083] The calculated similarity results are arranged in descending order, and the top N images with the highest similarity are selected as the final retrieval results and displayed to the user.

[0084] A spatial image retrieval system based on multi-source feature fusion, comprising: a preprocessing module, a feature extraction module, a feature weighting module, a feature fusion module, an index selection module and a result sorting module;

[0085] The preprocessing module preprocesses the input multimodal medical images, analyzes the details and structural characteristics of the images, decides on the image enhancement mode to ensure the image quality and subsequent processing quality, and sends the preprocessed images to the feature extraction module;

[0086] The feature extraction module uses a multi-scale convolutional neural network to extract multi-scale features from the pre-processed multi-modal medical images, obtains information of different scales of the image to capture details and global information in the image, and sends the multi-scale features to the feature weighting module;

[0087] Based on the extracted multi-scale features, the feature weighting module applies the spatial attention mechanism and the channel attention mechanism to weight the features to obtain weighted multimodal medical image features to highlight important features and suppress irrelevant information, and then sends the weighted features to the feature fusion module;

[0088] The feature fusion module fuses the weighted multimodal medical image features through a feature fusion algorithm to obtain a comprehensive feature vector containing multimodal information to enhance the information content and expression ability of the image, and sends the comprehensive feature vector to the index selection module;

[0089] The index selection module generates multi-dimensional feature fusion coefficients through support vector machine regression according to the data distribution and information volume of the comprehensive feature vector, selects the optimal index structure, quickly matches the database image features, retrieves the image most similar to the input image, and sends the retrieval results to the result sorting module;

[0090] The result sorting module sorts the retrieved images according to their similarity, selects several images with the highest similarity as the final retrieval results, and displays them to the user.

[0091] The technical effects and advantages of the spatial image retrieval system and retrieval method based on multi-source feature fusion of the present invention are as follows:

[0092] 1. The present invention realizes the adaptive selection of image enhancement mode by comprehensively considering the local contrast and overall complexity of the image, ensuring that the best preprocessing effect can be obtained in different types of images. It includes performing various preprocessing operations on the image to effectively improve the image quality and the stability of subsequent processing. By removing noise, enhancing image contrast and details, and standardizing image data, the image is made clearer and more consistent. On this basis, the local and overall information of the image is extracted and comprehensively evaluated, and the contrast complexity coefficient is constructed. The most suitable image enhancement mode is dynamically selected through the contrast complexity coefficient. It overcomes the shortcomings of the prior art that a single indicator cannot fully reflect the multidimensional characteristics of the image and ignores the balance between the local details and the global structure of the image, ensuring that the optimal enhancement algorithm can be adaptively selected when processing various types of images, thereby significantly improving the effect of image preprocessing and the overall processing efficiency.

[0093] 2. The present invention extracts low-level, intermediate and high-level features of multimodal medical images through a multi-scale convolutional neural network (MSCNN), comprehensively captures the details and global information of the image, and thus significantly improves the expressiveness and information content of the features. Through the combination of multiple convolutional layers and pooling layers, the multi-scale features of the image are gradually extracted to ensure that information of different scales is effectively captured and processed. In the extraction process, the global average pooling and maximum pooling operations are used to enhance the ability to capture the overall structure and significant features of the image, reduce the computational complexity and reduce the risk of overfitting. The multi-scale features not only retain the subtle changes and important details of the image, but also capture the overall structure of the image, providing a rich and comprehensive feature representation. These multi-scale features lay a solid foundation for the subsequent application of the attention mechanism and feature fusion, ensuring that when processing multimodal medical images, all useful information in the image can be fully utilized to improve the overall processing effect and accuracy. Through this process, high-quality feature input can be provided for subsequent image processing tasks, significantly improving the performance and efficiency of multimodal medical image processing.

[0094] 3. The present invention applies spatial attention mechanism and channel attention mechanism to perform weighted processing on multi-scale features, thereby effectively highlighting important information in the image and suppressing irrelevant information. Through the spatial attention mechanism, important spatial positions in the image can be accurately extracted and emphasized, and the spatial resolution of feature expression can be enhanced; while the channel attention mechanism improves the expressiveness of important channel features in the image and enhances the distinguishability of features by weighting the information of different channels. The weighted spatial feature map and channel feature map are added element by element, and the information is further fused through the 1x1 convolution layer to ensure the richness and accuracy of the comprehensive features. The normalization process ensures the consistency and stability of the feature values, eliminates the scale differences of the feature values, and improves the effect of model training. Together, the weighted multimodal medical image features are significantly improved in expressiveness and distinguishing ability, providing higher quality feature support for subsequent image processing tasks, thereby effectively improving the accuracy and efficiency of the overall processing.

[0095] 4. The present invention generates a comprehensive feature vector containing multimodal information by fusing weighted multimodal medical image features, thereby significantly enhancing the information content and expression ability of the image. Feature extraction and dimensionality reduction processing are performed through the fully connected layer, which effectively reduces the computational complexity while retaining important feature information. The spatial feature vector and the channel feature vector are spliced ​​to combine the advantages of different features to ensure the integrity and diversity of information. The feature fusion algorithm is applied to dynamically adjust the feature weights through the attention mechanism, highlight the key features, suppress irrelevant information, and further enhance the expression ability of the features. Normalization processing standardizes the data distribution of the comprehensive feature vector, eliminates the scale difference of the eigenvalues, and ensures the stability and consistency of the features. The final generated comprehensive feature vector not only contains rich multimodal information, but also has higher expression and differentiation capabilities, providing high-quality feature support for subsequent image processing tasks, thereby significantly improving the performance and efficiency of multimodal medical image processing.

[0096] 5. The present invention achieves high efficiency and accuracy in feature matching and retrieval through standardized processing of multimodal medical image features and calculation of multidimensional feature fusion coefficients. Support vector machine regression analysis is used to dynamically select the optimal retrieval index structure to ensure optimal performance in different data scenarios. The candidate features are quickly located through an efficient index structure, combined with a similarity measurement method to ensure the accuracy and reliability of feature matching. Overall, these measures significantly improve the efficiency and quality of multimodal medical image feature matching and retrieval, providing solid technical support for the rapid and accurate diagnosis and analysis of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 A schematic diagram of a flow chart of a spatial image retrieval method based on multi-source feature fusion according to the present invention;

[0098] Figure 2 It is a structural schematic diagram of a spatial image retrieval system based on multi-source feature fusion of the present invention. DETAILED DESCRIPTION

[0099] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0100] Example 1

[0101] Figure 1 The present invention provides a spatial image retrieval method based on multi-source feature fusion, comprising:

[0102] S1, preprocessing the input multimodal medical images, analyzing the details and structural characteristics of the images, and deciding the image enhancement mode to ensure the image quality and subsequent processing quality;

[0103] S2, uses a multi-scale convolutional neural network to extract multi-scale features from the preprocessed multi-modal medical images to obtain information of different scales of the image to capture details and global information in the image;

[0104] S3, based on the extracted multi-scale features, applies the spatial attention mechanism and channel attention mechanism to weight the features and obtain weighted multimodal medical image features to highlight important features and suppress irrelevant information;

[0105] S4, fusing the weighted multimodal medical image features through a feature fusion algorithm to obtain a comprehensive feature vector containing multimodal information to enhance the information content and expression ability of the image;

[0106] S5, based on the data distribution and information content of the comprehensive feature vector, generates multi-dimensional feature fusion coefficients through support vector machine regression, selects the optimal index structure, quickly matches the database image features, and retrieves the image most similar to the input image;

[0107] S6, sorting the retrieved images according to their similarity, selecting several images with the highest similarity as the final retrieval results, and displaying them to the user.

[0108] Step S1 includes the following contents:

[0109] S1-1, preprocessing the input multimodal medical images, including image denoising, enhancement and normalization operations, to ensure image quality and stability of subsequent processing.

[0110] S1-2, use algorithms such as mean filtering, Gaussian filtering or median filtering to denoise the image to reduce noise in the image and improve the image quality.

[0111] S1-3, enhance the image by using methods such as histogram equalization or adaptive histogram equalization (CLAHE) to improve the contrast and details of the image and make the image clearer.

[0112] Extract local equalization information and global equalization information of the image, where the local equalization information includes the local contrast index, and the global equalization information includes the complexity sensitivity factor. Construct a contrast complexity coefficient through the local contrast index and the complexity sensitivity factor, and select and decide whether to use histogram equalization or adaptive histogram equalization.

[0113] The local contrast enhancement mode makes the image details more obvious by improving the contrast of local areas of the image. It is suitable for situations where the image contrast is low but the details are rich. The adaptive histogram equalization (CLAHE) mode performs histogram equalization in the local area of ​​the image to enhance the local contrast. It is suitable for images with uneven contrast and can effectively improve the details and layering of the image.

[0114] There are many deficiencies in the existing technology in the mode selection of image preprocessing. First, most existing methods mainly rely on a single indicator, such as global contrast or single texture complexity. This single indicator cannot fully reflect the multidimensional characteristics of the image, resulting in the selected enhancement mode is not accurate enough. Secondly, these methods often ignore the balance between the local details and global structure of the image, which makes it difficult to process images with uneven contrast or rich details. In addition, the standard for mode selection in the existing technology is relatively simple, and lacks the ability to adapt to different image types. As a result, when processing multimodal medical images, the advantages of various enhancement algorithms cannot be fully utilized, resulting in unsatisfactory image preprocessing effects. In response to the above problems, a mode selection method based on contrast complexity index (CCX) is proposed. By combining the local contrast index (LCI) and the complexity sensitivity factor (CSF), the local contrast and overall complexity of the image are comprehensively evaluated, so as to dynamically select the most suitable image enhancement mode to ensure that the best preprocessing effect can be obtained in different types of images.

[0115] Among them, the logic of obtaining the local contrast index is as follows:

[0116] The image is divided into multiple local regions, each of which is Pixels;

[0117] For each local area, calculate the mean of the area ;

[0118] Calculate the local contrast index : ;

[0119] in, Represents a set of local regions in an image; is a small constant used to prevent division by zero; Represents the total number of pixels in the image; Indicates that the image is at position The pixel value of .

[0120] The local contrast index (LCI) measures the contrast of a local area of ​​an image by comparing the grayscale value of each pixel with the average grayscale value of the local area. The specific steps include dividing the image into multiple local areas, calculating the average grayscale value of the local area where each pixel is located, and calculating the difference between the grayscale value of each pixel and its local average value, and performing normalization processing to obtain the local contrast index of the entire image. The local contrast index can reflect the contrast level of the local area in the image. The higher the value, the stronger the local contrast of the image and the clearer the details.

[0121] The logic for obtaining the complexity sensitive factor is as follows:

[0122] Use the Laplace operator to detect edges and get edge images : ;

[0123] in, Indicates that the edge image is at position The edge strength value of Indicates that the image is The second derivative in direction indicates the horizontal edge change; Indicates that the image is The second derivative in direction, indicating vertical edge changes.

[0124] By calculating the second-order derivative of the image in the horizontal and vertical directions and taking its absolute value, the edge intensity value of each pixel is obtained. The second-order derivative can highlight the edges and contours in the image. The edge intensity value reflects the edge information at the pixel point. The greater the intensity, the more obvious the edge.

[0125] For each pixel, calculate the gray value probability distribution within its local window .

[0126] Calculate the local entropy value for each pixel : ;

[0127] in, Indicates at location The local entropy value of Represents pixel value Probability distribution within the surrounding window.

[0128] By calculating the probability distribution of the grayscale value of each pixel in its local window, and then calculating the entropy of these probability values, the entropy value measures the complexity of the grayscale distribution in the local area. The local entropy value reflects the texture complexity of the image in this area. The higher the entropy value, the more complex the texture of the area.

[0129] Combine edge strength and local entropy to calculate complexity sensitivity factor : ;

[0130] in, It represents the complexity sensitivity factor, which comprehensively measures the edge strength and texture complexity of the image;

[0131] By multiplying edge strength and local entropy, the complexity value of each pixel comprehensively considers the edge features and texture features at that point. The rationale for this is that the structural complexity of an image depends not only on significant edge information, but also on subtle texture information. Simple edge strength or local entropy alone may not be able to fully reflect the complexity of an image, while the combination of the two can take into account both the macroscopic structure (edge) and microscopic details (texture) of the image within a unified framework.

[0132] The complexity sensitivity factor is a global measure that reflects the average structural complexity of the entire image by summing up the complexity values ​​of all pixels and dividing by the total number of pixels. Normalization ensures that the complexity evaluation between images of different sizes is comparable, making the complexity sensitivity factor a stable indicator for comparing and analyzing the complexity of different images.

[0133] Combining the local contrast index and the complexity sensitivity factor, the contrast complexity coefficient is calculated using the complementary product method: ;

[0134] in, represents the contrast complexity coefficient;

[0135] , and is a tuning factor that ensures a balanced influence of the local contrast index and the complexity sensitivity factor.

[0136] Used to adjust the product weight between local contrast and complexity;

[0137] Used to smooth the sum of local contrast index and complexity sensitivity factor to prevent extreme value influence;

[0138] Used to balance the relative importance of the local contrast index and complexity sensitivity factor in the numerator and denominator.

[0139] The mode is selected according to the value of the contrast complexity coefficient: if the contrast complexity coefficient is greater than the contrast complexity threshold, the local contrast enhancement mode is selected; if the contrast complexity coefficient is less than or equal to the contrast complexity threshold, the adaptive histogram equalization mode is selected.

[0140] The image is processed accordingly according to the selected enhancement mode.

[0141] S1-4, normalize the image, scale the image pixel values ​​to a specific range (such as 0 to 1), and standardize the image data to facilitate subsequent processing and model training.

[0142] The present invention realizes the adaptive selection of image enhancement mode by comprehensively considering the local contrast and overall complexity of the image, ensuring that the best preprocessing effect can be obtained in different types of images. It includes performing various preprocessing operations on the image to effectively improve the image quality and the stability of subsequent processing. By removing noise, enhancing image contrast and details, and standardizing image data, the image is made clearer and more consistent. On this basis, the local and overall information of the image is extracted and comprehensively evaluated, and a contrast complexity coefficient is constructed. The most suitable image enhancement mode is dynamically selected through the contrast complexity coefficient. It overcomes the shortcomings of the prior art that a single indicator cannot fully reflect the multidimensional characteristics of the image and ignores the balance between the local details and the global structure of the image, ensuring that the optimal enhancement algorithm can be adaptively selected when processing various types of images, thereby significantly improving the effect of image preprocessing and the overall processing efficiency.

[0143] After completing the preprocessing of multimodal medical images, the noise of the image has been effectively removed, and the contrast and details have been significantly improved, ensuring the basic quality of the image. However, simple preprocessing cannot fully extract the rich information in the image. Especially when processing complex medical images, it is necessary to further extract multi-scale features to fully capture the details and global information in the image. Through the multi-scale convolutional neural network (MSCNN), low-level, medium-level and high-level features of the image can be extracted from different scales. These multi-scale features can not only reflect the subtle changes in the image, but also capture the overall structural information of the image, providing high-quality feature input for subsequent processing steps. The purpose of multi-scale feature extraction is to lay a solid foundation for weighted processing and feature fusion, ensuring that when processing multimodal medical images, all useful information in the image can be maximized, thereby improving the accuracy and comprehensiveness of feature expression. Through this step, rich and reliable feature data will be provided for the subsequent application of attention mechanism and feature fusion, further optimizing the effect of image processing.

[0144] Step S2 includes the following contents:

[0145] S2-1, build a multi-scale convolutional neural network (MSCNN) model, including multiple convolutional layers, pooling layers, and activation function layers to ensure that the network can capture the multi-scale features of the image. Build a network architecture containing multiple convolutional layers, each followed by a nonlinear activation function (such as ReLU) and a pooling layer. Through the combination of multiple convolutional layers and pooling layers, gradually extract low-level, mid-level, and high-level features of the image to obtain information at different scales.

[0146] When building a network architecture with multiple convolutional layers, we first designed a five-layer convolutional neural network, where each convolutional layer is followed by a ReLU nonlinear activation function and a pooling layer. In the first layer, 32 3x3 convolution kernels are used to convolve the preprocessed multimodal medical image to extract primary features, and then the ReLU activation function is applied to introduce nonlinearity, followed by a 2x2 max pooling layer to reduce the spatial dimension of the feature map. In the second layer, 64 3x3 convolution kernels are used to further extract intermediate features, and the ReLU activation function and 2x2 max pooling layer are still used. The third layer uses 128 3x3 convolution kernels, the fourth layer uses 256 3x3 convolution kernels, and the fifth layer uses 512 3x3 convolution kernels. All layers use ReLU activation functions and 2x2 max pooling layers to extract higher-level features of the image layer by layer. This combination of multi-layer convolution and pooling enables the network to gradually capture the low-level, mid-level and high-level features of the image, obtain information at different scales, fully extract the details and global information in multimodal medical images, and finally obtain a comprehensive feature representation in the last layer of the network, providing high-quality feature support for subsequent medical image processing tasks.

[0147] S2-2, data augmentation is performed on the preprocessed multimodal medical images, including random cropping, rotation, flipping, and scaling operations to increase the diversity of the data and improve the robustness and generalization ability of the model. The data augmentation step helps the network better learn and capture the details and global information in the image by generating more variant images during the training process.

[0148] S2-3, input the preprocessed multimodal medical image into the MSCNN model for forward propagation calculation. Each convolution layer extracts local features of the image through sliding window operation, and reduces the spatial dimension of the feature map through the pooling layer to reduce the amount of calculation and the risk of overfitting. In the process of forward propagation, the multi-scale features of the image are extracted layer by layer, and finally a comprehensive feature representation is obtained in the last layer of the network.

[0149] S2-4, during the training process, the back propagation algorithm and optimizer (such as Adam or SGD) are used to optimize the network parameters and minimize the loss function. The loss function can be selected as mean square error (MSE) or cross entropy loss, etc., and adjusted according to the specific task. Through iterative optimization, the convolution kernel parameters are gradually adjusted so that the model can extract image features more accurately.

[0150] S2-5, use the cross-validation method to evaluate the model performance, divide the data set into training set, validation set and test set, which are used for model training, validation and testing respectively. Through evaluation on the validation set, adjust the model's hyperparameters (such as learning rate, regularization parameters, etc.) to obtain the best model performance.

[0151] S2-6, after the model training is completed, multi-scale feature extraction is performed on the pre-processed multimodal medical images, and the images are input layer by layer into the trained MSCNN model to extract feature maps of different scales. The feature maps contain detailed information and global information of the image, which can be used for subsequent image fusion, classification or other tasks.

[0152] The detailed multi-scale feature extraction steps in step S2 ensure that the details and global information of the image can be fully captured when processing multimodal medical images, providing high-quality feature representation for subsequent image processing tasks. This method is innovative and significantly improves the effect and efficiency of image feature extraction by constructing and optimizing multi-scale convolutional neural networks and combining data enhancement and cross-validation techniques.

[0153] The present invention extracts low-level, intermediate and high-level features of multimodal medical images through a multi-scale convolutional neural network (MSCNN), comprehensively captures the details and global information of the image, and thus significantly improves the expressiveness and information content of the features. Through the combination of multiple convolutional layers and pooling layers, the multi-scale features of the image are gradually extracted to ensure that information of different scales is effectively captured and processed. In the extraction process, the global average pooling and maximum pooling operations are used to enhance the ability to capture the overall structure and significant features of the image, reduce the computational complexity and reduce the risk of overfitting. The multi-scale features not only retain the subtle changes and important details of the image, but also capture the overall structure of the image, providing a rich and comprehensive feature representation. These multi-scale features lay a solid foundation for the subsequent application of attention mechanisms and feature fusion, ensuring that when processing multimodal medical images, all useful information in the image can be fully utilized to improve the overall processing effect and accuracy. Through this process, high-quality feature input can be provided for subsequent image processing tasks, significantly improving the performance and efficiency of multimodal medical image processing.

[0154] After completing the preprocessing and multi-scale feature extraction of multimodal medical images, the obtained multi-scale features contain the details and global information of the image. However, these features still contain a lot of redundant and irrelevant information. Directly using these features for subsequent processing may affect the performance and accuracy of the model. Therefore, it is necessary to further optimize these features so that they can more effectively express the key information in the image. By introducing the attention mechanism, the weights of the features can be dynamically adjusted to highlight important features and suppress irrelevant information, thereby improving the expressiveness and distinguishing ability of the features. Specifically, the application of the spatial attention mechanism can enhance the important spatial position information in the feature map, while the channel attention mechanism can highlight the important channel information in the feature map. Combining these two attention mechanisms and weighting the features from different sources can significantly improve the quality of multimodal medical image features and provide more accurate and reliable feature support for subsequent image processing tasks. Through these optimization steps, it is ensured that when processing different types of multimodal medical images, the important information in the image can be effectively captured and the overall processing effect can be improved.

[0155] Step S3 includes the following contents:

[0156] S3-1, apply the spatial attention mechanism to the extracted multi-scale features for weighted processing. First, perform global average pooling on the multi-scale features to obtain the average pooling map of the feature map. This step is used to extract the average features of the entire image and highlight the main structural information of the image. Then, perform global maximum pooling on the multi-scale features to obtain the maximum pooling map of the feature map. The most significant feature information in the image is retained through the maximum pooling operation. The average pooling map and the maximum pooling map are spliced ​​along the channel dimension to form a new feature map, which integrates the global average information and the maximum feature information of the image. Next, the spliced ​​feature map is input into a 1x1 convolution layer, and the spatial attention map is extracted through the convolution operation. The convolution layer generates the spatial attention map by linearly combining the information of different channels. Finally, the original multi-scale features are element-by-element multiplied with the spatial attention map to generate a weighted spatial feature map. In this way, the spatial attention mechanism can highlight important spatial positions in the image, suppress irrelevant information, and improve the accuracy of feature expression.

[0157] S3-2, the channel attention mechanism is applied to the extracted multi-scale features for weighted processing. First, the multi-scale features are globally averaged pooled to obtain the average pooled value of each channel to form the first feature vector. The global information of each channel can be obtained through global average pooling. Then, the multi-scale features are globally max-pooled to obtain the maximum pooled value of each channel to form the second feature vector. The most significant information of each channel can be retained through global max-pooling. The first feature vector and the second feature vector are input into a shared multi-layer perceptron (MLP). The MLP consists of two fully connected layers and a ReLU activation function. The feature vector is nonlinearly transformed and weighted by the multi-layer perceptron to generate a channel attention vector. Finally, the original multi-scale features are multiplied by the channel attention vector channel by channel to generate a weighted channel feature map. The channel attention mechanism can highlight the important channel information in the image, suppress the information of irrelevant channels, and improve the ability to distinguish features by weighting the features of each channel.

[0158] S3-3, fuse the weighted spatial feature map and channel feature map to generate the final weighted multimodal medical image features. The specific steps are as follows: add the weighted spatial feature map and channel feature map element by element to generate a preliminary fused feature map. Element-by-element addition means that the values ​​at each position in the two feature maps are added correspondingly to form a new feature map. In this process, the feature information at each position will be taken into account at the same time, taking into account the influence of spatial attention and channel attention, thereby combining the advantages of the two attention mechanisms.

[0159] After the initial fusion feature map is generated, it is input into a 1x1 convolution layer to further fuse and refine the feature information. The role of the 1x1 convolution layer is to reweight and integrate the features by linearly combining the information of different channels. This step effectively reduces the redundant information between features and enhances the ability to express important features. Finally, a preliminary fusion feature map is generated, which has higher expressive and distinguishing capabilities and can better support subsequent image processing tasks. In this way, the effective enhancement of important information in the image is ensured, while irrelevant information is suppressed, and the quality of the overall features is improved.

[0160] S3-4, normalize the weighted multimodal medical image features. Perform batch normalization or layer normalization on the feature map. Batch normalization normalizes the features of each batch to eliminate the scale differences of the feature values ​​and ensure the stability of the features during training. Layer normalization normalizes the features of each layer to standardize the feature data and make the feature distribution more consistent. Normalization effectively prevents the problem of feature values ​​being too large or too small, improves the efficiency and effect of model training, and also improves the stability and consistency of the features, providing a reliable foundation for subsequent processing and model training.

[0161] The present invention performs weighted processing on multi-scale features by applying spatial attention mechanism and channel attention mechanism, thereby effectively highlighting important information in the image and suppressing irrelevant information. Through the spatial attention mechanism, important spatial positions in the image can be accurately extracted and emphasized, and the spatial resolution of feature expression can be enhanced; while the channel attention mechanism improves the expressiveness of important channel features in the image and enhances the distinguishability of features by weighting the information of different channels. The weighted spatial feature map and channel feature map are added element by element, and the information is further fused through the 1x1 convolution layer to ensure the richness and accuracy of the comprehensive features. The normalization process ensures the consistency and stability of the feature values, eliminates the scale difference of the feature values, and improves the effect of model training. Together, the weighted multimodal medical image features are significantly improved in expressiveness and distinguishing ability, providing higher quality feature support for subsequent image processing tasks, thereby effectively improving the accuracy and efficiency of the overall processing.

[0162] In multimodal medical image processing, rich multiscale features have been extracted through multiscale convolutional neural networks (MSCNN), and weighted processing has been performed through spatial and channel attention mechanisms to highlight important information and suppress irrelevant information. These weighted feature maps contain a lot of details and global information, and are high-quality inputs for subsequent processing. However, in order to further improve the expressiveness and information content of the features, these weighted multimodal features need to be fused. The purpose of feature fusion is to integrate the information in different modal images into a feature vector so that it contains as much useful information as possible while reducing redundancy. Through feature fusion, the advantages of different modalities can be complemented, so that the final feature vector can express the image content more comprehensively and accurately, providing a solid foundation for subsequent image classification, segmentation or other processing tasks. Feature fusion can not only enhance the information content of the image, but also improve the distinguishing ability of features, ensuring better results in complex medical image analysis. Therefore, implementing the feature fusion step is a crucial part of the multimodal medical image processing process, providing key support for improving overall processing performance and accuracy.

[0163] Step S4 includes the following contents:

[0164] S4-1, feature extraction and dimensionality reduction are performed on the weighted spatial feature map and channel feature map respectively. First, the fully connected layer is used to extract features from the weighted spatial feature map, and the high-dimensional feature map is converted into a reduced-dimensional spatial feature vector. The fully connected layer extracts important information from the spatial features and removes redundant information through linear transformation. Subsequently, the weighted channel feature map is processed in the same way, and the high-dimensional channel feature map is converted into a reduced-dimensional channel feature vector through the fully connected layer. The fully connected layer plays the role of compressing high-dimensional features into low-dimensional vectors here, ensuring that the important information of the original features is retained as much as possible while reducing the dimensionality. This processing step can reduce the computational complexity and lay the foundation for subsequent feature fusion.

[0165] S4-2, concatenate the reduced-dimensional spatial feature vector and channel feature vector. Concatenate the spatial feature vector and channel feature vector in the feature dimension to form a new high-dimensional feature vector. The concatenation operation directly connects the two vectors to ensure that the information of the spatial and channel features is not lost, while combining the advantages of the two features to enhance the expressiveness of the features. This processing step retains the integrity of the spatial and channel features by increasing the dimension of the feature vector, so that the comprehensive feature vector contains more information, improving the richness and diversity of the features.

[0166] S4-3, the feature fusion algorithm is applied to the concatenated high-dimensional feature vectors for fusion. First, the attention mechanism is used to weight the high-dimensional feature vectors to generate an attention weight vector. The attention mechanism dynamically adjusts the weights of each element in the feature vector by calculating the importance of each element in the feature vector, highlighting the key features and suppressing irrelevant information. The high-dimensional feature vector is multiplied element by element by the attention weight vector to obtain the weighted feature vector. This process ensures that the important information in the feature vector is strengthened. Next, a fully connected layer is used to fuse and reduce the dimension of the weighted feature vector to generate the final comprehensive feature vector. The fully connected layer further integrates information by linearly combining the features, reducing the dimension of the feature vector and improving the expressiveness of the features. Through these processing steps, the feature fusion algorithm realizes the effective integration of multimodal features, ensuring that the final feature vector has high information content and high expressiveness.

[0167] S4-4, normalize the fused comprehensive feature vector. Standardize the data distribution of the comprehensive feature vector through batch normalization or layer normalization. Normalization eliminates the scale differences of the feature values, making the features more stable and consistent during subsequent processing and model training. Batch normalization normalizes the feature vectors of each batch to ensure that the feature values ​​are within a reasonable range and eliminate the differences between different batches. Layer normalization normalizes the features of each layer to make the feature distribution more consistent. The normalization step ensures that the comprehensive feature vector is comparable and consistent in different images, improves the quality and stability of the features, and provides a solid foundation for subsequent processing.

[0168] Through the above feature fusion steps, the weighted multimodal medical image features are fused to generate a comprehensive feature vector containing multimodal information. The use of fully connected layers, splicing, attention mechanisms, and normalization processing technologies ensures that the comprehensive feature vector has higher information content and expression capabilities, providing high-quality feature support for subsequent image processing tasks. This method is innovative. By combining different feature fusion technologies and dynamically adjusting feature weights, it achieves effective integration of multimodal features and significantly improves image processing effects and information expression capabilities.

[0169] The present invention generates a comprehensive feature vector containing multimodal information by fusing weighted multimodal medical image features, thereby significantly enhancing the information content and expression ability of the image. Feature extraction and dimensionality reduction processing are performed through the fully connected layer, which effectively reduces the computational complexity while retaining important feature information. The spatial feature vector and the channel feature vector are spliced ​​to combine the advantages of different features to ensure the integrity and diversity of information. The feature fusion algorithm is applied to dynamically adjust the feature weights through the attention mechanism, highlight the key features, suppress irrelevant information, and further enhance the expression ability of the features. Normalization processing standardizes the data distribution of the comprehensive feature vector, eliminates the scale difference of the eigenvalues, and ensures the stability and consistency of the features. The final generated comprehensive feature vector not only contains rich multimodal information, but also has higher expression and differentiation capabilities, providing high-quality feature support for subsequent image processing tasks, thereby significantly improving the performance and efficiency of multimodal medical image processing.

[0170] After completing the extraction, weighted processing and fusion of multimodal medical image features, a comprehensive feature vector containing rich information is obtained. In order to quickly and accurately find the image most similar to the input image from a huge image database, it is necessary to use these comprehensive feature vectors for efficient matching and retrieval. By constructing an efficient feature retrieval index structure and applying appropriate similarity measurement methods, the most similar images can be quickly located in a large amount of image data, achieving efficient and accurate image retrieval. This process is of great significance for the diagnosis and analysis of medical images, and can help doctors and researchers quickly obtain relevant images for comparison and research, thereby improving the accuracy and efficiency of diagnosis.

[0171] Step S5 includes the following contents:

[0172] S5-1, construct a comprehensive feature vector data set based on the standardized comprehensive feature vector in step S4-4, and extract density distribution information and feature entropy information, wherein the density distribution information includes a multimodal density distribution index, and the feature entropy information includes a high-dimensional feature entropy index. In order to optimize the feature matching and retrieval performance of multimodal medical images, the multimodal density distribution index and the high-dimensional feature entropy index are subjected to support vector machine (SVM) regression to obtain a multidimensional feature fusion coefficient.

[0173] Through the multi-dimensional feature fusion coefficient, the inverted index, kd tree or LSH (local sensitive hashing) feature retrieval index structure is selected to quickly retrieve image features in the database.

[0174] The multimodal density distribution index (MMDD) is used to measure the distribution density and uniformity of the comprehensive feature vector in the high-dimensional space. By calculating the local density and global density distribution differences of the comprehensive feature vector, the ratio of the standard deviation of the feature vector to the average density is obtained. The high-dimensional feature entropy index (HFE) is used to measure the complexity and information content of the comprehensive feature vector. By calculating the probability distribution and entropy value of each feature dimension, the information complexity of the feature vector is obtained by combining the entropy values ​​of all feature dimensions. When the multimodal density distribution index is high, the feature vector is unevenly distributed, and it is suitable to select the inverted index to deal with large-scale and unevenly dense feature data; when the high-dimensional feature entropy index is high, the feature vector has a large amount of information, and it is suitable to select LSH for fast approximate nearest neighbor search of high-dimensional data; when the multimodal density distribution index and the high-dimensional feature entropy index are both moderate, it is suitable to select the kd tree to process small and medium-sized feature data with relatively uniform density. By comprehensively considering the multimodal density distribution index and the high-dimensional feature entropy index, the selection of the feature retrieval index structure can be dynamically adjusted to ensure the best retrieval performance and efficiency in different data scenarios.

[0175] Among them, the process of obtaining the multimodal density distribution index is as follows:

[0176] The multimodal density distribution index is used to measure the distribution density and uniformity of the comprehensive feature vector in high-dimensional space.

[0177] For each comprehensive feature vector, the Euclidean distance between it and the adjacent comprehensive feature vectors is calculated.

[0178] Counts the number of neighboring integrated feature vectors within a neighborhood threshold.

[0179] The local density is equal to the number of neighboring comprehensive feature vectors divided by the power of the dimension of the neighboring threshold;

[0180] The average density of all integrated eigenvectors is obtained by calculating the average of all local densities;

[0181] The standard deviation of the density distribution is obtained by calculating the average of the sum of squares of the differences between each local density and the mean density and then taking the square root;

[0182] The multimodal density distribution index is obtained by dividing the standard deviation of the density distribution by the mean density.

[0183] Among them, the process of obtaining the high-dimensional feature entropy index is as follows:

[0184] The high-dimensional feature entropy index is used to measure the complexity and information content of the feature vector.

[0185] The comprehensive feature vector is a high-dimensional data representation, where each dimension represents different feature information.

[0186] For each feature dimension, calculate the probability distribution of the corresponding value;

[0187] The entropy value of the feature dimension is obtained by calculating the probability of each eigenvalue multiplied by its negative logarithm and then summing all eigenvalues;

[0188] By calculating the average of the entropy values ​​of all feature dimensions, the entropy values ​​of all feature dimensions are averaged to obtain the comprehensive entropy value;

[0189] The entropy values ​​of all feature dimensions are combined to obtain the high-dimensional feature entropy index.

[0190] Through the support vector machine (SVM) regression analysis, a nonlinear model is constructed by combining the multimodal density distribution index and the high-dimensional feature entropy index to generate the multidimensional feature fusion coefficient;

[0191] Using historical datasets, including multimodal density distribution index and high-dimensional feature entropy index as input, and known optimal index structure as output;

[0192] Use nonlinear kernel functions (such as RBF kernel) to train the SVM regression model and fit the relationship between the two indicators and the optimal index structure;

[0193] Input the new multimodal density distribution index and high-dimensional feature entropy index value, use the trained SVM model to make predictions, and obtain the multi-dimensional feature fusion coefficient;

[0194] According to the size of the multi-dimensional feature fusion coefficient, the following feature retrieval index structure is selected:

[0195] If the multi-dimensional feature fusion coefficient is less than or equal to the first threshold, the inverted index is selected, which is suitable for large-scale comprehensive feature vectors with uneven density distribution.

[0196] If the multi-dimensional feature fusion coefficient is greater than the first threshold and less than or equal to the second threshold, the kd tree is selected, which is suitable for small and medium-sized comprehensive feature vectors with relatively uniform density distribution.

[0197] If the multi-dimensional feature fusion coefficient is greater than the second threshold, LSH is selected, which is suitable for comprehensive feature vectors with high dimensions and high information complexity.

[0198] Inverted index is an efficient data structure, mainly used in information retrieval systems. It records the correspondence between each keyword and the document to achieve fast query and retrieval, which is particularly applicable in large-scale databases. KD tree (k-dimensional tree) is a segmentation data structure used in multidimensional space. It recursively divides the space into hyper-rectangular areas to efficiently perform nearest neighbor search and range query. It is suitable for small and medium-sized databases. Locality-Sensitive Hashing (LSH) is an approximate nearest neighbor search algorithm for high-dimensional data. It maps similar feature vectors to the same bucket through hash functions, thereby achieving fast retrieval in high-dimensional space. It is particularly suitable for processing high-dimensional feature vectors.

[0199] S5-2, match the comprehensive feature vector with the image features in the database. First, quickly locate the set of candidate feature vectors similar to the input feature vector through the index structure. This process uses the efficiency of the index structure to narrow the search range and reduce the amount of calculation. Then, calculate the similarity between the comprehensive feature vector and the candidate feature vector one by one. Commonly used similarity measurement methods include Euclidean distance, cosine similarity, and Mahalanobis distance. Euclidean distance measures similarity by calculating the straight-line distance between two feature vectors, which is applicable to the case where feature vectors are on the same scale; cosine similarity measures similarity by calculating the cosine value of the angle between two feature vectors, which is applicable to the case where feature vectors are on different scales; Mahalanobis distance takes into account the covariance between feature vectors, which is applicable to the case where there is correlation between feature dimensions. By calculating the similarity, the similarity between the input image feature and each image feature in the database can be evaluated, providing a basis for subsequent sorting and retrieval.

[0200] The present invention achieves high efficiency and accuracy in feature matching and retrieval through standardized processing of multimodal medical image features and calculation of multidimensional feature fusion coefficients. Support vector machine regression analysis is used to dynamically select the optimal retrieval index structure to ensure optimal performance in different data scenarios. The candidate features are quickly located through an efficient index structure, combined with a similarity measurement method to ensure the accuracy and reliability of feature matching. Overall, these measures significantly improve the efficiency and quality of multimodal medical image feature matching and retrieval, providing solid technical support for the rapid and accurate diagnosis and analysis of medical images.

[0201] Step S6 includes the following contents:

[0202] Sort the calculated similarity results, select the images with the highest similarity as the search results, and display them to the user. Sort the similarity results from high to low, and select the first N images with the highest similarity as the final search results. Similarity sorting can be achieved through efficient algorithms such as quick sort or heap sort to ensure that the most similar images are quickly obtained in a large-scale database. The sorting process determines the similarity relationship between images by comparing similarity values ​​to ensure the accuracy and effectiveness of the search results. Selecting the images with the highest similarity can ensure the relevance of the search results and provide users with the closest image references.

[0203] Example 2

[0204] Figure 2 The present invention provides a spatial image retrieval system based on multi-source feature fusion, comprising: a preprocessing module, a feature extraction module, a feature weighting module, a feature fusion module, an index selection module and a result sorting module;

[0205] The preprocessing module preprocesses the input multimodal medical images, analyzes the details and structural characteristics of the images, decides on the image enhancement mode to ensure the image quality and subsequent processing quality, and sends the preprocessed images to the feature extraction module;

[0206] The feature extraction module uses a multi-scale convolutional neural network to extract multi-scale features from the pre-processed multi-modal medical images, obtains information of different scales of the image to capture details and global information in the image, and sends the multi-scale features to the feature weighting module;

[0207] Based on the extracted multi-scale features, the feature weighting module applies the spatial attention mechanism and the channel attention mechanism to weight the features to obtain weighted multimodal medical image features to highlight important features and suppress irrelevant information, and then sends the weighted features to the feature fusion module;

[0208] The feature fusion module fuses the weighted multimodal medical image features through a feature fusion algorithm to obtain a comprehensive feature vector containing multimodal information to enhance the information content and expression ability of the image, and sends the comprehensive feature vector to the index selection module;

[0209] The index selection module generates multi-dimensional feature fusion coefficients through support vector machine regression according to the data distribution and information volume of the comprehensive feature vector, selects the optimal index structure, quickly matches the database image features, retrieves the image most similar to the input image, and sends the retrieval results to the result sorting module;

[0210] The result sorting module sorts the retrieved images according to their similarity, selects several images with the highest similarity as the final retrieval results, and displays them to the user.

[0211] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0212] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0213] It should be noted that, in this article, if there are relational terms such as first and second, etc., they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0214] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A spatial image retrieval method based on multi-source feature fusion, characterized by: S1, preprocessing the input multimodal medical images, analyzing the details and structural characteristics of the images, and deciding the image enhancement mode to ensure the image quality and subsequent processing quality; S2, uses a multi-scale convolutional neural network to extract multi-scale features from the pre-processed multi-modal medical images to obtain information of different scales of the image to capture details and global information in the image; S3, based on the extracted multi-scale features, applies the spatial attention mechanism and channel attention mechanism to weight the features and obtain weighted multimodal medical image features to highlight important features and suppress irrelevant information; S4, fusing the weighted multimodal medical image features through a feature fusion algorithm to obtain a comprehensive feature vector containing multimodal information to enhance the information content and expression ability of the image; S5, based on the data distribution and information content of the comprehensive feature vector, generates multi-dimensional feature fusion coefficients through support vector machine regression, selects the optimal index structure, quickly matches the database image features, and retrieves the image most similar to the input image; S6, sorting the retrieved images according to their similarity, selecting several images with the highest similarity as the final retrieval results, and displaying them to the user; Step S1 includes the following contents: S1-1, preprocessing the input multimodal medical image, including image denoising, enhancement and normalization operations; S1-2, denoising the image using mean filtering to reduce the noise in the image; S1-3, extracting local equalization information and global equalization information of the image, wherein the local equalization information includes a local contrast index, and the global equalization information includes a complexity sensitivity factor, constructing a contrast complexity coefficient through the local contrast index and the complexity sensitivity factor, and selecting and deciding whether to use histogram equalization or adaptive histogram equalization; S1-4, normalizing the image, scaling the image pixel values ​​to a specific range, and standardizing the image data; Among them, the logic of obtaining the local contrast index is as follows: The image is divided into multiple local regions, each of which is Pixels; For each local area, calculate the mean ; Calculate the local contrast index: ; in, Represents a set of local regions in an image; is a constant used to prevent division by zero; Represents the total number of pixels in the image; Indicates that the image is at position The pixel value of The logic for obtaining the complexity sensitive factor is as follows: Use the Laplace operator to detect edges and get edge images : ; in, Indicates that the edge image is at position The edge strength value of Indicates that the image is The second derivative in direction indicates the horizontal edge change; Indicates that the image is The second derivative in direction indicates vertical edge changes; For each pixel, calculate the gray value probability distribution within its local window ; Calculate the local entropy value for each pixel: ; in, Indicates at location The local entropy value of Represents pixel value Probability distribution within the surrounding window; Combine edge strength and local entropy to calculate complexity sensitivity factor : ; in, It represents the complexity sensitivity factor, which comprehensively measures the edge strength and texture complexity of the image; Combining the local contrast index and the complexity sensitivity factor, the contrast complexity coefficient is calculated using the complementary product method: ; in, represents the contrast complexity coefficient; , and is a tuning factor that ensures a balanced influence of the local contrast index and the complexity sensitivity factor; Used to adjust the product weight between local contrast and complexity; Used to smooth the sum of local contrast index and complexity sensitivity factor to prevent extreme value influence; Used to balance the relative importance of the local contrast index and complexity sensitivity factor in the numerator and denominator; The mode is selected according to the value of the contrast complexity coefficient: if the contrast complexity coefficient is greater than the contrast complexity threshold, the local contrast enhancement mode is selected; if the contrast complexity coefficient is less than or equal to the contrast complexity threshold, the adaptive histogram equalization mode is selected; According to the selected enhancement mode, the image is processed accordingly; Step S3 includes the following contents: S3-1, applying spatial attention mechanism to weighted processing of the extracted multi-scale features; Perform global average pooling on multi-scale features to obtain the average pooling map of the feature map; perform global maximum pooling on multi-scale features to obtain the maximum pooling map of the feature map; splice the average pooling map and the maximum pooling map along the channel dimension to form a new feature map; input the spliced ​​feature map into the 1x1 convolution layer, extract the spatial attention map through the convolution operation, and generate the spatial attention map; multiply the original multi-scale features and the spatial attention map element by element to generate a weighted spatial feature map; S3-2, apply the channel attention mechanism to weighted processing of the extracted multi-scale features; Perform global average pooling on the multi-scale features to obtain the average pooling value of each channel to form a first feature vector; perform global maximum pooling on the multi-scale features to obtain the maximum pooling value of each channel to form a second feature vector; input the first feature vector and the second feature vector into a shared multi-layer perceptron to generate a channel attention vector; multiply the original multi-scale features by the channel attention vector channel by channel to generate a weighted channel feature map; Step S4 includes the following contents: S4-1, performing feature extraction and dimensionality reduction processing on the weighted spatial feature map and channel feature map respectively; The weighted spatial feature map is extracted using a fully connected layer to convert the high-dimensional feature map into a reduced-dimensional spatial feature vector. The weighted channel feature map is processed in the same way and the high-dimensional channel feature map is converted into a reduced-dimensional channel feature vector through a fully connected layer. S4-2, concatenate the reduced-dimensional spatial feature vector and the channel feature vector to form a new high-dimensional feature vector; S4-3, applying a feature fusion algorithm to fuse the concatenated high-dimensional feature vectors; The attention mechanism is used to weight the high-dimensional feature vector to generate an attention weight vector; the high-dimensional feature vector is multiplied element by element by the attention weight vector to obtain a weighted feature vector; the fully connected layer is used to fuse and reduce the dimension of the weighted feature vector to generate the final comprehensive feature vector; S4-4, normalize the fused comprehensive feature vector; Step S5 includes the following contents: S5-1, constructing a comprehensive feature vector data set according to the standardized comprehensive feature vector in step S4-4, extracting density distribution information and feature entropy information, wherein the density distribution information includes a multimodal density distribution index, and the feature entropy information includes a high-dimensional feature entropy index, and obtaining a multidimensional feature fusion coefficient by subjecting the multimodal density distribution index and the high-dimensional feature entropy index to a support vector machine regression method; Select the inverted index, kd tree or LSH feature retrieval index structure through the multi-dimensional feature fusion coefficient; Among them, the process of obtaining the multimodal density distribution index is as follows: For each comprehensive feature vector, calculate the Euclidean distance between it and the adjacent comprehensive feature vectors; Calculate the number of neighboring comprehensive feature vectors within the neighborhood threshold; The local density is equal to the number of neighboring comprehensive feature vectors divided by the power of the dimension of the neighboring threshold; The average density of all integrated eigenvectors is obtained by calculating the average of all local densities; The standard deviation of the density distribution is obtained by calculating the average of the sum of squares of the differences between each local density and the mean density and then taking the square root; The standard deviation of the density distribution is divided by the mean density to obtain the multimodal density distribution index; Among them, the process of obtaining the high-dimensional feature entropy index is as follows: For each feature dimension, calculate the probability distribution of the corresponding value; The entropy value of the feature dimension is obtained by calculating the probability of each eigenvalue multiplied by its negative logarithm and then summing all eigenvalues; By calculating the average of the entropy values ​​of all feature dimensions, the entropy values ​​of all feature dimensions are averaged to obtain the comprehensive entropy value; Combining the entropy values ​​of all feature dimensions, we can get the high-dimensional feature entropy index; Through support vector machine regression analysis, a nonlinear model is constructed by combining multimodal density distribution index and high-dimensional feature entropy index to generate multidimensional feature fusion coefficients; According to the size of the multi-dimensional feature fusion coefficient, different feature retrieval index structures are used: If the multi-dimensional feature fusion coefficient is less than or equal to the first threshold, select the inverted index; If the multi-dimensional feature fusion coefficient is greater than the first threshold and less than or equal to the second threshold, select the kd tree; If the multi-dimensional feature fusion coefficient is greater than the second threshold, LSH is selected; S5-2, matching the comprehensive feature vector with the image features in the database; The similarity between the comprehensive feature vector and the candidate feature vector is calculated one by one. By calculating the similarity, the similarity between the input image feature and each image feature in the database is evaluated.

2. The spatial image retrieval method based on multi-source feature fusion according to claim 1, characterized in that: Step S2 includes the following contents: S2-1, build a multi-scale convolutional neural network model, including multiple convolutional layers, pooling layers, and activation function layers, to ensure that the network can capture the multi-scale features of the image; S2-2, data augmentation is performed on the preprocessed multimodal medical images, including random cropping, rotation, flipping, and scaling operations; S2-3, input the preprocessed multimodal medical image into the multi-scale convolutional neural network model to perform forward propagation calculation; S2-4, during the training process, the back propagation algorithm and optimizer are used to optimize the network parameters and minimize the loss function; S2-5, use the cross-validation method to evaluate the model performance, divide the data set into training set, validation set and test set, which are used for model training, validation and testing respectively; S2-6, after the model training is completed, multi-scale feature extraction is performed on the preprocessed multimodal medical images, and the images are input layer by layer into the trained multi-scale convolutional neural network model to extract feature maps of different scales.

3. The spatial image retrieval method based on multi-source feature fusion according to claim 1, characterized in that: Step S6 includes the following contents: The calculated similarity results are arranged in descending order, and the top N images with the highest similarity are selected as the final retrieval results and displayed to the user.

4. A spatial image retrieval system based on multi-source feature fusion, used to implement a spatial image retrieval method based on multi-source feature fusion as claimed in any one of claims 1 to 3, comprising: Preprocessing module, feature extraction module, feature weighting module, feature fusion module, index selection module and result sorting module; The preprocessing module preprocesses the input multimodal medical images, analyzes the details and structural characteristics of the images, decides on the image enhancement mode to ensure the image quality and subsequent processing quality, and sends the preprocessed images to the feature extraction module; The feature extraction module uses a multi-scale convolutional neural network to extract multi-scale features from the pre-processed multi-modal medical images, obtains information of different scales of the image to capture details and global information in the image, and sends the multi-scale features to the feature weighting module; Based on the extracted multi-scale features, the feature weighting module applies the spatial attention mechanism and the channel attention mechanism to weight the features and obtain the weighted multimodal medical image features to highlight the important features and suppress the irrelevant information. The weighted features are sent to the feature fusion module. The feature fusion module fuses the weighted multimodal medical image features through a feature fusion algorithm to obtain a comprehensive feature vector containing multimodal information to enhance the information content and expression ability of the image, and sends the comprehensive feature vector to the index selection module; The index selection module generates multi-dimensional feature fusion coefficients through support vector machine regression according to the data distribution and information volume of the comprehensive feature vector, selects the optimal index structure, quickly matches the database image features, retrieves the image most similar to the input image, and sends the retrieval results to the result sorting module; The result sorting module sorts the retrieved images according to their similarity, selects several images with the highest similarity as the final retrieval results, and displays them to the user.

Images