Medical image super-resolution method based on dynamic attention and implicit neural representation

Abstract

The present invention relates to the fields of artificial intelligence and medical image super-resolution technology, and in particular to a medical image super-resolution method based on dynamic attention and implicit neural representation. The method comprises the following steps: obtaining a medical image to be processed; inputting the medical image to be processed into a preset medical image super-resolution model, and outputting a medical image super-resolution result. The medical image super-resolution model is trained using a training set, the training set comprising a plurality of training pairs each comprising a low-resolution medical image and a corresponding high-resolution medical image. The medical image super-resolution model is constructed based on a convolutional neural network that introduces a dynamic attention mechanism and implicit neural representation. The present invention obtains a high-resolution medical image based on implicit neural representation by learning inter-layer feature representations of medical images and applying a corresponding dynamic attention mechanism.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and medical image super-resolution technology, and in particular to a medical image super-resolution method based on dynamic attention and implicit neural representation. Background Art

[0002] Medical image super-resolution refers to the process of using image processing techniques to improve the spatial resolution of medical imaging data. Specifically, this task aims to reconstruct high-resolution (HR) images from low-resolution (LR) medical images to enhance image detail and quality, thereby improving doctors' interpretation and analysis of images.

[0003] Medical images collected in clinical scenarios usually have a significant problem, that is, the images have a large slice spacing, which leads to high in-plane resolution but low inter-plane resolution. In order to improve the inter-plane resolution of medical images to promote better visualization and computer-aided diagnosis, super-resolution technology is widely used. However, most existing super-resolution works usually perform super-resolution on the entire image, and the plane spacing of medical images is still relatively large, and the network is mostly trained under a fixed scale factor, which is not flexible enough and has poor adaptability when facing the changing slice spacing in medical image scans. At the same time, most super-resolution results have many artifacts, and the image is too smooth, which makes it lose its authenticity. This is often because the network's attention weights for each position are relatively consistent, ignoring the specificity of different areas of the image in actual applications. In addition, due to the large amount of three-dimensional data, the use of traditional networks often requires a large amount of computation. Therefore, the present invention provides a medical image super-resolution method based on dynamic attention and implicit neural representation. Summary of the Invention

[0004] The purpose of the present invention is to provide a medical image super-resolution method based on dynamic attention and implicit neural representation, which obtains high-resolution medical images based on implicit neural representation by learning the inter-layer feature representation of medical images and applying the corresponding dynamic attention mechanism.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] Medical image super-resolution methods based on dynamic attention and implicit neural representation, including:

[0007] Acquiring a medical image to be processed;

[0008] The medical image to be processed is input into a preset medical image super-resolution model, and a medical image super-resolution result is output, wherein the medical image super-resolution model is obtained by training with a training set, the training set includes several training pairs, and the training pairs include low-resolution medical images and corresponding high-resolution medical images. The medical image super-resolution model is constructed based on a convolutional neural network that introduces a dynamic attention mechanism and implicit neural representation.

[0009] Preferably, constructing the training set includes:

[0010] Acquire several high-resolution medical images through clinical equipment;

[0011] Downsampling is performed on the high-resolution medical image to generate a corresponding low-resolution medical image, and a plurality of training pairs including a low-resolution medical image and a high-resolution medical image are constructed to further construct the training set.

[0012] Preferably, the medical image super-resolution model includes:

[0013] A feature extraction module is used to extract features from the input image using a convolutional neural network to obtain a first feature map;

[0014] a pre-super-resolution module, configured to perform inter-layer super-resolution on the first feature map using an interpolation algorithm to obtain a second feature map;

[0015] A dynamic attention enhancement module, configured to enhance the second feature map using a dynamic window attention mechanism to obtain a third feature map;

[0016] The super-resolution module is used to map the third feature map and the coordinate information in the third feature map into the voxel intensity of the high-resolution image through the implicit neural representation decoding function, so as to reconstruct the high-resolution image.

[0017] Preferably, the feature extraction module uses a convolutional neural network to extract features from the input image to obtain a first feature map, including:

[0018] Map the input image to the feature space through the 3D convolution layer to obtain shallow features;

[0019] Then, local features are extracted through several residual dense blocks, the local features are combined with shallow features to obtain global features, and the first feature map is output.

[0020] Preferably, the residual dense blocks are connected via a ReLU activation function, and each residual dense block includes several densely connected convolutional layers.

[0021] Preferably, the pre-super-resolution module uses an interpolation algorithm to perform inter-layer super-resolution on the first feature map to obtain a second feature map, including:

[0022] Performing inter-layer super-resolution on the first feature map using a trilinear interpolation method to obtain a feature vector, and outputting the second feature map, wherein obtaining the feature vector includes:

[0023] , ;

[0024] in, is the nearest position corresponding to the query coordinate in the first feature map The eigenvector of are the query coordinates and eigenvectors The volume of the cuboid between the diagonal coordinates of is the total volume of the cuboid between the query coordinates and the diagonal coordinates of the eigenvector, is the eigenvector corresponding to the query coordinate in the first feature map.

[0025] Preferably, the dynamic attention enhancement module uses a dynamic window attention mechanism to enhance the second feature map to obtain a third feature map, including:

[0026] Input the feature vector into the dynamic receptive field unit to obtain the attention window size, and dynamically adjust the attention sliding window size according to the feature to determine the attention window range;

[0027] Within the attention window, attention calculation is performed on the second feature map to obtain a high-resolution feature vector, and the third feature map is output.

[0028] Preferably, the dynamic receptive field unit is composed of a lightweight depth-separable convolutional network, and the lightweight depth-separable convolutional network performs calculations through two three-dimensional convolutions to obtain the attention window size.

[0029] Preferably, performing the attention calculation includes:

[0030] ;

[0031] in, is the high-resolution feature vector obtained by attention calculation, is the eigenvector corresponding to the query coordinate in the first eigenmap, represents the learnable weight matrix in the linear layer of the neural network, Indicates that the dynamic window The feature vector extracted from the domain.

[0032] Preferably, the implicit neural representation decoding function is obtained by constructing a third feature map using a continuous function, and obtaining a mapping relationship between coordinate information in the third feature map and voxel density values ​​corresponding to the coordinates.

[0033] The beneficial effects of the present invention are:

[0034] The medical image super-resolution model constructed by the present invention performs the first step of feature extraction through a feature extraction module; the pre-super-resolution module uses a trilinear interpolation algorithm to integrate the features of two adjacent modules of a slice to obtain specific features at a certain coordinate; the dynamic attention enhancement module first predicts the dynamic window size based on the coordinate feature, then performs attention calculation to obtain the final feature vector; finally, the feature vector and coordinate representation are input into the super-resolution module based on the implicit neural representation network to obtain the predicted value at the super-resolution coordinate, and the mean square error loss is calculated with the true value of the paired data. By learning the inter-layer feature representation of medical images and applying the corresponding dynamic attention mechanism, the present invention obtains high-resolution medical images based on the implicit neural representation, which can achieve inter-layer resolution enhancement of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0036] Figure 1 Schematic diagram of the medical image super-resolution model structure and workflow according to an embodiment of the present invention;

[0037] Figure 2 This is a flow chart of a medical image super-resolution method based on dynamic attention and implicit neural representation according to an embodiment of the present invention;

[0038] Figure 3 This is a graph showing the super-resolution results of the test image 1 by the model and the comparison algorithm according to an embodiment of the present invention;

[0039] Figure 4 This is the result diagram after super-resolution of test image 2 by this model and the comparison algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] This embodiment provides a medical image super-resolution method based on dynamic attention and implicit neural representation, including:

[0043] Acquiring a medical image to be processed;

[0044] The medical image to be processed is input into a preset medical image super-resolution model, and a medical image super-resolution result is output, wherein the medical image super-resolution model is obtained by training with a training set, the training set includes several training pairs, and the training pairs include low-resolution medical images and corresponding high-resolution medical images. The medical image super-resolution model is constructed based on a convolutional neural network that introduces a dynamic attention mechanism and implicit neural representation.

[0045] Specifically, such as Figure 2 As shown, this embodiment achieves inter-layer resolution enhancement of medical images by constructing paired low- and high-resolution medical image datasets, designing a super-resolution algorithm based on implicit neural representation, training and optimizing the model, and systematically evaluating performance. Low- and high-resolution medical images are used as training pairs. The data is passed through a feature extraction module, a pre-super-resolution module, a dynamic attention enhancement module, and a super-resolution module to obtain the network output. The loss is then calculated using the original high-resolution data, and backpropagation is used to optimize model performance.

[0046] The medical image super-resolution model constructed in this embodiment performs the first step of feature extraction through a feature extraction module; the pre-super-resolution module uses a trilinear interpolation algorithm to integrate the features of two adjacent modules of a slice to obtain specific features at a certain coordinate; the dynamic attention enhancement module first predicts the dynamic window size based on the features of a certain coordinate, and then performs attention calculation to obtain the final feature vector; finally, the feature vector and coordinate representation are input into the super-resolution module based on the implicit neural representation network to obtain the predicted value at the super-resolution coordinate, and the mean square error loss is calculated with the true value of the paired data. This embodiment can achieve inter-layer resolution enhancement of medical images by learning the inter-layer feature representation of medical images and applying the corresponding dynamic attention mechanism to obtain high-resolution medical images based on implicit neural representation.

[0047] Furthermore, constructing the training set includes:

[0048] Acquire several high-resolution medical images through clinical equipment;

[0049] Downsampling is performed on the high-resolution medical image to generate a corresponding low-resolution medical image, and a plurality of training pairs including a low-resolution medical image and a high-resolution medical image are constructed to further construct the training set.

[0050] Specifically, this embodiment targets images acquired by clinical devices, including CT and MRI images. T1-weighted MRI sequences and CT images are selected as high-resolution (HR) images. Corresponding low-resolution (LR) images are generated through controllable downsampling operations (such as nearest neighbor interpolation, bilinear interpolation, or Gaussian downsampling). This simulates the low-resolution images acquired in real life, forming paired LR-HR datasets that are then divided into training and test sets.

[0051] Furthermore, the medical image super-resolution model includes:

[0052] A feature extraction module is used to extract features from the input image using a convolutional neural network to obtain a first feature map;

[0053] a pre-super-resolution module, configured to perform inter-layer super-resolution on the first feature map using an interpolation algorithm to obtain a second feature map;

[0054] A dynamic attention enhancement module, configured to enhance the second feature map using a dynamic window attention mechanism to obtain a third feature map;

[0055] The super-resolution module is used to map the third feature map and the coordinate information in the third feature map into the voxel intensity of the high-resolution image through the implicit neural representation decoding function, so as to reconstruct the high-resolution image.

[0056] Furthermore, the feature extraction module uses a convolutional neural network to extract features from the input image to obtain a first feature map, including:

[0057] Map the input image to the feature space through the 3D convolution layer to obtain shallow features;

[0058] Then, local features are extracted through several residual dense blocks, the local features are combined with shallow features to obtain global features, and the first feature map is output, wherein the residual dense blocks are connected by ReLU activation functions, and each residual dense block includes several densely connected convolutional layers.

[0059] Furthermore, the pre-super-resolution module uses an interpolation algorithm to perform inter-layer super-resolution on the first feature map to obtain a second feature map, including:

[0060] Performing inter-layer super-resolution on the first feature map using a trilinear interpolation method to obtain a feature vector, and outputting the second feature map, wherein obtaining the feature vector includes:

[0061] , ;

[0062] in, is the nearest position corresponding to the query coordinate in the first feature map The eigenvector of are the query coordinates and eigenvectors The volume of the cuboid between the diagonal coordinates of is the total volume of the cuboid between the query coordinates and the diagonal coordinates of the eigenvector, is the eigenvector corresponding to the query coordinate in the first feature map.

[0063] Furthermore, the dynamic attention enhancement module uses a dynamic window attention mechanism to enhance the second feature map to obtain a third feature map, including:

[0064] Input the feature vector into the dynamic receptive field unit to obtain the attention window size, and dynamically adjust the attention sliding window size according to the feature to determine the attention window range;

[0065] Within the attention window, attention calculation is performed on the second feature map to obtain a high-resolution feature vector, and the third feature map is output.

[0066] The dynamic receptive field unit is composed of a lightweight depth-separable convolutional network, and the lightweight depth-separable convolutional network is calculated through two three-dimensional convolutions to obtain the attention window size.

[0067] Performing the attention calculation includes:

[0068] ;

[0069] in, is the high-resolution feature vector obtained by attention calculation, is the eigenvector corresponding to the query coordinate in the first eigenmap, represents the learnable weight matrix in the linear layer of the neural network, Indicates that the dynamic window The feature vector extracted from the domain.

[0070] Furthermore, the implicit neural representation decoding function is obtained by constructing a third feature map using a continuous function, and obtaining a mapping relationship between coordinate information in the third feature map and voxel density values ​​corresponding to the coordinates.

[0071] Specifically, such as Figure 1 As shown, the workflow of the medical image super-resolution model constructed in this embodiment is as follows:

[0072] Step 1: Use the feature extraction Residual Dense Network (RDN) module to construct a regional feature map for each layer of medical images. Use a classic convolutional neural network to extract relevant features of medical images.

[0073] Step 2: Use an interpolation algorithm to perform simple pre-super-resolution of the image. To perform inter-layer super-resolution, it is necessary to generate several new layers of medical images based on each existing layer of images. New image features are obtained by combining the features of several similar layers of images.

[0074] Step 3: Dynamic Window-Based Attention Mechanism. A corresponding spatial attention operation is introduced to more accurately model the spatial dependencies between the query coordinate and its neighboring voxels. The attention window size is dynamically adjusted, and coordinate offset information is incorporated to enhance feature representation. This operation allows for a more appropriate receptive field, enabling the query coordinate to capture more contextual information from a larger neighborhood.

[0075] Step 4: High-resolution decoding and reconstruction: The implicit neural representation decoding function maps the implicit feature representation and query coordinates to the voxel intensity of the high-resolution image, thereby reconstructing HR images with arbitrary magnification and slice intervals.

[0076] Specifically, in the first step, the original image The feature extraction module extracts the feature representation of each voxel. First, the input image is mapped to the feature space through a 3D convolution layer. Assume that the number of output channels of the convolution layer is (e.g., 64), the initial feature extraction can be expressed as:

[0077] ;

[0078] in, is the initial feature map, with a size of .

[0079] The core of RDN is the residual dense block (RDB), each RDB contains multiple densely connected convolutional layers. Assume that each RDB contains convolutional layers, and the number of output channels of each convolutional layer is The number of output channels for each RDB is For the RDB, whose input is , the output is :

[0080] .

[0081] The internal structure of each RDB can be expressed as:

[0082] .

[0083] An activation function can be used between each two RDBs. In this embodiment, ReLU is used. The local features extracted by RDB will be reduced by 1*1*1 convolution to reduce redundant features and re-learn. After the local features are extracted, the RDN network will combine the outputs of multiple RDB structures and shallow features for global features. Assume that there are K RDBs in the network, and the number of output channels of each RDB gradually increases. Finally, the outputs of all RDBs are spliced ​​together to form a high-dimensional feature map. :

[0084] ;

[0085] In order to reduce the number of channels of the feature map and enhance the global features, a 3D convolution layer is used to extract the features of the original low-resolution image at the beginning, which is recorded as Finally, the initial shallowest features are combined with the features extracted from the subsequent K layers to obtain the most complete image information. :

[0086] .

[0087] In this embodiment, the size is set to W*H*D*128.

[0088] After obtaining the features extracted by the efficient RDN module, in the second step, this embodiment adopts the trilinear interpolation method to perform simple pre-super resolution. In this embodiment, the features extracted in the first step are used by trilinear interpolation method. and (The super-resolution result features of a certain slice must come from the combination of features of two adjacent slices of the original image) to obtain all the features of the higher resolution image. Specifically, the feature vector as follows:

[0089] , ;

[0090] in, (k=0,…,7) is for low-resolution feature maps The feature vectors of the eight nearest positions corresponding to a certain coordinate in the query, are the query coordinates and eigenvectors The volume of the cuboid between the diagonal coordinates of is the total volume.

[0091] The third step is an optimization of the pre-super-resolution result in the second step. The accuracy of the high-resolution features after trilinear interpolation is low (because it only considers the features of 8 positions around the adjacent slices of a certain coordinate). This embodiment uses an adaptive receptive field attention mechanism to improve the accuracy of its feature representation. Considering that if all neighborhoods use the attention mechanism with the same receptive field size, it is necessary to ensure that the information obtained is as comprehensive as possible, so it is necessary to use the largest possible receptive field, in order to avoid consuming additional computing resources by using a larger receptive field at all coordinates. In different image regions, the query coordinate super-resolution has different requirements for neighborhood information. In complex areas (such as tissue edges or areas with large intensity changes), a larger receptive field is required to capture more contextual information; in smooth areas, a smaller receptive field is sufficient.

[0092] Specifically, this embodiment further analyzes the pre-super-resolution results obtained in the second step. First, a certain coordinate The eigenvector of Input the dynamic field module DFM (Dynamic Field Module) to obtain the attention window size , dynamically adjust the attention sliding window size by features:

[0093] .

[0094] DFM is composed of a lightweight depth-wise separable convolutional network, which includes:

[0095] (1) Depthwise Separable Convolution Output :

[0096] ;

[0097] (2) 1×1 convolution output :

[0098] ;

[0099] (3) Get the attention window size :

[0100] .

[0101] This embodiment uses only two simple three-dimensional convolutions to perform the corresponding calculations, and obtains the window size of 3-13 as the attention used, which not only realizes the dynamic adjustment of attention, but also reduces the amount of calculation to a certain extent. After determining the attention range, in order to obtain a more accurate high-resolution image, it is necessary to perform attention calculation on the pre-super-resolution feature. The attention calculation is performed based on the feature value of the fixed range window of the adjacent slices of a certain slice, and the final high-resolution feature is obtained accordingly. The inter-layer attention mechanism used is as follows:

[0102] ;

[0103] in, Represents the learnable weight matrix of the linear layer in the neural network, which is dynamically updated through backpropagation. Indicates that the dynamic window The feature vector extracted from the domain.

[0104] After the above steps, this embodiment obtains the final predicted implicit features of the high-resolution image, and the following continuous function can be used to construct the implicit representation , coordinate information The density value of the voxel at this coordinate The mapping relationship, that is, the implicit neural representation, is calculated as follows:

[0105] .

[0106] This embodiment will use a multi-layer perceptron (MLP) to implement the continuous function .

[0107] Furthermore, the medical image super-resolution model was trained: based on the training set, the model was trained using the deep learning framework Pytorch. The number of training batches was 4, the epoch was set to 1500, and the learning rate was initialized to 1×10 -4 , iterate continuously until the loss function converges, and save the model every 300 epochs to obtain relevant data later until the final model is obtained.

[0108] Model performance evaluation: The relevant low-resolution images in the test set are input into this model and the comparison algorithm to obtain the corresponding medical image super-resolution output results as shown below: Figure 3 、 Figure 4 As shown in Table 1, the PSNR and SSIM values ​​of this model and the comparison algorithm on the test set are shown in Table 1 and Table 2, respectively, reflecting the excellent performance of this model.

[0109] Table 1

[0110]

[0111] Table 2

[0112]

[0113] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A medical image super-resolution method based on dynamic attention and implicit neural representation, characterized by: include: Acquiring a medical image to be processed; Inputting the to-be-processed medical image into a preset medical image super-resolution model and outputting a medical image super-resolution result, wherein the medical image super-resolution model is trained using a training set, the training set includes a plurality of training pairs, each of the training pairs including a low-resolution medical image and a corresponding high-resolution medical image, and the medical image super-resolution model is constructed based on a convolutional neural network that introduces a dynamic attention mechanism and implicit neural representation; The medical image super-resolution model includes: A feature extraction module is used to extract features from the input image using a convolutional neural network to obtain a first feature map; a pre-super-resolution module, configured to perform inter-layer super-resolution on the first feature map using an interpolation algorithm to obtain a second feature map; A dynamic attention enhancement module, configured to enhance the second feature map using a dynamic window attention mechanism to obtain a third feature map; a super-resolution module, configured to map the third feature map and the coordinate information in the third feature map to voxel intensities of a high-resolution image through an implicit neural representation decoding function, thereby reconstructing a high-resolution image; The dynamic attention enhancement module uses a dynamic window attention mechanism to enhance the second feature map to obtain a third feature map, including: Input the feature vector into the dynamic receptive field unit to obtain the attention window size, and dynamically adjust the attention sliding window size according to the feature to determine the attention window range; Performing attention calculation on the second feature map within the attention window to obtain a high-resolution feature vector and outputting the third feature map; The dynamic receptive field unit is composed of a lightweight depth-separable convolutional network, which calculates through two three-dimensional convolutions to obtain the attention window size; Performing the attention calculation includes: ； in, is the high-resolution feature vector obtained by attention calculation, is the eigenvector corresponding to the query coordinate in the first eigenmap, represents the learnable weight matrix in the linear layer of the neural network, Indicates that the dynamic window The feature vector extracted from the domain.

2. The medical image super-resolution method based on dynamic attention and implicit neural representation according to claim 1, characterized in that Constructing the training set includes: Acquire several high-resolution medical images through clinical equipment; Downsampling is performed on the high-resolution medical image to generate a corresponding low-resolution medical image, and a plurality of training pairs including a low-resolution medical image and a high-resolution medical image are constructed to further construct the training set.

3. The medical image super-resolution method based on dynamic attention and implicit neural representation according to claim 1, characterized in that The feature extraction module uses a convolutional neural network to extract features from the input image to obtain a first feature map, including: Map the input image to the feature space through the 3D convolution layer to obtain shallow features; Then, local features are extracted through several residual dense blocks, the local features are combined with shallow features to obtain global features, and the first feature map is output.

4. The medical image super-resolution method based on dynamic attention and implicit neural representation according to claim 3, characterized in that The residual dense blocks are connected by ReLU activation functions, and each residual dense block includes several densely connected convolutional layers.

5. The medical image super-resolution method based on dynamic attention and implicit neural representation according to claim 1, characterized in that The pre-super-resolution module uses an interpolation algorithm to perform inter-layer super-resolution on the first feature map to obtain a second feature map, including: Performing inter-layer super-resolution on the first feature map using a trilinear interpolation method to obtain a feature vector, and outputting the second feature map, wherein obtaining the feature vector includes: ， ； in, is the nearest position corresponding to the query coordinate in the first feature map The eigenvector of are the query coordinates and eigenvectors The volume of the cuboid between the diagonal coordinates of is the total volume of the cuboid between the query coordinates and the diagonal coordinates of the eigenvector, is the eigenvector corresponding to the query coordinate in the first feature map.

6. The medical image super-resolution method based on dynamic attention and implicit neural representation according to claim 1, characterized in that The implicit neural representation decoding function is obtained by constructing a third feature map using a continuous function, and obtaining a mapping relationship between coordinate information in the third feature map and voxel density values ​​corresponding to the coordinates.

Images