Image processing method, device, electronic device and storage medium

Abstract

The present disclosure provides an image processing method, device, electronic device and storage medium, which relate to the field of image processing technology. The method includes: obtaining an initial image. The initial image is input into a pre-trained image processing model to obtain a target image. The image processing model includes a feature extraction network, a self-attention convolutional network and a result mapping network connected in sequence; the feature extraction network is used to downsample the initial image and extract the first local feature map of the initial image; the self-attention convolutional network is used to capture the dependency between each position in the first local feature map to obtain a second local feature map; the result mapping network is used to upsample the second local feature map and map the upsampled result to the target image. The present disclosure introduces a self-attention convolutional network into the image processing model to capture the dependency between each position in the feature map, which can greatly improve the quality of the generated target image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] With the advancement of medical technology, medical imaging technology has become an important means of modern medical diagnosis. The acquisition methods and application scopes of medical images of different modalities are different.

[0003] In the related art, although the conversion between medical images of various modalities can be achieved through deep learning models, the quality of the images obtained after the conversion is poor and cannot meet expectations. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, device, electronic device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Get the initial image;

[0007] Inputting the initial image into a pre-trained image processing model to obtain a target image; wherein the target image and the initial image are medical images of different modalities;

[0008] The image processing model includes a feature extraction network, a self-attention convolutional network and a result mapping network connected in sequence; the feature extraction network is used to downsample the initial image and extract a first local feature map of the initial image; the self-attention convolutional network is used to capture the dependency between each position in the first local feature map to obtain a second local feature map; the result mapping network is used to upsample the second local feature map and map the upsampled result to a target image.

[0009] In some embodiments, the self-attention convolutional network includes: a depth convolution unit, a self-attention unit, and a first point-by-point convolution unit connected in sequence;

[0010] There is a residual connection between the input and output of the deep convolution unit, or there is a residual connection between the input of the deep convolution unit and the output of the self-attention unit.

[0011] In some embodiments, there are multiple self-attention convolutional networks, and multiple self-attention convolutional networks are connected in sequence.

[0012] In some embodiments, the result mapping network includes: a transposed convolution unit and a second point-by-point convolution unit connected in sequence;

[0013] The transposed convolution unit is used to upsample the second local feature map to obtain a complete feature map;

[0014] The second point-by-point convolution unit is used to extract nonlinear features of the complete feature map and reduce the number of channels of the complete feature map so as to map the complete feature map into a target image.

[0015] In some embodiments, the image processing model is trained in the following manner:

[0016] Acquire a sample set, wherein the sample set includes a plurality of groups of initial image samples and target image samples corresponding to each other;

[0017] Perform registration processing on each group of initial image samples and target image samples respectively;

[0018] Based on each group of registered initial image samples and target image samples, the image processing model is trained for multiple rounds;

[0019] In each round of training, the following operations are performed for each initial image sample after registration:

[0020] Randomly cropping the initial image sample into initial sub-image samples of a preset size;

[0021] Input the initial sub-image sample into the image processing model to obtain the predicted target sub-image sample;

[0022] Calculating the similarity loss between the predicted target sub-image sample and the real target sub-image sample; wherein the real target sub-image sample is obtained by cropping the target image sample corresponding to the initial image sample;

[0023] Based on the similarity loss, parameters of the image processing model are adjusted.

[0024] In some embodiments, inputting the initial image into a pre-trained image processing model to obtain a target image includes:

[0025] When the size of the initial image is larger than a preset size, dividing the initial image into a plurality of initial sub-images whose sizes are equal to the preset size;

[0026] Inputting a plurality of initial sub-images into a pre-trained image processing model respectively to obtain a plurality of target sub-images;

[0027] Multiple target sub-images are spliced ​​to obtain the target image.

[0028] In some embodiments, inputting the initial image into a pre-trained image processing model to obtain a target image includes:

[0029] When the size of the initial image is smaller than the preset size, edge expansion is performed on the initial image so that the size of the initial image is equal to the preset size;

[0030] The initial image after edge expansion is input into a pre-trained image processing model to obtain the target image.

[0031] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0032] An acquisition module, used for acquiring an initial image;

[0033] An input module, used for inputting the initial image into a pre-trained image processing model to obtain a target image; wherein the target image and the initial image are medical images of different modalities;

[0034] The image processing model includes a feature extraction network, a self-attention convolutional network and a result mapping network connected in sequence; the feature extraction network is used to downsample the initial image and extract a first local feature map of the initial image; the self-attention convolutional network is used to capture the dependency between each position in the first local feature map to obtain a second local feature map; the result mapping network is used to upsample the second local feature map and map the upsampled result to a target image.

[0035] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0036] Processor; and

[0037] A memory, configured to store executable instructions of the processor;

[0038] The processor is configured to execute the method described in the first aspect by executing the executable instructions.

[0039] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0040] The solution provided by the embodiment of the present disclosure includes: obtaining an initial image. Inputting the initial image into a pre-trained image processing model to obtain a target image. The target image and the initial image are medical images of different modalities. The image processing model includes a feature extraction network, a self-attention convolutional network, and a result mapping network connected in sequence; the feature extraction network is used to downsample the initial image and extract the first local feature map of the initial image; the self-attention convolutional network is used to capture the dependency between each position in the first local feature map to obtain a second local feature map; the result mapping network is used to upsample the second local feature map and map the upsampled result to the target image. The embodiment of the present disclosure introduces a self-attention convolutional network into the image processing model to capture the dependency between each position in the feature map, which can greatly improve the quality of the generated target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the system architecture of an image processing method in an embodiment of the present disclosure is shown.

[0042] Figure 2 A schematic flow chart of an image processing method in an embodiment of the present disclosure is shown.

[0043] Figure 3 A structural schematic diagram of an image processing model in an embodiment of the present disclosure is shown.

[0044] Figure 4 A flowchart of an image processing model training method in an embodiment of the present disclosure is shown.

[0045] Figure 5 A schematic structural diagram of an image processing device in an embodiment of the present disclosure is shown.

[0046] Figure 6 A schematic structural diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0048] In recent years, with the rapid development of medical imaging technology, various types of medical images have become an important basis for clinical diagnosis. Medical images of various modalities have different characteristics. For example, computed tomography (CT) images have high resolution and short scanning time in the imaging of tissues such as bones and lungs, but their soft tissue contrast is relatively low, and the use of X-rays has certain radiation risks. In contrast, magnetic resonance (MR) images have extremely high contrast in soft tissue imaging, but their scanning time is long and there are certain limitations for certain patient groups (such as patients with metal implants).

[0049] In this regard, researchers can use deep learning models to achieve conversion between medical images of various modalities to obtain more comprehensive imaging information. Although there are deep learning models that can achieve image conversion in related technologies, the image quality generated by these models has not met expectations and cannot be effectively used in clinical diagnosis.

[0050] In view of this, the image processing method provided in the embodiment of the present disclosure can obtain an initial image by inputting it into a pre-trained image processing model to obtain a target image. The image processing model includes a feature extraction network, a self-attention convolutional network, and a result mapping network connected in sequence. The feature extraction network is used to downsample the initial image and extract the first local feature map of the initial image. The self-attention convolutional network is used to capture the dependency between each position in the first local feature map to obtain a second local feature map. The result mapping network is used to upsample the second local feature map and map the upsampled result to the target image.

[0051] Therefore, the disclosed embodiment introduces a self-attention convolutional network into the image processing model to capture the dependencies between various positions in the feature map, which can greatly improve the quality of the generated target image.

[0052] For example, Figure 1 A schematic diagram of an exemplary system architecture that can be applied to an image processing method or an image processing device in an embodiment of the present disclosure is shown.

[0053] like Figure 1 As shown, the system architecture 100 includes an image acquisition device 101, a server 102 and a terminal 103. The image acquisition device 101 and the terminal 103 are connected to the server 102 via a network, such as a wired or wireless network connection.

[0054] Exemplarily, the image acquisition device 101 can be used to acquire an initial image of an object to be acquired that can show the subject. The subject can be a living thing or a non-living thing. For example, the subject can be a patient, an artificial object, an experimenter, etc. The object to be acquired can include a specific part, organ and / or tissue of the subject. For example, the object to be acquired can include the head, neck, chest, heart, stomach, blood vessels, soft tissue, tumor, nodule, etc. of the subject, or any combination thereof.

[0055] Exemplarily, the server 102 may be a single server, or a server cluster or a cloud server consisting of multiple servers.

[0056] An image processing model is deployed on the server 102. The server 102 can obtain the initial image captured by the image acquisition device, and execute the image processing method provided in the embodiment of the present disclosure to generate a target image.

[0057] In addition, the server 102 may also be used to send the generated target image to the terminal 103 so as to display the target image to the user.

[0058] The terminal 103 can receive the target image generated by the server 102. The terminal 103 may include a mobile phone, a smart TV, a tablet computer, a laptop, or a personal computer (PC). The terminal 103 may also be provided with a client, which may be an application client or a browser client.

[0059] Exemplarily, the server and the terminal may also be the same entity, that is, the entity may directly display the target image to the user after executing the image processing method provided by the present disclosure.

[0060] Those skilled in the art will know that Figure 1 The image acquisition device, server and terminal shown in the figure are merely illustrative and the present disclosure does not limit them.

[0061] This exemplary implementation will be described in detail below with reference to the accompanying drawings and embodiments.

[0062] An image processing method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device.

[0063] Figure 2 A schematic diagram of a process of an image processing method in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the image processing method provided in the embodiment of the present disclosure includes the following steps S201 to S202.

[0064] S201, obtaining an initial image.

[0065] In some embodiments, depending on the modality of the initial image, the initial image may be an ultrasound image, an X-ray image, a computed tomography image, a magnetic resonance image, a digital subtraction angiography (DSA) image, a positron emission tomography (PET) image, etc.

[0066] Accordingly, the initial image may be collected by medical equipment such as ultrasound equipment, X-ray equipment, nuclear magnetic resonance equipment, nuclear medicine equipment, medical optical equipment, and thermal imaging equipment, and the embodiments of the present disclosure are not limited to this.

[0067] S202, inputting the initial image into a pre-trained image processing model to obtain a target image.

[0068] The target image and the initial image are medical images of different modalities. For example, when the initial image is a CT image, the target image may be an MR image.

[0069] In some embodiments, the image processing model includes a feature extraction network, a self-attention convolutional network, and a result mapping network connected sequentially.

[0070] The feature extraction network is used to downsample the initial image and extract the first local feature map of the initial image. The self-attention convolution network is used to capture the dependency between the positions in the first local feature map to obtain the second local feature map. The result mapping network is used to upsample the second local feature map and map the upsampled result to the target image.

[0071] Based on this, the embodiment of the present disclosure introduces a self-attention mechanism in the process of converting medical images of different modalities, which can capture the long-range dependencies and global information in the medical images without losing local features, thereby improving the accuracy of the target image obtained after conversion.

[0072] It should be noted that different initial images may be used to show different objects. For example, the initial image may be used to show brain tissue, spinal cord, fundus, blood vessels, pancreas and lungs, etc. This results in different sizes of the initial images.

[0073] In some embodiments, when the size of the initial image is smaller than the preset size, the initial image may be edge-extended to make the size of the initial image equal to the preset size. The edge-extended initial image is input into a pre-trained image processing model to obtain a target image.

[0074] For example, since the edge area of ​​a medical image usually has no substantial content, the edge of the initial image can be extended by filling with a solid color, for example, filling the gap between the edge of the initial image and the preset size with black.

[0075] In some embodiments, when the size of the initial image is larger than a preset size, the initial image may be divided into a plurality of initial sub-images having a size equal to the preset size. A plurality of target sub-images may be obtained by inputting the plurality of initial sub-images into a pre-trained image processing model respectively. A target image may be obtained by splicing the plurality of target sub-images.

[0076] For example, among the multiple initial sub-images obtained by segmentation, each initial sub-image has a partial overlap region with the initial sub-images adjacent thereto. Thus, each target sub-image obtained may also have a partial overlap region with the target sub-images adjacent thereto, thereby avoiding gaps that may be generated during the splicing process of the multiple target sub-images, and improving the integrity and unity of the target image.

[0077] Exemplarily, among the multiple initial sub-images obtained by segmentation, there is no overlapping area between the initial sub-images, which can reduce the total amount of data that needs to be input into the image processing model to a certain extent, thereby improving the image processing efficiency.

[0078] In which, since the image input to the image processing model is limited by a preset size, the initial image may not be exactly divided into a plurality of initial sub-images that do not overlap with each other. In this case, it is preferred to ensure that the central area of ​​the initial image is divided into a plurality of initial sub-images of a size equal to the preset size, and in the manner described in the previous embodiment, the edge area of ​​the initial image is expanded so that the edge area can also be exactly divided into initial sub-images of a size equal to the preset size and do not overlap with each other.

[0079] It should be noted that the preset size involved in the above embodiments can be understood as the standard input size of the image processing model, which is the image size used to input the model during the training process of the image processing model.

[0080] Before introducing the training process of the image processing model, for ease of understanding, the following will combine Figure 3 First, the model structure of the image processing model in the embodiment of the present disclosure is introduced in detail.

[0081] Please refer to Figure 3 , the image processing model in the embodiment of the present disclosure includes: a feature extraction network, N self-attention convolution networks and a result mapping network. Wherein, N is a positive integer.

[0082] In some embodiments, the feature extraction network includes: an input layer, a two-dimensional convolutional layer, a batch normalization layer, and an activation layer connected in sequence.

[0083] The input layer is used to receive a single-channel initial image as the starting input of the image processing model, where the size of the initial image is a preset size, for example, 256×256.

[0084] The 2D convolution layer is used to downsample the input initial image. For example, the 2D convolution layer can use an 8×8 convolution kernel with a step size of 8 to perform a convolution layer operation on the initial image. Thus, the initial image of size 256×256 will be converted into a feature map of size 32×32 after convolution processing.

[0085] In the process of downsampling the initial image, the number of channels of the image can be increased, and the single-channel initial image can be converted into a multi-channel feature map, so that the features in the initial image can be retained as much as possible during the downsampling process. For example, the number of channels of the feature map obtained after convolution can be 256.

[0086] The feature map obtained after convolution can be passed through a batch normalization layer to standardize the feature map, thereby stabilizing the model training process and accelerating the convergence of the model.

[0087] The output of the batch normalization layer will be input into the activation layer (e.g., ReLU), thereby introducing nonlinear factors between layers and increasing the representation ability of the feature extraction network. The output result of the activation layer is the first local feature map mentioned above.

[0088] Please continue to refer to Figure 3 In some embodiments, the self-attention convolutional network includes: a depth convolution unit, a self-attention unit, and a first point-by-point convolution unit connected in sequence.

[0089] Specifically, the deep convolution unit includes: a deep convolution layer, a batch normalization layer, and an activation layer connected in sequence. The role of the batch normalization layer and the activation layer in the deep convolution unit is the same as that in the feature extraction network, and this embodiment of the disclosure will not be described in detail.

[0090] The deep convolution layer can perform convolution operations on each channel of the feature map independently. In other words, the deep convolution layer only focuses on one channel at a time, rather than the combination of all channels, so that the features of each channel of the feature map can be processed separately. Therefore, compared with the standard convolution layer, the number of parameters of the deep convolution layer will be greatly reduced, making the model more lightweight.

[0091] In addition, deep convolution can further extract local features of the image and effectively learn independent features between different channels of the image.

[0092] The self-attention unit consists of: a first reshape layer connected in sequence ( Figure 3 not shown), a self-attention layer, and a second reshape layer ( Figure 3 not shown).

[0093] The first reshape layer is used to reshape the two-dimensional feature map output by the deep convolution unit into a one-dimensional feature vector so that it can be input into the self-attention layer. For example, the 32×32 feature map input to the first reshape layer will be reshaped into a feature vector of length 1024, and the number of feature channels at each position remains unchanged.

[0094] The three inputs (query, key, and value) of the self-attention layer are all one-dimensional feature vectors output by the first reshaping layer. The self-attention layer can process the input one-dimensional feature vector based on the self-attention mechanism (or a multi-head self-attention mechanism), thereby capturing the long-range dependencies and global information between the positions of the two-dimensional feature map, so that the image processing model can better understand the complex mapping relationship between medical images of different modalities.

[0095] The second reshape layer is used to reshape the one-dimensional feature vector output by the self-attention layer into a two-dimensional feature map so that it can be input into the point-by-point convolution unit. For example, after a feature vector of length 1024 is input into the self-attention layer, the self-attention layer will output a feature vector of length 1024, which will be converted into a feature map of size 32×32 after being reshaped by the second reshape layer. In other words, the second reshape layer is actually an inverse transformation of the reshape process of the first reshape layer.

[0096] The self-attention unit and the deep convolution unit extract the features of the initial image from different angles respectively. The combination of the two enables the image processing model to learn features of different dimensions.

[0097] The first pointwise convolution unit includes: a first pointwise convolution layer, a batch normalization layer, and an activation layer connected in sequence. The batch normalization layer and the activation layer have the same role in the deep convolution unit as in the feature extraction network, and the embodiments of the present disclosure will not be described in detail.

[0098] The first point-by-point convolution layer is used in combination with the depth-wise convolution layer to implement deep separation convolution operations on feature maps, thereby reducing the amount of computation and the number of parameters of the image processing model and speeding up the inference speed.

[0099] In some embodiments, there is a residual connection between the input of the deep convolutional unit and the output of the self-attention unit (e.g. Figure 3As shown in the figure, this method can avoid the loss of local detail features when processing the feature map by the deep convolution unit as much as possible.

[0100] Exemplarily, the residual connection can also be set between the input and output of the depth convolution unit ( Figure 3 (not shown). In other words, the input of the depthwise convolution unit is connected to the input of the first point-by-point convolution unit through a residual connection, thereby avoiding the loss of a large number of local details of the feature map caused by two consecutive convolutions.

[0101] In some embodiments, an additional batch normalization layer may be set after the residual connection. The batch normalization layer can improve the stability of gradient transfer and accelerate the convergence process of the model.

[0102] In some embodiments, there are multiple self-attention convolutional networks, and the multiple self-attention convolutional networks are connected in sequence. In the multiple self-attention convolutional networks, the input of the first self-attention convolutional network is the first local feature, and the output of the last self-attention convolutional network is the second local feature. Each self-attention convolutional network can further perform convolution processing that integrates the self-attention mechanism and residual connection on the basis of the previous network, so that it can fully learn the complex mapping relationship between the initial image and the target image, and improve the quality of the final target image. At the same time, through the combination of deep convolutional units and point-by-point convolutional units, the parameters and calculation amount of the model can be reduced to ensure the reasoning speed of the model.

[0103] Please continue to refer to Figure 3 In some embodiments, the result mapping network includes: a transposed convolution unit and a second point-by-point convolution unit connected in sequence.

[0104] The transposed convolution unit is used to upsample the second local feature map to obtain a complete feature map.

[0105] Exemplarily, the transposed convolution unit includes: a transposed convolution layer and a batch normalization layer connected in sequence.

[0106] The transposed convolution layer is used to upsample the second local feature map so that the size of the upsampled feature map is the same as the size of the initial image input to the image processing model. At the same time, the transposed convolution layer is also used to reduce the number of channels of the second local feature map to a certain extent, thereby slowing down the speed at which the number of channels is reduced to 1 and improving the accuracy of the target image finally mapped.

[0107] For example, the size of the second local feature map output by the self-attention convolutional network is 32×32, and the number of channels is 256. After the transposed convolutional layer, the size of the complete feature map is 256×256, and the number of channels is 128. That is, while restoring the size of the feature map to the size of the initial image, the number of channels of the feature map is reduced by half.

[0108] The batch normalization layer is used to standardize the output of the transposed convolutional layer to obtain a complete feature map, thereby stabilizing the model training process and accelerating the convergence of the model.

[0109] Exemplarily, the second point-by-point convolution unit is used to extract nonlinear features of the complete feature map and reduce the number of channels of the complete feature map so as to map the complete feature map into a target image.

[0110] The second point-by-point convolution unit may include: a second point-by-point convolution layer. The second point-by-point convolution layer may act as a fully connected layer in the image processing model, and may establish a nonlinear mapping relationship between the complete feature map and the target image, and reduce the number of channels of the complete feature map to 1, thereby obtaining a single-channel target image.

[0111] The above is a detailed description of the results of the image processing model in the embodiment of the present disclosure. Next, Figure 4 , illustrating the training method of the image processing model in the embodiment of the present disclosure.

[0112] Figure 4 A flow chart of an image processing model training method in an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the image processing model training method provided in the embodiment of the present disclosure includes the following steps S401 to S403.

[0113] S401, obtaining a sample set, where the sample set includes a plurality of groups of corresponding initial image samples and target image samples.

[0114] In some embodiments, each set of initial image samples and target image samples can be obtained by acquiring images of the same subject in different modalities, for example, acquiring CT images and MR images of the same subject's brain, respectively.

[0115] S402, performing registration processing on each group of initial image samples and target image samples respectively.

[0116] In some embodiments, the initial image sample may be used as a reference, and the corresponding target image sample may be registered with the initial image sample through a registration algorithm.

[0117] The registration algorithm may include at least one of the following: affine transformation, mutual information (MI), gradient descent optimization, free-form deformation (FFD), elastic deformation, landmark-based deformation, etc. Of course, medical image registration software (eg, MIM) may also be used to achieve registration of the initial image sample with the target image sample.

[0118] The purpose of registration is to align a target image that is in a different position, size, or angle from the initial image with the initial image through rigid registration and deformable registration.

[0119] S403, performing multiple rounds of training on the image processing model based on each group of registered initial image samples and target image samples.

[0120] In each round of training, the following operations can be performed for each initial image sample after registration:

[0121] The initial image sample is randomly cropped into an initial sub-image sample of a preset size. The initial sub-image sample is input into an image processing model to obtain a predicted target sub-image sample. The similarity loss between the predicted target sub-image sample and the real target sub-image sample is calculated. The real target sub-image sample is obtained by cropping the target image sample corresponding to the initial image sample.

[0122] Exemplarily, before the initial sub-image samples are input into the image processing model, random data enhancement processing may be performed on the initial sub-image samples, such as injecting noise, adjusting contrast, performing random gamma correction, etc., so as to obtain diversified training samples.

[0123] For example, in each round of training, the cropping of the same initial image sample is random. In other words, the input of multiple rounds of training does not necessarily cover the entire area of ​​the same initial image sample. This method seems to be unable to use the entire area of ​​the initial image sample for training the image processing model, but this randomness can be combined with random data augmentation processing to enrich the model's training samples and improve the model's generalization ability.

[0124] For example, the similarity loss between the predicted target sub-image sample and the real target sub-image sample can be characterized by a structural similarity (SSIM) loss function or a multi-scale SSIM (MS-SSIM) loss function, so that the model can be iterated and optimized in the direction of higher structural similarity. The optimizer of the model can use the Adam function to accelerate the training speed of the model and improve the performance of the model.

[0125] In some embodiments, before each group of registered initial image samples and target image samples are used for model training, preprocessing operations may be performed on the initial image samples and the target image samples respectively.

[0126] Among them, the preprocessing operation may include:

[0127] (1) Data check: Check the consistency and integrity of the data to ensure that each set of initial image samples and target image samples are fully registered without obvious errors or losses.

[0128] (2) Data format conversion: Convert the initial image samples and the target image samples into a consistent format and ensure that they are in the same reference coordinate system.

[0129] (3) Resampling: The spatial resolution of each set of initial image samples and target image samples is resampled so that the spatial resolution of all image samples in the sample set is the same. This operation can be achieved by linear interpolation.

[0130] (4) Intensity normalization: The intensity values ​​of each image sample in the sample set are clipped to a certain range, and the intensity values ​​of each clipped image sample are normalized. For example, a Z-score normalization algorithm can be used to perform intensity normalization on each image sample.

[0131] The above preprocessing operations can ensure the correctness of the image processing model training and improve the accuracy of the image processing model to a certain extent.

[0132] In some embodiments, in the inference stage of the image processing model, the initial image used to input the model may also be subjected to similar preprocessing operations. That is, before executing the aforementioned S202, the initial image may be subjected to format conversion, resampling, and intensity normalization processing to keep it consistent with the model training stage.

[0133] Correspondingly, the image output by the image processing model needs to be inversely normalized to obtain the final target image.

[0134] Based on the same inventive concept, the present disclosure also provides an image processing device in the following embodiments. Figure 1 The method embodiment shown is similar to that shown in FIG. 1 , so the implementation of the image processing device embodiment can refer to the above Figure 1 The implementation of the illustrated method embodiment will not be described in detail in detail.

[0135] Figure 5 A schematic structural diagram of an image processing device in an embodiment of the present disclosure is shown.

[0136] Specifically, Figure 5 As shown, the image processing device 500 includes: an acquisition module 501 and a determination module 502 .

[0137] The acquisition module 501 is used to acquire an initial image.

[0138] The input module 502 is used to input the initial image into a pre-trained image processing model to obtain a target image; wherein the target image and the initial image are medical images of different modalities.

[0139] The image processing model includes a feature extraction network, a self-attention convolutional network and a result mapping network which are connected in sequence; the feature extraction network is used to downsample the initial image and extract the first local feature map of the initial image; the self-attention convolutional network is used to capture the dependency between each position in the first local feature map to obtain the second local feature map; the result mapping network is used to upsample the second local feature map and map the upsampled result to the target image.

[0140] In some embodiments, the self-attention convolutional network includes: a deep convolutional unit, a self-attention unit, and a first point-by-point convolutional unit connected in sequence; wherein there is a residual connection between the input and output of the deep convolutional unit, or there is a residual connection between the input of the deep convolutional unit and the output of the self-attention unit.

[0141] In some embodiments, there are multiple self-attention convolutional networks, and the multiple self-attention convolutional networks are connected in sequence.

[0142] In some embodiments, the result mapping network includes: a transposed convolution unit and a second point-by-point convolution unit connected in sequence; the transposed convolution unit is used to upsample the second local feature map to obtain a complete feature map; the second point-by-point convolution unit is used to extract the nonlinear features of the complete feature map and reduce the number of channels of the complete feature map so as to map the complete feature map to a target image.

[0143] In some embodiments, the image processing device 500 also includes: a training module (not shown in the figure), which is used to obtain a sample set, the sample set includes multiple groups of corresponding initial image samples and target image samples; each group of initial image samples and target image samples are registered respectively; based on each group of initial image samples and target image samples after registration, multiple rounds of training are performed on the image processing model.

[0144] In each round of training, the following operations are performed for each initial image sample after registration: the initial image sample is randomly cropped into an initial sub-image sample of a preset size; the initial sub-image sample is input into an image processing model to obtain a predicted target sub-image sample; the similarity loss between the predicted target sub-image sample and the real target sub-image sample is calculated; wherein the real target sub-image sample is obtained by cropping the target image sample corresponding to the initial image sample; and the parameters of the image processing model are adjusted based on the similarity loss.

[0145] In some embodiments, the input module 502 is specifically used to, when the size of the initial image is larger than a preset size, divide the initial image into multiple initial sub-images with a size equal to the preset size; input the multiple initial sub-images into a pre-trained image processing model respectively to obtain multiple target sub-images; and splice the multiple target sub-images to obtain a target image.

[0146] In some embodiments, the input module 502 is specifically used to, when the size of the initial image is smaller than a preset size, perform edge expansion on the initial image so that the size of the initial image is equal to the preset size; and input the initial image after edge expansion into a pre-trained image processing model to obtain a target image.

[0147] Refer to the following Figure 6 hereinafter, an electronic device 600 capable of implementing the embodiments of the present disclosure is described. Figure 6 The electronic device 600 shown is only an example and should not bring any limitation to the functions and applicable scope of the embodiments of the present disclosure.

[0148] like Figure 6 As shown, the electronic device 600 is in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processor 610, at least one memory 620, and a bus 630 connecting different system components (including the memory 620 and the processor 610).

[0149] The memory stores program codes, which can be executed by the processor 610, so that the processor 610 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of the present disclosure.

[0150] In some embodiments, the processor 610 may further perform the following steps of the above method embodiment:

[0151] Get the initial image;

[0152] Inputting the initial image into a pre-trained image processing model to obtain a target image; wherein the target image and the initial image are medical images of different modalities;

[0153] The image processing model includes a feature extraction network, a self-attention convolutional network and a result mapping network which are connected in sequence; the feature extraction network is used to downsample the initial image and extract the first local feature map of the initial image; the self-attention convolutional network is used to capture the dependency between each position in the first local feature map to obtain the second local feature map; the result mapping network is used to upsample the second local feature map and map the upsampled result to the target image.

[0154] The memory 620 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 6201 and / or a cache memory 6202 , and may further include a read-only memory (ROM) 6203 .

[0155] The memory 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0156] Bus 630 may be a bus representing one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0157] The electronic device 600 may also communicate with one or more external devices 640 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. Figure 6As shown, the network adapter 660 communicates with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0158] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0159] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above-mentioned method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above-mentioned "Exemplary Method" section of the present disclosure.

[0160] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. An image processing method, characterized in that: include: Get the initial image; Inputting the initial image into a pre-trained image processing model to obtain a target image; wherein the target image and the initial image are medical images of different modalities; The image processing model includes a feature extraction network, a self-attention convolution network and a result mapping network connected in sequence; the feature extraction network is used to downsample the initial image and extract a first local feature map of the initial image; the self-attention convolution network is used to capture the dependency between each position in the first local feature map to obtain a second local feature map; the result mapping network is used to upsample the second local feature map and map the upsampled result to a target image; The image processing model is trained in the following way: Acquire a sample set, wherein the sample set includes a plurality of groups of initial image samples and target image samples corresponding to each other; Perform registration processing on each group of initial image samples and target image samples respectively; Based on each group of registered initial image samples and target image samples, the image processing model is trained for multiple rounds; In each round of training, the following operations are performed for each initial image sample after registration: Randomly cropping the initial image sample into initial sub-image samples of a preset size; Input the initial sub-image sample into the image processing model to obtain the predicted target sub-image sample; Calculating the similarity loss between the predicted target sub-image sample and the real target sub-image sample; wherein the real target sub-image sample is obtained by cropping the target image sample corresponding to the initial image sample; Based on the similarity loss, parameters of the image processing model are adjusted.

2. The method according to claim 1, characterized in that The self-attention convolutional network comprises: a depth convolution unit, a self-attention unit and a first point-by-point convolution unit connected in sequence; There is a residual connection between the input and output of the deep convolution unit, or there is a residual connection between the input of the deep convolution unit and the output of the self-attention unit.

3. The method according to claim 1 or 2, characterized in that: There are multiple self-attention convolutional networks, and the multiple self-attention convolutional networks are connected in sequence.

4. The method according to claim 1, characterized in that: The result mapping network comprises: a transposed convolution unit and a second point-by-point convolution unit connected in sequence; The transposed convolution unit is used to upsample the second local feature map to obtain a complete feature map; The second point-by-point convolution unit is used to extract nonlinear features of the complete feature map and reduce the number of channels of the complete feature map so as to map the complete feature map into a target image.

5. The method according to claim 1, characterized in that: The step of inputting the initial image into a pre-trained image processing model to obtain a target image comprises: When the size of the initial image is larger than a preset size, dividing the initial image into a plurality of initial sub-images whose sizes are equal to the preset size; Inputting a plurality of initial sub-images into a pre-trained image processing model respectively to obtain a plurality of target sub-images; Multiple target sub-images are spliced ​​to obtain the target image.

6. The method according to claim 1, characterized in that The step of inputting the initial image into a pre-trained image processing model to obtain a target image comprises: When the size of the initial image is smaller than the preset size, edge expansion is performed on the initial image so that the size of the initial image is equal to the preset size; The initial image after edge expansion is input into a pre-trained image processing model to obtain the target image.

7. An image processing device, characterized in that: include: An acquisition module, used for acquiring an initial image; An input module, used for inputting the initial image into a pre-trained image processing model to obtain a target image; wherein the target image and the initial image are medical images of different modalities; The image processing model includes a feature extraction network, a self-attention convolution network and a result mapping network connected in sequence; the feature extraction network is used to downsample the initial image and extract a first local feature map of the initial image; the self-attention convolution network is used to capture the dependency between each position in the first local feature map to obtain a second local feature map; the result mapping network is used to upsample the second local feature map and map the upsampled result to a target image; The image processing model is trained in the following way: Acquire a sample set, wherein the sample set includes a plurality of groups of initial image samples and target image samples corresponding to each other; Perform registration processing on each group of initial image samples and target image samples respectively; Based on each group of registered initial image samples and target image samples, the image processing model is trained for multiple rounds; In each round of training, the following operations are performed for each initial image sample after registration: Randomly cropping the initial image sample into initial sub-image samples of a preset size; Input the initial sub-image sample into the image processing model to obtain the predicted target sub-image sample; Calculating the similarity loss between the predicted target sub-image sample and the real target sub-image sample; wherein the real target sub-image sample is obtained by cropping the target image sample corresponding to the initial image sample; Based on the similarity loss, parameters of the image processing model are adjusted.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images