Image processing method and storage medium

Abstract

This application relates to an image processing method and a storage medium. The method includes: inputting a first medical image into a preset first neural network model to obtain a second medical image with a resolution greater than that of the first medical image; the first neural network model is obtained by training a first initial neural network model based on processed sample image blocks corresponding to the sample image and a gold standard image corresponding to the sample image; the processed sample image blocks are obtained by performing a first expansion processing and / or a second expansion processing on each sample image block of the sample image. This method can improve the scalability of the first initial neural network model training, thereby improving the scalability of processing the first medical image using the trained first neural network model to obtain the second medical image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an image processing method and a storage medium. BACKGROUND

[0002] Medical images, as a way to obtain internal images of tissues in a non-invasive manner, play an important role in clinical diagnosis. Computer-aided diagnosis (CAD) analyzes medical images through a neural network model to give a preliminary diagnosis, so that doctors can make a diagnosis of the lesion area based on the preliminary diagnosis.

[0003] However, training a neural network model with high accuracy requires a large amount of medical data. Even if sufficient medical data can be obtained, due to the particularity of medical data, such as different imaging methods, different parameter settings for data collection, etc., different medical data will be collected, which greatly affects the performance of the model.

[0004] Therefore, the traditional training method of the neural network model has the problem of poor scalability. SUMMARY

[0005] Therefore, it is necessary to provide an image processing method and a storage medium capable of improving the scalability of a neural network model to solve the above technical problems.

[0006] In a first aspect, the present application provides an image processing method, which comprises:

[0007] obtaining a first medical image;

[0008] inputting the first medical image into a preset first neural network model to obtain a second medical image; wherein the resolution of the second medical image is greater than that of the first medical image; the first neural network model is obtained by training a first initial neural network model according to a processed sample image block corresponding to a sample image and a gold standard image corresponding to the sample image; the processed sample image block is obtained by performing first expansion processing and / or second expansion processing on each sample image block of the sample image; the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting the arrangement order of each sample image block, replacing a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block; the second expansion processing includes adding a blank image block to a feature map of each sample image block.

[0009] In one of the embodiments, if the processed sample image block is obtained by performing first expansion processing on each sample image block of the sample image, the training process of the first neural network model comprises:

[0010] performing the first expansion processing on each sample image block to obtain the processed sample image block;

[0011] inputting the processed image block into the first initial neural network model to obtain a third medical image;

[0012] obtaining a value of a first loss function according to the third medical image and the gold standard image, and training the first initial neural network model according to the value of the first loss function to obtain the first neural network model.

[0013] In one of the embodiments, if the first expansion processing is replacement processing on a target image block in each sample image block, the performing the first expansion processing on each sample image block to obtain the processed sample image block comprises:

[0014] arbitrarily selecting a plurality of first target image blocks from each sample image block;

[0015] replacing the plurality of first target image blocks with corresponding second target image blocks to obtain the processed sample image block; the resolution of the second target image block is higher than that of the first target image block.

[0016] In one of the embodiments, if the first expansion processing is deleting at least one sample image block other than a target image block in each sample image block, the performing the first expansion processing on each sample image block to obtain the processed sample image block comprises:

[0017] selecting a third target image block corresponding to a lesion area in the sample image from each sample image block;

[0018] deleting at least one sample image block other than the third target image block in each sample image block to obtain the processed sample image block.

[0019] In one of the embodiments, if the processed sample image block is obtained by performing second expansion processing on each sample image block of the sample image, the training process of the first neural network model comprises:

[0020] inputting a target sample image block in each sample image block into an encoding layer of the first initial neural network model to obtain a feature map corresponding to each target sample image block;

[0021] According to the order of each sample image block, a blank image block is added in the feature map corresponding to each target sample image block as the feature map corresponding to the sample image block except the target sample image block, to obtain a processed feature map;

[0022] The processed feature map is input into a decoding layer of the first initial neural network model to obtain a fourth medical image;

[0023] According to the fourth medical image and the gold standard image, a value of a second loss function is obtained, and the first initial neural network model is trained according to the value of the second loss function to obtain the first neural network model.

[0024] In one of the embodiments, the method further comprises:

[0025] The image block corresponding to the second medical image is input into a preset second neural network model to obtain an analysis result of the second medical image; the analysis result of the second medical image comprises any one of a classification result, a segmentation result and a detection result.

[0026] In one of the embodiments, the training process of the second neural network model comprises:

[0027] A gold standard analysis result corresponding to the gold standard image is obtained; the gold standard analysis result comprises any one of a gold standard classification result, a gold standard segmentation result and a gold standard detection result;

[0028] The image block corresponding to the second medical image is input into a second initial neural network model to obtain an analysis result of the second medical image;

[0029] According to the analysis result of the second medical image and the gold standard analysis result, a value of a third loss function is obtained;

[0030] According to the value of the third loss function, the first initial neural network model and the second initial neural network model are cascaded and trained to obtain the first neural network model and the second neural network model.

[0031] In one of the embodiments, the method further comprises:

[0032] According to the gold standard analysis result, a standard template image is obtained; the standard template image is used to represent the labeling information of the gold standard analysis result;

[0033] A class activation map output by a convolution layer of the second initial neural network model is obtained;

[0034] The class activation map is adjusted by using the standard template image to obtain the second neural network model.

[0035] In one of the embodiments, the class activation map is obtained by adjusting the feature map output by the convolutional layer according to the weight of the full connection layer of the second initial neural network model.

[0036] In one of the embodiments, the method further comprises:

[0037] resampling the sample images to obtain resampled sample images;

[0038] dividing the resampled sample images into blocks to obtain the sample image blocks.

[0039] In a second aspect, the present application further provides an image processing device, which comprises:

[0040] a first obtaining module configured to obtain a first medical image;

[0041] a first processing module configured to input the first medical image into a preset first neural network model to obtain a second medical image; wherein the resolution of the second medical image is greater than that of the first medical image; the first neural network model is obtained by training a first initial neural network model according to processed sample image blocks corresponding to sample images and gold standard images corresponding to the sample images; the processed sample image blocks are obtained by performing first expansion processing and / or second expansion processing on the sample image blocks corresponding to the sample images; the first expansion processing comprises at least one of adding a blank image block to each of the sample image blocks, adjusting the arrangement order of each of the sample image blocks, replacing a target image block in each of the sample image blocks, and deleting at least one sample image block other than the target image block in each of the sample image blocks; the second expansion processing comprises adding a blank image block to the feature map of each of the sample image blocks.

[0042] In a third aspect, the present application further provides a computer device, which comprises a memory and a processor; the memory stores a computer program; and the processor implements the following steps when executing the computer program:

[0043] obtaining a first medical image;

[0044] input the first medical image into a preset first neural network model to obtain a second medical image; wherein a resolution of the second medical image is greater than a resolution of the first medical image; the first neural network model is obtained by training a first initial neural network model according to processed sample image blocks corresponding to sample images and gold standard images corresponding to the sample images; the processed sample image blocks are obtained by performing first expansion processing and / or second expansion processing on each sample image block of the sample images; the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting an arrangement order of each sample image block, performing replacement processing on a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block; the second expansion processing includes adding a blank image block to a feature map of each sample image block.

[0045] In a fourth aspect, the present application further provides a computer readable storage medium, wherein the computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the following steps:

[0046] obtaining a first medical image;

[0047] inputting the first medical image into a preset first neural network model to obtain a second medical image; wherein a resolution of the second medical image is greater than a resolution of the first medical image; the first neural network model is obtained by training a first initial neural network model according to processed sample image blocks corresponding to sample images and gold standard images corresponding to the sample images; the processed sample image blocks are obtained by performing first expansion processing and / or second expansion processing on each sample image block of the sample images; the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting an arrangement order of each sample image block, performing replacement processing on a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block; the second expansion processing includes adding a blank image block to a feature map of each sample image block.

[0048] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and the computer program, when executed by a processor, implements the following steps:

[0049] obtaining a first medical image;

[0050] input the first medical image into a preset first neural network model to obtain a second medical image; a resolution of the second medical image is greater than a resolution of the first medical image; the first neural network model is obtained by training a first initial neural network model according to a processed sample image block corresponding to a sample image and a gold standard image corresponding to the sample image; the processed sample image block is obtained by performing first expansion processing and / or second expansion processing on each sample image block of the sample image; the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting an arrangement order of each sample image block, performing replacement processing on a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block; the second expansion processing includes adding a blank image block to a feature map of each sample image block.

[0051] The image processing method and the storage medium, since the first neural network model is obtained by training a first initial neural network model according to a processed sample image block corresponding to a sample image and a gold standard image corresponding to the sample image, and the processed sample image block is obtained by performing first expansion processing and / or second expansion processing on each sample image block of the sample image, the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting an arrangement order of each sample image block, performing replacement processing on a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block, and the second expansion processing includes adding a blank image block to a feature map of each sample image block, so that redundant information in the sample image is excluded, the first initial neural network model can learn more deep and more universal features of the sample image by the first expansion processing and / or the second expansion processing performed on each sample image block of the sample image, the richness of the processed sample image block obtained by combining different first expansion processing and second expansion processing is improved, so that the expansibility of the first initial neural network model training is improved, and the expansibility of processing the first medical image to obtain the second medical image by using the trained first neural network model is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 An application environment diagram of an image processing method in an embodiment;

[0053] Figure 2 A flowchart of an image processing method in an embodiment;

[0054] Figure 3 A process diagram of training a first initial neural network model by using a sample image block that has not been expanded in an embodiment;

[0055] Figure 4 A process diagram of training the first initial neural network model by adding a blank image block to each sample image block in an embodiment;

[0056] Figure 5 A process diagram of training the first initial neural network model by adjusting the arrangement order of each sample image block in an embodiment;

[0057] Figure 6 A process diagram of training the first initial neural network model by replacing a target image block in each sample image block in an embodiment;

[0058] Figure 7 A reconstruction result diagram of the first neural network model provided in an embodiment for first medical images with different layer thicknesses;

[0059] Figure 8 A reconstruction result diagram of the first neural network model provided in an embodiment for first medical images scanned by different scanning devices;

[0060] Figure 9 A flowchart of an image processing method in another embodiment;

[0061] Figure 10 A flowchart of an image processing method in another embodiment;

[0062] Figure 11 A process diagram of performing second expansion processing on each sample image block of a sample image in an embodiment;

[0063] Figure 12 A flowchart of an image processing method in another embodiment;

[0064] Figure 13 A flowchart of a classification task in an embodiment;

[0065] Figure 14 A flowchart of an image processing method in another embodiment;

[0066] Figure 15 A structural block diagram of an image processing apparatus in an embodiment. DETAILED DESCRIPTION

[0067] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0068] The image processing method provided by the embodiments of the present application can be applied to a computer device as shown in Figure 1 The computer device includes a processor and a memory connected through a system bus. The memory stores a computer program. When the processor executes the computer program, the steps of the method embodiments described below can be executed. Optionally, the computer device can further include a network interface, a display screen and an input device. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external terminal through a network connection. Optionally, the computer device can be a server, a personal computer, a personal digital assistant, or other terminal devices such as a tablet computer, a mobile phone, etc. The computer device can also be a cloud server or a remote server. The embodiments of the present application do not limit the specific form of the computer device.

[0069] In one embodiment, as shown in Figure 2 An image processing method is provided. The method is applied to a computer device as shown in Figure 1 The method includes the following steps:

[0070] S201, obtaining a first medical image.

[0071] The first medical image can be a brain medical image, a lung medical image, an abdominal medical image, a chest medical image, etc. Optionally, the first medical image can be a magnetic resonance image (MRI), a computed tomography (CT) image, a digit radiography (DR) image, or an image of any modality in a natural image. Optionally, the computer device can obtain the first medical image from a medical image device in real time, or obtain the first medical image from a picture archiving and communication systems (PACS) server. Optionally, the computer device can further preprocess the obtained first medical image, such as removing artifacts and noise. In addition, it should be noted that the image processing method provided by the embodiments of the present application is not only applicable to two-dimensional medical images, but also applicable to three-dimensional medical images.

[0072] S202, input the first medical image into a preset first neural network model to obtain a second medical image; wherein the resolution of the second medical image is greater than the resolution of the first medical image; the first neural network model is obtained by training a first initial neural network model according to a processed sample image block corresponding to a sample image and a gold standard image corresponding to the sample image; the processed sample image block is obtained by performing first expansion processing and / or second expansion processing on each sample image block of the sample image; the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting the arrangement order of each sample image block, replacing a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block; the second expansion processing includes adding a blank image block to a feature map of each sample image block.

[0073] In the embodiment, the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting the arrangement order of each sample image block, replacing a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block. For example, the first expansion processing is adding a blank image block to each sample image block, Figure 3 For a process diagram of training the first initial neural network model by using a sample image block without expansion processing in an embodiment, Figure 4 For a process diagram of training the first initial neural network model by adding a blank image block to each sample image block in an embodiment, Figure 4 In the embodiment, the image block without any combined image information is the added blank block. For example, the first expansion processing is adjusting the arrangement order of each sample image block, Figure 5 For a process diagram of training the first initial neural network model by adjusting the arrangement order of each sample image block in an embodiment, Figure 5 In the embodiment, the sample image block with a frame input into the encoder is the sample image block with the adjusted arrangement order. For example, the first expansion processing is replacing a target image block in each sample image block, in which case the target image block can be a sample image block with a higher resolution in each sample image block, or a sample image block that is a lesion part in each sample image block, Figure 6 For a process diagram of training the first initial neural network model by replacing a target image block in each sample image block in an embodiment, Figure 6 In the embodiment, the sample image block with a frame input into the encoder is the image block after the replacement processing of the target image block. For example, the first expansion processing is deleting at least one sample image block other than the target image block in each sample image block, in which case the target image block can be any image block in the sample image block except an image block corresponding to a lesion area.

[0074] Optionally, in the embodiment, the first initial neural network model can be any one of a convolutional neural network model, a fully convolutional network model, a deep neural network model, a generative adversarial network model, a recurrent neural network model, a deep residual network model, a long short-term memory network model, and all existing neural network models suitable for images. Optionally, the computer device can segment the sample image using a sliding window method to obtain each sample image block of the sample image; or the computer device can also segment the sample image using a segmentation template of a preset size to obtain each sample image block of the sample image. Optionally, the computer device can input the processed sample image block corresponding to the sample image into the first initial neural network model, obtain the value of the loss function using the output of the first initial neural network model and the gold standard image corresponding to the sample image, and train the first initial neural network model according to the value of the loss function to obtain the first neural network model.

[0075] Exemplarily, Figure 7 A reconstruction result diagram of the first neural network model provided by one embodiment for the first medical image with different layer thicknesses is shown in FIG. 6, Figure 8 A reconstruction result diagram of the first neural network model provided by one embodiment for the first medical image scanned by different scanning devices is shown in FIG. 7, which is obtained by Figure 7 and Figure 8 It can be seen that the first neural network model has high robustness in reconstructing the first medical image with different layer thicknesses and scanned by different scanning devices.

[0076] In the image processing method, the first neural network model is obtained by training the first initial neural network model according to the processed sample image block corresponding to the sample image and the gold standard image corresponding to the sample image, and the processed sample image block is obtained by performing the first expansion processing and / or the second expansion processing on each sample image block of the sample image. The first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting the arrangement order of each sample image block, performing replacement processing on a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block. The second expansion processing includes adding a blank image block to a feature map of each sample image block. In this way, the redundant information in the sample image is excluded. The first initial neural network model can learn more deep and more universal features of the sample image by performing the first expansion processing and / or the second expansion processing on each sample image block of the sample image. The richness of the processed sample image block is improved by combining different first expansion processing and second expansion processing, thereby improving the expansibility of training the first initial neural network model, and further improving the expansibility of processing the first medical image to obtain the second medical image by using the trained first neural network model.

[0077] In the above scenario of training the first neural network model, if the processed sample image block is obtained by performing the first expansion processing on each sample image block of the sample image, in an embodiment, as shown in Figure 9 the training process of the first neural network model includes:

[0078] S301, performing first expansion processing on each sample image block to obtain a processed image block.

[0079] Optionally, in this embodiment, the first expansion processing performed by the computer device on each sample image block can include at least one of adding a blank image block to each sample image block, adjusting the arrangement order of each sample image block, performing replacement processing on a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block. The first expansion processing is to replace the target image block in each sample image block and the first expansion processing is to delete at least one sample image block other than the target image block in each sample image block will be described in detail as follows:

[0080] Firstly, if the first expansion processing is to replace the target image block in each sample image block, the above S301 includes:

[0081] Step A: randomly selecting a plurality of first target image blocks from each sample image block.

[0082] Optionally, in the embodiment, the number of the selected first target image blocks can be determined according to actual application requirements, and then the first target image blocks of the number are selected from the sample image blocks. Optionally, the selected first target image blocks can be adjacent image blocks, or can be different image blocks with a plurality of image blocks in between.

[0083] Step B: replacing the plurality of first target image blocks with corresponding second target image blocks to obtain a processed image block, the resolution of the second target image block being higher than that of the first target image block.

[0084] Optionally, in the embodiment, after the first target image blocks are selected, the first target image blocks can be subjected to image enhancement processing to obtain second target image blocks with higher resolution than the first target image blocks. For example, the first target image blocks can be subjected to histogram enhancement processing to obtain the second target image blocks, and then the selected first target image blocks are replaced with the corresponding second target image blocks to obtain the processed image block.

[0085] Secondly, if the first expansion processing is to delete at least one sample image block other than the target image block in each sample image block, the S301 comprises:

[0086] Step C: selecting a third target image block corresponding to a lesion region in the sample image from the sample image blocks.

[0087] Optionally, the computer device can select a lesion region corresponding image block in the sample image from the sample image blocks, and determine the selected lesion region corresponding image block as the third target image block. For example, taking a brain image as the sample image, the computer device can determine a brain atrophy region corresponding image block in the brain image as the third target image block.

[0088] Step D: deleting at least one sample image block other than the third target image block in each sample image block to obtain a processed image block.

[0089] Optionally, in this embodiment, after the third target image block is selected from each sample image block of the sample image, the computer device can delete at least one sample image block other than the third target image block in each sample image block to obtain a processed image block, that is, the computer device can delete an image block other than the lesion area in each sample image block to obtain a processed image block. It should be noted that the lesion area in the medical image will affect the processing accuracy of the neural network model on the medical image, and the lesion area in the medical image is retained, and in the process of training the first initial neural network model using the sample image block including the lesion area, the first initial neural network model can learn the features of the lesion area, so that the first initial neural network model can process the lesion area, thereby reducing the processing accuracy of the neural network model on the medical image.

[0090] It should be noted that due to the anisotropy of the medical image, the resolution of the transverse section is generally higher, and the resolution of the sagittal and coronal sections is lower. In the scene of super-resolution reconstruction of the medical image, the several layers with high resolution can be deleted to let the network know that the several layers do not need to be reconstructed. Optionally, in an embodiment, the image block corresponding to the high-resolution image in the sample image can be selected from each sample image block, and the selected image block corresponding to the high-resolution image is deleted to obtain a processed sample image block.

[0091] S302, input the processed image block into the first initial neural network model to obtain a third medical image.

[0092] Optionally, the computer device can input the processed image block into the first initial neural network model in order according to the order of the processed image block to obtain the third medical image; or the computer device can also input all the processed image blocks into the first initial neural network model at the same time to obtain the third medical image.

[0093] S303, obtain the value of the first loss function according to the third medical image and the gold standard image, and train the first initial neural network model according to the value of the first loss function to obtain a first neural network model.

[0094] In this embodiment, the computer device can obtain the value of the first loss function according to the third medical image and the gold standard image corresponding to the sample medical image, and adjust the parameters of the first initial neural network model according to the value of the first loss function. The operation is repeated until the value of the first loss function reaches a minimum value or the value of the first loss function reaches a stable value, and the first initial neural network model at this time is taken as the first neural network model.

[0095] In this embodiment, the first extension processing is performed on each sample image block of the sample image, the redundant information in the sample image is excluded, the first initial neural network model can learn more deep and more universal features of the sample image, the richness of the processed sample image block is improved, and the expansibility of the training of the first initial neural network model is improved.

[0096] In the above scenario of training the first neural network model, if the processed sample image block is obtained by performing second extension processing on each sample image block of the sample image, in one embodiment, as shown in Figure 10 the training process of the first neural network model includes:

[0097] S401, inputting each target sample image block in the sample image block into the encoding layer of the first initial neural network model to obtain the feature map corresponding to each target sample image block.

[0098] Optionally, in this embodiment, the computer device can select part of the sample image blocks from each sample image block as the target sample image block, input the selected target sample image block into the encoding layer of the first initial neural network model, perform feature extraction on each target sample image block, and obtain the feature map corresponding to each target sample image block.

[0099] S402, according to the order of each sample image block, adding a blank image block as the feature map corresponding to the sample image block other than the target sample image block in the feature map corresponding to each target sample image block to obtain a processed feature map.

[0100] Optionally, in this embodiment, the computer device can add a blank image block in the corresponding place of the feature map corresponding to each target sample image block according to the order of each sample image block, replace the feature map corresponding to the sample image block other than the target sample image block with the blank image block, and obtain a processed feature map. It should be noted that the number of added blank image blocks corresponds to the number of sample image blocks other than the target sample image block in each sample image block, and is not random. Exemplarily, as shown in Figure 11 Figure 11 the first column of image blocks after the midbrain image is the sample image blocks corresponding to the brain image, the second column of image blocks is the target sample image block selected from each sample image block, the first column of image blocks after the encoder is the feature map corresponding to the target sample image block, and the second column of image blocks after the encoder is the feature map corresponding to each sample image block other than the target sample image block, that is, this column of image blocks is the feature map corresponding to all sample image blocks.

[0101] ​S403, input the processed feature map into a decoding layer of the first initial neural network model to obtain a fourth medical image.

[0102] Specifically, the computer device inputs the processed feature map obtained above into a decoding layer of the first initial neural network model, processes the input processed sample image by using the decoding layer of the first initial neural network model, and obtains a fourth medical image corresponding to the sample image.

[0103] S404, obtain a value of a second loss function according to the fourth medical image and a gold standard image, and train the first initial neural network model according to the value of the second loss function to obtain a first neural network model.

[0104] In this embodiment, the computer device obtains a value of a second loss function according to the fourth medical image obtained above and a gold standard image corresponding to the sample image, adjusts parameters of the first initial neural network model according to the value of the second loss function, repeatedly performs the operation until the value of the second loss function reaches a minimum value or the value of the second loss function reaches a stable value, and regards the first initial neural network model at this time as the first neural network model.

[0105] In this embodiment, by inputting each sample image block of the sample image into the encoding layer of the first initial neural network model, a feature map corresponding to each sample image block is obtained, the feature maps corresponding to the sample image blocks are sorted according to the order of the sample image blocks to obtain a sorted feature map, a blank image block is added to the sorted feature map, so that the processed sample image block is more abundant, and thus the processed sample image block is input into the decoding layer of the first initial neural network model, which can enable the first initial neural network model to learn more deep and more universal features of the sample image, thereby improving the expansibility of the training of the first initial neural network model.

[0106] In some scenarios, the obtained second medical image can also be analyzed to obtain an analysis result of the second medical image. In one embodiment, the above method further includes: inputting an image block corresponding to the second medical image into a preset second neural network model to obtain an analysis result of the second medical image; and the analysis result of the second medical image includes any one of a classification result, a segmentation result, and a detection result.

[0107] In this embodiment, after obtaining the second medical image with a resolution greater than the first medical image, the second medical image can also be analyzed, for example, the second medical image can be classified, segmented, detected, etc. Optionally, the second neural network model in this embodiment can be a classification model, or can be a segmentation model, or can also be a detection model. Optionally, the computer device can input the image blocks corresponding to the second medical image into the preset second neural network model in the arrangement order to obtain the analysis result of the second medical image; or the image blocks corresponding to the second medical image can also be simultaneously input into the preset second neural network model to obtain the analysis result of the second medical image. Optionally, the above-mentioned second neural network model can be obtained by the following training method:

[0108] Step E: obtaining a gold standard analysis result corresponding to the above-mentioned variable standard image; the gold standard analysis result includes any one of a gold standard classification result, a gold standard segmentation result, and a gold standard detection result.

[0109] It can be understood that when the second neural network model is used to classify the second medical image, the obtained gold standard analysis result is a gold standard classification result; when the second neural network model is used to segment the second medical image, the obtained gold standard analysis result is a gold standard segmentation result; and when the second neural network model is used to detect the second medical image, the obtained gold standard analysis result is a gold standard detection result.

[0110] Step F: inputting the image blocks corresponding to the second medical image into the second initial neural network model to obtain an analysis result of the second medical image.

[0111] Step G: obtaining a value of a third loss function according to the analysis result of the second medical image and the gold standard analysis result.

[0112] Step H: performing cascade training on the first initial neural network model and the second initial neural network model according to the value of the third loss function to obtain the first neural network model and the second neural network model.

[0113] Optionally, the computer device can return the parameters of the first initial neural network model and the parameters of the second initial neural network model according to the value of the third loss function, and perform cascade training on the first initial neural network model and the second initial neural network model, until the value of the third loss function reaches a minimum value or the value of the third loss function reaches a stable value, to obtain the first neural network model and the second neural network model.

[0114] In this embodiment, the computer device inputs the image block corresponding to the second medical image with improved resolution into the preset second neural network model, and can obtain the analysis result of the second medical image through the second neural network model. Since the resolution of the second medical image is improved, the detail information of the second medical image is more abundant, and the second neural network model can more accurately analyze the second medical image. Therefore, the accuracy of the analysis result of the second medical image obtained through the second neural network model is improved.

[0115] In the above scenario of training the second neural network model, in one embodiment, as shown in Figure 12 The above method further includes:

[0116] S501, obtaining a standard template image according to the gold standard analysis result; the standard template image is used to represent the annotation information of the gold standard analysis result.

[0117] The standard template image is an image used to represent the annotation information of the gold standard analysis result. Optionally, the annotation information corresponding to the gold standard analysis result can be determined according to the gold standard analysis result, and the annotation information corresponding to the gold standard analysis result is highlighted in the second medical image to obtain the standard template image. Taking the second neural network model as a classification model as an example, the annotation information corresponding to the gold standard analysis result is the range of the lesion area, and therefore the range of the lesion area can be highlighted in the standard template image.

[0118] S502, obtaining a class activation map of the convolution layer output of the second initial neural network model.

[0119] The class activation map is obtained by mapping the output of the second initial neural network model back to the second medical image, and can be used to represent which part of the second medical image has a greater impact on the final analysis result. Optionally, the class activation map is obtained by adjusting the feature map of the convolution layer output of the second neural network model according to the weights of the full connection layer of the second initial neural network model. For example, as shown in Figure 13 , Figure 13 is a schematic diagram of a classification task, Figure 13 The class activation map in the above is the class activation map obtained by adjusting the feature map of the convolution layer output according to the weights of the full connection layer of the second initial neural network model in this embodiment.

[0120] S503, adjusting the class activation map by using the standard template image to obtain the second neural network model.

[0121] Optionally, the error caused by the different data sources and the like can be reduced by constraining the same class in the class activation map and the same area in the standard template image within the error tolerance range, and the class activation map is adjusted by using the standard template image, so as to obtain the second neural network model. For example, please continue to refer to Figure 13 , Figure 13 The image pointed to by the classification result in the above embodiment is a standard template image, and the highlighted part in the class activation map shown in Figure 13 is constrained in the same area in the same class image in the standard template image within the error tolerance range, so as to reduce the error caused by the different data sources and the like in the classification task of the second initial neural network model, thereby improving the scalability of the second initial neural network model on each center data.

[0122] In the embodiment, the annotation information representing the gold standard analysis result can be obtained according to the gold standard analysis result corresponding to the gold standard image, and the class activation map output by the convolution layer of the second neural network model can be obtained, so as to constrain and adjust the class activation map output by the convolution layer of the second neural network model by using the standard template image, so that the same area in the same class image in the standard template image is within the error tolerance range, thereby reducing the error caused by the different data sources and the like in the analysis task of the network, thereby improving the scalability of the second initial neural network model on each center data.

[0123] In the above scenario of performing the first expansion processing and / or the second expansion processing on each sample image block of the sample image, each sample image block of the sample image needs to be obtained. In one embodiment, as shown in Figure 14 , the above method further includes:

[0124] S601, performing resampling processing on the sample image to obtain a resampled sample image.

[0125] Optionally, in the embodiment, the sample image can be resampled to a standard resolution space of 1x1x1mm 3 . It should be noted that the resampling processing is performed on the sample image, and the resampled sample image is input into the neural network model, so as to reduce the learning difficulty of the neural network model and improve the training efficiency of the neural network model. Optionally, if the sample image is a brain image, the skull and other non-brain parenchymal regions in the sample image can be removed.

[0126] S602, performing block processing on the resampled sample image to obtain each sample image block.

[0127] Optionally, the computer device can perform block processing on the resampled sample image by using a preset image block size to obtain each sample image block corresponding to the sample image; or the computer device can also perform block processing on the resampled sample image by using a sliding window method to obtain each sample image block corresponding to the sample image.

[0128] In this embodiment, the sample image is resampled to be a same resolution image, and then the resampled sample image is block processed, so that the consistency of each sample image block corresponding to the sample image is ensured.

[0129] It should be understood that, although each step in the flowchart involved in each of the above embodiments is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0130] Based on the same inventive concept, the embodiments of the present application also provide an image processing apparatus for implementing the above-mentioned image processing method. The solution provided by the apparatus is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more image processing apparatus embodiments provided below can refer to the limitations of the image processing method described above, and will not be repeated here.

[0131] In one embodiment, as shown in Figure 15 An image processing apparatus is provided, comprising: a first acquisition module and a first processing module, wherein:

[0132] The first acquisition module is configured to acquire a first medical image.

[0133] The first processing module is configured to input the first medical image into a preset first neural network model to obtain a second medical image, wherein a resolution of the second medical image is greater than a resolution of the first medical image; the first neural network model is obtained by training a first initial neural network model according to processed sample image blocks of sample images and ground truth images corresponding to the sample images; the processed sample image blocks are obtained by performing first expansion processing and / or second expansion processing on each sample image block of the sample images; the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting an arrangement order of each sample image block, performing replacement processing on a target image block in each sample image block, and deleting at least one sample image block in each sample image block except the target image block; and the second expansion processing includes adding a blank image block to a feature map of each sample image block.

[0134] The image processing apparatus provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0135] On the basis of the above-mentioned embodiments, optionally, if the processed sample image blocks are obtained by performing the first expansion processing on each sample image block of the sample images, the apparatus further includes a second processing module, a second acquisition module, and a first training module, wherein:

[0136] The second processing module is configured to perform the first expansion processing on each sample image block to obtain the processed sample image blocks.

[0137] The second acquisition module is configured to input the processed image blocks into the first initial neural network model to obtain a third medical image.

[0138] The first training module is configured to obtain a value of a first loss function according to the third medical image and the ground truth image, and train the first initial neural network model according to the value of the first loss function to obtain the first neural network model.

[0139] The image processing apparatus provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0140] On the basis of the above-mentioned embodiments, optionally, if the first expansion processing is the replacement processing on the target image block in each sample image block, the second processing module includes a first selection unit and a replacement unit, wherein:

[0141] The first selection unit is configured to select a plurality of first target image blocks from each sample image block.

[0142] A replacing unit is configured to replace the plurality of first target image blocks with corresponding second target image blocks to obtain processed sample image blocks, wherein the resolution of the second target image blocks is higher than the resolution of the first target image blocks.

[0143] The image processing apparatus provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0144] On the basis of the above-mentioned embodiment, optionally, if the first extension processing is to delete at least one sample image block other than the target image block in each sample image block, the second processing module comprises a second selection unit and a deletion unit, wherein:

[0145] The second selection unit is configured to select a third target image block corresponding to a lesion area in the sample image from each sample image block.

[0146] The deletion unit is configured to delete at least one sample image block other than the third target image block in each sample image block to obtain the processed sample image block.

[0147] The image processing apparatus provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0148] On the basis of the above-mentioned embodiment, optionally, if the processed sample image block is obtained by performing second extension processing on each sample image block of the sample image, the apparatus further comprises a third acquisition module, an ordering module, a fourth acquisition module and a second training module, wherein:

[0149] The third acquisition module is configured to input a target sample image block in each sample image block into an encoding layer of the first initial neural network model to obtain a feature map corresponding to each target sample image block.

[0150] The ordering module is configured to add a blank image block as a sample image block corresponding to the target sample image block in the feature map corresponding to each target sample image block according to the order of each sample image block to obtain a processed feature map.

[0151] The fourth acquisition module is configured to input the processed feature map into a decoding layer of the first initial neural network model to obtain a fourth medical image.

[0152] The second training module is configured to obtain a value of a second loss function according to the fourth medical image and a gold standard image, and train the first initial neural network model according to the value of the second loss function to obtain a first neural network model.

[0153] The image processing apparatus provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0154] Optionally, the apparatus further comprises a fifth obtaining module, wherein:

[0155] The fifth obtaining module is configured to input the image block corresponding to the second medical image into a preset second neural network model to obtain an analysis result of the second medical image; the analysis result of the second medical image comprises any one of a classification result, a segmentation result and a detection result.

[0156] The image processing apparatus provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0157] Optionally, the apparatus further comprises a sixth obtaining module, a seventh obtaining module, an eighth obtaining module and a third training module, wherein:

[0158] The sixth obtaining module is configured to obtain a gold standard analysis result corresponding to a gold standard image; the gold standard analysis result comprises any one of a gold standard classification result, a gold standard segmentation result and a gold standard detection result.

[0159] The seventh obtaining module is configured to input the image block corresponding to the second medical image into the second initial neural network model to obtain the analysis result of the second medical image.

[0160] The eighth obtaining module is configured to obtain a value of a third loss function according to the analysis result of the second medical image and the gold standard analysis result.

[0161] The third training module is configured to perform cascade training on the first initial neural network model and the second initial neural network model according to the value of the third loss function to obtain the first neural network model and the second neural network model.

[0162] The image processing apparatus provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0163] Optionally, the apparatus further comprises a ninth obtaining module, a tenth obtaining module and an adjusting module, wherein:

[0164] The ninth obtaining module is configured to obtain a standard template image according to the gold standard analysis result; the standard template image is used to represent annotation information of the gold standard analysis result.

[0165] The tenth obtaining module is configured to obtain a class activation map output by a convolution layer of the second initial neural network model.

[0166] The adjusting module is configured to constrain and adjust the class activation map by using the standard template image to obtain the second neural network model.

[0167] Optionally, the class activation map is obtained by adjusting the feature map output by the convolutional layer according to the weight of the full connection layer of the second initial neural network model.

[0168] The image processing apparatus provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0169] On the basis of the above-mentioned embodiment, the device further comprises a third processing module and a fourth processing module, wherein:

[0170] The third processing module is configured to perform resampling processing on the sample images to obtain resampled sample images.

[0171] The fourth processing module is configured to perform block processing on the resampled sample images to obtain sample image blocks.

[0172] The image processing apparatus provided in the embodiment can execute the method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0173] The modules in the image processing apparatus can be implemented by software, hardware or a combination thereof. The modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules.

[0174] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps:

[0175] Obtaining a first medical image;

[0176] Inputting the first medical image into a preset first neural network model to obtain a second medical image; the resolution of the second medical image is greater than that of the first medical image; the first neural network model is obtained by training a first initial neural network model according to processed sample image blocks corresponding to sample images and ground truth images corresponding to the sample images; the processed sample image blocks are obtained by performing first expansion processing and / or second expansion processing on the sample image blocks of the sample images; the first expansion processing includes at least one of adding a blank image block to the sample image blocks, adjusting the arrangement order of the sample image blocks, replacing a target image block in the sample image blocks, and deleting at least one sample image block in the sample image blocks except the target image block; the second expansion processing includes adding a blank image block to the feature maps of the sample image blocks.

[0177] In one embodiment, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, implements the following steps:

[0178] obtaining a first medical image;

[0179] inputting the first medical image into a preset first neural network model to obtain a second medical image; wherein the resolution of the second medical image is greater than the resolution of the first medical image; the first neural network model is obtained by training a first initial neural network model according to processed sample image blocks corresponding to sample images and gold standard images corresponding to the sample images; the processed sample image blocks are obtained by performing first expansion processing and / or second expansion processing on each sample image block of the sample images; the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting the arrangement order of each sample image block, performing replacement processing on a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block; the second expansion processing includes adding a blank image block to a feature map of each sample image block.

[0180] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the following steps:

[0181] obtaining a first medical image;

[0182] inputting the first medical image into a preset first neural network model to obtain a second medical image; wherein the resolution of the second medical image is greater than the resolution of the first medical image; the first neural network model is obtained by training a first initial neural network model according to processed sample image blocks corresponding to sample images and gold standard images corresponding to the sample images; the processed sample image blocks are obtained by performing first expansion processing and / or second expansion processing on each sample image block of the sample images; the first expansion processing includes at least one of adding a blank image block to each sample image block, adjusting the arrangement order of each sample image block, performing replacement processing on a target image block in each sample image block, and deleting at least one sample image block other than the target image block in each sample image block; the second expansion processing includes adding a blank image block to a feature map of each sample image block.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0184] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0185] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0186] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a first medical image; inputting the first medical image into a preset first neural network model to obtain a second medical image; wherein the resolution of the second medical image is higher than that of the first medical image; the first neural network model is obtained by training a first initial neural network model according to processed sample image blocks corresponding to sample images and gold standard images corresponding to the sample images; the processed sample image blocks are obtained by performing first expansion processing and second expansion processing on each sample image block of the sample images; the first expansion processing comprises at least one of adding a blank image block to each sample image block, adjusting the arrangement order of each sample image block, and deleting at least one sample image block other than a target image block in each sample image block; the second expansion processing comprises adding a blank image block to an image block in which no feature is extracted in a feature map of each sample image block; performing second expansion processing on each sample image block of the sample images to obtain a processed sample image, comprising: inputting a target sample image block in each sample image block into an encoding layer of the first initial neural network model to obtain a feature map corresponding to each target sample image block; according to the order of each sample image block, adding a blank image block as a feature map corresponding to a sample image block other than the target sample image block in each target sample image block to obtain a processed feature map.

2. The method of claim 1, wherein, If the processed sample image block is obtained by performing first expansion processing on each sample image block of the sample images, the training process of the first neural network model comprises: performing the first expansion processing on each sample image block to obtain the processed sample image block; inputting the processed image block into the first initial neural network model to obtain a third medical image; obtaining the value of a first loss function according to the third medical image and the gold standard image, and training the first initial neural network model according to the value of the first loss function to obtain the first neural network model.

3. The method of claim 2, wherein, If the first expansion processing is replacement processing on a target image block in each sample image block, the performing the first expansion processing on each sample image block to obtain the processed sample image block comprises: arbitrarily selecting a plurality of first target image blocks from each sample image block; replacing the plurality of first target image blocks with corresponding second target image blocks to obtain the processed sample image block; the resolution of the second target image block is higher than that of the first target image block.

4. The method of claim 2, wherein, If the first expansion processing is deleting at least one sample image block other than a target image block in each sample image block, the performing the first expansion processing on each sample image block to obtain the processed sample image block comprises: selecting a third target image block corresponding to a lesion area in the sample image from each sample image block; delete at least one sample image block in each of the sample image blocks except the third target image block to obtain the processed sample image block.

5. The method of claim 1, wherein, If the processed sample image block is obtained by performing second expansion processing on each sample image block of the sample image, the training process of the first neural network model comprises: inputting the processed feature map into a decoding layer of the first initial neural network model to obtain a fourth medical image; obtaining a value of a second loss function according to the fourth medical image and the gold standard image, and training the first initial neural network model according to the value of the second loss function to obtain the first neural network model.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: inputting the image block corresponding to the second medical image into a preset second neural network model to obtain an analysis result of the second medical image; the analysis result of the second medical image comprises any one of a classification result, a segmentation result and a detection result.

7. The method of claim 6, wherein, The training process of the second neural network model comprises: obtaining a gold standard analysis result corresponding to the gold standard image; the gold standard analysis result comprises any one of a gold standard classification result, a gold standard segmentation result and a gold standard detection result; inputting the image block corresponding to the second medical image into a second initial neural network model to obtain an analysis result of the second medical image; obtaining a value of a third loss function according to the analysis result of the second medical image and the gold standard analysis result; training the first initial neural network model and the second initial neural network model in cascade according to the value of the third loss function to obtain the first neural network model and the second neural network model.

8. The method of claim 7, wherein, The method further comprises: obtaining a standard template image according to the gold standard analysis result; the standard template image is used to represent the annotation information of the gold standard analysis result; obtaining a class activation map output by a convolution layer of the second initial neural network model; constraining and adjusting the class activation map by using the standard template image to obtain the second neural network model.

9. The method of claim 8, wherein, The class activation map is obtained by adjusting a feature map output by the convolution layer according to weights of a full connection layer of the second initial neural network model.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 9. The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 9.

Images