Image processing method, electronic device and non-transitory computer-readable storage medium

Abstract

The application provides an image processing method, comprising: acquiring first image data; converting the first image data into second image data according to a first threshold, wherein the first threshold is determined according to parameters of a deep learning model; acquiring first sub-image data in the second image data according to a second threshold, wherein the second threshold is determined according to the first threshold and a storage unit capacity of a GPU; adjusting the number of pixels in each dimension of the first sub-image data until a preset condition is met, to generate second sub-image data; and loading the second sub-image data into the storage unit of the GPU and performing inference through the deep learning model, wherein the deep learning model runs on the GPU. The image processing method provided by the application can occupy as much GPU storage space as possible in the process of one-time prediction (forward propagation) of the segmented sub-image, thereby reducing the inference times and improving the inference speed of the deep learning model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, an electronic device, and a non-transitory computer-readable storage medium. Background Technology

[0002] With the development of science and technology, more and more artificial intelligence technologies are being applied to the medical field, such as processing medical images through machine learning. Deep learning models are trained on a large amount of medical imaging data, learning complex model parameters and forming a certain level of "intelligence" in understanding this type of medical imaging data. When image data is forward-propagated through a deep learning model, the model can infer the corresponding optimization or processing results.

[0003] Deep learning models typically consist of Convolutional Neural Networks (CNNs) followed by several fully connected layers, mapping the feature maps generated by the convolutional layers into fixed-length feature vectors. Classical CNN architectures, exemplified by LeNet, are well-suited for image-level classification and regression tasks. In regression tasks, the desired output of the neural network is a numerical description of the entire input image. Classification tasks, building upon regression, normalize the output using a softmax function to obtain the desired probability distribution. However, CNNs cannot solve semantic-level image segmentation problems; therefore, fully convolutional networks were proposed.

[0004] Unlike classic convolutional neural networks (CNNs), which use fully connected layers after convolutional layers to obtain fixed-length feature vectors for classification (fully connected layer + softmax output), fully convolutional neural networks (WCNNs) can accept input images of any size. They use deconvolutional layers to upsample the feature map of the last convolutional layer, restoring it to the same size as the input image. This allows for a prediction for each pixel while preserving the spatial information of the original input image. Finally, pixel-by-pixel classification is performed on the upsampled feature map, and the softmax classification loss is calculated for each pixel. This is equivalent to each pixel corresponding to a training sample, thus completing pixel-level image segmentation.

[0005] Because fully convolutional networks can accept images of arbitrary size during inference, their memory usage increases significantly when the image size is too large. When the image's memory usage exceeds the available memory, to ensure the inference process continues, the input image needs to be divided into multiple patches, each inferred separately, and then the inference results of each patch are merged to generate the final inference result. This method has certain drawbacks. For example, the size of the image patches is usually fixed, the same as the size of the data blocks during training, and cannot be adaptively modified according to different hardware environments. Furthermore, the large number of image patches results in a higher number of inference / prediction operations for the convolutional neural network, slowing down the overall inference speed. Summary of the Invention

[0006] In view of at least one deficiency of the prior art, according to a first aspect of the present invention, the present invention provides an image processing method, comprising:

[0007] Acquire the first image data;

[0008] The first image data is converted into second image data according to a first threshold, wherein the first threshold is determined according to the parameters of the deep learning model;

[0009] The first sub-image data in the second image data is obtained according to the second threshold, wherein the second threshold is determined based on the first threshold and the storage unit capacity of the GPU;

[0010] Adjust the number of pixels in each dimension of the first sub-image data until the preset conditions are met, and then generate the second sub-image data;

[0011] The second sub-image data is loaded into the storage unit of the GPU and inferred through the deep learning model, wherein the deep learning model runs on the GPU.

[0012] According to a first aspect of the invention, the first threshold includes the number of pixels in each dimension of the minimum image data, the number of pixels in each dimension of the minimum image data being determined by the following process:

[0013] Load the deep learning model and parse the parameter file of the deep learning model;

[0014] Based on the parameter file, determine the number of pooling operations performed by the deep learning model during feedforward propagation;

[0015] The number of pixels in each dimension of the minimum image data is determined based on the number of pooling operations.

[0016] According to a first aspect of the invention, the second threshold includes the number of pixels in each dimension of the maximum image data that can be run on the GPU, the number of pixels in each dimension of the maximum image data being determined by the following process:

[0017] The first simulated image data is randomly generated based on the number of pixels in each dimension of the minimum image data;

[0018] The first simulated image data is loaded into the storage unit of the GPU and inferred through the deep learning model;

[0019] If the deep learning model is able to infer the first simulated image data, then the first simulated image data is expanded into second simulated image data and loaded into the storage unit of the GPU, and inference is performed through the deep learning model.

[0020] If the deep learning model cannot reason about the second simulated image data, then the first simulated image data is used as the maximum image data to determine the number of pixels in each dimension of the maximum image data.

[0021] According to a first aspect of the invention, the number of pixels in each dimension of the maximum image data is further determined by the following steps:

[0022] After deleting the first simulated image data from the GPU's storage unit, the second simulated image data is loaded.

[0023] According to a first aspect of the invention, the step of converting the first image data into second image data according to a first threshold comprises:

[0024] The first image data is interpolated by pixels to obtain the second image data, wherein the second image data can be segmented into one or more minimum image data corresponding to the first threshold.

[0025] According to a first aspect of the present invention, obtaining the first sub-image data in the second image data according to a second threshold includes:

[0026] The second image data is segmented along the first dimension to obtain the first sub-image data, wherein the number of pixels in the first sub-image data is the same as the number of pixels in the maximum image data corresponding to the second threshold, and the first dimension is the dimension with the largest number of pixels in the second image data.

[0027] According to a first aspect of the invention, the preset conditions include:

[0028] The ratio of the number of pixels in each dimension of the first sub-image data is less than a preset value.

[0029] According to a first aspect of the present invention, adjusting the number of pixels in each dimension of the first sub-image data until a preset condition is met to generate the second sub-image data includes:

[0030] Based on the first threshold, reduce the number of pixels in the first sub-image data along the second dimension;

[0031] The number of pixels in the first sub-image data is increased along the third dimension so that the total number of pixels in the first sub-image data remains unchanged. The second dimension is the dimension with the smallest number of pixels in the first sub-image data, and the third dimension is the dimension with the largest number of pixels in the first sub-image data.

[0032] According to a first aspect of the invention, the method further comprises:

[0033] Multiple second sub-image data are sequentially loaded into the storage unit of the GPU and inference is performed through the deep learning model;

[0034] Merge multiple inference results.

[0035] According to a first aspect of the invention, the method further comprises:

[0036] A zero matrix with the same number of pixels as the second image data is generated to store the merged inference results.

[0037] According to a first aspect of the invention, the deep learning model is implemented by a fully convolutional neural network, and the first image data includes images representing three-dimensional regions of the patient.

[0038] In a second aspect, the present invention also provides an electronic device, comprising:

[0039] Processor; and

[0040] A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in the first aspect of the invention.

[0041] Thirdly, the present invention also provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method as described in the first aspect of the present invention.

[0042] The image processing method provided by this invention pre-segments the original input image. Based on the limitations of GPU storage space for running the deep learning model, the parameter characteristics of the deep learning model itself, and the feature distribution and noise of the original input image, the image data during the forward propagation process occupies as much GPU storage space as possible, thereby reducing the number of inferences and accelerating inference; and achieving better feature extraction and noise reduction effects. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings, without exceeding the scope of protection claimed by this application.

[0044] Figure 1 An image processing method provided by an embodiment of the present invention is shown;

[0045] Figure 2 An embodiment of the present invention provides an image processing method in which a first threshold is determined based on the number of pooling operations of a neural network;

[0046] Figure 3 An embodiment of the present invention illustrates an image processing method that segments sub-image data from the original image data after insufflation.

[0047] Figure 4 The image processing method provided by one embodiment of the present invention is shown in which the segmented sub-image data is propagated forward in a neural network. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] By feeding known data or data pairs from an image database into a deep learning model and training it through backpropagation, the model parameters are obtained, essentially allowing the deep learning model to develop a certain type of "cognition." In actual use, the data to be processed / optimized is input into the deep learning model for forward propagation, enabling the model to infer or predict the learning outcome.

[0050] Deep learning models can be built using fully convolutional neural networks (LCNs). LCNs can accept image data of any size and use deconvolution layers to restore the final convolution result (feature map) to the same size as the original input image, thus completing pixel-by-pixel image segmentation. However, due to the storage limitations of the GPU running the deep learning model, to ensure smooth execution of the inference process, the original input image needs to be segmented into multiple sub-images (patch), inferred separately for each, and then the inference results for multiple sub-images (patch) are merged.

[0051] Segmenting the original input image data presents the following problems:

[0052] 1. The size of the segmented sub-images (patch size) is fixed, usually consistent with the size of the image data during training, and cannot be adaptively modified according to different hardware environments. This fixed segmentation method results in a large original input image being divided into many sub-images, which is not conducive to the extraction of contour features and may also generate more noise at the segmentation points.

[0053] 2. Under the same memory usage, the number of sub-images divided from the same original input image cannot be guaranteed to be the minimum. As the number of segmented sub-images increases, the number of predictions made by the deep learning model also increases accordingly, thus slowing down the overall inference speed.

[0054] This invention provides an image processing method that pre-segments the original input image. Based on the limitations of GPU storage space for running the deep learning model, the parameter characteristics of the deep learning model itself, and the feature distribution and noise of the original input image, the method ensures that the image data during the forward propagation process occupies as much GPU storage space as possible, thereby reducing the number of inferences, accelerating inference, and achieving better feature extraction and noise reduction effects.

[0055] According to one embodiment of the present invention, such as Figure 1 As shown, the present invention provides an image processing method 10, comprising steps S101 to S105. Wherein:

[0056] In step S101, first image data is acquired. The first image data is the original image data to be optimized / inferred. The first image dataset is loaded into the storage unit, and the learned deep learning model is loaded.

[0057] In step S102, the first image data is processed into second image data according to a first threshold, wherein the first threshold is determined based on the parameters of the learned deep learning model. Since deep learning models have certain parameter limitations, the original image data to be optimized / inferred cannot be segmented indefinitely; that is, the input image data to the deep learning model has a minimum image data size limitation. The parameters of the deep learning model include, for example, the number of pooling operations performed by the deep learning model on each dimension of the input image data. If the number of pixels in a certain dimension of the input image data is insufficient to support the deep learning model performing pooling operations on that dimension, the deep learning model cannot make correct inferences about the input image data. Therefore, the first threshold corresponds to the minimum image data size, and the first image data is processed into second image data according to the first threshold, for example, such that the second image data includes one or more sub-images of the minimum image data size.

[0058] In step S103, the first sub-image data in the second image data is obtained according to a second threshold, wherein the second threshold is determined based on the first threshold and the storage capacity of the GPU. The maximum image data size that can be processed when running the deep learning model on the GPU is determined based on the GPU's storage capacity. This maximum image data size is obtained based on the aforementioned minimum image data size and the GPU's storage capacity. Optionally, through experimentation, the input image data size is gradually increased until the deep learning model can no longer run on the GPU; the image data size corresponding to the last successful run is then the maximum image data size. Therefore, the second threshold corresponds to the maximum image data size, and the first sub-image data in the second image data is obtained according to the second threshold, for example, making the first sub-image data the maximum image data size, which, after adjustment, can be used as the actual input image data for the deep learning model.

[0059] In step S104, the number of pixels in each dimension of the first sub-image data is adjusted until a preset condition is met, generating the second sub-image data. The segmented sub-image data (i.e., the first sub-image data) is obtained based on a second threshold (corresponding to the maximum image data size). However, the segmented sub-image data should also meet the preset conditions proposed for the original image data. For example, if there is a lot of noise in the original image data, the shape of the segmented sub-image data should be close to the shape of the original image data; if the features in the original image data are obvious, the segmented sub-image data can be made closer to the maximum image data size without excessive shape adjustment. The number of pixels in each dimension of the first sub-image data is adjusted according to the preset conditions. Optionally, each adjustment is checked using the preset conditions until the preset conditions are met, obtaining the second sub-image data.

[0060] In step S105, the second sub-image data is loaded into the storage unit of the GPU and inferred using the deep learning model, which runs on the GPU. The final generated second sub-image data meets the maximum image data size that the deep learning model can process when running on the GPU, and the shape of the adjusted sub-image data meets the requirements for feature extraction and noise reduction of the original image data. The second sub-image data is then loaded into the GPU's storage unit, and the deep learning model is run on the GPU for inference. Optionally, the original image data is divided into multiple second sub-image data sets with pixel overlap. These sets are sequentially loaded into the GPU's storage unit, the deep learning model is run for inference, and the inference results are output. The multiple inference results are then merged to obtain the inference result for the original image data.

[0061] The image processing method provided by the above embodiments of the present invention obtains the basis for segmenting the original input image data based on the parameters of the deep learning model and the capacity limitations of the GPU storage unit. The segmented sub-image data can occupy as much GPU storage space as possible in one prediction (forward propagation) process, thereby reducing the number of inferences, speeding up the inference process, and making it more conducive to the extraction and noise reduction of the overall image features.

[0062] According to an embodiment of the present invention, in the image processing method 10 provided by the present invention, the first threshold includes the minimum number of pixels in each dimension of the image data, and determining the first threshold specifically includes:

[0063] Load the deep learning model and parse the parameter file of the deep learning model.

[0064] Based on the parameter file, determine the number of pooling operations performed by the deep learning model during feedforward propagation.

[0065] The number of pixels in each dimension of the minimum image data is determined based on the number of pooling operations.

[0066] The following explanation uses the nnUNet neural network as an example of a deep learning model.

[0067] First, prepare the data, including the trained fully convolutional network and its corresponding parameter file. The main parameters in the parameter file are as follows:

[0068]

[0069]

[0070]

[0071]

[0072] Load the trained nnUNet neural network model and parse the parameter file mentioned above.

[0073] Calculate a first threshold, which includes the number of pixels in the three dimensions of the minimum image data, wherein the minimum image data is determined based on the deep learning model used.

[0074] Taking the nnUNet 3D neural network model as an example, its neural network structure is as follows: Figure 2 As shown, during feedforward propagation, the 3D U-Net (a type of nnUNet) neural network performs 5 max pooling operations in two dimensions and 4 max pooling operations with a stride of 2 in the other dimension. Therefore, the minimum image data size that 3D U-Net can process is (if the image data is smaller, it cannot perform a specified number of pooling operations):

[0075] 2 4 ×2 5 ×2 5 =16×32×32

[0076] The first threshold is set based on the number of pixels in each dimension of the minimum image data. That is, for the segmentation of the original input image data, the number of pixels in any dimension cannot be less than 16, and apart from the dimension with the smallest number of pixels, the number of pixels in the other two dimensions cannot be less than 32.

[0077] According to an embodiment of the present invention, in the image processing method 10 provided by the present invention, the second threshold includes the number of pixels in each dimension of the maximum image data that can be processed on the GPU, and determining the second threshold specifically includes:

[0078] The first simulated image data is randomly generated based on the number of pixels in each dimension of the minimum image data.

[0079] The first simulated image data is loaded into the storage unit of the GPU and inferred through the deep learning model.

[0080] If the deep learning model is able to infer from the first simulated image data, then the first simulated image data is expanded into second simulated image data and loaded into the storage unit of the GPU for inference by the deep learning model.

[0081] If the deep learning model cannot reason about the second simulated image data, the first simulated image data is used to determine the maximum image data.

[0082] According to an embodiment of the present invention, in the image processing method 10 provided by the present invention, determining the second threshold further includes:

[0083] After deleting the first simulated image data from the GPU's storage unit, the second simulated image data is loaded.

[0084] Determine the maximum image data size that the GPU can support for running the deep learning model, including the following steps (taking a deep learning model that includes a fully convolutional neural network as an example):

[0085] 1. Randomly generate a tensor with the smallest image data size (corresponding to the tensor of the first simulated image data) and load this tensor into a fully convolutional neural network for result prediction. Here, we do not care about the prediction result, but only whether it can run.

[0086] 2. If the fully convolutional neural network can run, clear the data in the GPU memory (GPU storage unit) other than the fully convolutional neural network model.

[0087] 3. Enlarge the first simulated image data and re-apply the fully convolutional neural network for prediction, then monitor whether it can run. The method for enlarging the first simulated image data is as follows:

[0088] A. Determine the minimum number of pixels (denoted as min_data) in the minimum image data size corresponding to the first threshold, and the dimension in which the minimum number of pixels is located (denoted as min_dim).

[0089] B. Continuously increase the minimum pixel count min_data in the minimum image data size on the dimension min_dim where the minimum pixel count is located.

[0090] 4. Repeat the process of clearing the stored data in the video memory (GPU storage unit), amplifying and predicting the first simulated image data, until the video memory (GPU storage unit) can no longer support the current data operation.

[0091] 5. The size of the first simulated image data corresponding to the previous operation that could not be supported is taken as the maximum image data size and appended to the parameter file. Taking nnUNet as an example, the maximum image data size (max patch size) information corresponding to the second threshold supplemented in the parameter file is as follows:

[0092] "max_patch_size":[

[0093] 672,

[0094] 32, 32 ]

[0097] According to an embodiment of the present invention, in the image processing method 10 provided by the present invention, the step of processing the first image data into second image data according to a first threshold specifically includes:

[0098] The first image data is interpolated by pixels to obtain the second image data, wherein the second image data can be segmented into one or more minimum image data corresponding to the first threshold.

[0099] The first image data serves as the original image data to be optimized / inferred. The first image is interpolated according to the parameters in the parameter file to modify its size, ensuring that the number of pixels in each dimension of the first image data is greater than the smallest integer multiple of the minimum image data size. The specific calculation method is as follows:

[0100] final_size=ceil(img_size÷min_patch_size)×min_patch_size

[0101] Where final_size represents the size of the adjusted second image data, ceil represents the rounding up operation, img_size represents the size of the first image data (the size of the original image data), and min_patch_size represents the minimum image data size corresponding to the first threshold.

[0102] According to an embodiment of the present invention, in the image processing method 10 provided by the present invention, the step of obtaining the first sub-image data in the second image data according to the second threshold specifically includes:

[0103] The second image data is segmented along the first dimension to obtain the first sub-image data, wherein the number of pixels in the first sub-image data is the same as the number of pixels in the maximum image data corresponding to the second threshold, and the first dimension is the dimension with the largest number of pixels in the second image data.

[0104] The actual segmentation of the sub-image data (patch size) is planned, that is, the acquisition of the first sub-image data from the second image data is planned. The specific steps are as follows:

[0105] 1. Determine the maximum number of pixels (denoted as max_data) and the dimension (denoted as max_dim) of the second image data (current image data).

[0106] 2. Using the dimension max_dim where the maximum number of pixels max_data is located as the dividing direction, and based on the remaining two dimensions, try to set the number of pixels in the corresponding dimension of the actual segmented sub-image data to the number of pixels in the current image data.

[0107] From the perspective of physical data storage, the space occupied by data is related to the data type and the amount of data, but not to the number of dimensions of the data. Therefore, the number of pixels in the dimension max_dim where the maximum number of pixels max_data is located can be obtained by dividing the maximum image data size (max_patch_size) corresponding to the second threshold by the product of the number of pixels in the other two dimensions of the current image data.

[0108] For example: Figure 3 As shown, the current image data size is 416×416×208, and the maximum image data size (max_patch_size) corresponding to the second threshold is 672×32×32. Therefore, the maximum number of pixels in the current image data, max_data, is 416, and the dimension containing the maximum number of pixels is 0 (or 1). The planned actual sub-image data size (patchsize) is: 208×416×(672×32×32÷208÷416), obtaining the actual segmented sub-image data (i.e., the first sub-image data). The size (patch size) of this actual segmented sub-image data is the same as the maximum image data size (max_patch_size) corresponding to the first threshold. In the two dimensions other than the dimension corresponding to the maximum number of pixels in the current image data, the number of pixels is the same as the current image data. Therefore, segmentation is performed in the dimension corresponding to the maximum number of pixels.

[0109] According to an embodiment of the present invention, in the image processing method 10 provided by the present invention, the preset conditions include:

[0110] The ratio of the number of pixels in each dimension of the first sub-image data is less than a preset value.

[0111] According to an embodiment of the present invention, in the image processing method 10 provided by the present invention, adjusting the number of pixels in each dimension of the first sub-image data until a preset condition is met to generate the second sub-image data specifically includes:

[0112] Based on the first threshold, reduce the number of pixels in the first sub-image data along the second dimension;

[0113] The number of pixels in the first sub-image data is increased along the third dimension so that the total number of pixels in the first sub-image data remains unchanged. The second dimension is the dimension with the smallest number of pixels in the first sub-image data, and the third dimension is the dimension with the largest number of pixels in the first sub-image data.

[0114] Calculate the proportions of each dimension value in the current sub-image data (first sub-image data), and adjust the final sub-image data size according to the set minimum scale. Since the size and shape of the original image data are uncertain, the planned sub-image data may have too few pixels in the division direction, leading to poor final prediction results. Therefore, it is necessary to calculate the scale of each dimension value in the current sub-image data and adjust the final sub-image data size. The adjustment method is as follows:

[0115] 1. Calculate the proportion of pixels in each dimension of the current sub-image data (first sub-image data). If the smallest proportion is less than scale, proceed to step 2.

[0116] 2. Determine the maximum number of pixels (denoted as max_patch_data) and the dimension (denoted as max_patch_dim) of the current sub-image data (first sub-image data).

[0117] 3. Subtract the number of pixels in the dimension (max_patch_dim) containing the maximum number of pixels in the minimum image data size (min_patch_size) from the maximum number of pixels (max_patch_data).

[0118] 4. Repeat step 1 until the proportions of all dimensions of the current sub-image data are greater than scale, and then determine the second sub-image data.

[0119] According to an embodiment of the present invention, the image processing method 10 provided by the present invention further includes:

[0120] Multiple second sub-image data are sequentially loaded into the storage unit of the GPU and inference is performed through the deep learning model.

[0121] Merge multiple inference results.

[0122] like Figure 4 As shown, the final segmented sub-image data (i.e., the second sub-image data) is input into the deep learning model for result prediction. The remaining pixels in the second image data are obtained using the same sub-image data size. Optionally, there may be pixel overlap between multiple segmented sub-image data. After merging multiple inference / prediction results, an inference / prediction result for the original input image data is generated.

[0123] According to an embodiment of the present invention, the image processing method 10 provided by the present invention further includes:

[0124] A zero matrix with the same number of pixels as the second image data is generated to store the merged inference results.

[0125] Generate an all-zero matrix with the same size as the interpolated image data, and save the prediction results of the fully convolutional neural network for the corresponding pixels.

[0126] According to an embodiment of the present invention, in the image processing method 10 provided by the present invention, the deep learning model is implemented by a fully convolutional neural network, and the first image data includes an image representing a three-dimensional region of the patient.

[0127] Existing fully convolutional neural networks (LCNs) maintain the same input image block size as during training. However, in practical applications, the training and deployment environments of LCNs often differ significantly. Furthermore, to accelerate training, data is loaded into GPU memory in batches, resulting in small data block sizes. Consequently, in the deployment environment, GPU memory cannot be fully utilized, meaning the computing power of the graphics card cannot be fully leveraged. The image processing method provided by one or more embodiments of the present invention can obtain the maximum amount of data that GPU memory can load, fully utilizing the computing power of the graphics card and improving the inference speed of the LCN.

[0128] Existing fully convolutional networks do not adjust the sub-block division of the input image based on the size of each dimension of the image itself. That is, with the same amount of GPU memory, the number of input sub-blocks cannot be minimized (sub-blocks can overlap). The image processing method provided by one or more embodiments of the present invention can minimize the number of sub-blocks under the same GPU memory usage, reducing the number of inference iterations in the fully convolutional neural network and accelerating the overall inference speed of the fully convolutional neural network.

[0129] The present invention also provides an electronic device, comprising:

[0130] Processor; and

[0131] The memory stores a computer program that, when executed by the processor, causes the processor to perform the image processing method 10 as described in one or more embodiments above.

[0132] The present invention also provides a non-transitory computer-readable storage medium having stored computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the image processing method 10 as described in one or more embodiments above.

[0133] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, and on the specific implementation methods and application scope of this application, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: Acquire the first image data; The first image data is converted into second image data according to a first threshold, wherein the first threshold is determined according to the parameters of the deep learning model; The first sub-image data in the second image data is obtained according to the second threshold, wherein the second threshold is determined based on the first threshold and the storage unit capacity of the GPU; Adjust the number of pixels in each dimension of the first sub-image data until the preset conditions are met, and then generate the second sub-image data; The second sub-image data is loaded into the storage unit of the GPU and inferred through the deep learning model, wherein the deep learning model runs on the GPU; The first threshold includes the number of pixels in each dimension of the minimum image data, which is determined through the following process: Load the deep learning model and parse the parameter file of the deep learning model; Based on the parameter file, determine the number of pooling operations performed by the deep learning model during feedforward propagation; The number of pixels in each dimension of the minimum image data is determined based on the number of pooling operations. The second threshold includes the number of pixels in each dimension of the maximum image data that can be run on the GPU, and the number of pixels in each dimension of the maximum image data is determined through the following process: The first simulated image data is randomly generated based on the number of pixels in each dimension of the minimum image data; The first simulated image data is loaded into the storage unit of the GPU and inferred through the deep learning model; If the deep learning model is able to infer the first simulated image data, then the first simulated image data is expanded into second simulated image data and loaded into the storage unit of the GPU, and inference is performed through the deep learning model. If the deep learning model cannot reason about the second simulated image data, then the first simulated image data is used as the maximum image data to determine the number of pixels in each dimension of the maximum image data; The preset conditions include: The ratio of the number of pixels in each dimension of the first sub-image data is less than a preset value; The step of adjusting the number of pixels in each dimension of the first sub-image data until a preset condition is met to generate the second sub-image data includes: Based on the first threshold, reduce the number of pixels in the first sub-image data along the second dimension; The number of pixels in the first sub-image data is increased along the third dimension so that the total number of pixels in the first sub-image data remains unchanged. The second dimension is the dimension with the smallest number of pixels in the first sub-image data, and the third dimension is the dimension with the largest number of pixels in the first sub-image data.

2. The method as described in claim 1, wherein, The number of pixels in each dimension of the maximum image data is further determined through the following steps: After deleting the first simulated image data from the GPU's storage unit, the second simulated image data is loaded.

3. The method as described in claim 1 or 2, wherein, The step of converting the first image data into second image data according to the first threshold includes: The first image data is interpolated by pixels to obtain the second image data, wherein the second image data can be segmented into one or more minimum image data corresponding to the first threshold.

4. The method as described in claim 1 or 2, wherein, Obtaining the first sub-image data from the second image data according to the second threshold includes: The second image data is segmented along the first dimension to obtain the first sub-image data, wherein the number of pixels in the first sub-image data is the same as the number of pixels in the maximum image data corresponding to the second threshold, and the first dimension is the dimension with the largest number of pixels in the second image data.

5. The method of claim 1 or 2, further comprising: Multiple second sub-image data are sequentially loaded into the storage unit of the GPU and inference is performed through the deep learning model; Merge multiple inference results.

6. The method of claim 5, further comprising: A zero matrix with the same number of pixels as the second image data is generated to store the merged inference results.

7. The method as described in claim 1 or 2, wherein, The deep learning model is implemented using a fully convolutional neural network, and the first image data includes images representing three-dimensional regions of the patient.

8. An electronic device, characterized in that, include: processor; as well as A memory storing a computer program that, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

9. A non-transitory computer-readable storage medium having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1-7.

Images