Tumor image processing method and device based on deep learning, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and intelligent medical treatment, and provides a tumor image processing method, device and equipment based on deep learning and a medium, and the method comprises the steps: carrying out the spatial transformation of a to-be-recognized first tumor image based on a preset first spatial transformation network, and obtaining a second tumor image after the spatial transformation; outputting the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including a tumor boundary; performing spatial inverse transformation on the third tumor image based on a preset second spatial transformation network to obtain a target tumor image after spatial inverse transformation; the target tumor image comprises a tumor boundary and corresponds to the first tumor image. According to the invention, the tumor boundary in the tumor image can be accurately segmented, and the recognition accuracy of the focus area in the tumor image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology in artificial intelligence, and in particular to a tumor image processing method, device, equipment and medium based on deep learning. Background Art

[0002] In the medical field, especially in neuro-oncology, the evaluation of treatment response is crucial for monitoring disease progression and treatment efficacy. The Response Assessment in Neuro-Oncology (RANO) criteria is one of the commonly used criteria for evaluating brain tumor treatment response in clinical trials, which evaluates treatment efficacy by measuring changes in one or two dimensions of the tumor.

[0003] However, due to the influence of surgery and other treatments, tumor lesions may show irregularity and variability, which will lead to low recognition accuracy of tumor images and inability to accurately identify the lesion area in the tumor image. Or, when the image quality of the tumor image is poor, the lesion area in the tumor image cannot be accurately identified. Therefore, how to improve the recognition accuracy of the lesion area in the tumor image has become an urgent problem to be solved. Summary of the invention

[0004] The main purpose of the present application is to provide a tumor image processing method, device, equipment and medium based on deep learning, which can accurately segment the tumor boundaries in the tumor image, thereby improving the recognition accuracy of the lesion area in the tumor image.

[0005] In a first aspect, the present application provides a tumor image processing method based on deep learning, comprising:

[0006] Acquiring a first tumor image to be identified;

[0007] Performing spatial transformation on the first tumor image based on a preset first spatial transformation network to obtain a second tumor image after spatial transformation; wherein the first spatial transformation network is generated based on transformation parameters obtained by iterative training of multiple tumor images;

[0008] Outputting the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including a tumor boundary;

[0009] Based on a preset second spatial transformation network, the third tumor image is spatially inversely transformed to obtain a target tumor image after spatial inverse transformation; wherein the second spatial transformation network is generated according to the transformation parameters of the first spatial transformation network; and the target tumor image includes the tumor boundary and corresponds to the first tumor image.

[0010] In a second aspect, the present application further provides an image processing device, the image processing device comprising:

[0011] An image acquisition module, used for acquiring a first tumor image to be identified;

[0012] An image transformation module, configured to perform spatial transformation on the first tumor image based on a preset first spatial transformation network to obtain a second tumor image after spatial transformation; wherein the first spatial transformation network is generated by transformation parameters obtained by iterative training of multiple tumor images;

[0013] An image segmentation module, used for outputting the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including a tumor boundary;

[0014] An image inverse transformation module is used to perform spatial inverse transformation on the third tumor image based on a preset second spatial transformation network to obtain a target tumor image after spatial inverse transformation; wherein the second spatial transformation network is generated according to the transformation parameters of the first spatial transformation network; and the target tumor image includes the tumor boundary and corresponds to the first tumor image.

[0015] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the deep learning-based tumor image processing method as described above are implemented.

[0016] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the deep learning-based tumor image processing method as described above are implemented.

[0017] The present application provides a tumor image processing method, device, equipment and medium based on deep learning. The present application performs spatial transformation on the first tumor image to be identified through a preset first spatial transformation network to obtain a second tumor image after spatial transformation; the second tumor image is output to a preset tumor image segmentation model for processing to obtain a third tumor image including a tumor boundary; the third tumor image is spatially inversely transformed based on a preset second spatial transformation network to obtain a target tumor image after spatial inverse transformation; the target tumor image includes a tumor boundary and corresponds to the first tumor image. The embodiment of the present application first uses the first spatial transformation network to perform spatial transformation on the first tumor image to be identified, so that the lesion area in the transformed second tumor image is located at a position more conducive to segmentation, and then uses the tumor image segmentation model to segment the tumor boundary in the second tumor image, thereby determining the lesion area through the tumor boundary in the segmented third tumor image, and then uses the second spatial transformation network to perform spatial inverse transformation on the third tumor image, and the target tumor image corresponding to the first tumor image finally obtained can intuitively display the lesion area through a clear tumor boundary, greatly improving the recognition accuracy of the lesion area in the tumor image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of the steps of a tumor image processing method based on deep learning provided in an embodiment of the present application;

[0020] Figure 2 for Figure 1 Schematic diagram of the sub-steps of the deep learning-based tumor image processing method;

[0021] Figure 3 for Figure 1 A schematic flow chart of another sub-step of the deep learning-based tumor image processing method;

[0022] Figure 4 for Figure 1 A schematic diagram of another sub-step flow chart of the deep learning-based tumor image processing method;

[0023] Figure 5 A schematic block diagram of an image processing device provided in an embodiment of the present application;

[0024] Figure 6 for Figure 5A schematic block diagram of submodules of an image processing device in FIG.

[0025] Figure 7 for Figure 5 A schematic block diagram of submodules of an image processing device in FIG.

[0026] Figure 8 for Figure 5 A schematic block diagram of submodules of an image processing device in FIG.

[0027] Fig. 9 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0028] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0031] The embodiments of the present application provide a method, device, equipment and medium for tumor image processing based on deep learning. Among them, the tumor image processing method based on deep learning can be applied to a terminal device or a server, and the terminal device can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant and a wearable device; the server can be a single server or a server cluster composed of multiple servers. The following is an explanation of the application of the tumor image processing method based on deep learning to a server as an example.

[0032] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0033] Please refer to Figure 1 , Figure 1 A schematic flowchart of the steps of a tumor image processing method based on deep learning provided in an embodiment of the present application.

[0034] like Figure 1As shown, the tumor image processing method based on deep learning includes steps S101 to S104.

[0035] Step S101: Acquire a first tumor image to be identified.

[0036] Among them, the first tumor image to be identified may include multiple parts, such as the patient's skull or sternum and its surrounding soft tissues, etc. These parts may not contain tumor lesions, or may contain one or more tumor lesions. The first tumor image can be an archived tumor image extracted from an existing medical imaging database or image library, or it can be a tumor image directly captured using imaging equipment such as CT, MRI, X-ray, etc. The first tumor image can be one or more, and multiple first tumor images can be medical images taken of the same body part of the same patient. Multiple first tumor images can be multi-modal images, such as T2Flair, T1 enhancement, T2 and other modalities, or they can be single-modal images.

[0037] In one embodiment, if Figure 2 As shown, step S101 includes: sub-steps S1011 to S1015.

[0038] Sub-step S1011: acquiring a sample tumor image set.

[0039] The sample tumor image set may be any tumor image set used as a sample, and the sample tumor image set may include multiple sample tumor images of the same tumor, and the multiple sample tumor images may be single-modality tumor images or multi-modality mixed tumor images.

[0040] For example, the sample tumor image set is a set of multiple postoperative images of a patient with a brain tumor. The multiple postoperative images include magnetic resonance imaging images of T2Flair, T1 enhancement, T2 and other modalities, and each postoperative image contains a tumor lesion area.

[0041] Sub-step S1012: performing offset field correction on a plurality of sample tumor images in the sample tumor image set to adjust the plurality of sample tumor images to the same brightness, thereby obtaining a plurality of first sample tumor images.

[0042] It should be noted that some types of first sample tumor images, such as first sample tumor images obtained by nuclear magnetic resonance, are affected by non-uniform magnetic fields during the imaging process, resulting in different signal intensities for the same tumor lesion in different first sample tumor images. Therefore, offset field correction is used to eliminate the local brightness offset in the sample tumor image, so that the brightness of multiple sample tumor images is consistent, so as to highlight the tumor boundary in the sample tumor image, which is helpful for subsequent accurate identification.

[0043] In one embodiment, local regions are eliminated on a plurality of first sample tumor images. For example, a skull region of a first sample tumor image that is a brain tumor image is removed to reduce unnecessary regions in the first sample tumor image.

[0044] Sub-step S1013: resampling the multiple first sample tumor images to adjust the voxels of the multiple first sample tumor images to the same size, thereby obtaining multiple second sample tumor images.

[0045] It should be noted that by resampling the first sample tumor images to voxels of the same size, the spatial resolutions of the multiple first sample tumor images are made consistent, which facilitates the fusion and registration of subsequent images.

[0046] Sub-step S1014: spatially register the multiple second sample tumor images to adjust the coordinate dimensions of the multiple second sample tumor images to the same dimension.

[0047] It should be noted that the specific method of spatial registration can be selected, combined and adjusted according to the anatomical characteristics and image modality of the patient's body area corresponding to the second sample tumor image. For example, when the second sample tumor image is a tumor image of a part with more bones or hard tissues such as the spine and thorax, rigid registration is used, and the rigid registration rotates and translates the second sample tumor image while keeping its shape and size unchanged; when the second sample tumor image is a tumor image of a part with large soft tissue changes such as the brain, linear registration can be used, and the linear registration rotates, translates, scales and shears the second sample tumor image to retain the fine structure in the second sample tumor image to the greatest extent, ensuring that the multiple second sample tumor images can retain the fine structure such as irregular tumor boundaries in the image to a large extent after the coordinate dimension is adjusted to the same dimension.

[0048] Sub-step S1015: cropping or expanding the plurality of second sample tumor images in the same dimension to adjust the plurality of second sample tumor images to the same size, thereby obtaining a plurality of first tumor images to be identified.

[0049] It should be noted that the multiple second sample tumor images may come from different imaging devices and may have different image sizes, so it is necessary to adjust these second sample tumor images to the same image size by means of cropping or expansion, etc., to reduce the calculation error and affect the calculation efficiency caused by different image sizes when the images are subsequently input into the preset model. For example, multiple second sample tumor images of T2Flair, T1 enhancement, T2 and other modalities in the same dimension are cropped or expanded to obtain multiple first tumor images with the same height, depth and width.

[0050] Step S102: performing spatial transformation on the first tumor image based on a preset first spatial transformation network to obtain a second tumor image after spatial transformation.

[0051] Among them, the preset first transformation network can be used to perform spatial transformation on the image, and the transformation operation includes but is not limited to one or more of translation, rotation, scaling and shearing of the image, so that the tumor lesion area in the image is transferred to a position that is more conducive to identification through spatial transformation.

[0052] In some embodiments, the first transformation network includes but is not limited to a positioning network, a grid generator, and a sampler. The positioning network may be a convolutional neural network, which may be composed of one or more neural network layers.

[0053] In one embodiment, the training method of the first spatial transformation network includes: steps S21 to S27.

[0054] Step S21, obtaining multiple training sample data pairs.

[0055] The training sample data pair includes a training sample tumor image and a spatially transformed tumor image. The training sample tumor image may be any tumor image used as a sample, and the spatially transformed tumor image may be a tumor image obtained after the training sample tumor image undergoes one or more spatial transformation operations.

[0056] For example, the training sample data pair includes a training sample tumor image and a spatially transformed tumor image obtained by translating, rotating and scaling the training sample tumor image.

[0057] Step S22: extracting features from the training sample tumor images in the plurality of training sample data pairs to obtain a plurality of first feature vectors.

[0058] The training sample tumor images in the plurality of training sample data pairs may be input into the first space transformation network in the form of a three-dimensional or four-dimensional tensor, and feature extraction may be performed through the neural network layer in the first space transformation network.

[0059] For example, the training sample tumor images in multiple training sample data pairs are taken as a training sample tumor image set, and the training sample tumor image set is recorded as x(bs, 3, H, W, D), where bs is the number of training sample tumor images in the training sample tumor image set, 3 is the number of channels of the training sample tumor image (such as RGB), and H, W, and D are the height, width, and depth of the training sample tumor image, respectively. x is input into the first spatial transformation network, and features of x are extracted through a convolution layer with a step size of 4 and a convolution kernel of 7×7, and the first feature vector obtained is recorded as x1(bs, 64, H / 4, W / 4, D / 4).

[0060] Step S23: compress the multiple first feature vectors to obtain multiple target feature vectors.

[0061] The compression of the first feature vector can be performed by a convolutional neural layer, the compression can be performed once or multiple times, and can be performed by different convolutional neural layers. It should be noted that compression can further reduce some features in the first feature vector, thereby extracting a target feature vector containing more key feature information.

[0062] In one embodiment, a plurality of first eigenvectors are compressed to obtain a plurality of target eigenvectors, including: compressing the first eigenvectors in a spatial dimension to obtain a second eigenvector, compressing the second eigenvector in a spatial scale to obtain a third eigenvector, and compressing the third eigenvector in a channel dimension to obtain a target eigenvector.

[0063] For example, the first eigenvector x1 (bs, 64, H / 4, W / 4, D / 4) is compressed in the spatial dimension to rearrange the first eigenvector x1 into a one-dimensional vector to obtain the second eigenvector x1 (bs, 64, H*W*D / 4); the second eigenvector x2 is compressed in the spatial scale to obtain the third eigenvector x3 (bs, 64, H*W*D / 8); the third eigenvector x3 is compressed in the channel dimension to obtain the target eigenvector X (bs, 32, H*W*D / 8).

[0064] Step S24: predicting transformation parameters of multiple training sample tumor images according to multiple target feature vectors.

[0065] The target feature vector contains the main feature information after one or more compressions, and the target feature vector can be a compact representation of the original image (i.e., the training sample tumor image). The transformation parameters can be used for image transformation prediction, and their number is related to the dimension of the original image. For example, when the original image is a 3D image, the transformation parameters can be 9, and when the original image is a 2D image, the transformation parameters can be 6.

[0066] For example, the target feature vector X(bs, 32, H*W*D / 8) passes through an output layer, compresses X to 9 output nodes, and the 9 output nodes are output respectively to obtain 9 transformation parameters, including 3 translation parameters (d x ,d y ,d z ), 3 rotation parameters (α x ,α y ,α z ) and three scaling parameters (β x ,β y ,βz ).

[0067] Step S25: Generate a transformation matrix according to the transformation parameters of multiple training sample tumor images.

[0068] Among them, the multiple transformation parameters include various parameter types for predicting different transformations, such as translation parameters, rotation parameters, and scaling parameters. Transformation parameters of different parameter types can constitute different matrices, such as translation matrices, rotation matrices, etc. The final required transformation matrix can be obtained by multiplying these matrices.

[0069] For example, the three translation parameters (d x ,d y ,d z ), 3 rotation parameters (α x ,α y ,α z ) and three scaling parameters (β x ,β y ,β z ) construct matrices respectively and obtain the translation matrix T translate , rotation matrix Rx·Ry·Rz and scaling matrix S:

[0070]

[0071] The above translation matrix T translate , multiply the rotation matrix Rx·Ry·Rz and the scaling matrix S to obtain the transformation matrix T.

[0072] Step S26, determining a transformation grid of the transformation matrix, and using the transformation grid to resample a plurality of training sample tumor images to obtain a plurality of predicted spatially transformed tumor images.

[0073] For example, the training sample tumor image (H, W, D), the coordinate points on the image are defined as (x, y, z). First, a transformation grid P containing all coordinates is generated through the transformation matrix T. Each coordinate point (x, y, z) can be expressed as a homogeneous coordinate (x, y, z, 1) so that calculation can be performed in the 4x4 transformation matrix T. The transformation grid P is multiplied by the transformation matrix T to obtain the new coordinates of each coordinate point after spatial transformation, so that the predicted spatial transformation tumor image is obtained through all the new coordinates. The predicted spatial transformation tumor image is the tumor image after the training sample tumor image is spatially transformed.

[0074] Step S27: updating the transformation parameters in the transformation matrix according to the errors between the multiple predicted spatially transformed tumor images and the corresponding multiple spatially transformed tumor images until the transformation matrix converges to obtain a trained first spatial transformation network.

[0075] The spatially transformed tumor image can be a gold standard image drawn manually corresponding to the training sample tumor image, and the error between the predicted spatially transformed tumor image and the spatially transformed tumor image can be measured using the mean square error as a loss function. By adjusting the transformation parameters in the transformation matrix, the predicted transformed image is matched with the gold standard image as much as possible until the value of the loss function is within a preset error range, thereby obtaining a first spatial transformation network that has been trained.

[0076] In one embodiment, if Figure 3 As shown, step S102 includes: sub-step S1021 to sub-step S1022.

[0077] Sub-step S1021: input the first tumor image into a preset first spatial transformation network for processing to obtain a first transformation matrix.

[0078] The first tumor image may be a plurality of tumor images having the same image parameters after image preprocessing.

[0079] For example, a plurality of multimodal first tumor images with the same height, width and depth are input into a preset first spatial transformation network in the form of an image set X1 to obtain a first transformation matrix Τ1.

[0080] Sub-step S1022: resample the first tumor image using the transformation grid of the first transformation matrix to obtain a second tumor image after spatial transformation.

[0081] For example, according to the first tumor image X1 and the first transformation matrix T1, a transformation grid P1 is generated, and all first tumor images in X1 are resampled using the transformation grid P1 to obtain a plurality of second tumor images after spatial transformation.

[0082] Step S103: output the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including the tumor boundary.

[0083] The second tumor image may be one or more, and each second tumor image may not contain a lesion area, or may contain one or more lesion areas. It should be noted that after the tumor boundary in the second tumor image is identified by the preset tumor image segmentation model, the lesion area in the image can be confirmed according to the tumor boundary.

[0084] In some embodiments, a training method for a tumor image segmentation model includes: obtaining multiple training sample data groups, the training sample data groups including training sample spatially transformed tumor images and tumor segmentation images; dividing the training sample spatially transformed images in the multiple training sample data groups into blocks to obtain multiple training sample spatially transformed image blocks; performing feature extraction on the multiple training sample spatially transformed image blocks to obtain first sample feature data of the multiple training sample spatially transformed image blocks; performing feature information capture on the multiple first sample feature data to obtain dependency information between the multiple training sample spatially transformed image blocks; integrating and / or expanding the training sample spatially transformed image blocks based on the dependency information between the multiple training sample spatially transformed image blocks to obtain a predicted tumor segmentation image including a tumor boundary; and updating parameters of the block processing and / or feature extraction processing based on the errors between the multiple predicted tumor segmentation images and the corresponding multiple tumor segmentation images until the errors are within a preset error range to obtain a trained tumor image segmentation model.

[0085] The tumor segmentation image can be a tumor image obtained by manually identifying the tumor boundary of the training sample spatial transformation tumor image, the error between the predicted tumor segmentation image and the tumor segmentation image can be calculated by the Dice loss function and the weighted cross entropy, and the model parameters can be updated using the Adam optimizer. It should be noted that by training the tumor image segmentation model, the recognition accuracy of the model for the tumor boundary can be effectively improved, thereby obtaining a tumor image with higher accuracy.

[0086] In one embodiment, the tumor image segmentation model includes an image segmentation subnetwork, a feature extraction subnetwork, an information capture subnetwork, and an image restoration subnetwork. The image segmentation subnetwork is used to segment the image; the feature extraction subnetwork is used to generate features corresponding to the image blocks; the information capture subnetwork is used to obtain local information and global information of the image; and the image restoration subnetwork is used to restore the image to the same parameters as the input image based on the captured local information and global information.

[0087] In one embodiment, if Figure 4 As shown, step S103 includes: sub-steps S1031 to S1034.

[0088] Sub-step S1031: input the second tumor image into the image segmentation sub-network for segmentation processing to obtain a plurality of second tumor image blocks.

[0089] Wherein, the block processing may include segmenting the second tumor image according to the size of the required image block, and the second tumor image block may be represented by a vector. It should be noted that the second tumor image is divided into smaller image blocks so that the tumor image segmentation model can extract features from smaller image regions, which is convenient for subsequent more accurate identification of lesion regions with complex boundary conditions. For example, the size of the second tumor image is (H*W, the size of the required image block is h*w, and the second tumor image is gradually divided by sliding a window of size h*w, thereby obtaining a plurality of second tumor image blocks of size h*w.

[0090] In some embodiments, after obtaining the plurality of second tumor image blocks, the method further includes: flattening the second tumor image blocks into one-dimensional vectors. For example, flattening the second tumor image blocks with a size of 8*8 into one-dimensional vectors with a length of 64.

[0091] Sub-step S1032: input the plurality of second tumor image blocks into a feature extraction network for processing to obtain first feature data of the plurality of second tumor image blocks.

[0092] The feature extraction network may be a convolutional neural network, which includes one or more convolutional neural layers, and the first feature data may be represented by a vector. For example, each second tumor image block is passed through a linear layer to obtain its corresponding embedding vector.

[0093] Sub-step S1033: inputting the multiple first features into the information capture sub-network for processing to obtain dependency information between the multiple second tumor image blocks.

[0094] The information capture subnetwork may be a Swin-transformer structure, and the number of Swin-transformers may be one or more, which is used to adjust the extraction depth of the first feature. It should be noted that the dependency information between all the second tumor image blocks is captured by multiple first features, thereby obtaining the global dependency information of the entire second tumor image.

[0095] For example, the information capture subnetwork includes two Swin-transformers, the first feature is an embedding vector, and multiple embedding vectors are combined to form a feature sequence and input into the information capture subnetwork, which is then passed through two Swin-transformers in turn for feature extraction. The Swin-Transformer extracts each first feature through a sliding window self-attention mechanism, thereby capturing the local dependencies between the second tumor image blocks and even the global dependencies of the entire second tumor image.

[0096] Sub-step S1034: inputting the dependency information into the image restoration sub-network, integrating and / or expanding the plurality of second tumor image blocks according to the dependency information, and obtaining a third tumor image including the tumor boundary.

[0097] Among them, the dependency information can include a variety of information, including but not limited to: whether there is a tumor boundary, the similarity between blocks, the spatial distribution of tumor features, the contrast of different areas of the image, etc. There can be one image recovery subnetwork, and one image recovery subnetwork is set after the information capture subnetwork. There can also be multiple image recovery subnetworks, and multiple image recovery subnetworks are interlaced between multiple information capture subnetworks. It should be noted that the image recovery subnetwork can obtain the relationship between various parts of the image (including tumor boundaries and internal structures, etc.) through the dependency information, and then better integrate them into a complete image.

[0098] For example, the image restoration sub-network includes multiple block integration modules and multiple block expansion modules, and the multiple block integration modules and block expansion modules are interlaced after multiple Swin-Transformers.

[0099] In one embodiment, there is a parameter jump connection between the information capture subnetwork and the image restoration subnetwork to prevent information loss or gradient disappearance problems in the tumor image segmentation model.

[0100] Step S104: performing spatial inverse transformation on the third tumor image based on a preset second spatial transformation network to obtain a target tumor image after spatial inverse transformation; the target tumor image includes a tumor boundary and corresponds to the first tumor image.

[0101] Among them, the second spatial transformation network can be generated according to the transformation parameters of the first spatial transformation network. In this embodiment, the second spatial transformation network can adopt a structure similar to or the same as the first network transformation network, and the second spatial transformation network can obtain an inverse transformation matrix corresponding to the transformation matrix in the first spatial transformation network. The generation process of the inverse transformation matrix and its application process in the spatial inverse transformation can be specifically referred to the training process of the first spatial transformation network in the above embodiment, and will not be repeated here.

[0102] In one embodiment, after performing spatial inverse transformation on the third tumor image based on the preset second spatial transformation network to obtain the target tumor image after spatial inverse transformation, the method further includes: inputting the target tumor image into a preset tumor image analysis model for processing to obtain the pixel volume of the target tumor image and the number of pixels within the tumor boundary; and obtaining the tumor volume data corresponding to the first tumor image according to the product of the pixel volume and the number of pixels. It should be noted that performing pixel-level volume calculation on the tumor can obtain tumor volume data with higher accuracy.

[0103] In one embodiment, after performing spatial inverse transformation on the third tumor image based on the preset second spatial transformation network to obtain the target tumor image after spatial inverse transformation, the method further includes: inputting the target tumor image into a preset tumor image analysis model for processing to obtain the pixel area of ​​the target tumor image and the number of pixels within the tumor boundary; obtaining tumor area data corresponding to the first tumor image according to the product of the pixel area and the number of pixels; determining the first target tumor image with the largest tumor area from multiple target tumor images according to the tumor area, and obtaining the first straight line and the second straight line within the tumor boundary in the first target tumor image; wherein the first straight line is the longest straight line in the tumor boundary, and the second straight line is the longest straight line in the vertical direction of the first straight line in the tumor boundary; obtaining the first tumor area data corresponding to the first target tumor image according to the product of the first diameter length and the second diameter length. By measuring the product of the first diameter length and the second diameter length, the corresponding tumor volume can be roughly estimated according to the first tumor area data, so that the tumor volume data has a high consistency with the conventional analysis results in clinical practice.

[0104] The tumor image processing method based on deep learning provided in the above embodiment performs spatial transformation on the first tumor image to be identified through a preset first spatial transformation network to obtain a second tumor image after spatial transformation; outputs the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including a tumor boundary; performs spatial inverse transformation on the third tumor image based on a preset second spatial transformation network to obtain a target tumor image after spatial inverse transformation; the target tumor image includes a tumor boundary and corresponds to the first tumor image. The embodiment of the present application first uses the first spatial transformation network to perform spatial transformation on the first tumor image to be identified, so that the lesion area in the transformed second tumor image is located at a position more conducive to segmentation, and then uses the tumor image segmentation model to segment the tumor boundary in the second tumor image, thereby determining the lesion area through the tumor boundary in the segmented third tumor image, and then uses the second spatial transformation network to perform spatial inverse transformation on the third tumor image, and finally obtains a target tumor image corresponding to the first tumor image that can intuitively display the lesion area through a clear tumor boundary, greatly improving the recognition accuracy of the lesion area in the tumor image.

[0105] Please refer to Figure 5 , Figure 5 A schematic block diagram of an image processing device provided in an embodiment of the present application.

[0106] like Figure 5 As shown, the image processing device 200 includes:

[0107] The image acquisition module 201 is used to acquire a first tumor image to be identified.

[0108] The image transformation module 202 is used to perform spatial transformation on the first tumor image based on a preset first spatial transformation network to obtain a second tumor image after spatial transformation; wherein the first spatial transformation network is generated based on transformation parameters obtained by iterative training of multiple tumor images.

[0109] The image segmentation module 203 is used to output the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including the tumor boundary.

[0110] The image inverse transformation module 204 is used to perform spatial inverse transformation on the third tumor image based on a preset second spatial transformation network to obtain a target tumor image after spatial inverse transformation; wherein the second spatial transformation network is generated according to the transformation parameters of the first spatial transformation network; the target tumor image includes a tumor boundary and corresponds to the first tumor image.

[0111] In one embodiment, Figure 6 As shown, the image acquisition module 201 includes:

[0112] The sample acquisition submodule 2011 is used to acquire a sample tumor image set, where the sample tumor image set includes multiple sample tumor images of the same tumor, and the multiple sample tumor images include at least one image modality;

[0113] An offset field correction submodule 2012 is used to perform offset field correction on a plurality of sample tumor images in the sample tumor image set, so as to adjust the plurality of sample tumor images to the same brightness, thereby obtaining a plurality of first sample tumor images;

[0114] A resampling submodule 2013 is used to resample the multiple first sample tumor images to adjust the voxels of the multiple first sample tumor images to the same size, thereby obtaining multiple second sample tumor images;

[0115] A spatial registration submodule 2014 is used to perform spatial registration on the plurality of second sample tumor images to adjust the coordinate dimensions of the plurality of second sample tumor images to the same dimension;

[0116] The image adjustment submodule 2015 is used to crop or expand the multiple second sample tumor images in the same dimension to adjust the multiple second sample tumor images to the same size to obtain multiple first tumor images to be identified.

[0117] In one embodiment, the image acquisition module 201 is further used for:

[0118] Acquire a plurality of training sample data pairs, the training sample data pairs comprising a training sample tumor image and a spatially transformed tumor image;

[0119] Extracting features of the training sample tumor images in the plurality of training sample data pairs to obtain a plurality of first feature vectors;

[0120] Performing compression processing on the multiple first feature vectors to obtain multiple target feature vectors;

[0121] Predicting transformation parameters of multiple training sample tumor images according to multiple target feature vectors;

[0122] generating a transformation matrix according to transformation parameters of multiple training sample tumor images;

[0123] Determining a transformation grid of the transformation matrix, and using the transformation grid to resample multiple training sample tumor images to obtain multiple predicted spatially transformed tumor images;

[0124] According to the errors between the multiple predicted spatially transformed tumor images and the corresponding multiple spatially transformed tumor images, the transformation parameters in the transformation matrix are updated until the transformation matrix converges to obtain a trained first spatial transformation network.

[0125] In one embodiment, the image acquisition module 201 is further used for:

[0126] Compressing the first eigenvector in the spatial dimension to obtain a second eigenvector;

[0127] Compress the second eigenvector in the spatial scale to obtain the third eigenvector;

[0128] The third eigenvector is compressed in the channel dimension to obtain the target eigenvector.

[0129] In one embodiment, Figure 7 As shown, the image transformation module 202 includes:

[0130] The transformation matrix submodule 2021 is used to input the first tumor image into a preset first spatial transformation network for processing to obtain a first transformation matrix.

[0131] The transformation grid submodule 2022 is used to resample the first tumor image using the transformation grid of the first transformation matrix to obtain a second tumor image after spatial transformation.

[0132] In one embodiment, the tumor image segmentation model includes an image segmentation subnetwork, a feature extraction subnetwork, an information capture subnetwork, and an image restoration subnetwork. Figure 8 As shown, the image segmentation module 203 includes:

[0133] The image segmentation submodule 2031 is used to input the second tumor image into the image segmentation subnetwork for segmentation processing to obtain a plurality of second tumor image blocks.

[0134] The feature extraction submodule 2032 is used to input the multiple second tumor image blocks into the feature extraction network for processing to obtain the first feature data of the multiple second tumor image blocks.

[0135] The relationship capturing submodule 2033 is used to input the multiple first features into the information capturing subnetwork for processing to obtain dependency relationship information between the multiple second tumor image blocks.

[0136] The relationship integration submodule 2034 is used to input the dependency information into the image restoration subnetwork, and integrate and / or expand the multiple second tumor image blocks according to the dependency information to obtain a third tumor image including the tumor boundary.

[0137] In one embodiment, the image processing apparatus 200 further includes:

[0138] The image data acquisition module 205 is used to input the target tumor image into a preset tumor image analysis model for processing to obtain the pixel volume of the target tumor image and the number of pixels within the tumor boundary;

[0139] The image data calculation module 206 is used to obtain tumor volume data corresponding to the first tumor image according to the product of the pixel volume and the number of pixels.

[0140] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described device and each module and unit can refer to the corresponding process in the aforementioned deep learning-based tumor image processing method embodiment, and will not be repeated here.

[0141] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Fig. 9 Runs on the computer device shown.

[0142] See also Fig. 9 , Fig. 9 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0143] like Fig. 9 As shown, the computer device includes a processor, a memory and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory, and the storage medium may be non-volatile or volatile.

[0144] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any tumor image processing method based on deep learning.

[0145] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0146] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any tumor image processing method based on deep learning.

[0147] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0148] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0149] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0150] Acquiring a first tumor image to be identified;

[0151] Performing spatial transformation on the first tumor image based on a preset first spatial transformation network to obtain a second tumor image after spatial transformation; wherein the first spatial transformation network is generated by transformation parameters obtained by iterative training based on multiple tumor images;

[0152] Outputting the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including the tumor boundary;

[0153] Based on a preset second spatial transformation network, a spatial inverse transformation is performed on the third tumor image to obtain a target tumor image after spatial inverse transformation; wherein the second spatial transformation network is generated according to the transformation parameters of the first spatial transformation network; the target tumor image includes a tumor boundary and corresponds to the first tumor image.

[0154] In one embodiment, when acquiring the first tumor image to be identified, the processor is used to implement:

[0155] Acquire a sample tumor image set, the sample tumor image set including multiple sample tumor images of the same tumor, the multiple sample tumor images including at least one image modality;

[0156] Performing offset field correction on multiple sample tumor images in the sample tumor image set to adjust the multiple sample tumor images to the same brightness, thereby obtaining multiple first sample tumor images;

[0157] resampling the plurality of first sample tumor images to adjust the voxels of the plurality of first sample tumor images to the same size, thereby obtaining a plurality of second sample tumor images;

[0158] Performing spatial registration on the plurality of second sample tumor images to adjust the coordinate dimensions of the plurality of second sample tumor images to the same dimension;

[0159] The multiple second sample tumor images in the same dimension are cropped or expanded to adjust the multiple second sample tumor images to the same size, thereby obtaining multiple first tumor images to be identified.

[0160] In one embodiment, when implementing the training of the first spatial transformation network, the processor is used to implement:

[0161] Acquire a plurality of training sample data pairs, the training sample data pairs comprising a training sample tumor image and a spatially transformed tumor image;

[0162] Extracting features of the training sample tumor images in the plurality of training sample data pairs to obtain a plurality of first feature vectors;

[0163] Performing compression processing on the multiple first feature vectors to obtain multiple target feature vectors;

[0164] Predicting transformation parameters of multiple training sample tumor images according to multiple target feature vectors;

[0165] generating a transformation matrix according to transformation parameters of multiple training sample tumor images;

[0166] Determining a transformation grid of the transformation matrix, and using the transformation grid to resample multiple training sample tumor images to obtain multiple predicted spatially transformed tumor images;

[0167] According to the errors between the multiple predicted spatially transformed tumor images and the corresponding multiple spatially transformed tumor images, the transformation parameters in the transformation matrix are updated until the transformation matrix converges to obtain a trained first spatial transformation network.

[0168] In one embodiment, when the processor implements the compression processing on the first feature vector to obtain the target feature vector, it is used to implement:

[0169] Compressing the first eigenvector in the spatial dimension to obtain a second eigenvector;

[0170] Compress the second eigenvector in the spatial scale to obtain the third eigenvector;

[0171] The third eigenvector is compressed in the channel dimension to obtain the target eigenvector.

[0172] In one embodiment, when the processor performs spatial transformation on the first tumor image based on the preset first spatial transformation network to obtain the second tumor image after spatial transformation, it is used to implement:

[0173] Inputting the first tumor image into a preset first spatial transformation network for processing to obtain a first transformation matrix;

[0174] The first tumor image is resampled using the transformation grid of the first transformation matrix to obtain a second tumor image after spatial transformation.

[0175] In one embodiment, the tumor image segmentation model includes an image block subnetwork, a feature extraction subnetwork, an information capture subnetwork, and an image restoration subnetwork. When the processor outputs the second tumor image to the preset tumor image segmentation model for processing to obtain a third tumor image including a tumor boundary, the processor is used to implement:

[0176] Inputting the second tumor image into the image block subnetwork for block processing to obtain a plurality of second tumor image blocks;

[0177] Inputting the plurality of second tumor image blocks into a feature extraction network for processing to obtain first feature data of the plurality of second tumor image blocks;

[0178] Inputting the plurality of first features into the information capture subnetwork for processing to obtain dependency relationship information between the plurality of second tumor image blocks;

[0179] The dependency information is input into the image restoration sub-network, and the plurality of second tumor image blocks are integrated and / or expanded according to the dependency information to obtain a third tumor image including a tumor boundary.

[0180] In one embodiment, after the processor performs spatial inverse transformation on the third tumor image based on the preset second spatial transformation network to obtain the target tumor image after spatial inverse transformation, it is further configured to implement:

[0181] Inputting the target tumor image into a preset tumor image analysis model for processing to obtain the pixel volume of the target tumor image and the number of pixels within the tumor boundary;

[0182] Tumor volume data corresponding to the first tumor image is obtained according to the product of the pixel volume and the number of pixels.

[0183] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the computer device described above can refer to the corresponding process in the aforementioned deep learning-based tumor image processing method embodiment, and will not be repeated here.

[0184] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0185] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the tumor image processing method based on deep learning in the present application.

[0186] The computer-readable storage medium may be an internal storage unit of the computer device of the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0187] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of blockchain nodes, etc. The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain may include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0188] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0189] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0190] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A tumor image processing method based on deep learning, characterized in that: include: Acquiring a first tumor image to be identified; Performing spatial transformation on the first tumor image based on a preset first spatial transformation network to obtain a second tumor image after spatial transformation; wherein the first spatial transformation network is generated based on transformation parameters obtained by iterative training of multiple tumor images; Outputting the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including a tumor boundary; Based on a preset second spatial transformation network, the third tumor image is spatially inversely transformed to obtain a target tumor image after spatial inverse transformation; wherein the second spatial transformation network is generated according to the transformation parameters of the first spatial transformation network; and the target tumor image includes the tumor boundary and corresponds to the first tumor image.

2. The tumor image processing method based on deep learning according to claim 1, characterized in that: The step of acquiring a first tumor image to be identified includes: Acquire a sample tumor image set, wherein the sample tumor image set includes a plurality of sample tumor images of the same tumor, and the plurality of sample tumor images include at least one image modality; Performing offset field correction on a plurality of the sample tumor images in the sample tumor image set to adjust the plurality of the sample tumor images to the same brightness, thereby obtaining a plurality of first sample tumor images; resampling the plurality of first sample tumor images to adjust the voxels of the plurality of first sample tumor images to the same size, thereby obtaining a plurality of second sample tumor images; Performing spatial registration on the plurality of second sample tumor images to adjust the coordinate dimensions of the plurality of second sample tumor images to the same dimension; The plurality of second sample tumor images in the same dimension are cropped or expanded to adjust the plurality of second sample tumor images to the same size, thereby obtaining a plurality of first tumor images to be identified.

3. The tumor image processing method based on deep learning according to claim 1, characterized in that: The training method of the first space transformation network includes: Acquire a plurality of training sample data pairs, wherein the training sample data pairs include training sample tumor images and spatially transformed tumor images; Performing feature extraction on the training sample tumor images in the plurality of training sample data pairs to obtain a plurality of first feature vectors; Performing compression processing on the plurality of first feature vectors to obtain a plurality of target feature vectors; Predicting transformation parameters of a plurality of the training sample tumor images according to the plurality of the target feature vectors; generating a transformation matrix according to the transformation parameters of the plurality of training sample tumor images; Determining a transformation grid of the transformation matrix, and using the transformation grid to resample a plurality of the training sample tumor images to obtain a plurality of predicted spatially transformed tumor images; According to the errors between the multiple predicted spatially transformed tumor images and the corresponding multiple spatially transformed tumor images, the transformation parameters in the transformation matrix are updated until the transformation matrix converges to obtain a trained first spatial transformation network.

4. The tumor image processing method based on deep learning according to claim 3, characterized in that: The step of compressing the first feature vector to obtain a target feature vector comprises: Compressing the first eigenvector in a spatial dimension to obtain a second eigenvector; Compressing the second eigenvector in terms of spatial scale to obtain a third eigenvector; The third eigenvector is compressed in the channel dimension to obtain a target eigenvector.

5. The tumor image processing method based on deep learning according to any one of claims 1 to 4, characterized in that: The performing spatial transformation on the first tumor image based on a preset first spatial transformation network to obtain a second tumor image after spatial transformation includes: Inputting the first tumor image into a preset first spatial transformation network for processing to obtain a first transformation matrix; The first tumor image is resampled using the transformation grid of the first transformation matrix to obtain a second tumor image after spatial transformation.

6. The tumor image processing method based on deep learning according to claim 1, characterized in that: The tumor image segmentation model includes an image segmentation subnetwork, a feature extraction subnetwork, an information capture subnetwork and an image restoration subnetwork; Outputting the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including a tumor boundary includes: Inputting the second tumor image into the image block subnetwork for block processing to obtain a plurality of second tumor image blocks; Inputting the plurality of second tumor image blocks into the feature extraction network for processing to obtain the first feature data of the plurality of second tumor image blocks; Inputting the plurality of the first features into the information capture subnetwork for processing to obtain dependency relationship information between the plurality of the second tumor image blocks; The dependency information is input into the image restoration sub-network, and a plurality of the second tumor image blocks are integrated and / or expanded according to the dependency information to obtain a third tumor image including a tumor boundary.

7. The tumor image processing method based on deep learning according to claim 1, characterized in that: After performing spatial inverse transformation on the third tumor image based on the preset second spatial transformation network to obtain a target tumor image after spatial inverse transformation, the method further includes: Inputting the target tumor image into a preset tumor image analysis model for processing to obtain the pixel volume of the target tumor image and the number of pixels within the tumor boundary; Tumor volume data corresponding to the first tumor image is obtained according to the product of the pixel volume and the number of pixels.

8. An image processing device, characterized in that: The image processing device comprises: An image acquisition module, used for acquiring a first tumor image to be identified; An image transformation module, configured to perform spatial transformation on the first tumor image based on a preset first spatial transformation network to obtain a second tumor image after spatial transformation; wherein the first spatial transformation network is generated by transformation parameters obtained by iterative training of multiple tumor images; An image segmentation module, used for outputting the second tumor image to a preset tumor image segmentation model for processing to obtain a third tumor image including a tumor boundary; An image inverse transformation module is used to perform spatial inverse transformation on the third tumor image based on a preset second spatial transformation network to obtain a target tumor image after spatial inverse transformation; wherein the second spatial transformation network is generated according to the transformation parameters of the first spatial transformation network; and the target tumor image includes the tumor boundary and corresponds to the first tumor image.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the deep learning-based tumor image processing method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the deep learning-based tumor image processing method according to any one of claims 1 to 7 is implemented.

Images