Medical image segmentation method, device, computer equipment and readable storage medium

Abstract

The embodiment of the present application discloses a medical image segmentation method, device, computer equipment and readable storage medium, which are related to computer vision technology of artificial intelligence; specifically, a first medical image and a second medical image with a target object area marked can be obtained; the first medical image and the second medical image are respectively subjected to feature extraction to obtain first feature information of the first medical image and second feature information of the second medical image; optical flow motion information from the second medical image to the first medical image is obtained based on the first feature information and the second feature information; the target object in the first medical image is segmented based on the optical flow motion information and the marked target object area to obtain the segmentation result of the first medical image. This solution can greatly improve the accuracy and efficiency of medical image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a medical image segmentation method, device, computer equipment and readable storage medium. Background Art

[0002] With the development of artificial intelligence (AI), AI is increasingly being used in the medical field, especially in the segmentation of medical images. For example, taking liver segmentation as an example, a convolutional neural network can be trained using sample images of labeled livers, and then the three-dimensional (3D) liver images to be segmented, such as CT (Computed Tomography) images of the liver, are input into the trained convolutional neural network for segmentation to obtain organ segmentation results, such as liver regions, etc.

[0003] However, the current AI segmentation method is highly dependent on the accuracy of organ labeling and requires a large number of medical image samples with labeled organs to train the network. Therefore, organ segmentation is very time-consuming, inefficient, and has low segmentation accuracy. Summary of the invention

[0004] The embodiments of the present application provide a medical image segmentation method, apparatus, computer equipment and readable storage medium, which can improve the efficiency and accuracy of segmentation.

[0005] The present application embodiment provides a medical image segmentation method, including:

[0006] Acquire a first medical image and a second medical image with a target object area marked;

[0007] Extracting features from the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image;

[0008] Acquire optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information;

[0009] The target object in the first medical image is segmented according to the optical flow motion information and the marked target object region to obtain a segmentation result of the first medical image.

[0010] Accordingly, the embodiment of the present application further provides a medical image segmentation device, comprising:

[0011] An acquisition unit, used for acquiring a first medical image and a second medical image with a target object area marked;

[0012] A feature extraction unit, used to extract features from the first medical image and the second medical image respectively, to obtain first feature information of the first medical image and second feature information of the second medical image;

[0013] an optical flow unit, configured to obtain optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information;

[0014] The segmentation unit is used to segment the target object in the first medical image according to the optical flow motion information and the marked target object area to obtain a segmentation result of the first medical image.

[0015] In one embodiment, the segmentation unit comprises:

[0016] A reconstruction subunit, configured to reconstruct the target object according to the optical flow motion information and the marked target object region to obtain a reconstructed target object region;

[0017] The segmentation subunit is used to use the reconstructed target object area as the segmentation result of the first medical image.

[0018] In one embodiment, the reconstruction subunit is used to perform trilinear interpolation processing on the optical flow motion information to obtain processed optical flow motion information; based on the processed optical flow motion information reconstruction and the marked target object area, perform target object area reconstruction to obtain a reconstructed target object area.

[0019] In one embodiment, the optical flow unit includes:

[0020] an associating subunit, configured to associate the first characteristic information with the second characteristic information to obtain pixel position correspondence information between the second medical image and the first medical image;

[0021] The sampling subunit is used to perform an up-sampling operation on the pixel position correspondence information to obtain the optical flow motion information from the second medical image to the first medical image.

[0022] In one embodiment, the association subunit is specifically used for:

[0023] Performing a convolution operation on the first feature information of the first medical image to obtain first convolution feature information;

[0024] Performing a convolution operation on the second feature information of the second medical image to obtain second convolution feature information;

[0025] Fusing the first convolution feature information and the second convolution feature information to obtain first fused feature information;

[0026] The first fusion feature information and the first convolution feature information are fused with each other to obtain pixel position correspondence information between the second medical image and the first medical image.

[0027] In one embodiment, the optical flow unit is specifically used to: use the first generator in the trained first generative adversarial network to obtain optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information.

[0028] In one embodiment, the medical image segmentation device of the present application further includes:

[0029] A sample acquisition unit, used to acquire a first sample medical image and a second sample medical image with a target object area marked;

[0030] A first sample optical flow unit is used to obtain first sample optical flow motion information from the second sample medical image to the first sample medical image using a first generator in a first generative adversarial network.

[0031] A first reconstruction unit, configured to use the first reconstruction module in the first generative adversarial network to perform image reconstruction based on the first sample optical flow motion information, and obtain a first reconstructed medical image and a first segmentation result corresponding to the first sample medical image;

[0032] A second sample optical flow unit, configured to obtain second sample optical flow motion information from the first reconstructed medical image to the second sample medical image by using a second generator in a second generative adversarial network;

[0033] A second reconstruction unit is used to use a second reconstruction module in a second generative adversarial network to perform image reconstruction based on the second sample optical flow motion information, and obtain a second reconstructed medical image and a second segmentation result corresponding to the second sample medical image;

[0034] A training unit is used to train the first generative adversarial network according to the first reconstructed medical image, the first segmentation result, the second reconstructed medical image, and the second segmentation result to obtain a trained first generative adversarial network.

[0035] In one embodiment, the training unit may include:

[0036] A reconstruction loss acquisition subunit, used to acquire a reconstruction consistency loss in an image reconstruction space according to the first reconstructed medical image and the second reconstructed medical image;

[0037] A segmentation loss acquisition subunit, used to acquire a segmentation consistency loss in the segmentation space according to the first segmentation result, the second segmentation result and the target object labeling result of the second sample medical image;

[0038] The training subunit is used to train the first generative adversarial network according to the reconstruction consistency loss and the segmentation consistency loss to obtain the trained first generative adversarial network.

[0039] In one embodiment, the reconstruction consistency loss includes: image reconstruction consistency loss and trajectory consistency loss; the reconstruction loss acquisition subunit may be specifically used for:

[0040] According to the second reconstructed medical image and the second sample medical image, obtaining an image reconstruction consistency loss that is consistent with the second sample medical image and the second reconstructed medical image;

[0041] According to the first sample optical flow motion information and the second sample optical flow motion information, a trajectory consistency loss is obtained in which the first sample optical flow motion information and the second sample optical flow motion information present a consistent trajectory.

[0042] In one embodiment, the segmentation consistency loss includes image segmentation consistency loss, mask value loss, and semantic consistency loss;

[0043] The segmentation loss acquisition subunit is used to:

[0044] According to the second segmentation result and the target object labeling result, obtaining an image segmentation consistency loss in which the target object labeling result is consistent with the second segmentation result;

[0045] According to the first segmentation result, the second segmentation result and a preset segmentation mask value, respectively obtaining mask value losses of the first segmentation result and the second segmentation result relative to the preset segmentation mask value;

[0046] Based on the first segmentation result, a semantic consistency loss is constructed to maintain semantic consistency of the first segmentation result on the image.

[0047] In one embodiment, the segmentation loss acquisition subunit is used to:

[0048] Derivative the pixel values ​​in the first segmentation result along a preset direction to obtain a derivative result in the preset direction;

[0049] According to the derivation result in the preset direction and the first segmentation result, a semantic consistency loss is constructed to maintain semantic consistency of the first segmentation result on the image.

[0050] In one embodiment, the reconstruction loss acquisition subunit may be specifically used for:

[0051] Generate an original medical image;

[0052] Transforming the reference medical image according to the first pixel position mapping relationship information and the second sample optical flow motion information to obtain a transformed medical image;

[0053] Obtaining a transformation loss between the transformed medical image and the original medical image;

[0054] The transformation loss is used as a trajectory consistency loss in which the first sample optical flow motion information and the second sample optical flow motion information present a consistent trajectory.

[0055] In one embodiment, it further includes a counter-loss acquisition unit;

[0056] The adversarial loss acquisition unit is used to use the discriminator in the first generative adversarial network to discriminate the sample medical image and the first reconstructed medical image to obtain a discrimination result; and obtain the adversarial loss of the first generative adversarial network according to the first reconstructed medical image and the discrimination result.

[0057] The training unit is used to train the first generative adversarial network according to the adversarial loss, the reconstruction consistency loss, and the segmentation consistency loss to obtain the trained first generative adversarial network.

[0058] Correspondingly, the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in any one of the medical image segmentation methods provided in the embodiments of the present application are implemented.

[0059] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the steps in any one of the medical image segmentation methods provided in the embodiments of the present application are implemented.

[0060] The embodiment of the present application can obtain a first medical image and a second medical image with a marked target object area; perform feature extraction on the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image; obtain optical flow motion information from the second medical image to the first medical image based on the first feature information and the second feature information; segment the target object in the first medical image based on the optical flow motion information and the marked target object area to obtain a segmentation result of the first medical image. Therefore, the scheme can transfer objects in the marked image, such as organ segmentation, to the marked image by learning the transformation from the marked image to the unmarked image, and use the marked medical image to segment objects such as organs in the unmarked medical image, thereby greatly improving the accuracy and efficiency of medical image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 is a scene schematic diagram of the medical image segmentation method provided in an embodiment of the present application;

[0063] Figure 2a is a flow chart of a medical image segmentation method provided by an embodiment of the present application;

[0064] Figure 2b It is a schematic diagram of the framework of the generator provided in the embodiment of the present application;

[0065] Figure 2c It is a schematic diagram of the structure of the Non-local Block provided in the embodiment of the present application;

[0066] Figure 2d It is a schematic diagram of separation and separation provided in an embodiment of the present application;

[0067] Figure 3a is an architecture diagram of a segmented network provided in an embodiment of the present application;

[0068] Figure 3b It is a flowchart of the training method provided in the embodiment of the present application;

[0069] Figure 3c is a schematic diagram of consistency of optical flow motion information provided by an embodiment of the present application;

[0070] Figure 4a is another architecture diagram of the segmentation network provided in this embodiment;

[0071] Figure 4b is another schematic diagram of the process of medical image segmentation provided by an embodiment of the present application;

[0072] Figure 5a is a schematic diagram of the structure of a medical image segmentation device provided in an embodiment of the present application;

[0073] Figure 5b is another structural schematic diagram of the medical image segmentation device provided in an embodiment of the present application;

[0074] Figure 5c is another structural schematic diagram of the medical image segmentation device provided in an embodiment of the present application;

[0075] Figure 5dis another structural schematic diagram of the medical image segmentation device provided in an embodiment of the present application;

[0076] Figure 5e is another structural schematic diagram of the medical imaging device provided in an embodiment of the present application;

[0077] Figure 5f is another structural schematic diagram of the medical imaging device provided in an embodiment of the present application;

[0078] Figure 6 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0080] The embodiments of the present application provide a medical image segmentation method, apparatus, computer device and computer readable storage medium. The medical image segmentation apparatus can be integrated in a computer device, which can be a server or a terminal.

[0081] The medical image segmentation solution provided in the embodiment of the present application involves artificial intelligence computer vision technology (Computer Vision, CV). The artificial intelligence computer vision technology can be used to implement fundus image segmentation and obtain segmentation results.

[0082] Among them, Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image segmentation, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0083] In the embodiment of the present application, the so-called image segmentation refers to the technology and process of dividing an image into several specific regions with unique properties and proposing a target of interest. In the embodiment of the present application, it mainly refers to segmenting a three-dimensional (3D) medical image and finding the required target objects such as organs. For example, the liver region, heart region, atrium region, etc. can be segmented from a 3D medical image. The segmented target objects such as organs can be subsequently analyzed by medical staff or other medical experts for further operations.

[0084] For example, see Figure 1 Taking the medical image segmentation device integrated in a computing device as an example, the computing device can obtain a first medical image and a second medical image with a target object region marked; extract features from the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image; obtain optical flow motion information from the second medical image to the first medical image based on the first feature information and the second feature information; segment the target object in the first medical image based on the optical flow motion information and the marked target object region to obtain the segmentation result of the first medical image. For example, the heart region can be segmented from an unmarked atrium image.

[0085] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0086] This embodiment will be described from the perspective of a medical image segmentation device, which can be specifically integrated into a computer device, which can be a server or a terminal. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro-processing box, or other devices.

[0087] like Figure 2a As shown, the specific process of the medical image segmentation method can be as follows:

[0088] 201. Obtain a first medical image and a second medical image with a target object region marked.

[0089] The medical image can be provided to the medical image segmentation device after the medical image acquisition equipment acquires the image of biological tissues, such as the heart or liver, etc. The medical image acquisition equipment can include electronic devices such as a magnetic resonance imaging device (MRI), a computer tomography device (CT), a colposcope or an endoscope.

[0090] The first medical image is a medical image of an unlabeled target object and is a medical image to be segmented, and can be acquired by various medical image acquisition devices.

[0091] Among them, the target object can be a biological organ such as liver, heart, tumor, etc.

[0092] The second medical image is a medical image with a marked target object, such as a heart image with a marked heart area.

[0093] The purpose of the method in the embodiment of the present application is to segment the target object area from the first medical image using the labeled second medical image.

[0094] In the embodiment of the present application, the first and second are merely for distinction and have no limitations in terms of timing etc.

[0095] 202. Perform feature extraction on the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image.

[0096] For example, convolution operations may be performed on the first medical image and the second medical image respectively to extract first feature information of the first medical image and second feature information of the second medical image.

[0097] In one embodiment, for the sake of feature extraction accuracy and efficiency, a siamese network may be used to extract feature information, for example, Figure 2b As shown, a twin convolutional neural network can be used to perform convolution operations on the first medical image and the second medical image respectively to extract the features of the first and second medical images. For example, the labeled medical image x and the unlabeled image y are decomposed and convolution operations are performed to extract their respective feature maps.

[0098] Among them, the convolutional neural network can include multiple convolutional layers, each of which is composed of a convolution of a preset size (such as a 3*3*3 convolution), a batch normalization layer (Batch Normalization) and a nonlinear activation layer ReLu. That is, each convolutional layer performs convolution, normalization, and activation function processing on its input.

[0099] Taking the first medical image as an example, the first medical image can be input into the first convolutional layer of the convolutional neural network, and convolution, normalization, and activation function processing are performed in sequence in the first convolutional layer to output feature information, and then the output feature information is input into the next convolutional layer for processing, and so on until the last convolutional layer outputs the final feature information. Among them, the first feature information of the first medical image can be the feature information output by the last convolutional layer in the convolutional neural network, such as a feature map.

[0100] 203. Obtain optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information.

[0101] Optical flow is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. It is a method that uses the change of pixels in the time domain in the image sequence and the correlation between adjacent images to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent images. Generally speaking, optical flow is caused by the movement of the foreground object itself in the scene, the movement of the camera, or the joint movement of the two.

[0102] Simply put, optical flow is the "instantaneous speed" of the pixel motion of a moving object in space on the observation imaging plane. The study of optical flow uses the time domain changes and correlations of pixel intensity data in image sequences to determine the "motion" of each pixel position. The purpose of studying the optical flow field is to approximate the real motion field that cannot be directly obtained from the image sequence.

[0103] The optical flow motion information is the motion information of a pixel point at a certain position in the second medical image moving to a target position in the first medical image, including motion parameters such as a moving direction and a moving speed.

[0104] For example, if the position of pixel A in the t-th medical image is (x1, y1), then when we find point A in the t+1-th medical image, if its position is (x2, y2), we can determine the movement of pixel A: (ux, vy) = (x2, y2) - (x1, y1).

[0105] In one embodiment, pixel position association information between the images can be learned based on feature information of the two medical images, and optical flow motion information can be learned therefrom. Specifically, the step of "obtaining optical flow motion information from the second medical image to the first medical image based on the first feature information and the second feature information" may include:

[0106] Associating the first feature information with the second feature information to obtain pixel position correspondence information between the second medical image and the first medical image;

[0107] An upsampling operation is performed on the pixel position correspondence information to obtain optical flow motion information from the second medical image to the first medical image.

[0108] In the embodiment of the present application, in order to obtain the pixel position correspondence relationship between medical images, the feature information of the medical images can be associated or fused to associate the two images, so as to obtain the pixel position correspondence relationship information between the medical images.

[0109] There may be multiple ways of associating features. For example, in one embodiment, the step of “associating the first feature information with the second feature information to obtain pixel position correspondence information between the second medical image and the first medical image” may include:

[0110] Performing a convolution operation on the first feature information of the first medical image to obtain first convolution feature information;

[0111] Performing a convolution operation on the second feature information of the second medical image to obtain second convolution feature information;

[0112] Fusing the first convolution feature information with the second convolution feature information to obtain first fused feature information;

[0113] The first fusion feature information and the first convolution feature information are fused with each other to obtain pixel position correspondence information between the second medical image and the first medical image.

[0114] In one embodiment, the step of “merging the first convolution feature information with the second convolution feature information to obtain first fused feature information” includes:

[0115] The first convolution feature information and the second convolution feature information are fused to obtain fused convolution feature information;

[0116] Fusing the fused convolution feature information with the first convolution feature information to obtain first fused feature information;

[0117] In one embodiment, the step of “merging the first convolution feature information with the second convolution feature information to obtain first fused feature information” includes:

[0118] Performing a convolution operation on the first fused feature information to obtain convolution feature information of a predetermined size, wherein the size of the fourth convolution feature information is the same as the size of the first feature information;

[0119] The convolution feature information of a predetermined size is fused with the first feature information to obtain pixel position correspondence information between the second medical image and the first medical image.

[0120] In one embodiment, in order to accurately or quickly obtain pixel position correlation information between images, a non-local operation may be used to obtain spatial and temporal correlation information between pixels of a medical image. Figure 2b , a non-local block can be used to perform non-local operations and learn the position correspondence at the pixel level.

[0121] like Figure 2cAs shown, it is a schematic diagram of the structure of the non-local block. Assume that the feature map of the labeled medical image is y, and the feature map of the unlabeled medical image is x', where the dimensions of x' and y' are T*H*W*1024, where T is the number of video frames (corresponding to the number of images in this application), and the feature map size is T×H×W×1024, that is, there are 1024 channels.

[0122] The dimensions of the input x' and y' are T*H*W*1024, and then they are convolved with 512 convolution kernels of size 1*1*1 to obtain the outputs of the three branches, all of which have dimensions T*H*W*512. Then, after flat (smoothing) and trans (transformation) operations, the outputs of THW*512, 512*THW and THW*512 are obtained. The two outputs of the first two branches are matrix multiplied to obtain the output of THW*THW, which is processed by softmax (normalization function) and then matrix multiplied with the output of the third branch to obtain the output of THW*512 dimension. The output is reshaped into the output of T*H*W*512 dimension and passed through a convolution layer with 1024 convolution kernels and size 1*1*1, and the element-wise sum is performed with the original y' feature map T*H*W*1024 to obtain the final output result. Based on this output result, the pixel position correspondence information between the labeled image x and the unlabeled image y can be obtained.

[0123] refer to Figure 2b After the pixel position correspondence information between the labeled image x and the unlabeled image y is learned by using the Non-local Block, the pixel position correspondence information can be upsampled in the upsampling layer to obtain the optical flow motion information Δp.

[0124] 204. Segment the target object in the first medical image according to the optical flow motion information and the marked target object region to obtain a segmentation result of the first medical image.

[0125] In one embodiment, after obtaining the optical flow motion information Δp (such as the direction and speed of pixel motion, etc.), the target object region can be reconstructed based on the optical flow motion information, and the reconstructed region is used as the unlabeled image segmentation result.

[0126] For example, step 204 may include: reconstructing (e.g., migrating) the target object region according to the optical flow motion information and the labeled target object region to obtain a reconstructed target object region; and using the reconstructed target object region as the segmentation result of the first medical image. In the embodiment of the present application, the target object region reconstruction may include the migration of the target object region, etc. The region is reconstructed by migrating the target object region.

[0127] There are many ways of reconstruction, for example, pixels in the labeled target object area can be moved according to the optical flow motion information to obtain the moved pixels; and the target object area can be reconstructed according to the moved pixels. For example, based on the optical flow motion information Δp, pixels in the labeled organ area can be migrated in the segmentation space to form a migrated area, which is used as the organ segmentation area of ​​the unlabeled image.

[0128] In order to enable the optical flow motion information Δp to be applied to the original medical image (since the optical flow motion information Δp may be obtained under a downsampling setting) and to improve the reconstruction speed; in one embodiment, the optical flow motion information may be subjected to trilinear interpolation processing, and an unlabeled first sample medical image may be reconstructed based on the processed optical flow motion information to obtain a first reconstructed medical image. That is, the step of "segmenting the target object in the first medical image according to the optical flow motion information and the labeled target object area to obtain the segmentation result of the first medical image" may include:

[0129] Perform trilinear interpolation processing on the optical flow motion information to obtain processed optical flow motion information;

[0130] Based on the processed optical flow motion information reconstruction and the labeled target object region, the target object in the first medical image is segmented to obtain the segmentation result of the first medical image. Specifically, the optical flow motion information can be processed by trilinear interpolation to obtain the processed optical flow motion information; based on the processed optical flow motion information reconstruction and the labeled target object region, the target object region is reconstructed to obtain the reconstructed target object region; and the reconstructed target object region is used as the segmentation result of the first medical image.

[0131] For example, based on the trilinear interpolation kernel F, the optical flow motion information can be interpolated in each preset direction (such as the xyz axis direction) to obtain processed optical flow motion information, which is the optical flow motion information implemented by trilinear interpolation. Specifically, the trilinear interpolation process can be described in detail with reference to the following embodiments.

[0132] The embodiment of the present application provides a solution based on migrating image-to-image mapping to segmentation space. The solution sets the image-to-image mapping modeling as pixel-level optical flow estimation, and reconstructs or migrates the segmentation results of the unlabeled image from the labeled results in the labeled image by learning the optical flow motion information such as the movement direction and size of the pixel. Figure 2d shown.

[0133] like Figure 2dAs shown in the figure, taking the medical image of the atrium as an example, a labeled image (Labeled) with the heart area labeled and an unlabeled image (Unlabeled) are first provided; then, the optical flow motion information Δp from the labeled image to the unlabeled image is learned, and the heart area is reconstructed (or migrated) through the labeled heart area and the optical flow motion information Δp. The reconstructed heart area (or migrated heart area) is the heart segmentation result of the unlabeled image.

[0134] From the above, it can be seen that the embodiment of the present application can transfer the object segmentation of the labeled image, such as organ segmentation, to the labeled image by learning the transformation from the labeled image to the unlabeled image, and use the labeled medical image to perform object segmentation, such as organ segmentation, on the unlabeled medical image, thereby avoiding the traditional solution of training the segmentation model without a large amount of labeled data. Only by providing a labeled image and an image to be segmented, the image to be segmented can be segmented, which greatly improves the accuracy and efficiency of medical image segmentation.

[0135] The segmentation scheme provided in the embodiment of the present application can be applied to any image segmentation scenario. For example, it can be applied to the annotation scenario of medical images. Since the current pixel / voxel level annotation is time-consuming and labor-intensive, and requires the strong support of doctors, it is very difficult to obtain a large number of annotations. In this context, using the scheme provided in the embodiment of the present application, the doctor only needs to accurately annotate one case of data, and the remaining data can be annotated using the scheme of the embodiment of the present application. For doctors, doctors only need to annotate one target object such as an organ, and they can perform accurate target object segmentation such as an organ on other unlabeled data, thereby saving a lot of doctor's annotation time.

[0136] The medical image segmentation method in the embodiment of the present application can be implemented by a deep learning network of AI. For example, after acquiring the first medical image and the second medical image, the target object such as organ segmentation can be implemented by a deep learning segmentation network. Specifically, the segmentation network can perform the above steps 202-203, etc. The specific structure of the generator can refer to Figure 2b .

[0137] The segmentation network may be a segmentation network trained with a large number of labeled data sets. There are many specific training methods. For example, the segmentation network may be trained by back propagation based on a large number of medical imaging data sets with labeled organs. Specifically, the segmentation loss may be calculated based on the target object segmentation result and the labeling result of the network (such as by calculating the loss function), and the network may be trained based on the segmentation loss.

[0138] Further consideration is that since medical imaging datasets are usually collected by different medical imaging devices, there are large differences between domains of different datasets, which seriously hinders the generalization performance of deep networks; and traditional training methods will lead to poor learning accuracy of optical flow motion information, resulting in low segmentation accuracy.

[0139] In order to ensure that the algorithm can truly play a role in assisting diagnosis in clinical practice, the generalization performance and segmentation accuracy of the model need to be further improved. At the same time, supervised training methods require additional vascular annotations, which is very time-consuming and expensive. It is impractical to perform large-scale vascular classification and segmentation in the diagnosis of eye diseases. Therefore, unsupervised training methods that do not require additional annotations need to be designed in clinical practice.

[0140] In response to the above-mentioned problems, the embodiments of the present application provide a segmentation network training method using unsupervised domain adaptation technology, which can learn the feature structure on an existing annotated data set and transfer the knowledge to a new data set, providing more accurate target object segmentation such as organ segmentation for new unlabeled data sets, and effectively improving the generalization performance of deep networks (such as blood segmentation networks) on other data sets.

[0141] In order to improve the accuracy of segmentation, the unsupervised adaptive training method in the embodiment of the present application can be performed by training a generative adversarial network including a segmentation network (i.e., a generator); then, the generator in the trained generative adversarial network is used to obtain the optical flow motion information from the labeled image to the unlabeled image, thereby realizing the segmentation of the unlabeled medical image, etc.

[0142] Specifically, the labeled sample images and the unlabeled sample images can be used to train the image segmentation network including the first generative adversarial network and the second generative adversarial network, wherein the first generative adversarial network includes the first object segmentation network (i.e., the first generator) and the first generative adversarial network includes the second object segmentation network (i.e., the second generator). Then, in actual image segmentation, the first generator is used to learn the optical flow motion information.

[0143] refer to Figure 3a , provides a segmentation network, the segmentation network includes a first generation adversarial network and a second generation adversarial network. The first generation adversarial network includes a first object segmentation network (i.e., a first generator G A ), a discriminator, a first reconstruction module, a second reconstruction module; the second generative adversarial network includes a second object segmentation network (i.e., a second generator G B ), a third reconstruction module and a fourth reconstruction module.

[0144] like Figure 3b As shown, the specific process of a network training method is as follows:

[0145] 301. Obtain a first sample medical image and a second sample medical image with a target object region marked.

[0146] Among them, the first sample medical image y is a medical image of an unlabeled target object (such as an organ) area, which can be acquired by a medical image acquisition device; the second sample medical image x is a medical image of an labeled target object (such as an organ) area, which can be obtained by medical personnel by labeling the target object such as an organ on the medical image.

[0147] 302. Use a first generator in a first generative adversarial network to obtain first sample optical flow motion information from a second sample medical image to a first sample medical image.

[0148] For example, we can use the generator G A , learn the pixel interaction relationship between the labeled image x and the unlabeled image y, and learn the optical flow motion information Δp from the labeled image x to the unlabeled image y A

[0149] The specific process of the first generator acquiring the first sample optical flow motion information can refer to the process of acquiring the optical flow motion information described in the above embodiment. Figure 2b .

[0150] 303. Use the first reconstruction module in the first generative adversarial network to perform image reconstruction based on the first sample optical flow motion information to obtain a first reconstructed medical image and a first segmentation result corresponding to the first sample medical image.

[0151] Specifically, the image reconstruction may refer to the description of the above embodiments.

[0152] For example, according to the first sample optical flow motion information Δp A Move the pixels in the labeled second sample medical image y to reconstruct the unlabeled first sample medical image to obtain the first reconstructed medical image According to the first sample optical flow motion information Δp A Move the target object labeled area xs pixels in the labeled medical image x to reconstruct the target object segmentation area of ​​the unlabeled image

[0153] In order to enable the optical flow motion information Δp to be applied to the original medical image (since the optical flow motion information Δp may be obtained under a downsampling setting) and to improve the reconstruction speed, in one embodiment, the optical flow motion information can be subjected to trilinear interpolation processing, and an unlabeled first sample medical image is reconstructed based on the processed optical flow motion information to obtain a first reconstructed medical image.

[0154] For example, based on the trilinear interpolation kernel, the optical flow motion information can be interpolated in each preset direction to obtain processed optical flow motion information, and the processed optical flow motion information is the optical flow motion information implemented by trilinear interpolation.

[0155] Due to Δp A It is often very small. The present embodiment uses trilinear interpolation to achieve it. For each position p0 in the original image, the target position is p=p0+Δp. The present embodiment expresses this process as:

[0156]

[0157] Where F is the kernel of trilinear interpolation, which operates on the x-axis, y-axis, and z-axis simultaneously. It is expressed as follows:

[0158] F(q,p)=f(q x , p x )·f(q y , p y )·f(q z , p z )

[0159] Here we set f(a,b) = max(0,1-|ab|). Since most elements of F(q,p) are 0, the calculation speed is very fast.

[0160] In an embodiment of the present application, each time an image is reconstructed (such as reconstructing an unlabeled image or reconstructing a labeled image), the learned optical flow motion information Δp can be subjected to trilinear interpolation processing, and the image can be reconstructed based on the processed information to improve the reconstruction speed and efficiency.

[0161] Similarly, in an embodiment of the present application, each time a segmented object area is reconstructed (such as reconstructing a segmented object area of ​​an unlabeled image, or reconstructing a segmented object area of ​​a labeled image), the learned optical flow motion information Δp can be subjected to trilinear interpolation processing, and then, the segmented object area can be reconstructed based on the processed information, thereby improving the reconstruction speed and efficiency.

[0162] 304. Use a second generator in a second generative adversarial network to obtain second sample optical flow motion information from the first reconstructed medical image to the second sample medical image.

[0163] Similarly, the manner in which the second generator obtains the optical flow motion information and the structure of the generator may refer to the introduction of the above embodiment.

[0164] For example, we can use the generator G A , learn to label the image x and reconstruct the image The pixel interaction relationship is learned to reconstruct the image The optical flow motion information Δp to the labeled image x B .

[0165] 305. Use the second reconstruction module in the second generative adversarial network to perform image reconstruction based on the second sample optical flow motion information to obtain a second reconstructed medical image and a second segmentation result corresponding to the second sample medical image.

[0166] For example, according to the second sample optical flow motion information Δp B First reconstruction of medical images The pixels in the image are moved to reconstruct the labeled image x and obtain the second reconstructed medical image. According to the first sample optical flow motion information Δp B Segment the reconstructed target object The pixels are moved to reconstruct the target object segmentation area of ​​the labeled image x

[0167] 306. Train a first generative adversarial network according to the first reconstructed medical image, the first segmentation result, the second reconstructed medical image, and the second segmentation result to obtain a trained first generative adversarial network.

[0168] Theoretically, the generator G A It can capture the optical flow motion information Δp from the labeled image x to the unlabeled image y A However, since there are thousands of possible ways to go from X to Y, in order to ensure the uniqueness of this mapping, the present application embodiment hopes to reconstruct the image by learning The data relationship with the labeled image x is used to reconstruct the labeled image. Similarly, After that, we can use the generator G B Learning to reconstruct images The pixel interaction relationship with the annotated image x is used to learn the optical flow motion information Δp B . Through the optical flow motion information Δp B , the reconstructed labeled image is

[0169] Specifically, the image reconstruction process is expressed by the following formula:

[0170]

[0171] Similarly, the embodiment of the present application can also apply the optical flow motion information to the segmentation space to obtain the segmentation of the reconstructed unlabeled image. And reconstruct the segmentation of the labeled image The formula is as follows:

[0172]

[0173] In the embodiment of the present application, the G can be improved by reconstructing the segmentation results of the segmented space for multiple times and training the network. A Learn the accuracy of optical flow motion information. During the training process, in order to improve the accuracy of image reconstruction and segmentation reconstruction, and thus improve the accuracy of captured motion information of the generator after training, the accuracy of image reconstruction can be ensured by constructing a reconstruction consistency loss in the image reconstruction space, and the accuracy of image segmentation can be ensured by constructing a segmentation consistency loss in the segmentation space.

[0174] For example, in one embodiment, the step of “training a first generative adversarial network according to the first reconstructed medical image, the first segmentation result, the second reconstructed medical image, and the second segmentation result to obtain a trained first generative adversarial network” may include:

[0175] S1. Obtaining a reconstruction consistency loss in an image reconstruction space according to a first reconstructed medical image and a second reconstructed medical image.

[0176] The reconstruction consistency loss includes at least one of the following losses: image reconstruction consistency loss in the image reconstruction process, trajectory consistency loss of optical flow motion information, and the like.

[0177] For example, reference Figure 3a , from the labeled image x to the reconstructed unlabeled image Then reconstruct the labeled image (Right now ), the embodiment of the present application hopes to reconstruct the labeled image is consistent with the original annotated image x; therefore, an image reconstruction consistency loss is constructed.

[0178] Specifically, the image consistency loss of the second sample medical image and the second reconstructed medical image can be obtained according to the second reconstructed medical image and the second sample medical image. Figure 3a , the reconstructed labeled image can be calculated based on the consistency loss function The image reconstruction consistency loss between the labeled image x.

[0179] For example, reference Figure 3a , in order to constrain and The optical flow motion information of this application embodiment hopes that Δp A With Δp B Therefore, a method to ensure Δp A With Δp B The trajectory consistency loss L is used to keep the trajectory consistent.tra .

[0180] According to the first sample optical flow motion information and the second sample optical flow motion information, the trajectory consistency loss of the first sample optical flow motion information and the second sample optical flow motion information showing the same trajectory is obtained. For example, according to the preset trajectory consistency loss function, Δp is calculated A With Δp B The trajectory consistency loss L between tra .

[0181] In one embodiment, in order to accurately and conveniently calculate the trajectory consistency loss, an image can be generated, and then two optical flow motion information are applied to the image to perform image transformation, and the transformation loss between the transformed image and the original image is calculated, and the trajectory consistency loss is obtained based on the loss. Specifically, the step of "obtaining the trajectory consistency loss of the first sample optical flow motion information and the second sample optical flow motion information presenting a consistent trajectory based on the first sample optical flow motion information and the second sample optical flow motion information" may include:

[0182] Generate an original medical image;

[0183] According to the first pixel position mapping relationship information and the second sample optical flow motion information, the reference medical image is transformed to obtain a transformed medical image;

[0184] Obtaining the transformation loss between the transformed medical image and the original medical image;

[0185] The transformation loss is taken as the trajectory consistency loss when the first sample optical flow motion information and the second sample optical flow motion information present a consistent trajectory.

[0186] like Figure 3c In order to achieve Δp A With Δp B The trajectory consistency constraint information is presented at the pixel level, and a random image r is generated. h*w*c ~N(0,I), the embodiment of the present application converts the optical flow motion information Δp A With Δp B It is hoped that after this pair of transformations, the reconstructed image It is consistent with the original image r.

[0187] In addition, in some embodiments, other losses that maintain image construction consistency may be constructed to improve the accuracy of image reconstruction and thereby improve segmentation accuracy.

[0188] S2. Obtain a segmentation consistency loss in the segmentation space according to the first segmentation result, the second segmentation result, and the target object labeling result of the second sample medical image.

[0189] The segmentation consistency loss is the loss of the target object segmentation result reconstructed in the segmentation space being consistent with the labeled target object segmentation result, wherein the reconstructed target object segmentation result and the labeled segmentation result can be consistent in terms of image segmentation, mask value, semantics, etc. That is, the segmentation consistency loss includes image segmentation consistency loss, mask value loss, semantic consistency loss, etc.

[0190] For example, refer to Figure 3a , in order to achieve consistency in the segmented space So that x s and To keep it consistent, we can design the image segmentation consistency loss. Specifically, we can obtain the image segmentation consistency loss that the target object annotation result is consistent with the second segmentation result based on the second segmentation result and the target object annotation result. For example, we can calculate x s and The difference between them is used to obtain the image segmentation consistency loss.

[0191] For another example, in order to ensure that the segmentation mask value obtained by using the optical flow information in the segmentation space is close to the preset segmentation mask value such as {0,1}, a mask value loss can be introduced to encourage the style mask value of the segmentation result to be close to the preset segmentation mask value such as 0 or 1. In one embodiment, since the segmentation mask value is close to 0 or 1, the mask value loss can be called a hash loss.

[0192] The segmentation mask value may include a pixel value of a segmentation result, such as a pixel value of an organ segmentation image.

[0193] In one embodiment, to ensure that the segmentation mask reconstructed based on the optical flow motion information is With the original segmentation x s The size is consistent, so it can also be Specifically, a constraint condition is constructed that the first segmentation result, the second segmentation result and the target object annotation result are consistent in size, and the constraint condition corresponding to the mask numerical loss is obtained; during training, the network can be trained according to the mask numerical loss and its corresponding constraint condition.

[0194] In one embodiment, it is also desirable to learn segmentation results such as segmentation masks. Semantic consistency is also expressed in the image. For example, for learning segmentation masks The pixel values ​​in the image are kept consistent, so that the target object segmentation area, such as the organ area, can be easily identified during observation. To this end, the embodiment of the present application introduces a semantic consistency loss. Specifically, based on the first segmentation result, a semantic consistency loss can be constructed to maintain the semantic consistency of the first segmentation result in the image. In one embodiment, semantic consistency can be regarded as appearance smoothness, so semantic consistency loss can also be called appearance smoothness loss.

[0195] In one embodiment, in order to maintain semantic consistency, it is necessary to ensure that the numerical distribution of the target object region, such as the interior of an organ, is consistent. Therefore, the pixel values ​​in the segmentation result image can be derived to calculate the loss of ensuring the consistency of the numerical distribution of the target object region in the segmentation result image.

[0196] For example, the pixel values ​​in the first segmentation result are differentiated along a preset direction to obtain a derivative result in the preset direction; based on the derivative result in the preset direction and the first segmentation result, a semantic consistency loss is constructed to maintain semantic consistency of the first segmentation result in the image.

[0197] For example, for the segmentation mask We can derive Δy from the x-axis, y-axis, and z-axis respectively. x-axis ,Δy y-axis ,Δy z-axis , the appearance smoothness loss is calculated based on the derivative results.

[0198] It should be understood that the segmentation consistency loss in the segmentation space and the reconstruction consistency loss in the image reconstruction space of the embodiments of the present application are not limited to the types introduced above, and other types of reconstruction consistency loss or segmentation consistency loss can also be designed according to needs.

[0199] S3. According to the reconstruction consistency loss and the segmentation consistency loss, the first generative adversarial network is trained to obtain the trained first generative adversarial network.

[0200] In one embodiment, in order to improve the segmentation accuracy of the network, the first generative adversarial network may also be trained by generative adversarial training; for example, the method of the embodiment of the present application may also include:

[0201] Using the discriminator in the first generative adversarial network, the sample medical image and the first reconstructed medical image are discriminated to obtain a discrimination result;

[0202] Obtaining the adversarial loss of the first generative adversarial network according to the first reconstructed medical image and the discrimination result;

[0203] At this time, in the embodiment of the present application, the first generative adversarial network can be trained according to the adversarial loss, the reconstruction consistency loss, and the segmentation consistency loss to obtain the trained first generative adversarial network.

[0204] In one embodiment, the adversarial loss may include maximizing adversarial loss and minimizing adversarial loss; specifically, the reconstruction loss may be calculated based on the first reconstructed medical image and the first sample medical image, the discrimination loss may be calculated based on the judgment result, and the minimizing adversarial loss and maximizing adversarial loss of the generative adversarial network may be calculated based on the reconstruction loss and the discrimination loss. At this point, the embodiment of the present application may iteratively train the first generative adversarial network based on minimizing adversarial loss and maximizing adversarial loss, reconstruction consistency loss, and segmentation consistency loss to obtain a trained generative adversarial network.

[0205] For example, refer to Figure 3a , generator G A We hope to accurately learn the optical flow motion information between the labeled image and the unlabeled image, while the discriminator D hopes to distinguish the reconstructed image. And the real unlabeled image y. So the generator G A A confrontation is formed between the generator G and the discriminator D. A We hope to reconstruct with the minimum loss while the discriminator hopes to distinguish with the maximum loss. Therefore, we can train the network with the maximum loss and the minimum loss.

[0206] From the above, it can be seen that the embodiment of the present application proposes to use unsupervised generative adversarial training to train the image segmentation network, which can effectively improve the generalization performance of the model on other data sets, and can also enable the generator to accurately learn the optical flow motion information, thereby improving the accuracy of image segmentation.

[0207] For example, after the first generative adversarial network is trained as described above, the generator in the first generative adversarial network can be used to accurately learn the optical flow motion information from the labeled image to the unlabeled image, so that the target object in the unlabeled image can be segmented based on the optical flow motion information, greatly improving the segmentation accuracy.

[0208] The method described in the above embodiment is further described in detail below with examples.

[0209] In this embodiment, the medical image segmentation device is specifically integrated into a computer device as an example for description.

[0210] (A) Training of segmentation network.

[0211] According to the above Figure 3a Description, such as Figure 4a A schematic diagram of the architecture of a segmentation network is proposed, including a first generator GA, GB, multiple Warp (reconstruction) modules, etc.

[0212] (1) First, the computer device obtains a labeled image x with a target object such as an organ, and an unlabeled image y without a label. Figure 4a , agree that x∈R h*w*c is a medical image with annotations, x s ∈R h*w*c is its corresponding annotation, h, w, c are the dimensions of the medical image in the coronal, sagittal and axial directions respectively. It is a large number of unlabeled medical image datasets.

[0213] use Figure 4a In the network shown, the first generator learns the pixel interaction relationship between the labeled image x and the unlabeled image y, thereby learning the optical flow motion information Δp A ∈R h*w*c*3 , however, through Warp based on the optical flow motion information Δp A , the unlabeled image is reconstructed as Warp is based on the optical flow motion information Δp A , the segmented image of the unlabeled image is reconstructed as

[0214] After obtaining the reconstructed unlabeled image After that, we can use the generator G B Learning to reconstruct images The pixel interaction relationship with the annotated image x is used to learn the optical flow motion information Δp B . Warp based on optical flow motion information Δp B , the reconstructed labeled image is Through Warp based on optical flow motion information Δp B Reconstruct the segmented image of the labeled image x

[0215] (2) The computer equipment calculates the loss based on the reconstruction results

[0216] For example, the loss is calculated based on a preset loss function and reconstruction results. In the above embodiments, the reconstruction consistency loss of the image reconstruction space and the segmentation consistency loss in the segmentation space are introduced. The embodiments of this application introduce the following loss and loss function:

[0217] (2.1) Countermeasure Loss

[0218] In order to make the reconstructed image close enough to the target domain, the embodiment of the present application introduces adversarial loss. The specific formula is as follows:

[0219]

[0220] Here the generator G A We hope to accurately learn the optical flow motion information between the labeled image and the unlabeled image, while the discriminator D hopes to distinguish the reconstructed image. And the real unlabeled image y. So the generator G A A confrontation is formed between the generator G and the discriminator D. A We hope to reconstruct with the minimum loss, while the discriminator hopes to distinguish with the maximum loss, that is,

[0221] (2.2), cycle consistency loss

[0222] From the labeled image x to the reconstructed unlabeled image Then reconstruct the labeled image (Right now ), we hope to reconstruct the labeled image It is consistent with the original labeled image x. In addition, we also hope to achieve consistency in the segmentation space The cycle consistency loss is introduced as follows

[0223]

[0224] (2.2), trajectory consistency loss

[0225] In addition, to constrain and The optical flow motion information of this application embodiment hopes that Δp A With Δp B There is trajectory consistency between them at the pixel level, such as Figure 3c To implement this constraint information, we can randomly construct a generated image r h*w*c ~N(0,I), the embodiment of the present application converts the optical flow motion information Δp A With Δp B Applied above, the embodiment of the present application hopes that after this pair of transformations, the reconstructed image It is consistent with the original image r. The whole process is expressed by the following formula.

[0226]

[0227] (2.3), Regularization loss

[0228] In order to ensure that the segmentation mask value obtained by using the optical flow information in the segmentation space belongs to {0,1}, the embodiment of the present application introduces hash loss to encourage it to be as close to 0,1 as possible. In addition, the segmentation mask that the embodiment of the present application hopes to learn With the original segmentation x s The size is consistent, and the embodiment of the present application is The whole formula is expressed as follows.

[0229]

[0230] In addition, the segmentation mask that the present application embodiment hopes to learn The semantic consistency is also expressed in the image. For this reason, the embodiment of the present application observes that the numerical distribution inside the organ is consistent. The embodiment of the present application takes the derivative of the image along the x-axis, y-axis, and z-axis respectively, and obtains Δy x-axis ,Δy y-axis ,Δy z-axis The present application embodiment hopes to segment the mask The image is consistent with the organ differentiation. To this end, the following loss is designed in the embodiment of the present application:

[0231]

[0232] (3) The computer device trains the network using the loss calculated above to obtain the first generative adversarial network after training.

[0233] (ii) After training, the generator in the first generative adversarial network can segment the target objects such as organs in the medical images to be segmented.

[0234] like Figure 4b As shown, taking the target object as an organ as an example, a medical image segmentation method, the specific process is as follows:

[0235] 401. The computer device obtains the labeled medical images and the unlabeled medical images.

[0236] The labeled medical image is a medical image with a labeled target organ, and the unlabeled medical image may come from an unlabeled image dataset, etc.

[0237] 402. The computing device uses a generator to extract features from the labeled medical image and the unlabeled medical image respectively to obtain first feature information of the unlabeled medical image and second feature information of the labeled medical image.

[0238] For example, refer to Figure 2b The generator can extract features from the labeled medical images and the unlabeled medical images through multiple convolutional layers. For the specific feature extraction process, refer to the description of the above embodiment.

[0239] 403. The computing device uses a generator to obtain optical flow motion information from the labeled medical image to the unlabeled medical image based on the first feature information and the second feature information.

[0240] Specifically, the specific process of the generator learning optical flow motion information can refer to the description of the above embodiment, such as performing non-local operations, etc. Figure 2b The process shown.

[0241] 404. The computer device uses a reconstruction module to reconstruct the target organ region according to the optical flow motion information and the marked organ region to obtain a reconstructed target organ region.

[0242] For example, based on the optical flow motion information Δp, the pixels in the labeled organ region can be moved in the segmentation space to form a reconstructed target organ region, which can be used as the organ segmentation region of the unlabeled image.

[0243] 405. The computer device uses the reconstructed target organ region as the organ segmentation result of the unlabeled medical image.

[0244] From the above, it can be seen that the embodiment of the present application can transfer the object segmentation of the labeled image, such as organ segmentation, to the labeled image by learning the transformation from the labeled image to the unlabeled image, and use the labeled medical image to perform object segmentation, such as organ segmentation, on the unlabeled medical image, thereby avoiding the traditional solution of training the segmentation model without a large amount of labeled data. Only by providing a labeled image and an image to be segmented, the image to be segmented can be segmented, which greatly improves the accuracy and efficiency of medical image segmentation.

[0245] In order to better implement the above method, an embodiment of the present application further provides a medical image segmentation device, which can be integrated into a computer device, such as a server or a terminal.

[0246] For example, Figure 5a As shown, the medical image segmentation device may include an acquisition unit 501, a feature extraction unit 502, an optical flow unit 503, and a segmentation unit 504, as follows:

[0247] An acquisition unit 501 is used to acquire a first medical image and a second medical image with a target object area marked;

[0248] A feature extraction unit 502 is used to extract features from the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image;

[0249] An optical flow unit 503, configured to obtain optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information;

[0250] The segmentation unit 504 is configured to segment the target object in the first medical image according to the optical flow motion information and the marked target object region to obtain a segmentation result of the first medical image.

[0251] In one embodiment, reference Figure 5b , the segmentation unit 504 comprises:

[0252] The reconstruction subunit 5041 is used to reconstruct the target object region according to the optical flow motion information and the marked target object region to obtain a reconstructed target object region;

[0253] The segmentation subunit 5042 is used to use the reconstructed target object area as the segmentation result of the first medical image.

[0254] In one embodiment, the reconstruction subunit 5041 is used to perform trilinear interpolation processing on the optical flow motion information to obtain processed optical flow motion information; based on the processed optical flow motion information reconstruction and the marked target object area, perform target object area reconstruction to obtain a reconstructed target object area.

[0255] In one embodiment, reference Figure 5c , the optical flow unit 503 may include:

[0256] An association subunit 5031 is used to associate the first feature information with the second feature information to obtain pixel position correspondence information between the second medical image and the first medical image;

[0257] The sampling subunit 5032 is used to perform an up-sampling operation on the pixel position correspondence information to obtain the optical flow motion information from the second medical image to the first medical image.

[0258] In one embodiment, the association subunit 5031 may be specifically used for:

[0259] Performing a convolution operation on the first feature information of the first medical image to obtain first convolution feature information;

[0260] Performing a convolution operation on the second feature information of the second medical image to obtain second convolution feature information;

[0261] Fusing the first convolution feature information and the second convolution feature information to obtain first fused feature information;

[0262] The first fusion feature information and the first convolution feature information are fused with each other to obtain pixel position correspondence information between the second medical image and the first medical image.

[0263] In one embodiment, the optical flow unit 503 is specifically used to: use the first generator in the trained first generative adversarial network to obtain optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information.

[0264] In one embodiment, reference Figure 5d , the medical image segmentation device of the present application also includes:

[0265] A sample acquisition unit 505 is used to acquire a first sample medical image and a second sample medical image with a target object area marked;

[0266] A first sample optical flow unit 506 is used to obtain first sample optical flow motion information from the second sample medical image to the first sample medical image using a first generator in a first generative adversarial network.

[0267] A first reconstruction unit 507 is configured to use the first reconstruction module in the first generative adversarial network to perform image reconstruction based on the first sample optical flow motion information, and obtain a first reconstructed medical image and a first segmentation result corresponding to the first sample medical image;

[0268] A second sample optical flow unit 508 is used to obtain second sample optical flow motion information from the first reconstructed medical image to the second sample medical image by using a second generator in a second generative adversarial network;

[0269] A second reconstruction unit 509 is configured to use a second reconstruction module in a second generative adversarial network to perform image reconstruction based on the second sample optical flow motion information, and obtain a second reconstructed medical image and a second segmentation result corresponding to the second sample medical image;

[0270] The training unit 510 is used to train the first generative adversarial network according to the first reconstructed medical image, the first segmentation result, the second reconstructed medical image, and the second segmentation result to obtain a trained first generative adversarial network.

[0271] In one embodiment, reference Figure 5e , the training unit 510 may include:

[0272] The reconstruction loss acquisition subunit 5101 is used to acquire the reconstruction consistency loss in the image reconstruction space according to the first reconstructed medical image and the second reconstructed medical image;

[0273] The segmentation loss acquisition subunit 5102 is used to acquire the segmentation consistency loss in the segmentation space according to the first segmentation result, the second segmentation result and the target object labeling result of the second sample medical image;

[0274] The training subunit is used to train the first generative adversarial network according to the reconstruction consistency loss and the segmentation consistency loss to obtain the trained first generative adversarial network.

[0275] In one embodiment, the reconstruction consistency loss includes: image reconstruction consistency loss and trajectory consistency loss; the reconstruction loss acquisition subunit 5101 may be specifically used for:

[0276] According to the second reconstructed medical image and the second sample medical image, obtaining an image reconstruction consistency loss that is consistent with the second sample medical image and the second reconstructed medical image;

[0277] According to the first sample optical flow motion information and the second sample optical flow motion information, a trajectory consistency loss is obtained in which the first sample optical flow motion information and the second sample optical flow motion information present a consistent trajectory.

[0278] In one embodiment, the segmentation consistency loss includes image segmentation consistency loss, mask value loss, and semantic consistency loss;

[0279] The segmentation loss acquisition subunit 5102 is used to:

[0280] According to the second segmentation result and the target object labeling result, obtaining an image segmentation consistency loss in which the target object labeling result is consistent with the second segmentation result;

[0281] According to the first segmentation result, the second segmentation result and a preset segmentation mask value, respectively obtaining mask value losses of the first segmentation result and the second segmentation result relative to the preset segmentation mask value;

[0282] Based on the first segmentation result, a semantic consistency loss is constructed to maintain semantic consistency of the first segmentation result on the image.

[0283] In one embodiment, the segmentation loss obtaining subunit 5102 is used to:

[0284] Derivative the pixel values ​​in the first segmentation result along a preset direction to obtain a derivative result in the preset direction;

[0285] According to the derivation result in the preset direction and the first segmentation result, a semantic consistency loss is constructed to maintain semantic consistency of the first segmentation result on the image.

[0286] In one embodiment, the reconstruction loss obtaining subunit 5101 may be specifically used for:

[0287] Generate an original medical image;

[0288] Transforming the reference medical image according to the first pixel position mapping relationship information and the second sample optical flow motion information to obtain a transformed medical image;

[0289] Obtaining a transformation loss between the transformed medical image and the original medical image;

[0290] The transformation loss is used as a trajectory consistency loss in which the first sample optical flow motion information and the second sample optical flow motion information present a consistent trajectory.

[0291] In one embodiment, reference Figure 5f , the medical image segmentation device also includes an adversarial loss acquisition unit 511;

[0292] The adversarial loss acquisition unit 511 is used to use the discriminator in the first generative adversarial network to discriminate the sample medical image and the first reconstructed medical image to obtain a discrimination result; and obtain the adversarial loss of the first generative adversarial network according to the first reconstructed medical image and the discrimination result.

[0293] The training unit 512 is used to train the first generative adversarial network according to the adversarial loss, the reconstruction consistency loss, and the segmentation consistency loss to obtain a trained first generative adversarial network.

[0294] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0295] As can be seen from the above, the embodiment of the present application can acquire a first medical image and a second medical image with a marked target object area through an acquisition unit 501; a feature extraction unit 502 performs feature extraction on the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image; an optical flow unit 503 obtains optical flow motion information from the second medical image to the first medical image based on the first feature information and the second feature information; a segmentation unit 504 segments the target object in the first medical image based on the optical flow motion information and the marked target object area to obtain a segmentation result of the first medical image. . This scheme can transfer objects in the labeled image, such as organ segmentation, to the labeled image by learning the transformation from the labeled image to the unlabeled image, and use the labeled medical image to perform object segmentation on the unlabeled medical image, such as organ segmentation, thereby greatly improving the segmentation accuracy and efficiency of medical images.

[0296] The present application also provides a computer device, such as Figure 6As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0297] The computer device may include components such as a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a power supply 603, and an input unit 604. Those skilled in the art will appreciate that Figure 6 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0298] The processor 601 is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 602, and calling data stored in the memory 602, the processor 601 performs various functions of the computer device and processes data. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 601.

[0299] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0300] The computer device also includes a power supply 603 for supplying power to various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 603 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0301] The computer device may further include an input unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0302] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 601 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 602 according to the following instructions, and the processor 601 will run the application programs stored in the memory 602, thereby realizing various functions, as follows:

[0303] Acquire a first medical image and a second medical image with a target object area marked; perform feature extraction on the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image; obtain optical flow motion information from the second medical image to the first medical image based on the first feature information and the second feature information; segment the target object in the first medical image based on the optical flow motion information and the marked target object area to obtain a segmentation result of the first medical image.

[0304] For details of the above operations, please refer to the previous embodiments, which will not be described in detail here.

[0305] As can be seen from the above, the computer device of this embodiment is after a first medical image and a second medical image with a target object area marked; feature extraction is performed on the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image; optical flow motion information from the second medical image to the first medical image is obtained according to the first feature information and the second feature information; the target object in the first medical image is segmented according to the optical flow motion information and the marked target object area to obtain the segmentation result of the first medical image. Therefore, this scheme can transfer objects in the labeled image, such as organ segmentation, to the labeled image by learning the transformation from the labeled image to the unlabeled image, and use the labeled medical image to perform objects such as organs on the unlabeled medical image, thereby greatly improving the accuracy and efficiency of medical image segmentation.

[0306] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0307] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute the steps in any medical image segmentation method provided in the embodiment of the present application. For example, the computer program can execute the following steps:

[0308] Acquire a first medical image and a second medical image with a target object area marked; perform feature extraction on the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image; obtain optical flow motion information from the second medical image to the first medical image based on the first feature information and the second feature information; segment the target object in the first medical image based on the optical flow motion information and the marked target object area to obtain a segmentation result of the first medical image.

[0309] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0310] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0311] Since the instructions stored in the computer-readable storage medium can execute the steps in any medical image segmentation method provided in the embodiments of the present application, the beneficial effects that can be achieved by any medical image segmentation method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0312] The above is a detailed introduction to a medical image segmentation method, device, computer equipment and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A medical image segmentation method, characterized in that: include: Acquire a first medical image and a second medical image with a target object area marked; Extracting features from the first medical image and the second medical image respectively to obtain first feature information of the first medical image and second feature information of the second medical image; Acquire optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information; Using a first generator in a trained first generative adversarial network, obtaining optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information; The medical impact segmentation method further comprises: Acquire a first sample medical image and a second sample medical image with a target object region marked; Using a first generator in a first generative adversarial network to obtain first sample optical flow motion information from the second sample medical image to the first sample medical image; Using the first reconstruction module in the first generative adversarial network to perform image reconstruction based on the first sample optical flow motion information, to obtain a first reconstructed medical image and a first segmentation result corresponding to the first sample medical image; Using a second generator in a second generative adversarial network, obtaining second sample optical flow motion information from the first reconstructed medical image to the second sample medical image; Using a second reconstruction module in a second generative adversarial network, reconstruct an image based on the second sample optical flow motion information to obtain a second reconstructed medical image and a second segmentation result corresponding to the second sample medical image; The first generative adversarial network is trained according to the first reconstructed medical image, the first segmentation result, the second reconstructed medical image, and the second segmentation result to obtain a trained first generative adversarial network.

2. The medical image segmentation method according to claim 1, characterized in that: Segmenting the target object in the first medical image according to the optical flow motion information and the marked target object region to obtain a segmentation result of the first medical image, including: Reconstructing the target object region according to the optical flow motion information and the marked target object region to obtain a reconstructed target object region; The reconstructed target object region is used as the segmentation result of the first medical image.

3. The medical image segmentation method according to claim 2, characterized in that: Reconstructing the target object region according to the optical flow motion information and the marked target object region includes: Performing trilinear interpolation processing on the optical flow motion information to obtain processed optical flow motion information; Based on the processed optical flow motion information and the marked target object region, the target object region is reconstructed to obtain a reconstructed target object region.

4. The medical image segmentation method according to claim 1, characterized in that: Acquiring optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information includes: Associating the first feature information with the second feature information to obtain pixel position correspondence information between the second medical image and the first medical image; An upsampling operation is performed on the pixel position correspondence information to obtain optical flow motion information from the second medical image to the first medical image.

5. The medical image segmentation method according to claim 4, characterized in that: Associating the first feature information with the second feature information to obtain pixel position correspondence information between the second medical image and the first medical image includes: Performing a convolution operation on the first feature information of the first medical image to obtain first convolution feature information; Performing a convolution operation on the second feature information of the second medical image to obtain second convolution feature information; Fusing the first convolution feature information and the second convolution feature information to obtain first fused feature information; The first fusion feature information and the first convolution feature information are fused with each other to obtain pixel position correspondence information between the second medical image and the first medical image.

6. The medical image segmentation method according to claim 1, characterized in that: The first generative adversarial network is trained according to the first reconstructed medical image, the first segmentation result, the second reconstructed medical image, and the second segmentation result to obtain a trained first generative adversarial network, including: Obtaining a reconstruction consistency loss in an image reconstruction space according to the first reconstructed medical image and the second reconstructed medical image; Obtaining a segmentation consistency loss in the segmentation space according to the first segmentation result, the second segmentation result, and a target object labeling result of the second sample medical image; The first generative adversarial network is trained according to the reconstruction consistency loss and the segmentation consistency loss to obtain a trained first generative adversarial network.

7. The medical image segmentation method according to claim 6, characterized in that: The reconstruction consistency loss includes: image reconstruction consistency loss and trajectory consistency loss; According to the first reconstructed medical image and the second reconstructed medical image, a reconstruction consistency loss in the image reconstruction space is obtained, including: According to the second reconstructed medical image and the second sample medical image, obtaining an image reconstruction consistency loss that is consistent with the second sample medical image and the second reconstructed medical image; According to the first sample optical flow motion information and the second sample optical flow motion information, a trajectory consistency loss is obtained in which the first sample optical flow motion information and the second sample optical flow motion information present a consistent trajectory.

8. The medical image segmentation method according to claim 6, characterized in that: The segmentation consistency loss includes image segmentation consistency loss, mask value loss, and semantic consistency loss; According to the first segmentation result, the second segmentation result, and the target object labeling result of the second sample medical image, the segmentation consistency loss in the segmentation space is obtained, including: According to the second segmentation result and the target object labeling result, obtaining an image segmentation consistency loss in which the target object labeling result is consistent with the second segmentation result; According to the first segmentation result, the second segmentation result and a preset segmentation mask value, respectively obtaining mask value losses of the first segmentation result and the second segmentation result relative to the preset segmentation mask value; Based on the first segmentation result, a semantic consistency loss is constructed to maintain semantic consistency of the first segmentation result on the image.

9. The medical image segmentation method according to claim 8, characterized in that: Based on the first segmentation result, a semantic consistency loss is constructed to maintain semantic consistency of the first segmentation result on the image, including: Derivative the pixel values ​​in the first segmentation result along a preset direction to obtain a derivative result in the preset direction; According to the derivation result in the preset direction and the first segmentation result, a semantic consistency loss is constructed to maintain semantic consistency of the first segmentation result on the image.

10. The medical image segmentation method according to claim 7, characterized in that: According to the first sample optical flow motion information and the second sample optical flow motion information, obtaining a trajectory consistency loss in which the first sample optical flow motion information and the second sample optical flow motion information present a consistent trajectory, including: Generate an original medical image; According to the first pixel position mapping relationship information and the second sample optical flow motion information, the reference medical image is transformed to obtain a transformed medical image; Obtaining a transformation loss between the transformed medical image and the original medical image; The transformation loss is used as a trajectory consistency loss in which the first sample optical flow motion information and the second sample optical flow motion information present a consistent trajectory.

11. The medical image segmentation method according to claim 6, characterized in that: The method further comprises: Using a discriminator in a first generative adversarial network to discriminate the sample medical image and the first reconstructed medical image to obtain a discrimination result; Obtaining the adversarial loss of the first generative adversarial network according to the first reconstructed medical image and the discrimination result; The first generative adversarial network is trained according to the reconstruction consistency loss and the segmentation consistency loss to obtain a trained first generative adversarial network, including: The first generative adversarial network is trained according to the adversarial loss, the reconstruction consistency loss, and the segmentation consistency loss to obtain a trained first generative adversarial network.

12. A medical image segmentation device, characterized in that: include: An acquisition unit, used for acquiring a first medical image and a second medical image with a target object area marked; A feature extraction unit, used to extract features from the first medical image and the second medical image respectively, to obtain first feature information of the first medical image and second feature information of the second medical image; an optical flow unit, configured to obtain optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information; a segmentation unit, configured to use the first generator in the trained first generative adversarial network to obtain optical flow motion information from the second medical image to the first medical image according to the first feature information and the second feature information; The medical impact segmentation device further comprises: A sample acquisition unit, used to acquire a first sample medical image and a second sample medical image with a target object area marked; A first sample optical flow unit, configured to obtain first sample optical flow motion information from the second sample medical image to the first sample medical image by using a first generator in a first generative adversarial network; A first reconstruction unit, configured to use the first reconstruction module in the first generative adversarial network to perform image reconstruction based on the first sample optical flow motion information, and obtain a first reconstructed medical image and a first segmentation result corresponding to the first sample medical image; A second sample optical flow unit, configured to obtain second sample optical flow motion information from the first reconstructed medical image to the second sample medical image by using a second generator in a second generative adversarial network; A second reconstruction unit is used to use a second reconstruction module in a second generative adversarial network to perform image reconstruction based on the second sample optical flow motion information, and obtain a second reconstructed medical image and a second segmentation result corresponding to the second sample medical image; A training unit is used to train the first generative adversarial network according to the first reconstructed medical image, the first segmentation result, the second reconstructed medical image, and the second segmentation result to obtain a trained first generative adversarial network.

13. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 11 are implemented.

Images