Image segmentation method, device and storage medium

Abstract

The present invention provides an image segmentation method, device and storage medium, the method comprising: obtaining a pre-trained image segmentation model, the image segmentation model comprising a first network branch, a second network branch and a spatial transformation network layer; the output of the first network branch and the output of the second network branch are input into the spatial transformation network layer; the first network branch generates a first deformation field corresponding to the first image data; the second network branch generates a second deformation field corresponding to the second image data; the deformation directions of the first deformation field and the second deformation field are opposite; the spatial transformation network layer is used to generate a segmentation mask corresponding to the first image data based on the first image data, the second image data, the first deformation field and the second deformation field; the first image data and the second image data are input into the image segmentation model to obtain a segmented image. The present invention can solve the problem that the current mainstream deep learning method has poor multi-target segmentation effect in the multi-target segmentation task of complex images.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to an image segmentation method, device and storage medium. Background Art

[0002] The principle of automatic multi-target image segmentation is to classify multiple objects with semantic labels voxel by voxel (pixel for two-dimensional images), that is, to classify all voxels of the image using a set of object categories to segment and describe multiple objects of interest in the image. In the medical field, image segmentation refers to the extraction of voxels of organs or lesions from medical images such as CT or MRI. This technology can help doctors diagnose diseases more accurately and develop more effective treatment plans, and it is also one of the most challenging tasks in medical image analysis. In medical image analysis, the accuracy and reliability of image segmentation are crucial, which is directly related to subsequent clinical decisions and treatment effects, and even higher-level tasks such as object classification. Therefore, automatic multi-target image segmentation is the most basic and critical process to help describe, characterize and visualize regions of interest in medical images.

[0003] Currently, the multi-target segmentation method of medical images includes: training a segmentation model in the form of deep learning, and using the segmentation model to segment the medical image.

[0004] However, the current mainstream deep learning methods have good performance in single-target segmentation, but have poor multi-target segmentation effects in multi-target segmentation tasks of complex images. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide an image segmentation method, device and storage medium to eliminate or improve one or more defects in the prior art. The current mainstream deep learning method has good performance in single-target segmentation, but has poor multi-target segmentation effect in multi-target segmentation tasks of complex images.

[0006] One aspect of the present invention provides an image segmentation method, the method comprising the following steps:

[0007] Acquire first image data and second image data, wherein the first image data refers to image data obtained by magnetic resonance imaging, and the second image data refers to reference image data corresponding to the first image data; the second image data includes a segmentation mask of at least one target area in the second image data;

[0008] Obtain a pre-trained image segmentation model, the image segmentation model includes a first network branch, a second network branch and a spatial transformation network layer; the output of the first network branch and the output of the second network branch are input into the spatial transformation network layer; the first network branch is used to generate a first deformation field corresponding to the first image data; the second network branch is used to generate a second deformation field corresponding to the second image data; the deformation directions of the first deformation field and the second deformation field are opposite; the spatial transformation network layer is used to generate a segmentation mask corresponding to the first image data based on the first image data, the second image data, the first deformation field and the second deformation field;

[0009] The first image data and the second image data are input into an image segmentation model to obtain a segmented image.

[0010] Optionally, the first network branch includes a first encoder, a decoder and a first deformation field module; the second network branch includes a second encoder, a decoder and a second deformation field module;

[0011] The first encoder is used to generate a first feature map corresponding to the first image data; the second encoder is used to generate a second feature map corresponding to the second image data;

[0012] The decoder is used to generate a third feature map based on the first feature map and the second feature map;

[0013] The first deformation field module is used to generate a first deformation field based on the third feature map, the first image data and the second image data;

[0014] The second deformation field module is used to generate a second deformation field based on the third feature map, the first image data and the second image data.

[0015] Optionally, the weights of the first encoder are the same as the weights of the second encoder.

[0016] Optionally, the decoder is used to recursively use feature maps of high-level semantic information to extract feature maps with low-level detail information in the form of skip-layer connections until a third feature map with the same resolution as the first image data and the second image data is obtained.

[0017] Optionally, obtain a pre-trained image segmentation model, including:

[0018] Acquire training data, the training data including: sample magnetic resonance image data, and sample reference image data corresponding to the sample magnetic resonance image data;

[0019] Obtaining a pre-created initial segmentation model; the initial segmentation model has the same model structure as the image segmentation model; the initial segmentation model includes a first initial network branch and a second initial network branch;

[0020] Inputting sample magnetic resonance image data into a first initial network branch, inputting sample reference image data into a second initial network branch, and obtaining first training results output by a first encoder in the first initial network branch and a second encoder in the second initial network branch respectively;

[0021] Inputting the first training result into a preset loss function to obtain a first loss function value;

[0022] Iteratively train the first encoder in the first initial network branch and the second encoder in the second initial network branch using the first loss function value to obtain a first intermediate network branch and a second intermediate network branch;

[0023] Inputting sample magnetic resonance image data into a first intermediate network branch, inputting sample reference image data into a second intermediate network branch, and outputting second training results respectively by a first deformation field module in the first intermediate network branch, a second deformation field module in the second intermediate network branch, and a spatial transformation network layer;

[0024] Inputting the second training result, the sample reference image data and the sample magnetic resonance image data into a preset loss function to obtain a second loss function value;

[0025] Iteratively train the first deformation field module in the first intermediate network branch and the second deformation field module in the second intermediate network branch using the second loss function value to obtain a first iterative network branch and a second iterative network branch;

[0026] Inputting sample magnetic resonance image data into the first iterative network branch and inputting sample reference image data into the second iterative network branch, respectively obtaining third training results output by the first deformation field module in the first iterative network branch and the second deformation field module in the second iterative network branch;

[0027] Inputting the third training result and the gradient corresponding to the third training result into a preset loss function to obtain a third loss function value;

[0028] The third loss function value is used to iteratively train the first deformation field module in the first iterative network branch and the second deformation field module in the second iterative network branch to obtain the first network branch and the second network branch.

[0029] Optionally, the loss function includes a first loss function, a second loss function and a third loss function;

[0030] The first loss function is used to amplify the similarity between the first training sub-result and the second training sub-structure in the first training result, and to reduce the difference between the first training sub-result and the second training sub-result; the first training sub-result is a feature sequence output by the first encoder in the first initial network branch; the second training sub-result is a feature sequence output by the second encoder in the first initial network branch;

[0031] The second loss function is used to reduce the appearance difference between the sample reference image data, the sample magnetic resonance image data and the second training result; the second training result includes a registered image corresponding to the sample magnetic resonance image data and a registered image corresponding to the sample reference image data;

[0032] The third loss function is used to reduce the difference between adjacent positions in the third training result; the third training result includes the deformation field output by the first deformation field module in the first iterative network branch and the deformation field output by the second deformation field module in the second iterative network branch.

[0033] Optionally, the spatial transformation network layer includes a grid generator and a sampler;

[0034] The grid generator is used to divide the first image data into a plurality of grid units, and determine the position of each grid unit in the second image data according to the first deformation field; divide the second image data into a plurality of grid units, and determine the position of each grid unit in the first image data according to the second deformation field;

[0035] The sampler is used to perform linear interpolation on a number of grid cells corresponding to the first image data to generate a first registration image corresponding to the first image data; perform linear interpolation on the grid cells corresponding to the segmentation mask in the second image data to generate a segmentation mask corresponding to the first image data and apply it to the first image data to obtain a segmented image.

[0036] Optionally, performing linear interpolation on a plurality of grid units corresponding to the first image data includes: performing linear interpolation on values ​​of eight adjacent voxels of any voxel in the plurality of grid units; and each grid unit includes at least one voxel.

[0037] Another aspect of the present invention provides an image segmentation device, which includes: a processor and a memory, characterized in that computer instructions are stored in the memory, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the above-mentioned image segmentation method.

[0038] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the above-mentioned image segmentation method are implemented.

[0039] The image segmentation method, device and storage medium of the present invention can input the first image data and the second image data into the image segmentation model to obtain a segmented image; it can solve the problem that the current mainstream deep learning method has poor multi-target segmentation effect in the multi-target segmentation task of complex images; firstly, the first image data to be registered and the second image data as the reference image are input into the first network branch and the second network branch as a registration pair, and the feature maps of the two image data are extracted and integrated respectively, and the output deformation field is calculated; finally, through the spatial transformation network, the segmentation mask corresponding to at least one target area in the second image data is aligned to the spatial coordinate system of the first image data, and the segmentation mask corresponding to at least one target area in the first image data is obtained in an unsupervised manner, and applied to the first image data. In this way, multiple target areas of interest can be segmented from the first image data, which can improve the effect of multi-target segmentation.

[0040] Furthermore, by combining the image-level registration system and feature-level contrastive learning, contrastive learning is embedded into the unsupervised registration and segmentation architecture, and labeled benchmark datasets and visualization results are used to evaluate the segmentation accuracy and generalization ability of the model. The introduction of contrastive learning can greatly improve the performance of the unsupervised registration and segmentation system, and at the same time, narrow the performance gap between supervised and unsupervised segmentation.

[0041] Furthermore, using cross-correlation loss instead of the mean square error loss commonly used in existing medical image segmentation methods as the unsupervised reconstruction loss function can improve the unsupervised segmentation performance.

[0042] Furthermore, by training the model with reference images, the image segmentation model can be trained without a labeled data set, avoiding the large amount of time and cost to obtain high-quality labeled data and reducing the difficulty of training the image segmentation model.

[0043] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0044] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0046] Figure 1 A flowchart of an image segmentation method provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the structure of an image segmentation model provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the structure of a space transformation network layer provided by an embodiment of the present invention;

[0049] Figure 4 A flowchart of an image segmentation model training method provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0051] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0052] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0053] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0054] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0055] The following is an introduction to the image segmentation method provided by this application.

[0056] Optionally, the executor of the image segmentation method provided in the present application is an electronic device, which may be a terminal such as a computer, a mobile phone, a tablet computer, a camera, or a server. This embodiment does not limit the implementation method of the electronic device.

[0057] This embodiment provides an image segmentation method, such as Figure 1 As shown, the method comprises at least steps S101 to S103:

[0058] Step S101, obtaining first image data and second image data, wherein the first image data refers to image data obtained by magnetic resonance imaging, and the second image data refers to reference image data corresponding to the first image data; the second image data includes a segmentation mask of at least one target area in the second image data, wherein the target area includes but is not limited to the hippocampus area, the lateral ventricle area, or the caudate nucleus area.

[0059] In this embodiment, the first image data is an image file acquired by magnetic resonance imaging (MRI) technology, including but not limited to an image file in NifTI format or an image file in MINC format.

[0060] Magnetic resonance imaging has become the preferred method for exploring changes in the internal structure of the brain due to its non-invasiveness and ability to provide high-resolution images. It has extremely important clinical application value for the early identification and prognosis of neurological and brain-related diseases. Common brain diseases are usually accompanied by morphological changes in local or multiple brain structures during their progression. For example, the hippocampus, temporal lobe, and frontal lobe of patients with Alzheimer's Disease (AD) will shrink to a greater extent than those of normally aged elderly people. Structural magnetic resonance imaging is often used clinically to observe the internal tissue structure of the brain. Therefore, segmenting these areas of interest (such as the hippocampus) from magnetic resonance images and quantitatively observing them is helpful to assist doctors in clinical diagnosis.

[0061] The second image data and the first image data are image data of magnetic resonance imaging of the same part, and the second image data includes a segmentation mask.

[0062] The second image data and the first image data may be image data of magnetic resonance imaging of the same part of the same target, or may be image data of magnetic resonance imaging of the same part of different targets.

[0063] For example: when the targets include target A and target B, the first image data may be image data obtained by magnetic resonance imaging of the brain of target A, and the second image data may be image data obtained by magnetic resonance imaging of the brain of target A; or, the first image data may be image data obtained by magnetic resonance imaging of the brain of target A, and the second image data may be image data obtained by magnetic resonance imaging of the brain of target B.

[0064] In this embodiment, the number of second image data corresponding to different parts is one copy. In actual implementation, the number of second image data may be adjusted to at least two copies according to actual conditions.

[0065] Step S102, obtaining a pre-trained image segmentation model, the image segmentation model includes a first network branch, a second network branch and a spatial transformation network layer; the output of the first network branch and the output of the second network branch are input into the spatial transformation network layer; the first network branch is used to generate a first deformation field corresponding to the first image data; the second network branch is used to generate a second deformation field corresponding to the second image data; the deformation directions of the first deformation field and the second deformation field are opposite; the spatial transformation network layer is used to generate a segmentation mask corresponding to the first image data based on the first image data, the second image data, the first deformation field and the second deformation field.

[0066] refer to Figure 2 In the image segmentation model shown, the input of the first network branch 21 is the first image data, so that the network branch 21 generates a first deformation field corresponding to the first image data based on the first image data; the input of the second network branch 22 is the second image data, so that the second network branch 22 generates a second deformation field corresponding to the second image data based on the second image data. The spatial transformation network layer 23 uses the first image data, the second image data, the first deformation field and the second deformation field to generate a segmentation mask corresponding to the first image data.

[0067] like Figure 2 As shown, the first network branch 21 includes a first encoder, a decoder and a first deformation field module; the second network branch 22 includes a second encoder, a decoder and a second deformation field module. The first encoder is used to generate a first feature map corresponding to the first image data; the second encoder is used to generate a second feature map corresponding to the second image data; the decoder is used to generate a third feature map based on the first feature map and the second feature map; the first deformation field module is used to generate a first deformation field based on the third feature map, the first image data and the second image data; the second deformation field module is used to generate a second deformation field based on the third feature map, the first image data and the second image data.

[0068] In this embodiment, the weight of the first encoder is the same as the weight of the second encoder. The first encoder and the second encoder each include a three-dimensional convolutional neural network (3D CNN).

[0069] Among them, the first encoder and the second encoder are the structures of autoencoder (AE), which is a generative model. In actual implementation, they can be replaced by other generative models, such as variational autoencoder (VAE), generative adversarial network (GAN), etc.

[0070] The first image data and the second image data are input as a registration pair into two three-dimensional convolutional neural networks with the same weights to generate a first feature map corresponding to the first image data and a second feature map corresponding to the second image data. The resolution of the first feature map and the second feature map is the same as the resolution of the input image. The first feature map and the second feature map are integrated through a decoder.

[0071] Specifically, the first feature map and the second feature map are concatenated and then input into the decoder, which is used to recursively use the feature map of high-level semantic information (low resolution) to extract the feature map with low-level detail information (high resolution) in the form of skip-layer connection until a third feature map with the same resolution as the first image data and the second image data is obtained.

[0072] The decoder uses the high-level semantic information saved in the encoder stage and combines it with other lower-level feature maps to reconstruct the details of the input data. This structure can help the 3D convolutional neural network better learn and retain the detailed features of the input image data, thereby improving the performance of the image segmentation model in image segmentation tasks.

[0073] In this embodiment, the first feature map corresponding to the first image data and the second feature map corresponding to the second image data are decoded to obtain a third feature map. The third feature map is input into the deformation field calculation module, and the deformation field calculation module calculates the target transformation parameter. The target transformation parameter is applied to the first image data to obtain the first deformation field; the target transformation parameter is applied to the second image data to obtain the second deformation field. Among them, the deformation field calculation module is a localization network, which is used to receive the input feature map and regress the target transformation parameter φ.

[0074] For example: taking the first image data as image x and the second image data as image y as an example, after image x is input into the first encoder, feature map A is output; after image y is input into the second encoder, feature map B is output; after concatenating feature map A and feature map B, feature map AB is obtained; feature map AB is input into the decoder and feature map C is output; feature map C is input into the deformation field calculation module to obtain the target transformation parameter φ; the target transformation parameter φ is applied to image x and image y respectively to obtain deformation field φ_xy and deformation field φ_yx.

[0075] The deformation field can be viewed as a non-rigid deformation of the image, where each voxel point corresponds to a new position. By applying the deformation field, voxels from one image can be mapped to corresponding positions in another image, thereby achieving image registration or deformation.

[0076] The first deformation field is used to deform the first image data, convert the first image data into the second image data, realize the registration of the first image data, and obtain the registered image corresponding to the first image data. The first image data and the second image data are aligned so that they achieve the best match in space. Correspondingly, the second deformation field is used to deform the second image data and obtain the registered image corresponding to the second image data.

[0077] In this embodiment, the registration image corresponding to the second image data is used to determine the difference between the registration image corresponding to the first image data during the training process, and to learn the optimal parameter value and optimize the deformation field calculation module by minimizing the difference between the registration image corresponding to the second image data and the registration image corresponding to the first image data.

[0078] In this embodiment, Figure 2 As shown, the deformation field calculation module includes a first deformation field calculation module and a second deformation field calculation module, and the first deformation field calculation module and the second deformation field calculation module are the same function. Through the calculation of the deformation field calculation module, deformation fields in two directions can be obtained, including the first deformation field and the second deformation field, which can play an important role in tasks such as image registration and image reconstruction, and realize accurate matching and deformation of images.

[0079] After the deformation field calculation module outputs the first deformation field and the second deformation field, the first deformation field and the second deformation field are input into the space transformation network layer 23. Figure 3 As shown, the spatial transformation network layer 23 includes a grid generator and a sampler. The grid generator is used to apply the deformation field to the input image. The sampler is used as an interpolator to construct the final output image.

[0080] The grid generator is used to divide the first image data into a plurality of grid units, and determine the position of each grid unit in the second image data according to the first deformation field; divide the second image data into a plurality of grid units, and determine the position of each grid unit in the first image data according to the second deformation field.

[0081] The sampler is used to perform linear interpolation on a number of grid cells corresponding to the first image data to generate a first registration image corresponding to the first image data; perform linear interpolation on the grid cells corresponding to the segmentation mask in the second image data to generate a segmentation mask corresponding to the first image data and apply it to the first image data to obtain a segmented image.

[0082] The linear interpolation is performed on a plurality of grid units corresponding to the first image data, including: in the plurality of grid units, linear interpolation is performed on the values ​​of eight adjacent voxels of any voxel; each grid unit includes at least one voxel.

[0083] Correspondingly, linear interpolation is performed on the grid cells corresponding to the segmentation mask in the second image data, and linear interpolation is performed on the values ​​of eight adjacent voxels of any voxel in the plurality of grid cells corresponding to the segmentation mask.

[0084] In this embodiment, in order to implement a standard gradient-based method, in this embodiment, a differentiable operation based on a spatial transformation network layer is used to calculate an image after the first image data and the second image data are registered.

[0085] For each voxel, this embodiment calculates the position of (sub-voxel) in the first image data, which is expressed by the following formula:

[0086] p'=p+u(p)

[0087] Wherein, p is the voxel in the first image data; u(p) represents the first deformation field; and p' represents the voxel after the first deformation field acts on the first image data.

[0088] Since image values ​​are defined only in the integer range, the image segmentation model linearly interpolates the values ​​of the eight neighboring voxels, expressed as follows:

[0089]

[0090] Where m represents the first image data; p' represents the voxel after the first deformation field acts on the first image data; Z(p') is the adjacent voxel of p'; d is the voxel in the three-dimensional Euclidean space (R 3 ) iteration; p' d represents the (x, y, z) value of the spatial three-dimensional coordinates of voxel p'; q represents any voxel in the adjacent voxel Z(p') of p', q d The (x, y, z) value representing the three-dimensional coordinates of voxel q.

[0091] The introduction of the spatial transformer network layer 23 makes the gradient computable, so that the error can be back-propagated during the optimization process.

[0092] Step S103: input the first image data and the second image data into an image segmentation model to obtain a segmented image.

[0093] Specifically, the first image data is input into the first network branch 21, and the second image is input into the second network branch 22, and after calculation by the image segmentation model, a segmented image corresponding to the first image data is obtained.

[0094] refer to Figure 4 This embodiment provides a method for training an image segmentation model, the method at least comprising steps S401 to S411:

[0095] Step S401 : acquiring training data, where the training data includes: sample magnetic resonance image data and sample reference image data corresponding to the sample magnetic resonance image data.

[0096] Optionally, acquiring training data includes: acquiring magnetic resonance image data of the same part of different target objects as sample magnetic resonance image data; and determining a piece of magnetic resonance image data from the sample magnetic resonance image data as sample reference image data.

[0097] Herein, a piece of image data may be randomly selected from the sample magnetic resonance image data; or, image data with the highest image quality may be screened out from the sample magnetic resonance image data.

[0098] Step S402: obtaining a pre-created initial segmentation model.

[0099] The initial segmentation model has the same model structure as the image segmentation model; the initial segmentation model includes a first initial network branch and a second initial network branch.

[0100] Step S403, inputting the sample magnetic resonance image data into the first initial network branch, inputting the sample reference image data into the second initial network branch, and obtaining first training results output by the first encoder in the first initial network branch and the second encoder in the second initial network branch respectively.

[0101] Step S404: input the first training result into a preset loss function to obtain a first loss function value.

[0102] Optionally, the loss function includes a first loss function, a second loss function and a third loss function. The first loss function is a contrast loss function; the second loss function is a reconstruction loss function; and the third loss function is a smoothing loss function. Specifically, the loss function can be expressed by the following formula:

[0103] L total =L recon +αL smooth +βL contrast

[0104] Where, L total represents the loss function; L recon represents the reconstruction loss function; L smooth represents the smooth loss function; L contrast represents the contrast loss function; α and β are the balance L recon , L smooth and L contrast The hyperparameters are set to 1 and 0.01.

[0105] In this embodiment, the first loss function is used to amplify the similarity between the first training sub-result and the second training sub-result in the first training result, and to reduce the difference between the first training sub-result and the second training sub-result.

[0106] The first training sub-result is a feature sequence output by the first encoder in the first initial network branch; and the second training sub-result is a feature sequence output by the second encoder in the first initial network branch.

[0107] In this embodiment, the first loss function is formally, given a set of images, the first image data and the second image data are regarded as enhanced image pairs, and other images in the set of images are regarded as negative samples. Specifically, the first loss function can be expressed by the following formula:

[0108]

[0109] Where, f(x) represents the first feature sequence corresponding to the first image data; f(y) represents the second feature sequence corresponding to the second image data; f(i) represents the feature sequence of other images in a given set of images; i≠x ∈{0,1} is an identifier that takes the value 1 when i≠x; τ is a hyperparameter; sim(f(x),f(y)) represents the cosine similarity between f(x) and f(y), which can be expressed as follows:

[0110]

[0111] Accordingly, sim(f(x),f(i)) is the cosine similarity between f(x) and f(i), which can be expressed as follows:

[0112]

[0113] In this embodiment, the first encoder is connected to the decoder via a projection layer, and the second encoder is connected to the decoder via another projection layer. The two projection layers are used to calculate the contrast loss to maximize the consistency between the image and the enhanced view, thereby improving the unsupervised performance.

[0114] The second loss function is used to reduce the appearance difference between the sample reference image data, the sample magnetic resonance image data and the second training result; accordingly, the second training result includes the registration image corresponding to the sample magnetic resonance image data and the registration image corresponding to the sample reference image data.

[0115] For the second loss function, in this embodiment, in order to improve the unsupervised performance, the cross-correlation function (Cross-Correlation loss) is used for experiments. This function is more suitable for processing the registration problem between different scanners and different data sets and has better robustness. and It represents the image obtained by averaging the grayscale values ​​of the voxels in a certain area around each voxel in the image. f and The local cross-correlation of is recorded as:

[0116]

[0117] Wherein, m represents the first image data, f represents the second image data (i.e., the reference image), It means that the deformation field φ is applied to the first image data; and represents the image obtained by averaging the grayscale values ​​of voxels in a certain area around each voxel in the image; p represents a voxel; p i represents the i-th voxel (where i = 1, 2, ... n), n is the number of voxels in the image; Ω∈R 3 , which represents the three-dimensional Euclidean space where n voxels are located.

[0118] The higher the local cross-correlation (CC) value, the better the alignment effect. Based on the above CC calculation formula, the second loss function is expressed as:

[0119]

[0120] In the formula, represents f and The local cross-correlation of; f represents the second image data (ie, the reference image); It means that the deformation field φ is applied to the first image data.

[0121] In addition, in this embodiment, using cross-correlation loss (Cross-Correlation Loss) as the reconstruction loss of the image segmentation model can achieve the best effect. In actual implementation, other commonly used reconstruction loss functions can be selected as a substitute; for example, mean square error loss (Mean Square Error Loss), cross entropy loss (Cross-Entropy Loss), etc.

[0122] Step S405: Iteratively train the first encoder in the first initial network branch and the second encoder in the second initial network branch using the first loss function value to obtain a first intermediate network branch and a second intermediate network branch.

[0123] Step S406, input the sample magnetic resonance image data into the first intermediate network branch, input the sample reference image data into the second intermediate network branch, and the first deformation field module in the first intermediate network branch, the second deformation field module in the second intermediate network branch and the spatial transformation network layer respectively output the second training result.

[0124] Step S407: input the second training result, the sample reference image data and the sample magnetic resonance image data into a preset loss function to obtain a second loss function value.

[0125] Step S408: Use the second loss function value to iteratively train the first deformation field module in the first intermediate network branch and the second deformation field module in the second intermediate network branch to obtain a first iterative network branch and a second iterative network branch.

[0126] Step S409, inputting the sample magnetic resonance image data into the first iterative network branch, and inputting the sample reference image data into the second iterative network branch, to obtain the third training results output by the first deformation field module in the first iterative network branch and the second deformation field module in the second iterative network branch respectively.

[0127] In this embodiment, the third loss function is used to reduce the difference between adjacent positions in the third training result; the third training result includes the deformation field output by the first deformation field module in the first iterative network branch, and the deformation field output by the second deformation field module in the second iterative network branch.

[0128] Minimizing the third loss function will make Aligned with f, but may produce a physically unrealistic non-smooth deformation field φ. Therefore, a smooth loss function is introduced to act on the deformation field to ensure good registration. The third loss function is expressed by the following formula:

[0129]

[0130] Where p represents a voxel; φ represents the deformation field; Used to represent the gradient of voxel p, Ω∈R 3 , which represents the three-dimensional Euclidean space where n voxels are located.

[0131] Step S410: input the third training result and the gradient corresponding to the third training result into a preset loss function to obtain a third loss function value.

[0132] Step S411, using the third loss function value to iteratively train the first deformation field module in the first iterative network branch and the second deformation field module in the second iterative network branch to obtain the first network branch and the second network branch.

[0133] In this embodiment, the image segmentation model embeds the contrastive learning mechanism into the registration system to extract feature maps with richer information, thereby improving the performance of unsupervised segmentation. Specifically, this embodiment follows the four component principles of the contrastive learning process. The first part is the input registration pair, in which the unregistered first image data and the second image data used as a reference are regarded as images sampled from different enhanced views. The second part is two weight-shared CNN encoders. In this way, it can be ensured that the CNN-based encoder can extract unified CNN features from the unregistered first image data and the second image data used as a reference. The third part uses a fully connected layer as a projection layer to map the CNN features to the latent space. Finally, according to the standard definition of the contrastive learning process, the fourth component is the contrast loss.

[0134] In summary, the image segmentation method provided in this embodiment inputs the first image data and the second image data into the image segmentation model to obtain a segmented image; it can solve the problem that the current mainstream deep learning method has poor multi-target segmentation effect in the multi-target segmentation task of complex images; first, the first image data to be registered and the second image data as the reference image are input as a registration pair into the first network branch and the second network branch, and the feature maps of the two image data are extracted and integrated respectively, and the output deformation field is calculated; finally, through the spatial transformation network, the segmentation mask corresponding to at least one target area in the second image data is aligned to the spatial coordinate system of the first image data, and the segmentation mask corresponding to at least one target area in the first image data is obtained in an unsupervised manner, and applied to the first image data. In this way, multiple target areas of interest can be segmented from the first image data, which can improve the effect of multi-target segmentation.

[0135] Furthermore, by combining the image-level registration system and feature-level contrastive learning, contrastive learning is embedded into the unsupervised registration and segmentation architecture, and labeled benchmark datasets and visualization results are used to evaluate the segmentation accuracy and generalization ability of the model. The introduction of contrastive learning can greatly improve the performance of the unsupervised registration and segmentation system, and at the same time, narrow the performance gap between supervised and unsupervised segmentation.

[0136] Furthermore, using cross-correlation loss instead of the mean square error loss commonly used in existing medical image segmentation methods as the unsupervised reconstruction loss function can improve the unsupervised segmentation performance.

[0137] Furthermore, by training the model with reference images, the image segmentation model can be trained without a labeled data set, avoiding the large amount of time and cost to obtain high-quality labeled data and reducing the difficulty of training the image segmentation model.

[0138] Corresponding to the above method, the present invention also provides an image segmentation method device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the image segmentation method as described above.

[0139] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned image segmentation method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0140] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0141] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0142] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0143] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image segmentation method, characterized in that: The method comprises the following steps: Acquire first image data and second image data, wherein the first image data refers to image data obtained by magnetic resonance imaging, and the second image data refers to reference image data corresponding to the first image data; the second image data includes a segmentation mask of at least one target area in the second image data; A pre-trained image segmentation model is obtained, wherein the image segmentation model includes a first network branch, a second network branch and a spatial transformation network layer; the output of the first network branch and the output of the second network branch are input into the spatial transformation network layer; the first network branch is used to generate a first deformation field corresponding to the first image data; the second network branch is used to generate a second deformation field corresponding to the second image data; the deformation direction of the first deformation field is opposite to that of the second deformation field; the spatial transformation network layer is used to generate a segmentation mask corresponding to the first image data based on the first image data, the second image data, the first deformation field and the second deformation field; the spatial transformation network layer includes a network A grid generator and a sampler; the grid generator is used to divide the first image data into a plurality of grid units, and determine the position of each grid unit in the second image data according to the first deformation field; divide the second image data into a plurality of grid units, and determine the position of each grid unit in the first image data according to the second deformation field; the sampler is used to perform linear interpolation on the plurality of grid units corresponding to the first image data to generate a first registration image corresponding to the first image data; perform linear interpolation on the grid units corresponding to the segmentation mask in the second image data to generate a segmentation mask corresponding to the first image data and apply it to the first image data to obtain a segmented image; The first image data and the second image data are input into the image segmentation model to obtain the segmented image.

2. The image segmentation method according to claim 1, characterized in that: The first network branch includes a first encoder, a decoder and a first deformation field module; the second network branch includes a second encoder, the decoder and a second deformation field module; The first encoder is used to generate a first feature map corresponding to the first image data; The second encoder is used to generate a second feature map corresponding to the second image data; The decoder is used to generate a third feature map based on the first feature map and the second feature map; The first deformation field module is used to generate the first deformation field based on the third feature map, the first image data and the second image data; The second deformation field module is used to generate the second deformation field based on the third feature map, the first image data and the second image data.

3. The image segmentation method according to claim 2, characterized in that: The weight of the first encoder is the same as the weight of the second encoder.

4. The image segmentation method according to claim 2, characterized in that: The decoder is used to recursively use the feature map of high-level semantic information to extract the feature map with low-level detail information in the form of skip-layer connection until the third feature map with the same resolution as the first image data and the second image data is obtained.

5. The image segmentation method according to claim 2, characterized in that: The obtaining of a pre-trained image segmentation model comprises: Acquire training data, the training data comprising: sample magnetic resonance image data, and sample reference image data corresponding to the sample magnetic resonance image data; Acquire a pre-created initial segmentation model; the initial segmentation model has the same model structure as the image segmentation model; the initial segmentation model includes a first initial network branch and a second initial network branch; Inputting the sample magnetic resonance image data into the first initial network branch, and inputting the sample reference image data into the second initial network branch, to obtain first training results output by a first encoder in the first initial network branch and a second encoder in the second initial network branch respectively; Inputting the first training result into a preset loss function to obtain a first loss function value; Iteratively train the first encoder in the first initial network branch and the second encoder in the second initial network branch using a first loss function value to obtain a first intermediate network branch and a second intermediate network branch; Input the sample magnetic resonance image data into the first intermediate network branch, input the sample reference image data into the second intermediate network branch, and the first deformation field module in the first intermediate network branch, the second deformation field module in the second intermediate network branch, and the spatial transformation network layer respectively output second training results; Inputting the second training result, the sample reference image data and the sample magnetic resonance image data into the preset loss function to obtain a second loss function value; Iteratively training the first deformation field module in the first intermediate network branch and the second deformation field module in the second intermediate network branch using the second loss function value to obtain a first iterative network branch and a second iterative network branch; Inputting the sample magnetic resonance image data into the first iterative network branch, and inputting the sample reference image data into the second iterative network branch, to obtain third training results output by the first deformation field module in the first iterative network branch and the second deformation field module in the second iterative network branch respectively; Inputting the third training result and the gradient corresponding to the third training result into the preset loss function to obtain a third loss function value; The first deformation field module in the first iterative network branch and the second deformation field module in the second iterative network branch are iteratively trained using the third loss function value to obtain the first network branch and the second network branch.

6. The image segmentation method according to claim 5, characterized in that: The loss function includes a first loss function, a second loss function and a third loss function; The first loss function is used to amplify the similarity between the first training sub-result and the second training sub-structure in the first training result, and to reduce the difference between the first training sub-result and the second training sub-result; the first training sub-result is a feature sequence output by the first encoder in the first initial network branch; the second training sub-result is a feature sequence output by the second encoder in the first initial network branch; The second loss function is used to reduce the appearance difference between the sample reference image data, the sample magnetic resonance image data and the second training result; the second training result includes a registered image corresponding to the sample magnetic resonance image data and a registered image corresponding to the sample reference image data; The third loss function is used to reduce the difference between adjacent positions in the third training result; the third training result includes the deformation field output by the first deformation field module in the first iterative network branch, and the deformation field output by the second deformation field module in the second iterative network branch.

7. The image segmentation method according to claim 1, characterized in that: The performing linear interpolation on a plurality of grid units corresponding to the first image data includes: performing linear interpolation on values ​​of eight adjacent voxels of any voxel in the plurality of grid units; and each grid unit includes at least one voxel.

8. An image segmentation device, comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the image segmentation method as claimed in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the image segmentation method according to any one of claims 1 to 7 are implemented.

Images