Image segmentation method, model training method, device, equipment and medium

Abstract

The present invention discloses an image segmentation method, a model training method, an apparatus, a device and a medium, wherein the image segmentation method comprises: sequentially acquiring initial magnetic resonance images under multiple sequences, wherein the initial magnetic resonance images include a lesion area to be segmented; performing a registration operation on the initial magnetic resonance images under a target sequence to obtain a magnetic resonance registration image; inputting the initial magnetic resonance image and the magnetic resonance registration image into a pre-trained lesion segmentation model to obtain an initial segmentation feature matrix, wherein the lesion segmentation model is obtained by training based on sample magnetic resonance images and lesion annotation images under multiple sequences; based on the initial segmentation feature matrix, determining a target lesion segmentation image of the lesion area to be segmented. The technical solution of the embodiment of the present invention achieves the effect of improving the lesion segmentation efficiency and segmentation accuracy by adding lesion features in the target direction and fusing the segmentation results.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, a model training method, a device, equipment and a medium. Background Art

[0002] With the popularization of medical imaging equipment and the increasing demand for brain imaging to assist in the diagnosis of brain diseases, the processing of brain images has become increasingly important. Automatic segmentation technology based on magnetic resonance imaging sequences can pre-label the lesion area of ​​brain images and reduce the workload of medical staff.

[0003] However, unlike the scientific research sequences used in various competition data, medical staff in real scenarios need to consider the joint labeling of multiple sequence information. In addition, compared with scientific research sequences, clinical sequences have the characteristics of strong anisotropy and unclear features, which may lead to problems such as the lesion area segmentation results not fully integrating the information of multiple sequences and poor segmentation effects. Summary of the invention

[0004] The present invention provides an image segmentation method, a model training method, a device, an equipment and a medium to achieve accurate segmentation of lesion feature areas in magnetic resonance images under multiple sequences.

[0005] According to one aspect of the present invention, there is provided an image segmentation method, the method comprising:

[0006] Sequentially acquiring initial magnetic resonance images under multiple sequences, wherein the initial magnetic resonance images include the lesion area to be segmented;

[0007] Performing a registration operation on the initial magnetic resonance image under the target sequence to obtain a magnetic resonance registration image;

[0008] Inputting the initial magnetic resonance image and the magnetic resonance registration image into a pre-trained lesion segmentation model to obtain an initial segmentation feature matrix, wherein the lesion segmentation model is trained based on sample magnetic resonance images and lesion annotated images under multiple sequences;

[0009] Based on the initial segmentation feature matrix, a target lesion segmentation image of the lesion area to be segmented is determined.

[0010] In a second aspect, an embodiment of the present invention further provides a model training method, the method comprising:

[0011] Acquire at least one sample magnetic resonance image under multiple sequences, and annotate each sample magnetic resonance image to obtain at least one lesion annotated image, so as to construct at least one set of training samples based on the sample magnetic resonance images and the lesion annotated images, wherein the training samples include a sample registration image corresponding to the sample magnetic resonance image under any of the sequences;

[0012] For each training sample, input the current training sample into the model to be trained to obtain at least one actual output image corresponding to the current training sample;

[0013] Determining a first model loss based on the lesion annotation image, the actual output image under any sequence, and the corresponding registered output image;

[0014] Determining a second model loss based on a target sequence output image corresponding to the registered output image and other sequence output images except the target sequence output image;

[0015] Based on the first model loss, the second model loss and the corresponding weights, a target model loss is determined, so as to adjust the model parameters of the training model based on the target model loss to obtain a lesion segmentation model;

[0016] The lesion segmentation model is used to segment the lesion area in the magnetic resonance images under multiple sequences to obtain a lesion segmentation image.

[0017] In a third aspect, an embodiment of the present invention further provides an image segmentation device, the device comprising:

[0018] An image acquisition module, used for sequentially acquiring initial magnetic resonance images under multiple sequences, wherein the initial magnetic resonance images include the lesion area to be segmented;

[0019] An image registration module is used to perform a registration operation on the initial magnetic resonance image under the target sequence to obtain a magnetic resonance registration image;

[0020] A feature matrix determination module is used to input the initial magnetic resonance image and the magnetic resonance registration image into a pre-trained lesion segmentation model to obtain an initial segmentation feature matrix, wherein the lesion segmentation model is trained based on sample magnetic resonance images and lesion annotation images under multiple sequences;

[0021] The lesion segmentation image determination module is used to determine the target lesion segmentation image of the lesion area to be segmented based on the initial segmentation feature matrix.

[0022] In a fourth aspect, an embodiment of the present invention further provides a model training device, the device comprising:

[0023] A training sample construction module, used for acquiring at least one sample magnetic resonance image under multiple sequences, and annotating each sample magnetic resonance image to obtain at least one lesion annotated image, so as to construct at least one set of training samples based on the sample magnetic resonance image and the lesion annotated image, wherein the training samples include a sample registration image corresponding to the sample magnetic resonance image under any sequence;

[0024] An actual output image determination module is used to input the current training sample into the model to be trained for each training sample, and obtain at least one actual output image corresponding to the current training sample;

[0025] A first model loss determination module, used to determine a first model loss based on the lesion annotation image, the actual output image under any sequence, and the corresponding registered output image;

[0026] A second model loss determination module, configured to determine a second model loss based on a target sequence output image corresponding to the registration output image and other sequence output images except the target sequence output image;

[0027] A lesion segmentation model determination module is used to determine a target model loss based on the first model loss, the second model loss and corresponding weights, so as to adjust the model parameters of the training model based on the target model loss to obtain a lesion segmentation model;

[0028] The lesion segmentation model is used to segment the lesion area in the magnetic resonance images under multiple sequences to obtain a lesion segmentation image.

[0029] In a fifth aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0030] at least one processor; and

[0031] a memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image segmentation method or model training method described in any embodiment of the present invention.

[0033] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image segmentation method or model training method described in any embodiment of the present invention when executed.

[0034] The technical solution of the embodiment of the present invention obtains the initial magnetic resonance images under multiple sequences and aligns the magnetic resonance images under the target sequence to obtain the magnetic resonance registration image. Furthermore, the initial magnetic resonance image and the magnetic resonance registration image are processed based on the lesion segmentation model to determine the target lesion segmentation image of the lesion area to be segmented. This solves the problem that in the prior art, compared with the scientific research sequence, the clinical sequence has the characteristics of strong anisotropy and unclear features, which may cause the lesion area segmentation result to fail to fully integrate the information of multiple sequences and the segmentation effect is poor. By increasing the lesion features in the target direction and fusing the segmentation results, the effect of improving the lesion segmentation efficiency and segmentation accuracy is achieved.

[0035] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 is a flowchart of an image segmentation method provided according to Embodiment 1 of the present invention;

[0038] Figure 2 is a flow chart of a model training method provided according to Embodiment 2 of the present invention;

[0039] Figure 3 is a schematic diagram of a model training method provided according to Embodiment 2 of the present invention;

[0040] Figure 4 is a structural schematic diagram of an image segmentation device provided according to Embodiment 3 of the present invention;

[0041] Figure 5 is a structural schematic diagram of a model training device provided according to Embodiment 4 of the present invention;

[0042] Figure 6 It is a structural schematic diagram of an electronic device for implementing the image segmentation method or model training method of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0045] Embodiment 1

[0046] Figure 1 A flowchart of an image segmentation method is provided for the first embodiment of the present invention. This embodiment is applicable to the case of segmenting the lesion area of ​​the magnetic resonance image of the clinical magnetic resonance sequence. The method can be performed by an image segmentation device. The image segmentation device can be implemented in the form of hardware and / or software. The image segmentation device can be configured in a terminal and / or a server. Figure 1 As shown, the method includes:

[0047] S110 , sequentially acquiring initial magnetic resonance images under multiple sequences.

[0048] Among them, multiple sequences can be understood as different sampling pulse sequences during the acquisition process of magnetic resonance images. Exemplarily, multiple sequences may include but are not limited to T1 weighted imaging (T1 Weighted Imaging, T1WI) sequence, T2 weighted imaging (T2 Weighted Imaging, T2WI) sequence, diffusion weighted imaging (Diffusion Weighted Imaging, DWI) sequence or fluid attenuated inversion recovery (Fluid Attenuated Inversion Recovery, FLAIR) sequence, etc. Each sampling pulse sequence corresponds to a different magnetic resonance image. In practical applications, the magnetic resonance image corresponding to the T1WI sequence is an image close to tissue anatomy, which can be used to observe anatomical structures; the magnetic resonance image corresponding to the T2WI sequence is an image that highlights the lesion area and can be used to observe tissue lesions.

[0049] In this embodiment, the initial magnetic resonance image can be understood as image information obtained after scanning the target area of ​​the target object by a magnetic resonance imaging scanning system.

[0050] It should be noted that the magnetic resonance image can be used to determine the lesion area in the target object, so when acquiring the initial magnetic resonance image, the acquired magnetic resonance image can include the lesion area to be determined. Optionally, the initial magnetic resonance image includes the lesion area to be segmented. The lesion area to be segmented can be understood as the target area in the initial magnetic resonance image that needs to be studied for the lesion.

[0051] It should also be noted that the initial magnetic resonance image can be obtained in real time from a medical imaging device based on magnetic resonance imaging (MRI) technology, or can be obtained from an image database, or can be received by transmitting image scanning data from an external device, etc. This embodiment does not specifically limit this.

[0052] Specifically, when a clinical study is required on a lesion region of a target object, initial magnetic resonance images of the lesion region under different sequences need to be acquired so that further study of the lesion region can be performed based on the initial magnetic resonance images.

[0053] S120 , performing a registration operation on the initial magnetic resonance image in the target sequence to obtain a magnetic resonance registered image.

[0054] Generally speaking, magnetic resonance images under different sequences play different roles in the process of lesion research. When it is necessary to conduct specific research on magnetic resonance images under a certain sequence, the sequence can be used as the target sequence for subsequent operations.

[0055] In practical applications, the thickness of the image layer of the magnetic resonance image used for clinical research is large, and the number of layers is small, and subsequent image processing is not implemented. Therefore, it is necessary to perform a registration operation on the initial magnetic resonance image under the target sequence so that the model can be used to process the image later. The registration operation can be understood as performing a registration and alignment process on the image so that the image can be standardized and the difficulty of subsequent image processing can be reduced. Exemplarily, the registration operation can be to register the initial magnetic resonance image to the MNI standard space so that the image thickness in the z direction is constant, and the number of image layers changes from the initial 20 layers to more than 100 layers, thereby achieving an increase in the disease characteristics of the magnetic resonance image in the z direction. Among them, the registration method can include linear registration and nonlinear registration, and the linear registration can include but is not limited to coordinate rotation, translation, overall and local scaling, etc.

[0056] In this embodiment, the magnetic resonance registration image can be understood as a magnetic resonance image obtained after image registration processing.

[0057] Optionally, performing a registration operation on the initial magnetic resonance image under the target sequence to obtain a magnetic resonance registered image includes: processing the initial magnetic resonance image under the target sequence according to a preset registration space to obtain a registration matrix; and determining the magnetic resonance registered image based on the registration matrix.

[0058] The preset registration space may be a pre-set standard space for implementing image registration operations. Exemplarily, the preset registration space may be an MNI space or a Talairach space, etc. The MNI space is a coordinate system established based on a series of magnetic resonance images of a normal human brain. The registration matrix may be a transformation matrix for implementing conversion from an initial magnetic resonance image to a magnetic resonance registration image.

[0059] In a specific implementation, after acquiring the initial magnetic resonance images under different sequences, in order to facilitate statistical analysis of the features in the magnetic resonance images, the initial magnetic resonance images under the target sequence need to be registered and standardized to a standard template. The initial magnetic resonance images under the target sequence can be processed based on a preset registration space to obtain a corresponding registration matrix. Furthermore, based on the registration matrix, the initial magnetic resonance images are registered into magnetic resonance registered images.

[0060] S130, inputting the initial magnetic resonance image and the magnetic resonance registration image into a pre-trained lesion segmentation model to obtain an initial segmentation feature matrix.

[0061] In the present embodiment, the lesion segmentation model can be understood as a pre-trained neural network model for achieving target lesion segmentation. Optionally, the lesion segmentation model can be a neural network model trained based on an encoding-decoding method. Among them, the encoding-decoding method can be understood as first extracting features from the image, and then restoring the image after feature extraction to an output image with the same dimension as the original image. For example, the lesion segmentation model can be a U-Net neural network model, etc. The initial segmentation feature matrix can be understood as a three-dimensional matrix obtained after the initial magnetic resonance image is processed by the lesion segmentation model, which is used to reflect the lesion features. It should be noted that the initial segmentation feature matrix includes each segmentation feature matrix corresponding to the initial magnetic resonance image under each sequence and the segmentation feature matrix corresponding to the magnetic resonance registration image.

[0062] In practical applications, after obtaining the initial magnetic resonance images and magnetic resonance registration images under each sequence, each image is input into the lesion segmentation model, and the image is encoded and decoded by the lesion segmentation model to obtain the initial segmentation feature matrix corresponding to the initial magnetic resonance images and magnetic resonance registration images under each sequence.

[0063] S140 , determining a target lesion segmentation image of the lesion area to be segmented based on the initial segmentation feature matrix.

[0064] In this embodiment, the target lesion segmentation image can be understood as a lesion mask image used to identify the specific location and contour of the lesion in the magnetic resonance image. In practical applications, the target lesion segmentation image can be characterized by setting the pixel points of the lesion area to 1 and the pixel points of other areas to 0, so that the lesion area is distinguished and displayed as a region of interest.

[0065] It should be noted that since the initial segmentation feature matrix includes the lesion feature segmentation matrix corresponding to the initial magnetic resonance image under each sequence and the disease feature segmentation matrix corresponding to the standard magnetic resonance image, when determining the target lesion segmentation image, it is necessary to determine it based on all lesion feature segmentation matrices included in the initial segmentation feature matrix.

[0066] Optionally, based on the initial segmentation feature matrix, a target lesion segmentation image of the lesion area to be segmented is determined, including: performing feature fusion processing on the initial segmentation feature matrix to obtain a feature fusion matrix; performing mean processing on the feature fusion matrix to obtain a target lesion segmentation matrix, so as to determine the target lesion segmentation image based on the target lesion segmentation matrix.

[0067] In this embodiment, the feature fusion matrix can be understood as a matrix containing all feature information in the initial segmentation feature matrix. The target lesion segmentation matrix can be a matrix containing target lesion feature information.

[0068] In a specific implementation, after obtaining the initial segmentation feature matrix, the various lesion feature matrices contained in the initial segmentation feature can be added together to obtain a feature fusion matrix containing all lesion feature information. Furthermore, according to the number of lesion feature segmentation matrices contained in the initial segmentation feature matrix, the feature fusion matrix is ​​mean processed to obtain the target lesion segmentation matrix, thereby determining the target lesion segmentation image based on the target lesion segmentation matrix.

[0069] It should be noted that, since the number of layers of the magnetic resonance image increases after the registration operation, it is not equal to the number of layers of the magnetic resonance images under other sequences, and the standard space after registration is different from the cranial space displayed in the initial magnetic resonance image. Therefore, after obtaining the lesion feature segmentation matrix corresponding to the magnetic resonance registration image, it is also necessary to restore it to the matrix of the original number of layers, that is, to restore it to the matrix corresponding to the cranial space displayed in the initial magnetic resonance image. The advantage of performing an inverse registration operation on the segmentation feature registration matrix is ​​that the segmentation feature registration matrix obtained after registration cannot be matched and analyzed with the feature matrix corresponding to the initial magnetic resonance image. By restoring the segmentation feature registration matrix to the feature matrix corresponding to the initial magnetic resonance image through an inverse registration operation, the lesion area in the cranial space displayed in the initial magnetic resonance image can be determined based on the segmentation result.

[0070] On the basis of the above technical solution, before the initial segmentation feature matrix is ​​subjected to feature fusion processing to obtain the feature fusion matrix, it also includes: based on the registration matrix, performing an inverse registration operation on the segmentation feature registration matrix in the initial segmentation feature matrix to obtain the segmentation feature matrix to be fused; based on the segmentation feature matrix to be fused, updating the initial segmentation feature matrix.

[0071] The segmentation feature registration matrix can be understood as the lesion feature segmentation matrix obtained after the magnetic resonance registration image is processed by the lesion segmentation model.

[0072] Specifically, after obtaining the initial segmentation feature matrix, the segmentation feature registration matrix in the initial segmentation feature matrix can be inversely registered according to the registration matrix obtained in the registration operation to restore it to the matrix corresponding to the cranial space displayed in the initial magnetic resonance image, and obtain the segmentation feature matrix to be fused. Furthermore, the segmentation feature registration matrix is ​​replaced by the segmentation feature matrix to realize the update of the initial segmentation feature matrix.

[0073] The technical solution of the embodiment of the present invention obtains the initial magnetic resonance images under multiple sequences and aligns the magnetic resonance images under the target sequence to obtain the magnetic resonance registration image. Furthermore, the initial magnetic resonance image and the magnetic resonance registration image are processed based on the lesion segmentation model to determine the target lesion segmentation image of the lesion area to be segmented. This solves the problem that in the prior art, compared with the scientific research sequence, the clinical sequence has the characteristics of strong anisotropy and unclear features, which may cause the lesion area segmentation result to fail to fully integrate the information of multiple sequences and the segmentation effect is poor. By increasing the lesion features in the target direction and fusing the segmentation results, the effect of improving the lesion segmentation efficiency and segmentation accuracy is achieved.

[0074] Embodiment 2

[0075] Figure 2 This is a flow chart of a model training method provided in Embodiment 2 of the present invention. This embodiment is applicable to the case of segmenting the lesion area of ​​the magnetic resonance images of the clinical magnetic resonance sequence. The method can be executed by a model training device, which can be implemented in the form of hardware and / or software, and can be configured in a terminal and / or a server. Among them, the technical terms that are the same or corresponding to the above embodiments are not repeated here.

[0076] like Figure 2 As shown, the method specifically comprises the following steps:

[0077] S210, obtaining at least one sample magnetic resonance image under multiple sequences, and annotating the sample magnetic resonance image to obtain an annotated lesion image, so as to construct at least one group of training samples based on the sample magnetic resonance image and the annotated lesion image.

[0078] In this embodiment, the sample magnetic resonance image can be an image acquired in real time based on a medical image acquisition device, or an image pre-stored in a medical image storage space, etc. The lesion annotated image can be understood as pre-annotated label image information that can identify the target lesion area. It should be noted that the disease annotated image can use a manual annotation program to annotate the sample magnetic resonance image, or other methods can be used to implement the annotation process of the lesion annotated image, etc., which is not specifically limited in this embodiment. Among them, the training sample includes a sample registration image corresponding to the sample magnetic resonance image under any sequence. The sample registration image can be understood as a registration image obtained after the sample magnetic resonance image under a certain sequence is registered.

[0079] In practical applications, sample magnetic resonance images corresponding to multiple sequences are obtained, and lesion features are annotated on the sample magnetic resonance images to obtain lesion annotated images. Then, the sample magnetic resonance images and lesion annotated images under each sequence are used as training samples to train the model to be trained.

[0080] It should be noted that before training the model to be trained, training samples need to be obtained first, so as to train the model based on the training samples. In order to improve the accuracy of the model, as many and rich training samples as possible can be obtained.

[0081] On the basis of the above technical solution, it also includes: for each training sample, performing a registration operation on the sample magnetic resonance image to obtain a sample registered image.

[0082] Specifically, after the sample magnetic resonance image is obtained, the sample magnetic resonance image under the target sequence is determined according to the current user demand, and a registration operation is performed on it to obtain a sample registered image.

[0083] S220: For each training sample, input the current training sample into the model to be trained to obtain at least one actual output image corresponding to the current training sample.

[0084] It should be noted that, for each training sample, the method of S220 can be used to train it, so as to obtain a lesion segmentation model.

[0085] The model to be trained may be a model in which the model parameters are initial parameters or default parameters. The actual output image is the image output after the sample magnetic resonance images and the sample registration images under each sequence are input into the model to be trained.

[0086] Specifically, the sample magnetic resonance images and sample registration images corresponding to each sequence are output to the model to be trained, and each input image is processed based on the model structure in the model to be trained, so as to output the actual output image corresponding to the sample magnetic resonance images and sample registration images under each sequence included in the current training sample.

[0087] S230, determining a first model loss based on the lesion annotation image, the actual output image in any sequence, and the corresponding registered output image.

[0088] In this embodiment, the registered output image can be understood as an image output after the sample registered image is input into the model to be trained. The first model loss can be understood as a loss value that corrects the model parameters of the model to be trained.

[0089] In practical applications, after obtaining the actual output image, since the output images corresponding to each sequence in the actual output image have differences in the representation of disease characteristics, it is necessary to process each image contained in the actual output image separately to obtain different model losses.

[0090] Optionally, based on the lesion annotation image, the actual output image in any sequence and the corresponding registered output image, a first model loss is determined, including: determining a first loss to be superimposed based on the lesion annotation image, the actual output image in any sequence and a first preset loss function; determining a second loss to be superimposed based on the lesion annotation image, the registered output image and the first preset loss function; and superimposing the first loss to be superimposed and the second loss to be superimposed to obtain the first model loss.

[0091] The first preset loss function may be understood as a predetermined function used to express the degree of difference between the actual output and the theoretical output. In this embodiment, the first preset loss function may be a Dice coefficient loss function.

[0092] Specifically, the difference value between the lesion annotation image and the actual output image in any sequence is determined, the difference value is substituted into the first preset loss function to obtain a loss value, and the loss value is used as the first loss to be superimposed.

[0093] Furthermore, the difference value between the lesion annotation image and the registered output image is determined, and the difference value is substituted into the first preset loss function to obtain the loss value, and the loss value is used as the second loss to be superimposed. Finally, the first loss to be superimposed and the second loss to be superimposed are added to obtain the first model loss.

[0094] Exemplarily, the first model loss may be determined based on the following formula:

[0095]

[0096] Among them, L1 represents the first model loss, DL represents the Dice coefficient loss function, and Y represents the lesion annotation image. represents the actual output image under the T2WI sequence, Represents the registration standard image.

[0097] S240. Determine a second model loss based on the target sequence output image corresponding to the registered output image and other sequence output images except the target sequence output image.

[0098] The target sequence output image can be understood as the image output after the sample magnetic resonance image that needs to be registered is input into the model to be trained. Correspondingly, the other sequence output images can be understood as the image output after the sample magnetic resonance image that does not need to be registered is input into the model to be trained.

[0099] Optionally, based on the target sequence output image corresponding to the registered output image and other sequence output images except the target sequence output image, a second model loss is determined, including: determining at least one loss to be fused based on the target sequence output image, other sequence output images and a second preset loss function; and superimposing each loss to be fused to obtain a second model loss.

[0100] The second preset loss function may be a predetermined function used to express the difference between the actual output and the theoretical output. In this embodiment, the second preset loss function may be a mean square error (MSE) loss function.

[0101] Specifically, the difference values ​​between the target sequence output image and the output images of other sequences are determined respectively, and each difference value is substituted into the second preset loss function respectively to obtain at least one loss value, and the obtained loss value is used as the loss to be fused. Further, each loss to be fused is added together to obtain the second model loss.

[0102] Exemplarily, the second model loss may be determined based on the following formula:

[0103]

[0104] Among them, L2 represents the second model loss, MSE represents the mean square error loss function, represents the actual output image under the T1WI sequence, Represents the actual output image under the Flair sequence.

[0105] S250, based on the first model loss, the second model loss and the corresponding weights, determine the target model loss, adjust the model parameters of the to-be-trained model based on the target model loss, and obtain a lesion segmentation model.

[0106] The target loss can be understood as the loss value based on which the model to be trained modifies the model parameters. The lesion segmentation model is used to segment the lesion area in the magnetic resonance images under multiple sequences to obtain a lesion segmentation image.

[0107] Specifically, the weights corresponding to the first model loss and the second model loss are determined respectively. Further, the first model loss and its corresponding weight are multiplied, and the second model loss and its corresponding weight are multiplied. The multiplied losses are added together to finally obtain the target model loss, and the model parameters of the model to be trained are adjusted based on the target model loss to finally obtain the lesion segmentation model.

[0108] It should be noted that the convergence of the loss function corresponding to the target model loss can be used as a training target for model training. Specifically, the training error of the loss function corresponding to the target model loss, that is, the loss parameter, can be used as a condition for detecting whether the current loss function has reached convergence. For example, whether the training error is less than the preset error or whether the error change trend tends to be stable, or whether the current number of model iterations is equal to the preset number, etc. If the detection reaches the convergence condition, it indicates that the training of the current model to be trained is completed, and the iterative training can be stopped at this time; if it is detected that the convergence condition is not met at present, the current training sample can be further obtained to train the model to be trained until the training error of the loss function is within the preset range. When the training error of the loss function reaches convergence, the model to be trained obtained by the current training can be used as a lesion segmentation model.

[0109] Exemplarily, the target model loss may be determined based on the following formula:

[0110] L=αL1+βL2

[0111] Among them, L represents the target model loss, α represents the weight corresponding to the first model loss, and β represents the weight corresponding to the second model loss.

[0112] For example, see Figure 3 As shown, the training process of the model to be trained can be specifically described: sample magnetic resonance images under T1WI sequence, T2WI sequence and Flair sequence are obtained, and the sample magnetic resonance images under T2WI sequence are registered to obtain sample registered images, and the sample magnetic resonance images under T1WI sequence, T2WI sequence and Flair sequence and the sample registered images are input into the model to be trained to obtain different actual output images respectively. Furthermore, based on the actual output image under T2WI sequence, the sample magnetic resonance image, the lesion annotation image and the Dice coefficient loss function, the first model loss is determined, and based on the actual output image under T1WI sequence, the actual output image under T2WI sequence, the actual output image under Flair sequence and the mean square error loss function, the second model loss is determined. Finally, the target model loss is determined based on the first model loss, the second model loss and the corresponding weights, so as to train the model to be trained based on the target model loss.

[0113] The technical solution of the embodiment of the present invention trains the training model and determines the first model loss and the second model loss respectively. Then, based on the first model loss, the second model loss and the corresponding weights, the target model loss is determined, and the model parameters of the training model are adjusted based on the target model loss, so as to obtain a lesion segmentation model. This solves the problem that in the prior art, compared with scientific research sequences, clinical sequences have strong anisotropy and unclear features, which may lead to the failure of the lesion area segmentation result to fully integrate the information of multiple sequences and the poor segmentation effect. By separately determining different model losses and weighted integration processing of each model loss, the segmentation efficiency and accuracy of the model for the target feature are effectively improved.

[0114] Embodiment 3

[0115] Figure 4 FIG. 1 is a schematic diagram of the structure of an image segmentation device provided in Embodiment 3 of the present invention. Figure 4 As shown, the device includes: an image acquisition module 310, an image registration module 320, a feature matrix determination module 330 and a lesion segmentation image determination module 340.

[0116] An image acquisition module 310 is used to sequentially acquire initial magnetic resonance images under multiple sequences, wherein the initial magnetic resonance images include the lesion area to be segmented;

[0117] An image registration module 320 is used to perform a registration operation on the initial magnetic resonance image under the target sequence to obtain a magnetic resonance registered image;

[0118] A feature matrix determination module 330 is used to input the initial magnetic resonance image and the magnetic resonance registration image into a pre-trained lesion segmentation model to obtain an initial segmentation feature matrix, wherein the lesion segmentation model is trained based on sample magnetic resonance images and lesion annotated images under multiple sequences;

[0119] The lesion segmentation image determination module 340 is used to determine a target lesion segmentation image of the lesion area to be segmented based on the initial segmentation feature matrix.

[0120] The technical solution of the embodiment of the present invention obtains the initial magnetic resonance images under multiple sequences and aligns the magnetic resonance images under the target sequence to obtain the magnetic resonance registration image. Furthermore, the initial magnetic resonance image and the magnetic resonance registration image are processed based on the lesion segmentation model to determine the target lesion segmentation image of the lesion area to be segmented. This solves the problem that in the prior art, compared with the scientific research sequence, the clinical sequence has the characteristics of strong anisotropy and unclear features, which may cause the lesion area segmentation result to fail to fully integrate the information of multiple sequences and the segmentation effect is poor. By increasing the lesion features in the target direction and fusing the segmentation results, the effect of improving the lesion segmentation efficiency and segmentation accuracy is achieved.

[0121] Optionally, the image registration module 320 is further configured to process the initial magnetic resonance image under the target sequence according to a preset registration space to obtain a registration matrix; and determine a magnetic resonance registration image based on the registration matrix.

[0122] Optionally, the lesion segmentation image determination module 340 is also used to perform feature fusion processing on the initial segmentation feature matrix to obtain a feature fusion matrix; perform mean processing on the feature fusion matrix to obtain a target lesion segmentation matrix, so as to determine a target lesion segmentation image based on the target lesion segmentation matrix.

[0123] Optionally, before performing feature fusion processing on the initial segmentation feature matrix to obtain a feature fusion matrix, the device further includes: a matrix inverse registration module and a matrix update module.

[0124] A matrix inverse registration module is used to perform an inverse registration operation on the segmentation feature registration matrix in the initial segmentation feature matrix based on the registration matrix to obtain the segmentation feature matrix to be fused;

[0125] The matrix updating module is used to update the initial segmentation feature matrix based on the segmentation feature matrix to be fused.

[0126] Optionally, the lesion segmentation model is a neural network model trained based on an encoding-decoding method.

[0127] The image segmentation device provided in the embodiment of the present invention can execute the image segmentation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0128] Embodiment 4

[0129] Figure 5 This is a schematic diagram of the structure of a model training device provided in Embodiment 4 of the present invention. Figure 5As shown, the device includes: a training sample construction module 410, an actual output image determination module 420, a first model loss determination module 430, a second model loss determination module 440 and a lesion segmentation model determination module 450.

[0130] The training sample construction module 410 is used to obtain at least one sample magnetic resonance image under multiple sequences, and annotate each sample magnetic resonance image to obtain at least one lesion annotated image, so as to construct at least one set of training samples based on the sample magnetic resonance image and the lesion annotated image, wherein the training samples include a sample registration image corresponding to the sample magnetic resonance image under any sequence;

[0131] The actual output image determination module 420 is used for inputting the current training sample into the model to be trained for each training sample to obtain at least one actual output image corresponding to the current training sample;

[0132] A first model loss determination module 430, configured to determine a first model loss based on the lesion annotation image, the actual output image under any sequence, and the corresponding registered output image;

[0133] A second model loss determination module 440, configured to determine a second model loss based on a target sequence output image corresponding to the registration output image and other sequence output images except the target sequence output image;

[0134] The lesion segmentation model determination module 450 is used to determine the target model loss based on the first model loss, the second model loss and the corresponding weights, so as to adjust the model parameters of the training model based on the target model loss to obtain the lesion segmentation model; wherein the lesion segmentation model is used to segment the lesion area in the magnetic resonance images under multiple sequences to obtain the lesion segmentation image.

[0135] The technical solution of the embodiment of the present invention trains the training model and determines the first model loss and the second model loss respectively. Then, based on the first model loss, the second model loss and the corresponding weights, the target model loss is determined, and the model parameters of the training model are adjusted based on the target model loss, so as to obtain a lesion segmentation model. This solves the problem that in the prior art, compared with scientific research sequences, clinical sequences have strong anisotropy and unclear features, which may lead to the failure of the lesion area segmentation result to fully integrate the information of multiple sequences and the poor segmentation effect. By separately determining different model losses and weighted integration processing of each model loss, the segmentation efficiency and accuracy of the model for the target feature are effectively improved.

[0136] Optionally, the device further comprises: a sample image registration module, configured to perform a registration operation on the sample magnetic resonance image for each training sample to obtain a sample registered image.

[0137] Optionally, the first model loss determination module 430 is also used to determine the first loss to be superimposed based on the lesion annotation image, the actual output image under any sequence and the first preset loss function; determine the second loss to be superimposed based on the lesion annotation image, the registration output image and the first preset loss function; and superimpose the first loss to be superimposed and the second loss to be superimposed to obtain the first model loss.

[0138] Optionally, the second model loss determination module 440 is also used to determine at least one loss to be fused based on the target sequence image, other sequence output images and a second preset loss function; and superimpose the losses to be fused to obtain the second model loss.

[0139] The model training device provided in the embodiment of the present invention can execute the model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0140] Embodiment 5

[0141] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0142] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0143] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0144] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as an image segmentation method or a model training method.

[0145] In some embodiments, the image segmentation method or the model training method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image segmentation method or the model training method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image segmentation method or the model training method in any other appropriate manner (e.g., by means of firmware).

[0146] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0148] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0150] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0151] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0152] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0153] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image segmentation method, characterized in that: include: Sequentially acquiring initial magnetic resonance images under multiple sequences, wherein the initial magnetic resonance images include the lesion area to be segmented; Performing a registration operation on the initial magnetic resonance image under the target sequence to obtain a magnetic resonance registered image, including: processing the initial magnetic resonance image under the target sequence according to a preset registration space to obtain a registration matrix; Based on the registration matrix, determining the magnetic resonance registration image; Inputting the initial magnetic resonance image and the magnetic resonance registration image into a pre-trained lesion segmentation model to obtain an initial segmentation feature matrix, wherein the lesion segmentation model is trained based on sample magnetic resonance images and lesion annotated images under multiple sequences; Determining a target lesion segmentation image of the lesion area to be segmented based on the initial segmentation feature matrix includes: performing feature fusion processing on the initial segmentation feature matrix to obtain a feature fusion matrix; Performing mean processing on the feature fusion matrix to obtain a target lesion segmentation matrix, so as to determine the target lesion segmentation image based on the target lesion segmentation matrix; Before performing feature fusion processing on the initial segmentation feature matrix to obtain a feature fusion matrix, the method further includes: Based on the registration matrix, an inverse registration operation is performed on the segmentation feature registration matrix in the initial segmentation feature matrix to obtain a segmentation feature matrix to be fused; Based on the segmentation feature matrix to be fused, updating the initial segmentation feature matrix; The feature fusion matrix is ​​a matrix containing all feature information in the initial segmentation feature matrix.

2. The method according to claim 1, characterized in that The lesion segmentation model is a neural network model trained based on an encoding-decoding method.

3. A model training method, characterized in that: include: Acquire at least one sample magnetic resonance image under multiple sequences, and annotate each of the sample magnetic resonance images to obtain at least one lesion annotated image, so as to construct at least one set of training samples based on the sample magnetic resonance images and the lesion annotated images, wherein the training samples include a sample registration image corresponding to the sample magnetic resonance image under any of the sequences; For each training sample, input the current training sample into the model to be trained to obtain at least one actual output image corresponding to the current training sample; Determining a first model loss based on the lesion annotated image, an actual output image under any sequence, and a corresponding registered output image; Determining a second model loss based on a target sequence output image corresponding to the registered output image and other sequence output images except the target sequence output image; Based on the first model loss, the second model loss and corresponding weights, a target model loss is determined, so as to adjust the model parameters of the model to be trained based on the target model loss to obtain a lesion segmentation model; The lesion segmentation model is used to segment the lesion area in the magnetic resonance images under multiple sequences to obtain a lesion segmentation image; The determining of the second model loss based on the target sequence output image corresponding to the registered output image and other sequence output images except the target sequence output image comprises: Determine at least one loss to be fused based on the target sequence output image, other sequence output images and a second preset loss function; Superimposing the losses to be fused to obtain the second model loss; The second preset loss function is a predetermined function used to express the degree of difference between the actual output and the theoretical output.

4. The method according to claim 3, characterized in that Also includes: For each training sample, a registration operation is performed on the sample magnetic resonance image to obtain a sample registered image.

5. The method according to claim 3, characterized in that: The determining of the first model loss based on the lesion annotated image, the actual output image in any sequence, and the corresponding registered output image comprises: Determine a first loss to be superimposed based on the lesion annotated image, an actual output image in any sequence, and a first preset loss function; Determining a second loss to be superimposed based on the lesion annotated image, the registered output image and the first preset loss function; The first loss to be superimposed and the second loss to be superimposed are superimposed to obtain the first model loss.

6. An image segmentation device, characterized in that: include: An image acquisition module, used for sequentially acquiring initial magnetic resonance images under multiple sequences, wherein the initial magnetic resonance images include the lesion area to be segmented; An image registration module is used to perform a registration operation on the initial magnetic resonance image under the target sequence to obtain a magnetic resonance registration image; A feature matrix determination module is used to input the initial magnetic resonance image and the magnetic resonance registration image into a pre-trained lesion segmentation model to obtain an initial segmentation feature matrix, wherein the lesion segmentation model is trained based on sample magnetic resonance images and lesion annotation images under multiple sequences; A lesion segmentation image determination module is used to determine a target lesion segmentation image of the lesion area to be segmented based on the initial segmentation feature matrix; The image registration module is further used to process the initial magnetic resonance image under the target sequence according to a preset registration space to obtain a registration matrix; Based on the registration matrix, determining the magnetic resonance registration image; The lesion segmentation image determination module is further used to perform feature fusion processing on the initial segmentation feature matrix to obtain a feature fusion matrix; Performing mean processing on the feature fusion matrix to obtain a target lesion segmentation matrix, so as to determine the target lesion segmentation image based on the target lesion segmentation matrix; Before performing feature fusion processing on the initial segmentation feature matrix to obtain a feature fusion matrix, the device further includes: Matrix inverse registration module and matrix update module; The matrix inverse registration module is used to perform an inverse registration operation on the segmentation feature registration matrix in the initial segmentation feature matrix based on the registration matrix to obtain the segmentation feature matrix to be fused; The matrix updating module is used to update the initial segmentation feature matrix based on the segmentation feature matrix to be fused; The feature fusion matrix is ​​a matrix containing all feature information in the initial segmentation feature matrix.

7. A model training device, characterized in that: include: A training sample construction module, used for acquiring at least one sample magnetic resonance image under multiple sequences, and annotating each of the sample magnetic resonance images to obtain at least one lesion annotated image, so as to construct at least one group of training samples based on the sample magnetic resonance images and the lesion annotated images, wherein the training samples include a sample registration image corresponding to the sample magnetic resonance image under any of the sequences; An actual output image determination module is used to input the current training sample into the model to be trained for each training sample, and obtain at least one actual output image corresponding to the current training sample; A first model loss determination module, used to determine a first model loss based on the lesion annotation image, the actual output image under any sequence, and the corresponding registered output image; A second model loss determination module, configured to determine a second model loss based on a target sequence output image corresponding to the registered output image and other sequence output images except the target sequence output image; A lesion segmentation model determination module, used to determine a target model loss based on the first model loss, the second model loss and corresponding weights, so as to adjust model parameters of the model to be trained based on the target model loss to obtain a lesion segmentation model; The lesion segmentation model is used to segment the lesion area in the magnetic resonance images under multiple sequences to obtain a lesion segmentation image; The second model loss determination module is further used to determine at least one loss to be fused based on the target sequence output image, other sequence output images and a second preset loss function; and to superimpose the losses to be fused to obtain a second model loss; The second preset loss function is a predetermined function used to express the degree of difference between the actual output and the theoretical output.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image segmentation method described in any one of claims 1-2 or the model training method described in any one of claims 3-5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image segmentation method described in any one of claims 1-2 or the model training method described in any one of claims 3-5 when executed.

Images