Constraint-based image segmentation method, system, device and medium

Abstract

The present invention relates to the field of medical image processing. In view of the fact that the existing deep learning model for medical image segmentation cannot avoid various image quality and image data defects such as poor image quality, many interference factors, blurred boundaries, and difficulty in distinguishing, as well as the difficulty in accurately segmenting and identifying images, the present invention proposes a constraint-based small lesion image segmentation method, system, device and medium, which aims to solve the technical problem of how to use the spatial relationship between the lesion and the organ in which it is located when the sample size is small, and reduce the hypothesis space of the sample by adding a deep learning model with geometric feature constraints, so as to improve the precision of small lesion image segmentation. To this end, the method of the present invention constructs a cascaded multi-task segmentation network and a loss function based on spatial position geometric constraints, and after network training, predicts the medical image to be segmented to obtain the lesion target that needs to be segmented, so as to effectively avoid interference and improve segmentation accuracy, quality and recognition.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a constraint-based image segmentation method, system, device and medium. Background Art

[0002] In the field of medical image processing, medical image segmentation is a task of great clinical significance in medical imaging. Traditional image segmentation methods require manual adjustment of algorithm parameters for each image; in recent years, deep learning methods have also been widely used in the field of image processing. However, in the medical image segmentation stage, the poor image quality caused by the physical noise of the medical equipment itself, metal artifacts during imaging, errors during image reconstruction, etc., the differences in individual body shape and anatomical structure of the patients themselves, and the chest movement caused by the patient's own movement and breathing during the imaging process, all bring great difficulties to medical image segmentation. In addition, the similar characteristics of some soft tissues and organs that are in contact with each other, the low differentiation of image performance, the blurred boundaries, and the relatively small target area also bring great difficulties to image segmentation. In addition, in addition to image quality, the number of medical images is a bigger problem. The difficulty in obtaining data, the extremely high cost of manual annotation, the imbalance of the number of samples of different categories, etc., make it difficult for deep neural networks, which already require a large amount of data to train, to play their role and have no practical effect in medical image processing, especially medical image segmentation. For example, the classic deep learning-based image segmentation model uses the U-Net encoding and decoding architecture. However, it is difficult to achieve good performance when using the U-Net segmentation method directly for small samples, high complexity, and high difficulty such as medical images. Similarly, for medical images, targets of different sizes require receptive fields of different sizes, and multi-task learning models with shared parameters are not applicable.

[0003] Therefore, based on the above-mentioned defects of the prior art, a new medical image segmentation solution is needed that can effectively apply deep learning to medical image tasks with small data volume, high data complexity, many interference factors, high precision requirements and high data difficulty. Summary of the invention

[0004] In response to the above-mentioned defects in the prior art, the present invention provides a technical solution of a constraint-based image segmentation method, system, device and medium, aiming to solve the technical problem of how to utilize the spatial relationship between the lesion and the organ in which it is located when the sample size is small, and to reduce the hypothesis space of the sample by adding a deep learning model with geometric feature constraints to improve the precision of small lesion image segmentation.

[0005] In order to solve the above technical problems, the first aspect of the present invention proposes a constraint-based image segmentation method, including: constructing a cascaded multi-task segmentation network and constructing a loss function based on spatial position geometric constraints; training the cascaded multi-task segmentation network according to the loss function; inputting the image to be segmented into the trained cascaded multi-task segmentation network to obtain a predicted segmentation target area.

[0006] Preferably, the construction of a cascaded multi-task segmentation network specifically includes: constructing at least two subtasks to perform multi-region segmentation of an image; the sub-segmentation network used in each subtask is based on a U-Net encoding and decoding network architecture; wherein the feature map sampling method used by the U-Net encoding and decoding network architecture is: 3 downsamplings and 3 upsamplings, or 4 downsamplings and 4 upsamplings; wherein, the binary matrix of the first target area predicted by the sub-segmentation network of the previous subtask is multiplied by the elements of the corresponding positions of the matrix of the original image, and then the sub-segmentation network of the next subtask predicts a second target area limited to the first target area.

[0007] Preferably, constructing a loss function based on spatial position geometric constraints specifically includes: the loss function of the cascaded multi-task segmentation network is defined as:

[0008]

[0009] Among them, seg (1) ,seg (2) Represents the sub-loss function of the sub-segmentation network of the corresponding different sub-tasks, and β is a hyperparameter for adjusting the weights between multiple tasks; among them, each sub-loss function is combined with the binary cross entropy BCE and the Dice loss function. definition:

[0010]

[0011] Among them, L BCE is the binary cross entropy loss function, L Dice is the Dice loss function, α1 is the adjustment L BCE , L Dice The hyperparameters of the weights of these two loss functions, n represents the corresponding nth subtask and n is a natural number greater than or equal to 1.

[0012] Preferably, constructing a loss function based on spatial position geometric constraints specifically further includes: optimizing the optimized loss function Loss of the cascaded multi-task segmentation network constructed, which is defined as:

[0013]

[0014]

[0015] Among them, L loc To optimize the loss function, α2 is a hyperparameter for adjusting the weights of these three loss functions, L contour It is a contour loss function constrained according to the spatial position geometric feature relationship, which is defined as:

[0016]

[0017] Where dist(x,y) represents the L2 distance between pixel points x and y; is the boundary of the image P representing the predictions of different subtasks to indicate the set to which the pixel points x and y belong.

[0018] Preferably, the cascaded multi-task segmentation network is trained according to the loss function, specifically including adopting a distributed training method: first inputting annotated image samples in a training set to perform a preset M-times iterative training on the sub-segmentation network of the previous sub-task in the cascaded multi-task segmentation network, wherein the loss function used in the training is the sub-loss function After the number of training times reaches M times, the labeled image samples in the training set are input to perform preset NM iterative training on the cascaded multi-task segmentation network as a whole, wherein the optimized loss function Loss is used during training; wherein the preset number of iterations of the training of the cascaded multi-task segmentation network is N times, N and M are natural numbers greater than or equal to 1 and N is greater than M; wherein 1 iteration refers to a single epoch iteration.

[0019] Preferably, the image to be segmented is input into the trained cascaded multi-task segmentation network to obtain a predicted segmentation target area, specifically comprising: inputting the actual medical image to be segmented into the trained cascaded multi-task segmentation network; first predicting the organ area in the medical image through the sub-segmentation network of the previous sub-task in the cascaded multi-task segmentation network; multiplying the binary matrix of the organ area with the elements at corresponding positions of the matrix of the original image of the medical image to obtain the original image restricted to the organ area; inputting the original image restricted to the organ area into the sub-segmentation network of the next sub-task of the cascaded multi-task segmentation network; and then predicting the lesion area in the organ area through the sub-segmentation network of the next sub-task.

[0020] Preferably, the image to be segmented includes a CT image of a gallbladder polyp; the predicted segmentation target area includes a multi-target area prediction; wherein the cascaded multi-task segmentation network is used to predict the gallbladder area in its previous subtask, and then based on the spatial restriction of the gallbladder area, the polyp area under the restriction of the gallbladder area is predicted in its subsequent subtask.

[0021] In order to solve the above technical problems, the second aspect of the present invention proposes a constraint-based image segmentation system, including: a model building module, used to construct a cascaded multi-task segmentation network; a loss function building module, used to construct a loss function based on spatial position geometric constraints; a training module, used to train the cascaded multi-task segmentation network according to the loss function; a prediction module, used to input the image to be segmented into the trained cascaded multi-task segmentation network to obtain a predicted segmentation target area.

[0022] In order to solve the above technical problem, the third aspect of the present invention proposes an electronic device, which includes a processor and a memory storing computer-executable instructions, and when the computer-executable instructions are executed, the processor executes any one of the methods described in the first aspect.

[0023] In order to solve the above technical problem, the fourth aspect of the present invention proposes a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by a processor, the method described in any one of the above first aspects is implemented.

[0024] One embodiment of the present invention realizes the segmentation of medical images by combining multi-task learning and utilizing the geometric features of the spatial position relationship between organs and lesions. In particular, by combining multi-task learning, medical images, especially small lesion images, are segmented from organs and lesions, and a cascaded U-Net model is designed by utilizing the inclusion relationship between the spatial position of organs and lesions. After the segmentation of the organ area is obtained, it is used as a bounding area (an area surrounded by a non-fixed shape, which can be seen in Fig.10 The lesion area is then segmented in this area. This provides a priori information for the segmentation of lesions with small target foregrounds, allowing the deep learning network to find the lesion location more effectively, that is, to achieve constraint-based small lesion image segmentation.

[0025] In addition, in a further preferred method, for the segmentation task of more complex small lesion images, such as gallbladder polyp images, a loss function based on the spatial geometric distance between the organ and the lesion area contour is proposed, which can make full use of the positional relationship between such small lesions and organs, such as information such as polyps growing on the inner wall of the gallbladder, thereby constraining the network to predict the contour of the area to be more refined. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to make the technical problems solved by the present invention, the technical means adopted and the technical effects achieved more clearly, the specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, it should be noted that the drawings described below are only drawings of exemplary embodiments of the present invention, and those skilled in the art can obtain drawings of other embodiments based on these drawings without creative work.

[0027] Figure 1 is a schematic diagram of the main steps of an embodiment of the constraint-based image segmentation method according to the present invention;

[0028] Figure 2 It is a schematic diagram of the basic principle of the U-Net decoding and encoding network architecture adopted in an embodiment of the technical solution of the present invention;

[0029] Figure 3 is a schematic diagram of the architecture of U-Net further optimized according to an embodiment of the technical solution of the present invention;

[0030] Figure 4 is a schematic diagram of a network architecture of a cascaded multi-task segmentation network according to an embodiment of the technical solution of the present invention;

[0031] Figure 5 is a schematic diagram of an example of a distributed training experiment according to an embodiment of the technical solution of the present invention;

[0032] Figure 6 is a structural block diagram of an embodiment of a constraint-based image segmentation system according to the present invention;

[0033] Figure 7 is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention;

[0034] Figure 8 is a schematic diagram of a logical exemplary embodiment of a computer readable medium according to the present invention;

[0035] Fig. 9 , 10 It is a schematic diagram of the comparison results of the experimental comparison effects between the technical solution according to the present invention and the prior art. DETAILED DESCRIPTION

[0036] The exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. Although each exemplary embodiment can be implemented in a variety of specific ways, it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, these exemplary embodiments are provided to make the content of the present invention more complete and to facilitate the comprehensive communication of the inventive concept to those skilled in the art.

[0037] Under the premise of conforming to the technical concept of the present invention, the structure, performance, effect or other characteristics described in a specific embodiment may be combined with one or more other embodiments in any appropriate manner.

[0038] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0039] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.

[0040] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0041] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0042] In the existing field of medical image processing, deep learning models in medical image segmentation tasks have various image quality and image data defects such as small data volume, poor image quality, many interference factors, blurred boundaries, and difficulty in distinguishing, as well as difficulties in accurately segmenting and identifying images. Therefore, the technical solution of the present invention is proposed, which uses the spatial relationship between the lesion and the organ in which it is located to reduce the sample hypothesis space by adding a deep learning model with geometric feature constraints for medical images with high difficulty and high requirements under the condition of small sample size, and effectively improves the segmentation precision of small lesion images.

[0043] In one embodiment of the present invention provided to overcome the above-mentioned defects, the image to be segmented is mainly input into a cascaded multi-task segmentation network, the hierarchical relationship of the multiple tasks is fully utilized to learn the spatial information of the target in the image to be segmented, and the constraints of the loss function of the constructed segmentation network are combined to obtain the segmentation result corresponding to the target.

[0044] [Example 1]

[0045] Combine the following Figure 1 The main steps of an embodiment shown in the figure illustrate the implementation of the constraint-based image segmentation method of the present invention.

[0046] Step S110, constructing a cascaded multi-task segmentation network.

[0047] In one embodiment, in the cascaded multi-task of the cascaded multi-task segmentation network, each cascaded multi-task learning can adopt a deep neural network. Further, the deep neural network can adopt, for example, a U-Net encoding and decoding architecture as a basic architecture for image segmentation.

[0048] An example of a deep neural network with a U-Net encoder-decoder architecture can be an end-to-end encoder-decoder image segmentation network with 4 downsampling and 4 upsampling, such as Figure 2 shown.

[0049] Among them, core components such as the jump connection between downsampling and upsampling combine shallow and deep semantic features without increasing network parameters, enabling the network to understand the image from the whole to the part.

[0050] Specifically, the image to be segmented is input and feature extraction is performed. For example, the downsampling part (for example, 4 downsamplings), that is, the feature extraction part, is similar to VGG. The feature extraction is performed on the image to be segmented. Each downsampling includes a convolutional layer with a convolution kernel size of 3×3, batch normalization, activation function (ReLU, SoftMax, etc.) and a maximum pooling layer (for example, 2×2). Furthermore, in the downsampling part, each pooling layer has a scale, including the original image scale, for a total of 5 scales; and each downsampling reduces the size of the feature map by half, while the number of channels increases exponentially.

[0051] Specifically, in the upsampling part, each upsampling also includes the convolution kernel conv size of 3×3, batch normalization, activation function and max-pooling layer (for example: 2×2). Furthermore, in the upsampling part, the conv and deconv corresponding to the downsampling are upsampled 4 times to obtain 4 scale images, including the final output segmentation image of 5 scales; and, corresponding to the downsampling part, and just the opposite of the downsampling part, each upsampling (feature map) is fused with the same scale as the number of channels corresponding to the feature extraction part. Preferably, the feature map crop can be cut before fusion (for example, copy copy + cut crop feature map). Wherein, the fusion is splicing.

[0052] See also Figure 2 In the schematic example, in the left downsampling / downsampling part, the convolution kernel conv is 3×3, batch normalization, activation function, and maximum pooling. Each downsampling generates a channel. After copying and cutting the feature map of each downsampling, in the upsampling part on the right, each upsampling, the same convolution kernel deconv is 3×3, batch normalization, activation function, and maximum pooling. Each upsampling is combined with the feature map after cutting the crop, and the scale is the same as the number of channels. For example, cutting to 392×392, 200×200, 104×104, etc. Further, the last conv can be 1×1, and then the segmented image with increased resolution is output.

[0053] Preferably, for the example of image segmentation of medical images, 3 downsampling and 3 upsampling can be used. If the image to be segmented is an image of a small lesion (e.g., an image of a gallbladder polyp), for segmenting such targets, a neural network of a U-Net encoding and decoding framework with 3 downsampling and 3 upsampling can be used as the basic neural network, i.e., the basic framework, for each task in the cascaded multi-task. In this way, in order to adapt to the size of the image to be segmented, the depth of each U-Net encoding and decoding neural network in the cascade is reduced by one layer, i.e., one downsampling and one upsampling are reduced, so that the model is simpler and the parameters are reduced, and while reducing the amount of parameters, overfitting can be prevented to a greater extent. The preferred structural hierarchy of the U-Net encoding and decoding network is, for example Figure 3 shown.

[0054] In one embodiment, see Figure 4The cascaded multi-task structure shown, for example, can adopt a schematic diagram of two cascaded tasks. Specifically, the first task Task1, inputs the image to be segmented into the first encoding and decoding neural network (such as U-Net) of the cascade, segments the structural area where the target is located, and processes the segmented structural area where the target is located in combination with the image to be segmented (original image) (for example, limiting it to the original image), and then enters the next task Task2, that is, inputs it into the second encoding and decoding neural network (such as U-Net) of the cascade, and performs segmentation prediction, such as segmenting the target by segmenting again. The structure of the cascaded multi-task neural network in this embodiment makes full use of the hierarchical relationship between the two tasks, that is, the mask of the structural area where the target is located contains the mask of the target, and the target segmentation task is divided into two steps, so that the network can learn the spatial position information between the structural area where the target is located and the target, such as the spatial position information between small lesions (such as polyps) and the organs (such as gallbladder) in medical images, thereby improving the accuracy of target (small lesions such as polyps, etc.) segmentation.

[0055] Step S120, constructing a loss function of the cascaded multi-task segmentation network.

[0056] In one embodiment, a corresponding loss function is constructed for the cascaded multi-task segmentation network. The loss function is a function that measures the error of network prediction. When training a neural network model, the loss function plays a decisive role. The training process of the neural network is the process of updating the network parameters by optimizing the loss function. Specifically, in the cascaded multi-task segmentation network, a binary cross entropy BCE and / or Dice loss function can be constructed.

[0057] For example, in supervised learning, we can give an image and its corresponding groundtruth annotation, denoted as X and G respectively, and the neural network model P = f(X, θ), where P is the network prediction (for example: the segmentation prediction of the segmentation network U-Net used in the first cascade task); θ is the network parameter, denoted as H and W, respectively, the height and width of the image, and the binary cross entropy loss function BCE is defined as:

[0058]

[0059] Wherein, i and j are both natural numbers greater than or equal to 1.

[0060] When the target foreground is very small, BCE tends to predict the background and leads to overfitting.

[0061] Since the Dice coefficient is used as a metric to evaluate segmentation performance, especially in medical image segmentation, it can be used as a loss function in the segmentation network to evaluate or update model parameters, which can show good performance, especially when the target foreground is small. Unlike cross entropy, since the Dice coefficient acts on the region (such as the structural region where the target is located) rather than on each pixel, it is basically not affected by the proportion of the target foreground. Therefore, the Dice coefficient can be used to measure the degree of overlap between the ground truth and the network prediction results, that is, the degree of overlap is defined as:

[0062]

[0063] Then, according to the prediction result of the desired network, Dice is made larger, and the Dice loss function is constructed, which is defined as:

[0064] L Dice =1-Dice

[0065] Therefore, the cross entropy loss function can be used to help the back propagation of the neural network optimization process during training, while the Dice loss function can better handle the segmentation problem of small target foreground, taking advantage of their respective advantages. Preferably, for each segmentation task, that is, the segmentation network (the network of the U-Net infrastructure) in each of the two tasks in the aforementioned embodiment, the weighted sum of the BCE and Dice loss functions can be used as the loss function L of the segmentation network. seg :

[0066] L seg =L BCE +α1L Dice

[0067] Among them, α1 is a hyperparameter for adjusting the weights of these two loss functions.

[0068] Furthermore, the loss function of the entire cascaded multi-task segmentation network is the loss function of multiple tasks to The sum of (n is a natural number greater than or equal to 1), preferably, the loss function of the segmentation network of the above two tasks is:

[0069]

[0070] where seg (1) ,seg (2) They represent different tasks respectively. β is a hyperparameter for adjusting the weights between tasks.

[0071] Furthermore, the cross entropy and Dice loss functions have certain advantages when applied to segmentation tasks, especially when used in combination. However, when there are various difficulties such as small sample size, small target foreground, low foreground-background contrast, unclear boundaries, etc., for specific tasks, more special loss functions can be designed based on the two to constrain the network's segmentation prediction in order to achieve better and more accurate segmentation effects.

[0072] In a preferred embodiment, for multi-region segmentation, such as the example of medical image segmentation of small lesions in the above example, it is hoped to make full use of the relationship between two task target areas, such as the relationship between small lesions and organs. Specifically, taking medical images as an example, lesions usually have a certain geometric characteristic relationship of spatial position with organs, such as: in the data set of gallbladder polyps, gallbladder polyps grow on the inner wall of the gallbladder, which contains the location information of the gallbladder and the polyps, that is, the boundary contour of the polyp is basically close to the boundary contour of the gallbladder. Based on this condition, it is hoped that the segmentation prediction result of the network can meet this condition as much as possible, and a loss function can be given as a penalty:

[0073]

[0074] Where dist(x,y) represents the distance between pixel points x and y. For example, the L2 distance is used. is the boundary of the image P, indicating the set to which the pixel belongs (the boundary of the set). (1) and (2) represent different corresponding tasks.

[0075] Continuing with the above example: P = f(X, θ), P(prediction) is the predicted image (for medical images, it can be the organ region, lesion region, etc. segmented after prediction by the neural network model / segmentation network of the two tasks in the solution of the present invention), P (1) , P (2) That is, the area predicted by the first subtask (U-Net of the previous subtask 1) is such as the location area of ​​the gallbladder, and the area predicted by the second subtask (U-Net of the next subtask 2) is such as the location area of ​​the polyp.

[0076] In this way, the actual loss function Loss of the entire segmentation network is the sum of the multi-task loss function and the conditional loss function:

[0077]

[0078] Step S130, training the cascaded multi-task segmentation network.

[0079] In one embodiment, the constructed cascaded multi-task segmentation network is iteratively trained, and the parameters of the network are updated according to the loss function of each iteration until a preset number of iterations is reached, and the iteration is terminated to complete the training. This training can be end-to-end training, that is, multiple iterations of the entire process from inputting an image to be segmented to outputting a segmentation prediction image each time.

[0080] A specific example of training is mainly small lesions and organ images of medical images:

[0081] In the constructed cascade multi-task segmentation network, the organ area (e.g., gallbladder area) is used as the bounding area of ​​the small lesion area (e.g., polyp area). The boundary area here is the area formed by the predicted / annotated border with a non-fixed shape edge line.

[0082] Furthermore, at the beginning of neural network training, the parameters of the entire segmentation network are randomly initialized. In this way, the segmented image predicted by the segmentation network may contain irregular areas without complete contours, or some areas that are not connected to each other, or completely wrong organ areas. Therefore, using organ areas to limit small lesion areas in the early stage will not only make the training unstable, but also cause the network to learn wrong results. In addition, the penalty L contour The loss function also acts between the organ contour (such as the gallbladder contour) and the small lesion contour (such as the polyp contour). When the organ contour is not predicted well, the effect of the loss function in constraining the small lesion will be reduced.

[0083] Therefore, preferably, the cascaded multi-task segmentation network can be trained in a step-by-step training manner. More specifically, for example: Step SX1: Perform N iterations on the training set, and train the segmentation network in the first task in the first M iterations (for example, only perform segmentation prediction on the target structure area), wherein, As the loss function for training; Step SX2: Starting from NM iterations until the Nth iteration, train the entire cascaded multi-task segmentation network (for example: the network of the first task and the second task that completes the segmentation prediction process), where the use As the loss function for training. In the preferred method, the network training process and the subsequent segmentation prediction process will be more stable and more able to make L contour It plays a more effective constraint role and has higher segmentation accuracy. At the same time, the entire segmentation network (i.e., the overall model) has also completed end-to-end training.

[0084] Among them, epochs is defined as a single training iteration of all batches in forward and backward propagation, that is, 1 cycle is a single forward and backward pass of the entire input data. For example: if the training set has 1000 samples, then: it takes 1 epoch to train the entire sample set.

[0085] For example, for medical images such as gallbladder polyps, image segmentation is performed, and model training is performed through the constructed model and loss function. The model is the cascaded multi-task segmentation network constructed above. Figure 5 In the training process shown in the figure, after 40 epochs, the network has performed well in predicting gallbladders, and the average Dice coefficient can reach above 0.85. In this way, it can be determined how many epochs (iterations) of training the network has completed relatively good segmentation of the gallbladder, and the outline of the gallbladder has been roughly determined and will not change much. Using the step-by-step training method, N can be set according to actual needs. In this example, the first step of training is (1) in the first 40 epochs (i.e., M), only the gallbladder segmentation model is trained, i.e., Task 1 (Encoder-Decoder-1), using As the loss function, the second step is to train the entire model after (2), that is, the entire task 1 to task 2 after M (both Encoder-Decoder-1 and Encoder-Decoder-2 are included), using The training is performed as the loss function. M and N are natural numbers greater than or equal to 1, and N is greater than M.

[0086] Step S140: performing target segmentation prediction on the image to be segmented based on the cascaded multi-task segmentation network.

[0087] Specifically, the image to be segmented (i.e., the image from which the target in the image needs to be segmented) is input into the trained cascaded multi-task segmentation network to perform segmentation prediction to obtain the target segmented result in the image to be segmented.

[0088] For example, when performing constraint-based image segmentation processing on medical images, the gallbladder organ polyp lesions are targeted, and the gallbladder region is obtained after the first task of the segmentation network, i.e., the first encoding-decoding architecture network, and the gallbladder region and the original image to be segmented are processed together. For example, after the first task, a binary matrix of the gallbladder region is obtained, and then multiplied with the corresponding position elements (i.e., matrix elements) of the original image matrix, the original image restricted to the gallbladder position area is obtained, and input into the second task, i.e., the second encoding-decoding architecture network, to obtain the polyp region of the gallbladder position area, i.e., the segmentation prediction of the target polyp. Such a network structure makes full use of the hierarchical relationship between the two tasks, i.e., the gallbladder mask contains the polyp mask, and the polyp segmentation task is divided into two steps, so that the network can learn the spatial position information between the gallbladder and the polyp, thereby improving the accuracy of polyp segmentation.

[0089] [Example 2]

[0090] Combine the following Figures 1 to 5 According to an embodiment of the actual application scenario of the constraint-based image segmentation of the present invention in medical image segmentation, the implementation of constraint-based small lesion image segmentation is described. Here, the small lesion is exemplified as a polyp, and the organ where the small lesion is located is exemplified as a gallbladder, that is, the gallbladder polyp is taken as the target for segmentation prediction, and the polyp area is found.

[0091] First, a cascaded multi-task segmentation network is constructed, and a loss function of the cascaded multi-task segmentation network is constructed.

[0092] In one embodiment, the cascaded multi-task segmentation network can adopt two tasks, divided into two levels, and two sub-segmentation networks, that is, the first task is a sub-segmentation network (the first sub-segmentation network), and the second task is another sub-segmentation network (the second sub-segmentation network). Therefore, after the entire model of the segmentation network is constructed and the corresponding loss function is set, the model is trained, and the image to be segmented is input for segmentation prediction. For example, the specific segmentation prediction process is: the image to be segmented is input into the first task and the first sub-segmentation network completes the preliminary segmentation prediction, and the image to be segmented is input into the second task and the second sub-segmentation network further completes the target segmentation prediction, and the target segmentation result is output.

[0093] Preferably, 1. The network architecture of the overall segmentation network model:

[0094] The sub-segmentation network in each subtask uses the U-Net encoder-decoder architecture as the basic framework of the neural network. U-Net is a high-precision, end-to-end encoder-decoder image segmentation network. Its core component is the jump connection between downsampling and upsampling. Without increasing network parameters, the jump connection combines shallow and deep semantic features, allowing the network to understand the image from the whole to the part.

[0095] In an example, U-Net performs 4 downsampling and 4 upsampling. Each downsampling and upsampling includes a convolution layer conv with a convolution kernel size of 3×3, batch normalization (which is similar to ordinary data normalization. It is a way to unify scattered data, which can optimize neural networks. Data with uniform specifications can make machine learning easier to learn the rules in the data), activation functions (such as ReLU, Softmax, etc.) and max-pooling layers. Each downsampling reduces the size of the feature map by half, while the number of channels increases exponentially. Upsampling is just the opposite.

[0096] In another more preferred example, in order to adapt to the size of medical images, such as medical CT images and X-ray images of gallbladder polyps, the depth of the network (or model) of the U-Net encoding and decoding architecture in the aforementioned example is reduced by one layer, that is, one downsampling and one upsampling are reduced (three downsamplings and three upsamplings are performed), so that the number of model parameters is reduced, the operation speed is accelerated, and the model is simplified, while at the same time, overfitting can be prevented to a large extent.

[0097] For the segmentation tasks with small sample size and high difficulty, the direct use of U-Net segmentation cannot achieve a good effect. Therefore, it is directly applied to the image segmentation task of small lesions. For example, in the segmentation task of gallbladder polyps, there are often defects such as small area of ​​small lesions (polyps), low contrast with surrounding tissues, unclear boundaries, etc., and the scheme of the present invention effectively optimizes the segmentation model of machine learning / neural network in the prior art through the reconstructed cascade multi-task segmentation network such as a neural network model. Further, the area proportion of organs (such as gallbladder, etc.) in medical images such as CT is much larger than that of small lesions (such as polyps, etc.), and the contrast with surrounding tissues is high. The segmentation task of organs (gallbladder, etc.) is much simpler than that of polyps. Therefore, by designing the neural network model of the scheme of the present invention, that is, designing a multi-region segmentation network structure, the organ (gallbladder) and small lesions (gallbladder polyps) are segmented at the same time, so that the segmentation task of small lesions (polyps) is assisted by the segmentation task of the organ (gallbladder). The method of multi-task learning (such as 2-task learning) further improves the performance of segmentation and improves the segmentation accuracy.

[0098] For example, the segmentation of the CT image of gallbladder polyps uses the network structure of the cascaded multi-task segmentation network of the present invention, i.e., an embodiment of the neural network model: in the first step, the CT image is input into the first encoding and decoding architecture (task 1), and the entire gallbladder is segmented first. In the second step, the segmentation result of the gallbladder is obtained, and the output binary matrix of the gallbladder segmentation network (task 1) is multiplied by the corresponding position elements of the original image (i.e., the input CT image) matrix to obtain the original image restricted to the gallbladder position area (called the restricted original image), and then the restricted original image is input into the second encoding and decoding architecture (task 2) to obtain the segmentation prediction of the polyp. Such a network structure makes full use of the hierarchical relationship between the two tasks, that is, the mask of the gallbladder contains the mask of the polyp, and the polyp segmentation task is divided into two steps, so that the network can learn the spatial position information between the gallbladder and the polyp, thereby improving the accuracy of polyp segmentation. Moreover, such a cascaded multi-task segmentation network can also be trained end-to-end.

[0099] Preferably, 2. Design a loss function for the corresponding segmentation network constructed above:

[0100] The loss function is a function that measures the error of network prediction. When training a neural network model, the loss function plays a decisive role in updating and especially determining the model parameters. The training process of the neural network model is the process of updating the parameters of the neural network model by optimizing the loss function, and then obtaining the trained model. The image segmentation target can be predicted online or offline, such as target segmentation prediction.

[0101] The image segmentation network may use a loss function such as binary cross entropy (BCE), Dice loss function, etc. In a preferred embodiment, the loss function corresponding to the aforementioned segmentation network may be constructed by combining BCE and Dice to train the segmentation network.

[0102] For the BCE loss function: given an image data and its corresponding ground truth, denoted as X, G respectively, the neural network model is such as the sub-segmentation network in this application, the segmentation prediction result: P = f(X, θ), where P is the network prediction, θ is the network parameter, and H, W are the height and width of the image respectively, the binary cross entropy (BCE) loss function can be defined as:

[0103]

[0104] However, when the target foreground is small, BCE tends to predict the background and leads to overfitting.

[0105] Dice, or the Dice coefficient, is usually used as a metric to evaluate segmentation performance. This evaluation metric can be applied to medical image segmentation, and it can also be used as a loss function. It can also show good performance in experimental calculations, especially for cases where the target foreground is small. Dice is different from cross entropy BCE. Since the Dice coefficient acts on the region rather than on each pixel, it is basically not affected by the proportion of the target foreground. The Dice coefficient measures the degree of overlap between the ground truth and the segmentation network prediction results, and is defined as:

[0106]

[0107] We hope that the prediction results of the segmentation network can make Dice larger, that is, the better the network prediction effect, so the Dice loss function is defined as:

[0108] L Dice =1-Dice

[0109] In this way, the gradient form of the cross entropy loss function BCE defined above is better and is conducive to the back propagation characteristics of the neural network optimization process. Combined with the characteristics of the Dice loss function defined above, it can better handle the segmentation problem of small target foreground. Combining the advantages of both, for each segmentation task, that is, in each sub-segmentation network, the weighted sum of cross entropy and Dice is used as the loss function of the sub-segmentation network:

[0110] L seg =L BCE +α1L Dice

[0111] Among them, α1 is a hyperparameter that adjusts the weights of these two loss functions. Furthermore, for a multi-task (such as two tasks) segmentation network, the sum of the loss functions of each subtask or sub-segmentation network is the loss function of the overall segmentation network:

[0112]

[0113] where seg (1) ,seg (2) They represent different tasks respectively. β is a hyperparameter for adjusting the weights between tasks.

[0114] Further, optimize the loss function:

[0115] Although the aforementioned cross entropy BCE and Dice loss functions are constructed and applied to the segmentation task, when there are various difficulties such as small sample size, small target foreground, low foreground-background contrast, unclear boundaries, etc., the loss function of the segmentation network constructed only based on the two loss functions of BCE and Dice is not very effective. It is further preferred to optimize the loss function of the entire model / segmentation network. For specific tasks, corresponding special loss functions can be designed to constrain the network's prediction, thereby further realizing constraint-based image segmentation.

[0116] For example, for multi-region segmentation of medical images, it is hoped that full use can be made of the relationship between two task target areas in multiple tasks, such as Example 1 of the solution of the present invention. In medical images, lesions usually have a certain geometric characteristic relationship of spatial position with organs. For example, the relationship between organ regions and small lesion regions is the relationship between two task target regions. Although the main target of segmentation is small lesions, with the help of the segmentation and restriction of the organ region where the small lesions are located in the previous subtask, the target, i.e., small lesions, can be more accurately segmented in the latter subtask. Taking the CT image of gallbladder polyps as an example, in the image data set of gallbladder polyps, gallbladder polyps grow on the inner wall of the gallbladder, which contains the location information of the gallbladder and the polyps, that is, the boundary contour of the polyp is basically close to the boundary contour of the gallbladder. It is hoped that the prediction result of the segmentation network can meet this condition as much as possible. Therefore, when constructing the loss function, a punitive loss function is further optimized, that is, the loss function L is given. contour As a punishment

[0117]

[0118] Where dist(x,y) represents the distance between pixel points x and y. is the boundary of the image P, indicating the set where the pixel points are located (the boundary of the set), (1) and (2) represent different corresponding tasks, such as the image P (prediction) predicted by the U-Net segmentation of the first subtask 1, such as the organ region, and the image P predicted by the U-Net segmentation of the second subtask 2, such as the lesion region. In one example, the distance function used is the L2 distance, i.e., the Euclidean distance. Finally, the actual loss function used in the training after optimization is Loss:

[0119]

[0120] Among them, α2 is a hyperparameter for adjusting the weights of the three loss functions.

[0121] Preferably, 3. training the cascaded multi-task segmentation network:

[0122] Based on the network architecture of the segmentation network constructed above and the designed loss function, the segmentation network is trained as a network / model. The training is end-to-end training, which is mainly performed by setting the number of iterations of the training set, such as N times, such as training for N epochs, and converging and ending the network training after N iterations, and then the trained segmentation network can be used.

[0123] In the segmentation network constructed above, ie, the network architecture of the neural network, the gallbladder region segmented in the first subtask is used as the border (bounding area) of the polyp region, ie, the constraint or restriction.

[0124] In the early stage of training the neural network / segmentation network, the parameters are randomly initialized. However, the initialized parameters may exist in the target area (organ area, such as gallbladder area) predicted by the segmentation network: it may be an irregular area without a complete contour, some areas that are not connected to each other, and / or it may be a completely wrong gallbladder area, etc. If such gallbladder areas are used to restrict the polyp area, the training will not only be unstable, but also the network may learn wrong results. In addition, the contour loss function L contour It also acts between the gallbladder contour and the polyp contour. When the gallbladder contour is not predicted well, the effect of such a loss function on the polyp becomes worse. Then, the training process or iteration method is optimized: step-by-step training is performed according to multiple tasks. The sub-segmentation network is first trained in the previous sub-task and then used After a predetermined number of iterations M (M is set according to actual needs), the entire segmentation network is trained as a whole and Loss is used (i.e., the training set is input into the sub-segmentation networks of the two subtasks). For example, the entire training set is input for iteration (from M+1 iterations to the preset N times).

[0125] For example, the training set includes 1,000 medical images (1,000 CT images of gallbladder polyps), and it is assumed to be supervised training, that is, each training sample (image) in the training set is labeled (such as labeled data ground truth). In the experiment, after 40 epochs, the entire segmentation network has performed well in predicting the gallbladder, which is the final small lesion target. The average Dice coefficient can reach above 0.85. It is considered that the network has completed the segmentation of the gallbladder relatively well at this time, and the outline of the gallbladder has been roughly determined and will not change much. Therefore, the step-by-step training method can be used: (1) In the first 40 epochs, only the gallbladder segmentation model, namely Encoder-Decoder-1, is trained. As the loss function for training; (2) After that (starting from the 41st epochs), the entire segmentation network is trained, that is, the cascaded multi-task segmentation network formed by Encoder-Decoder-1 and Encoder-Decoder-2, using As a loss function, the training is performed. This makes the parameters obtained during the training process and the target results predicted during the training more stable, and also makes it easier to optimize the contour loss function L used in the loss function. contour It plays an effective constraint role, that is, the organ area constrains the small lesion area, and the entire segmentation network is also trained end-to-end.

[0126] The trained cascaded multi-task segmentation network is used to predict the actual image to be segmented, such as a gallbladder polyp CT image, online or offline. The first sub-segmentation network predicts the organ target such as the gallbladder area through two sub-networks, and then the second sub-segmentation network predicts the final main small lesion target such as the polyp area, and outputs the prediction result, such as an image with a clear segmentation of the polyp (polyp location).

[0127] In the above embodiment of a specific application scenario of the solution of the present invention, multi-task learning is combined to perform segmentation of organs and lesions at the same time, and the inclusion relationship between the spatial positions of organs and lesions is utilized to design a cascaded U-Net model. After the segmentation of the organ area is obtained, it is used as a bounding area, and the lesion area is further segmented within this area. This provides a priori information for the segmentation prediction of lesions with smaller target foregrounds, allowing the network to find the lesion location more effectively. In addition, the gallbladder polyp segmentation task in the above example further proposes a loss function L based on the spatial geometric distance between the regional contours of the organ and the lesion contour , making full use of the information that the polyp grows on the inner wall of the gallbladder to constrain the network to predict the contour of the area more finely.

[0128] Therefore, according to one embodiment of the solution of the present invention, a constraint-based small lesion segmentation deep learning model / network is implemented to perform target prediction of medical images (for example, segmenting the target contour). For some small lesions on human organs, the situation in which it is difficult to accurately segment using general deep learning methods when the sample size is relatively small is changed. By utilizing the spatial position relationship between the lesion and the organ, a deep learning model with added geometric feature constraints is designed, which effectively reduces the hypothesis space of the sample and obtains more refined segmentation results, thereby obtaining better detection results in the detection of small lesions in small samples, namely predicting the target area or segmenting the target area.

[0129] [Example 3]

[0130] Combine the following Figure 6 The structural block diagram shown in the figure describes an embodiment of the constraint-based image segmentation system according to the present invention, and further illustrates the implementation of the present invention. The system corresponds to an embodiment of the method of the present invention. The details described in the system embodiment of the present invention should be regarded as a supplement to the above method embodiment; the details not disclosed in the system embodiment of the present invention can be implemented with reference to the above method embodiment. Furthermore, the system can also realize the prediction in the above-mentioned practical application scenario, that is, the example described in Example 2.

[0131] In this embodiment of the system, at least:

[0132] Model building module 610: used to build a cascaded multi-task segmentation network.

[0133] In one embodiment, in the cascaded multi-task of the cascaded multi-task segmentation network, each cascaded multi-task learning can adopt a deep neural network. Further, the deep neural network can adopt, for example, a U-Net encoding and decoding architecture as a basic architecture for image segmentation.

[0134] An example of a deep neural network with a U-Net encoder-decoder architecture can be an end-to-end encoder-decoder image segmentation network with 4 downsampling and 4 upsampling, such as Figure 2 shown.

[0135] Among them, core components such as the jump connection between downsampling and upsampling combine shallow and deep semantic features without increasing network parameters, enabling the network to understand the image from the whole to the part.

[0136] Specifically, the image to be segmented is input and feature extraction is performed. For example, the downsampling part (for example, 4 downsamplings), that is, the feature extraction part, is similar to VGG. The feature extraction is performed on the image to be segmented. Each downsampling includes a convolutional layer with a convolution kernel size of 3×3, batch normalization, activation function (ReLU, SoftMax, etc.) and a maximum pooling layer (for example, 2×2). Furthermore, in the downsampling part, each pooling layer has a scale, including the original image scale, for a total of 5 scales; and each downsampling reduces the size of the feature map by half, while the number of channels increases exponentially.

[0137] Specifically, in the upsampling part, each upsampling also includes the convolution kernel conv size of 3×3, batch normalization, activation function and max-pooling layer (for example: 2×2). Furthermore, in the upsampling part, the conv and deconv corresponding to the downsampling are upsampled 4 times to obtain 4 scale images, including the final output segmentation image of 5 scales; and, corresponding to the downsampling part, and just the opposite of the downsampling part, each upsampling (feature map) is fused with the same scale as the number of channels corresponding to the feature extraction part. Preferably, the feature map crop can be cut before fusion (for example, copy copy + cut crop feature map). Wherein, the fusion is splicing.

[0138] See also Figure 2 In the schematic example, in the left downsampling / downsampling part, the convolution kernel conv is 3×3, batch normalization, activation function, and maximum pooling. Each downsampling generates a channel. After copying and cutting the feature map of each downsampling, in the upsampling part on the right, each upsampling, the same convolution kernel deconv is 3×3, batch normalization, activation function, and maximum pooling. Each upsampling is combined with the feature map after cutting the crop, and the scale is the same as the number of channels. For example, cutting to 392×392, 200×200, 104×104, etc. Further, the last conv can be 1×1, and then the segmented image with increased resolution is output.

[0139] Preferably, for the example of image segmentation of medical images, 3 downsampling and 3 upsampling can be used. If the image to be segmented is an image of a small lesion (e.g., an image of a gallbladder polyp), for segmenting such targets, a neural network of a U-Net encoding and decoding framework with 3 downsampling and 3 upsampling can be used as the basic neural network, i.e., the basic framework, for each task in the cascaded multi-task. In this way, in order to adapt to the size of the image to be segmented, the depth of each U-Net encoding and decoding neural network in the cascade is reduced by one layer, i.e., one downsampling and one upsampling are reduced, so that the model is simpler and the parameters are reduced, and while reducing the amount of parameters, overfitting can be prevented to a greater extent. The preferred structural hierarchy of the U-Net encoding and decoding network is, for example Figure 3 shown.

[0140] In one embodiment, see Figure 4The cascaded multi-task structure shown, for example, can adopt a schematic diagram of two cascaded tasks. Specifically, the first task Task1, inputs the image to be segmented into the first encoding and decoding neural network (such as U-Net) of the cascade, segments the structural area where the target is located, and processes the segmented structural area where the target is located in combination with the image to be segmented (original image) (for example, limiting it to the original image), and then enters the next task Task2, that is, inputs it into the second encoding and decoding neural network (such as U-Net) of the cascade, and performs segmentation prediction, such as segmenting the target by segmenting again. The structure of the cascaded multi-task neural network in this embodiment makes full use of the hierarchical relationship between the two tasks, that is, the mask of the structural area where the target is located contains the mask of the target, and the target segmentation task is divided into two steps, so that the network can learn the spatial position information between the structural area where the target is located and the target, such as the spatial position information between small lesions (such as polyps) and the organs (such as gallbladder) in medical images, thereby improving the accuracy of target (small lesions such as polyps, etc.) segmentation.

[0141] Loss function construction module 620: constructs the loss function of the cascaded multi-task segmentation network.

[0142] In one embodiment, a corresponding loss function is constructed for the cascaded multi-task segmentation network. The loss function is a function that measures the error of network prediction. When training a neural network model, the loss function plays a decisive role. The training process of the neural network is the process of updating the network parameters by optimizing the loss function. Specifically, in the cascaded multi-task segmentation network, a binary cross entropy BCE and / or Dice loss function can be constructed.

[0143] For example, in supervised learning, we can give an image and its corresponding groundtruth annotation, denoted as X and G respectively, and the neural network model P = f(X, θ), where P is the network prediction (for example: the segmentation prediction of the segmentation network U-Net used in the first cascade task); θ is the network parameter, denoted as H and W, respectively, the height and width of the image, and the binary cross entropy loss function BCE is defined as:

[0144]

[0145] Wherein, i and j are both natural numbers greater than or equal to 1.

[0146] When the target foreground is very small, BCE tends to predict the background and leads to overfitting.

[0147] Since the Dice coefficient is used as a metric to evaluate segmentation performance, especially in medical image segmentation, it can be used as a loss function in the segmentation network to evaluate or update model parameters, which can show good performance, especially when the target foreground is small. Unlike cross entropy, since the Dice coefficient acts on the region (such as the structural region where the target is located) rather than on each pixel, it is basically not affected by the proportion of the target foreground. Therefore, the Dice coefficient can be used to measure the degree of overlap between the ground truth and the network prediction results, that is, the degree of overlap is defined as:

[0148]

[0149] Then, according to the prediction result of the desired network, Dice is made larger, and the Dice loss function is constructed, which is defined as:

[0150] L Dice =1-Dice

[0151] Therefore, the cross entropy loss function can be used to help the back propagation of the neural network optimization process during training, while the Dice loss function can better handle the segmentation problem of small target foreground, taking advantage of their respective advantages. Preferably, for each segmentation task, that is, the segmentation network (the network of the U-Net infrastructure) in each of the two tasks in the aforementioned embodiment, the weighted sum of the BCE and Dice loss functions can be used as the loss function L of the segmentation network. seg :

[0152] L seg =L BCE +α1L Dice

[0153] Among them, α1 is a hyperparameter for adjusting the weights of these two loss functions.

[0154] Furthermore, the loss function of the entire cascaded multi-task segmentation network is the loss function of multiple tasks to The sum of (n is a natural number greater than or equal to 1), preferably, the loss function of the segmentation network of the above two tasks is:

[0155]

[0156] where seg (1) ,seg (2) They represent different tasks respectively. β is a hyperparameter for adjusting the weights between tasks.

[0157] Furthermore, the cross entropy and Dice loss functions have certain advantages when applied to segmentation tasks, especially when used in combination. However, when there are various difficulties such as small sample size, small target foreground, low foreground-background contrast, unclear boundaries, etc., for specific tasks, more special loss functions can be designed based on the two to constrain the network's segmentation prediction in order to achieve better and more accurate segmentation effects.

[0158] In a preferred embodiment, for multi-region segmentation, such as the example of medical image segmentation of small lesions in the above example, it is hoped to make full use of the relationship between two task target areas, such as the relationship between small lesions and organs. Specifically, taking medical images as an example, lesions usually have a certain geometric characteristic relationship of spatial position with organs, such as: in the data set of gallbladder polyps, gallbladder polyps grow on the inner wall of the gallbladder, which contains the location information of the gallbladder and the polyps, that is, the boundary contour of the polyp is basically close to the boundary contour of the gallbladder. Based on this condition, it is hoped that the segmentation prediction result of the network can meet this condition as much as possible, and a loss function can be given as a penalty:

[0159]

[0160] Where dist(x,y) represents the distance between pixel points x and y. For example, the L2 distance is used. The boundaries of the image P predicted for different tasks (subtasks) indicate the set to which the pixel belongs. In this way, the actual loss function Loss of the entire segmentation network is the sum of the multi-task loss function and the conditional loss function:

[0161]

[0162] Training module 630: training the cascaded multi-task segmentation network.

[0163] In one embodiment, the constructed cascaded multi-task segmentation network is iteratively trained, and the parameters of the network are updated according to the loss function of each iteration until a preset number of iterations is reached, and the iteration is terminated to complete the training. This training can be end-to-end training, that is, multiple iterations of the entire process from inputting an image to be segmented to outputting a segmentation prediction image each time.

[0164] A specific example of training is mainly small lesions and organ images of medical images:

[0165] In the constructed cascade multi-task segmentation network, the organ region (e.g., gallbladder region) is used as the bounding area of ​​the small lesion region (e.g., polyp region).

[0166] Furthermore, at the beginning of neural network training, the parameters of the entire segmentation network are randomly initialized. In this way, the segmented image predicted by the segmentation network may contain irregular areas without complete contours, or some areas that are not connected to each other, or completely wrong organ areas. Therefore, using organ areas to limit small lesion areas in the early stage will not only make the training unstable, but also cause the network to learn wrong results. In addition, the penalty L contour The loss function also acts between the organ contour (such as the gallbladder contour) and the small lesion contour (such as the polyp contour). When the organ contour is not predicted well, the effect of the loss function in constraining the small lesion will be reduced.

[0167] Therefore, preferably, the cascaded multi-task segmentation network can be trained in a step-by-step training manner. More specifically, for example: Step SX1: Perform N iterations on the training set, and train the segmentation network in the first task in the first M iterations (for example, only perform segmentation prediction on the target structure area), wherein, As the loss function for training; Step SX2: Starting from NM iterations until the Nth iteration, train the entire cascaded multi-task segmentation network (for example: the network of the first task and the second task that completes the segmentation prediction process), where the use As the loss function for training. In the preferred method, the network training process and the subsequent segmentation prediction process will be more stable and more able to make L contour It plays a more effective constraint role and has higher segmentation accuracy. At the same time, the entire segmentation network (i.e., the overall model) has also completed end-to-end training.

[0168] Among them, epochs is defined as a single training iteration of all batches in forward and backward propagation, that is, 1 cycle is a single forward and backward pass of the entire input data. For example: if the training set has 1000 samples, then: it takes 1 epoch to train the entire sample set.

[0169] For example, for medical images such as gallbladder polyps, image segmentation is performed, and model training is performed through the constructed model and loss function. The model is the cascaded multi-task segmentation network constructed above. Figure 5In the training process shown in the figure, after 40 epochs, the network has performed well in predicting gallbladders, and the average Dice coefficient can reach above 0.85. In this way, it can be determined how many epochs (iterations) of training the network has completed relatively good segmentation of the gallbladder, and the outline of the gallbladder has been roughly determined and will not change much. Using the step-by-step training method, N can be set according to actual needs. In this example, the first step of training is (1) in the first 40 epochs (i.e., M), only the gallbladder segmentation model is trained, i.e., Task 1 (Encoder-Decoder-1), using As the loss function, the second step is to train the entire model after (2), that is, the entire task 1 to task 2 after M (both Encoder-Decoder-1 and Encoder-Decoder-2 are included), using The training is performed as the loss function. M and N are natural numbers greater than or equal to 1, and N is greater than M.

[0170] The prediction module 640 performs target segmentation prediction on the image to be segmented based on the cascaded multi-task segmentation network.

[0171] Specifically, the image to be segmented (i.e., the image from which the target in the image needs to be segmented) is input into the trained cascaded multi-task segmentation network to perform segmentation prediction to obtain the target segmented result in the image to be segmented.

[0172] For example, when performing constraint-based image segmentation processing on medical images, the polyp lesions of the gallbladder organ are taken as the target. After the first task of the segmentation network, i.e., the first encoding-decoding architecture network, the gallbladder region is obtained, and the gallbladder region and the original image to be segmented are processed together. For example, after the first task, a binary matrix of the gallbladder region is obtained, and then multiplied with the corresponding position elements of the original image matrix, the original image restricted to the gallbladder position area is obtained, which is input into the second task, i.e., the second encoding-decoding architecture network, to obtain the polyp region of the gallbladder position area, i.e., the segmentation prediction of the target polyp. Such a network structure makes full use of the hierarchical relationship between the two tasks, i.e., the mask of the gallbladder contains the mask of the polyp, and the polyp segmentation task is divided into two steps, so that the network can learn the spatial position information between the gallbladder and the polyp, thereby improving the accuracy of polyp segmentation.

[0173] The following experimental comparison data illustrate that the image segmentation performed by the above-mentioned embodiments of the solution of the present invention improves the accuracy of medical images compared with the image segmentation performed by the existing neural network model:

[0174] (1) Comparative experiment

[0175] The results of polyp segmentation using the cascaded multi-task network technology proposed in the present invention and the U-Net segmentation network were compared through experiments. In order to verify the stability of the model, the original data set was randomly divided into three data sets, and then the training set and the test set were divided into a 9:1 ratio and experimented. The best performing model was selected for each experiment. When comparing, for each model, the parameters were adjusted to make the network perform best in polyp segmentation.

[0176] Table 1 shows the comparison results: As can be seen from Table 1, due to the difficulty of the data set itself, the final performance of directly using U-Net to segment polyps is poor, and the average Dice coefficient on the test set is only 0.5724. The constraint-based multi-region segmentation method we proposed has learned additional spatial information, and its performance is much higher than that of directly using U-Net for segmentation. The accuracy is significantly improved. The Dice coefficient of gallbladder segmentation reaches 0.9154 and the Dice coefficient of meat segmentation reaches 0.7467.

[0177] Table 1:

[0178]

[0179] See also Fig. 9 The example of the segmentation result comparison experiment shown is: (a) original image; (b) polyp ground truth; (c) direct use of U-Net segmentation result; (d) cascade network segmentation result. From the segmentation results of the two networks shown in the figure, it can be seen that the cascaded multi-task segmentation network of the present invention has a significant improvement over the direct use of U-Net segmentation. When the task is relatively simple and the network can find the location of the polyp, the segmentation result of the cascaded multi-task segmentation network is more refined. When the task is more difficult, there is a large difference between the U-Net segmentation result and the annotated ground truth (the target or predicted target area that should be segmented out of the sample image in the training set manually annotated during training), that is, the accuracy is low, while the cascaded multi-task segmentation network still performs well, and the advantage is more obvious at this time. In addition, it is very easy to overfit when directly using U-Net to segment polyps. Although its network can perform well on the training set, it performs poorly on the test set. The technical solution of the image segmentation based on constraints of the present invention, through multi-region segmentation based on constraints, adds additional loss functions for constraints when optimizing the loss function when constructing the loss function, which greatly alleviates the overfitting phenomenon, and the training is more stable than U-Net.

[0180] (2) Ablation experiment

[0181] In order to further verify the effectiveness of the loss function optimized and constructed in the technical solution of the constraint-based image segmentation of the present invention, an ablation experiment was conducted to compare the results of training using different loss functions. As shown in Table 2, the Dice coefficient comparison (%) is compared by comparing the predicted contour and the ground truth contour. In Table 2, when the loss function without introducing spatial position constraints is used, L contour After the constraint of , the performance of the network is significantly improved. Since the gallbladder segmentation task is relatively simple, the overall improvement is not very obvious. For polyps that are more difficult to segment, after the spatial position constraint is introduced, the Dice coefficient is increased by about 6 percentage points on average. This fully demonstrates the effectiveness of the optimized loss function on difficult samples.

[0182]

[0183] Furthermore, in order to more clearly see the advantages of the loss function constructed after optimization in the technical solution of the present invention, the outline of the network prediction and the outline of the ground truth are drawn in the same figure for comparison. In order to explicitly show the superiority of the loss function constructed after optimization, the ground truth of the gallbladder area is specially drawn as a reference. Fig.10 Comparison of segmentation results. The black part represents the polyp contour predicted by the network, and the gray part represents the polyp ground truth contour, i.e. the annotated contour. (a) Using L seg as a loss function for training; (b) using L seg +L contour As the loss function for training; (c) Gallbladder groundtruth. The figure shows L contour The black line represents the polyp contour predicted by the network, and the gray line represents the polyp ground truth contour. (a) is the case without L contour The segmentation result of (b) is the one with L added contour The segmentation result of (c) is the gallbladder region ground truth. By comparison, we can see that after adding L contour After optimization, the polyp boundary predicted by the network in (b) can be closer to the gallbladder boundary, which refines the predicted polyp contour and reduces the probability of false negative prediction. This fully verifies the geometric constraint effect of the loss function constructed after optimization.

[0184] The technical solution of the present invention combines multi-task learning through a cascaded multi-task segmentation network based on U-Net and a loss function based on spatial position geometric constraints, and simultaneously performs segmentation of organs and lesions. Specifically, the inclusion relationship between the spatial position of the organ and the lesion is utilized, and a cascaded U-Net model is designed to provide a priori information for the segmentation problem of lesions with smaller target prospects, which can more effectively find the location of the lesion; further, a loss function based on the spatial geometric distance between the regional contours of the organ and the lesion (gallbladder and polyp) is proposed, which makes full use of the information that the polyp grows on the inner wall of the gallbladder to constrain the predicted contour to be more refined. As a result, the segmentation of medical images is improved and the segmentation accuracy is effectively improved. Furthermore, the technical solution of the present invention also preferably calculates the distance between different target contours as the loss function for optimizing the segmentation network; a step-by-step training method is adopted during the network training process, which can greatly improve the prediction accuracy of the network.

[0185] [Example 4]

[0186] Specifically, an embodiment of an electronic device is also included, which includes a processor and a memory storing computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to perform the embodiment steps of any method described in the aforementioned embodiment 1.

[0187] An embodiment of the electronic device of the present invention is described below, and the electronic device can be regarded as a physical implementation of the method and device embodiments of the present invention described above. The details described in this embodiment of the electronic device of the present invention should be regarded as a supplement to the embodiments of the above method or device / system; details not disclosed in the embodiment of the electronic device of the present invention can be implemented with reference to the above method or device / system embodiments.

[0188] Figure 7 is a structural block diagram of an exemplary embodiment of an electronic device according to the present invention. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0189] like Figure 7 As shown, the electronic device 200 of this exemplary embodiment is in the form of a general data processing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, etc.

[0190] The storage unit 220 stores a computer-readable program, which may be a source program or a code of a read-only program. The program may be executed by the processing unit 210, so that the processing unit 210 performs the steps of various embodiments of the present invention. For example, the processing unit 210 may perform the various steps of the methods of the aforementioned embodiments 2 to 5.

[0191] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 2201 and / or a cache storage unit 2202, and may further include a read-only storage unit (ROM) 2203. The storage unit 220 may also include a program / utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0192] Bus 230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0193] The electronic device 200 may also communicate with one or more external devices 300 (e.g., keyboard, display, network device, Bluetooth device, etc.) so that a user can interact with the electronic device 200 via these external devices 300, and / or the electronic device 200 can communicate with one or more other data processing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 250, and may also be performed through a network adapter 260 with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet). The network adapter 260 may communicate with other modules of the electronic device 200 through the bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0194] [Example 5]

[0195] Specifically, it also includes a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by a processor, the embodiment steps of any method described in the above-mentioned embodiment 1 are implemented.

[0196] Figure 8Schematic diagram of a computer readable medium embodiment of the present invention. Figure 8 As shown, the computer program can be stored on one or more computer-readable media. The computer-readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. When the computer program is executed by one or more data processing devices, the computer-readable medium is enabled to implement the above method of the present invention, namely: the configuration service management provided by the server generates a configuration file related to the domain name; when a domain name failure occurs during the request for a business service, the client implements the automatic switching of domain name access by updating the configuration according to the configuration file.

[0197] Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present invention can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation mode of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a data processing device (which can be a personal computer, a server, or a network device, etc.) to perform the above method according to the present invention.

[0198] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0199] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0200] In summary, the present invention can be implemented by a method, system, electronic device or computer readable medium that executes a computer program. In practice, a general data processing device such as a microprocessor or a digital signal processor (DSP) can be used to implement some or all functions of the present invention.

[0201] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A constraint-based image segmentation method, characterized in that: include: Construct a cascaded multi-task segmentation network and a loss function based on spatial position geometric constraints; Training the cascaded multi-task segmentation network according to the loss function; Inputting the image to be segmented into the trained cascade multi-task segmentation network to obtain a predicted segmentation target area, including: inputting the actual medical image to be segmented into the trained cascade multi-task segmentation network, first predicting the organ area in the medical image through the sub-segmentation network of the previous sub-task in the cascade multi-task segmentation network, multiplying the binary matrix of the organ area with the elements of the matrix corresponding to the original image of the medical image to obtain the original image limited to the organ area, inputting the original image limited to the organ area into the sub-segmentation network of the next sub-task of the cascade multi-task segmentation network, and then predicting the lesion area in the organ area through the sub-segmentation network of the next sub-task; Constructing a loss function based on spatial position geometric constraints specifically includes: The loss function of the cascaded multi-task segmentation network is defined as: ; Among them, seg (1) ,seg (2) represents the sub-loss function of the sub-segmentation network corresponding to different sub-tasks, and β is the hyperparameter for adjusting the weights between multiple tasks; Among them, each sub-loss function is combined according to the binary cross entropy BCE and Dice loss function Defined as: ; Among them, L BCE is the binary cross entropy loss function, L Dice is the Dice loss function, α1 is the adjustment L BCE , L Dice The hyperparameters of the weights of these two loss functions, n represents the corresponding nth subtask and n is a natural number greater than or equal to 1; The optimized loss function Loss of the cascaded multi-task segmentation network is defined as: ; Among them, L loc To optimize the loss function, α2 is a hyperparameter for adjusting the weights of these three loss functions, L contour It is a contour loss function constrained according to the spatial position geometric feature relationship, which is defined as: ; Where dist(x,y) represents the L2 distance between pixel points x and y; is the boundary of the image P representing the predictions of different subtasks to indicate the set to which the pixel points x and y belong.

2. The method according to claim 1, characterized in that The construction of the cascaded multi-task segmentation network specifically includes: Construct at least two subtasks to perform multi-region segmentation of the image; The sub-segmentation network used in each subtask is based on the U-Net encoder-decoder network architecture; The feature map sampling method adopted by the U-Net encoding and decoding network architecture is: 3 downsampling and 3 upsampling, or 4 downsampling and 4 upsampling; Among them, the binary matrix of the first target area predicted by the sub-segmentation network of the previous subtask is multiplied by the elements of the corresponding positions of the matrix of the original image, and then the sub-segmentation network of the next subtask predicts the second target area limited to the first target area.

3. The method according to claim 1 or 2, characterized in that The cascaded multi-task segmentation network is trained according to the loss function, specifically including adopting a distributed training method: First, input the labeled image samples in the training set to perform a preset M-times iterative training on the sub-segmentation network of the previous sub-task in the cascaded multi-task segmentation network, wherein the loss function used in the training is the sub-loss function ; After the number of training times reaches M times, the labeled image samples in the training set are input to perform preset NM times of iterative training on the cascaded multi-task segmentation network as a whole, wherein the optimized loss function Loss is used during the training; The preset number of training iterations of the cascaded multi-task segmentation network is N, where N and M are natural numbers greater than or equal to 1 and N is greater than M; Among them, iteration 1 refers to a single epoch iteration.

4. The method according to claim 1 or 2, characterized in that: The image to be segmented includes a gallbladder polyp CT image; The predicted segmentation target area includes multi-target area prediction; The cascaded multi-task segmentation network is used to predict the gallbladder region in the previous subtask, and then based on the spatial restriction of the gallbladder region, the polyp region within the restriction of the gallbladder region is predicted in the subsequent subtask.

5. The method according to claim 3, characterized in that The image to be segmented includes a gallbladder polyp CT image; The predicted segmentation target area includes multi-target area prediction; The cascaded multi-task segmentation network is used to predict the gallbladder region in the previous subtask, and then based on the spatial restriction of the gallbladder region, the polyp region within the restriction of the gallbladder region is predicted in the subsequent subtask.

6. A constraint-based image segmentation system, characterized in that: include: Model building module for building cascaded multi-task segmentation networks; The loss function construction module is used to construct a loss function based on spatial position geometric constraints, including: The loss function of the cascaded multi-task segmentation network is defined as: ; Among them, seg (1) ,seg (2) represents the sub-loss function of the sub-segmentation network corresponding to different sub-tasks, and β is the hyperparameter for adjusting the weights between multiple tasks; Among them, each sub-loss function is combined according to the binary cross entropy BCE and Dice loss function Defined as: ; Among them, L BCE is the binary cross entropy loss function, L Dice is the Dice loss function, α1 is the adjustment L BCE , L Dice The hyperparameters of the weights of these two loss functions, n represents the corresponding nth subtask and n is a natural number greater than or equal to 1; The optimized loss function Loss of the cascaded multi-task segmentation network is defined as: ; Among them, L loc To optimize the loss function, α2 is a hyperparameter for adjusting the weights of these three loss functions, L contour It is a contour loss function constrained according to the spatial position geometric feature relationship, which is defined as: ' Where dist(x,y) represents the L2 distance between pixel points x and y; is the boundary of the image P representing the predictions of different subtasks to indicate the set to which the pixels x and y belong; A training module, used for training the cascaded multi-task segmentation network according to the loss function; The prediction module is used to input the image to be segmented into the trained cascaded multi-task segmentation network to obtain the predicted segmentation target area, including: inputting the actual medical image to be segmented into the trained cascaded multi-task segmentation network, first predicting the organ area in the medical image through the sub-segmentation network of the previous sub-task in the cascaded multi-task segmentation network, multiplying the binary matrix of the organ area with the elements of the matrix of the original image of the medical image at corresponding positions to obtain the original image restricted to the organ area, inputting the original image restricted to the organ area into the sub-segmentation network of the next sub-task of the cascaded multi-task segmentation network, and then predicting the lesion area in the organ area through the sub-segmentation network of the next sub-task.

7. An electronic device comprising a processor and a memory storing computer executable instructions, characterized in that: The computer executable instructions, when executed, cause the processor to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images