An image segmentation method based on loss function of complementary information

Abstract

This application provides an image segmentation method based on a complementary information loss function. The method comprises: obtaining an image to be segmented; inputting the image to be segmented into a multi-layer deep learning model to obtain a mask including the lesion area; wherein the multi-layer deep learning model is supervised by a loss function, wherein the loss function includes a complementary information-based false positive-negative loss function, which is used to characterize the complementary information between feature maps of different layers. This solution fully utilizes the complementary information between feature maps of different layers, performs targeted optimization on each layer of feature maps to improve segmentation results, and has higher detection accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to an image segmentation method based on a loss function of complementary information. Background Art

[0002] Automatically and accurately segmenting lesions in ultrasound images using computers is crucial for computer-assisted clinical examination and treatment. This task can be formulated as a binary labeling problem for a single ultrasound image, i.e., using a computer-assisted system to automatically perform pixel-level labeling on lesions in ultrasound data.

[0003] Recently, deep learning models, leveraging deep network architectures and a large number of trainable parameters, have demonstrated outstanding performance in image processing. While many researchers have proposed novel and high-performing multi-layer deep learning model architectures, there has been little innovation in the training methods for deep learning models.

[0004] Currently, a common approach to training multi-layer deep learning models is to use a loss function for deep supervision. Specifically, this directly constrains the feature maps of each layer. However, the cross-entropy loss function only quantifies the difference between each layer's feature maps and the ground truth; it cannot constrain the relationships between feature maps. Consequently, deep learning models cannot optimize each feature map based on the complementary information between them, resulting in unsatisfactory segmentation accuracy. Summary of the Invention

[0005] The purpose of the embodiments of this specification is to provide an image segmentation method based on a loss function of complementary information.

[0006] To solve the above technical problems, the embodiments of the present application are implemented in the following ways:

[0007] This application provides an image segmentation method based on a loss function of complementary information, the method comprising:

[0008] Obtain the image to be segmented;

[0009] The image to be segmented is input into a multi-layer deep learning model to obtain a mask including the lesion area;

[0010] Among them, a loss function is used to supervise the training of the multi-layer deep learning model. The loss function includes a false positive-negative loss function based on complementary information. The false positive-negative loss function based on complementary information is used to characterize the complementary information between feature maps of different layers.

[0011] In one embodiment, the false positive-negative loss function based on complementary information includes a false positive loss function and a false negative loss function;

[0012] Among them, the false positive loss function is used to suppress the feature map of deep low spatial resolution to reduce the non-lesion area from being mistaken for the lesion area;

[0013] The false negative loss function is used to constrain the shallow high spatial resolution feature map to reduce the misclassification of lesion areas as non-lesion areas.

[0014] In one embodiment, the false positive-negative loss function based on complementary information is determined by weighting the false positive loss function and the false negative loss function.

[0015] In one embodiment, the i-th layer feature map F i The false positive loss function L FPi for:

[0016]

[0017] Among them, GT represents the real lesion area corresponding to the current image to be segmented, and It is a reversal of GT, FP g is a feature map F from a deeper layer relative to the current layer feature map g Get the false positive segmentation mask in L Dice False positive segmentation mask FP representing the feature map g and The Dice loss function value between .

[0018] In one embodiment, the false positive segmentation mask FP g Through the feature map F g The 1×1 convolution layer and Sigmoid function on it are calculated as follows:

[0019] FP g =Sigmoid(Conv(F g \GT))

[0020] Among them, Conv represents the feature map F g 1×1 convolution layer on top; Sigmoid() represents the sigmoid activation function; F g \GT represents the feature map F g and the ground truth lesion region.

[0021] In one embodiment, the false negative loss function L FNi for:

[0022] L FNi =L Dice (FN i ,R g )

[0023] Among them, FN i is the feature map F of the i-th layer i Corresponding false negative segmentation template; R g Represents the feature map F g Segmentation result of L Dice Indicates FN i and R g The Dice loss function value between .

[0024] In one embodiment, the false negative segmentation template FN i Through the real lesion area GT corresponding to the current image to be segmented and the current i-th layer feature map F i Calculation yields:

[0025] FN i =GT-GT∩Sigmoid(Conv(F i ))

[0026] Among them, Sigmoid() represents the sigmoid activation function, and Conv represents the 1×1 convolutional layer.

[0027] In one embodiment, the loss function further includes a segmentation loss function.

[0028] In one embodiment, the segmentation loss function includes a Dice loss function and a CE loss function; the Dice loss function and the CE loss function are arranged in each layer of the multi-layer deep learning model to constrain the difference between the feature map of the current layer and the ground truth result.

[0029] In one embodiment, the loss function is determined by weighting a false positive-negative loss function based on complementary information and a segmentation loss function.

[0030] It can be seen from the technical solution provided in the above embodiments of this specification that this solution: fully utilizes the complementary information between feature maps of different layers, performs targeted optimization on the feature maps of each layer to improve the segmentation results, and has higher detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1A flowchart of the image segmentation method based on the complementary information loss function provided in this application;

[0033] Figure 2 A visual diagram of the feature map generation process in the false positive-negative loss function based on complementary information provided in this application;

[0034] Figure 3 Schematic diagram of the training method for the multi-layer deep learning model provided in this application. DETAILED DESCRIPTION

[0035] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0036] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0037] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments described herein without departing from the scope or spirit of the present application. Other embodiments will be apparent to those skilled in the art from the present description. The present description and examples are intended to be illustrative only.

[0038] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0039] Unless otherwise specified, "parts" in this application are calculated by mass.

[0040] In related technologies, deep learning models trained using the Deep Supervision training method have an encoder that generates four feature maps (F1, F2, F3, F4) of different spatial resolutions for a multi-layer deep learning model. The decoder of the multi-layer deep learning model then uses these four feature maps (F1, F2, F3, F4) as input and sequentially generates three decoder feature maps (D1, D2, D3) through upsampling and convolution operations. For deep supervision training, each decoder feature map (D1, D2, D3) is measured against the ground truth using a loss function, generating three corresponding loss function values ​​(L1, L2, L3). The multi-layer deep learning model then uses these three loss function values ​​(L1, L2, L3) to update the model parameters and optimize the network parameters.

[0041] The above-mentioned deep supervision deep learning model training method is proposed for natural images; and because this training method measures the feature map (D1, D2, D3) of each layer separately with the ground truth result (GroundTruth), it cannot characterize the correlation between feature maps of different layers, and the deep learning model cannot mine the feature information missing from the feature map of the current layer from the feature map of other layers.

[0042] Based on the above-mentioned defects, this application combines the characteristics of the lesion area (or called lesion area) in the ultrasound image and proposes an image segmentation method based on the loss function of complementary information. It can specifically constrain the relationship between the feature maps of different layers in the multi-layer deep learning model, and then in the training process, the deep learning model optimizes and completes the feature maps of each layer in a targeted manner, greatly improving the accuracy of lesion segmentation, and providing more possibilities for the application of computer-assisted systems based on deep learning in the field of clinical medical imaging.

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0044] Reference Figure 1 , which shows a flow chart of an image segmentation method based on a loss function of complementary information applicable to an embodiment of the present application.

[0045] like Figure 1 As shown in FIG, the image segmentation method based on the loss function of complementary information may include:

[0046] S110: Obtain the image to be segmented.

[0047] Specifically, the image to be segmented may be an ultrasound image, which may be an image in a stored data set or an image collected clinically.

[0048] S120, inputting the image to be segmented into a multi-layer deep learning model to obtain a mask including the lesion area;

[0049] Among them, a loss function is used to supervise the training of the multi-layer deep learning model. The loss function includes a false positive-negative loss function based on complementary information. The false positive-negative loss function based on complementary information is used to characterize the complementary information between feature maps of different layers.

[0050] Specifically, the false positive-negative loss function based on complementary information enables a multi-layer deep learning model (or simply a deep learning model, or an ultrasound image segmentation network based on a false positive-negative loss function of complementary information, or an ultrasound image segmentation network, etc.) to further suppress non-lesion tissue noise in the shallow feature map and enhance more target liver lesion tissue details in the deep feature map.

[0051] The false positive-negative loss function based on complementary information used when training multi-layer deep learning models can characterize the complementary information between feature maps of different layers, helping the multi-layer deep learning model to perform targeted optimization and completion of each layer of feature maps, which is used to improve the speed and accuracy of automatic detection of lesions in computer-assisted ultrasound images.

[0052] Among them, the false positive-negative loss function based on complementary information includes a false positive loss function and a false negative loss function;

[0053] Among them, the false positive loss function is used to suppress the feature maps of deep layers with low spatial resolution to minimize the possibility that non-lesion areas are mistaken for lesion areas;

[0054] The false negative loss function is used to constrain the shallow high spatial resolution feature maps to reduce the possibility that the lesion area is misclassified as a non-lesion area.

[0055] like Figure 2 The following is a visualization of the feature map generation process in the false positive-negative loss function based on complementary information (or simply the false positive-negative loss function): FP g 、 R g and FN i Among them, the yellow rectangle represents the feature map F of a deeper layer relative to the current layer feature map g Tumor segmentation area, the blue rectangle represents the feature map F of the current layer (i-th layer) i The segmented tumor area is shown in Figure 1, and the green rectangle is the ground truth tumor area. Note that only the pixel values ​​in the pink area are set to 1, while the other pixel values ​​are set to 0.

[0056] The false positive - negative loss function L based on complementary information proposed in this application FPN is used to complement the missing tumor lesion regions in each layer of feature map F i (1 < i < 4) of the multi - layer deep learning model and suppress the non - tumor region parts located in the potential tumor regions.

[0057] Specifically, we first obtain the false positive segmentation mask FP g from the feature map F g one layer deeper than the current layer's feature map (see Figure 2 ). FP g can be calculated by using a 1×1 convolutional layer and the Sigmoid function on F g :

[0058] FP g = Sigmoid(Conv(F g \GT))

[0059] where Conv represents the 1×1 convolutional layer on the feature map F g . Sigmoid() represents the sigmoid activation function. F g \GT represents the set difference operation between the feature map F g and the ground truth tumor region. Then, we use the classic Dice metric in image segmentation to represent the difference between FP g and to guide the training of the deep learning model, thereby eliminating the non - liver regions at FP g , denoted by letters as

[0060] By doing so, the non - liver regions in the feature map F i are further suppressed. The definition of the loss function at the i - th layer is given by the following formula: The loss function at the i - th layer is defined as follows:

[0061]

[0062] where GT represents the true tumor region corresponding to the current ultrasound image (i.e., the image to be segmented), and is the reverse of GT, that is: the places where the pixel value in GT is 1 become 0, and the places where the pixel value in GT is 0 become 1. L Dice represents the Dice loss function value between the feature map FP g and .

[0063] We further use GT and the current i - th layer feature map F iCalculate and get F i Corresponding false negative segmentation template FN i Specifically, FN i It is to use GT minus GT and F i The mathematical formula is as follows:

[0064] FN i =GT-GT∩Sigmoid(Conv(F i ))

[0065] Among them, Conv represents a 1×1 convolution layer. In addition, we use R g To express F g The segmentation result, so R g can be calculated as:

[0066] R g =Sigmoid(Conv(F g ))

[0067] Then, we use the classic indicator Dice in image segmentation to represent the FN in the i-th layer i With R g The difference between In this way, we can guide the deep learning model to better complete the feature map F of layer i i The missing tumor lesion area.

[0068] Mathematically, The calculation formula can be expressed as:

[0069]

[0070] Among them, L Dice Indicates FN i and R g The Dice loss function value between .

[0071] Finally, we take the loss function value corresponding to each layer of feature map in the network and The weighted addition gives the entire false positive-negative loss function L FPN :

[0072]

[0073] The weight λ1 can be set according to experimental results, for example, to 1.

[0074] It is understandable that the Dice loss function used in the loss function of the embodiment of the present application can adopt the loss function of other processes, such as cross entropy (CE).

[0075] In one embodiment, the loss function also includes a segmentation loss function. Seg Including Dice loss function and CE loss function; Dice loss function and CE loss function are arranged in each layer of the multi-layer deep learning model to constrain the difference between the current layer feature map and the ground truth result (GroundTruth), and guide the training of the multi-layer deep learning model.

[0076] By adding L FPN And the segmentation loss function L Seg Adding together, we can get the overall loss function L of the network Total :

[0077] L Total =L FPN +λ2L Seg

[0078] Among them, the weight λ2 is used to balance L FPN and L Seg , λ2 can be set according to experimental results, for example, it can be set to 1 (ie, λ2=1).

[0079] This application minimizes the overall loss function L Total .

[0080] like Figure 3 The figure shows a schematic diagram of the training method of a multi-layer deep learning model. First, the input image is input into a multi-layer encoder, and the encoder generates feature maps (F1, F2, F3, F4) of different spatial resolutions in turn. Next, the feature maps (F1, F2, F3, F4) generated by the encoder are input into a multi-layer decoder in turn, and the decoder generates multiple feature maps (D1, D2, D3) of different spatial resolutions. In each layer of the decoder, the Dice loss function and the cross entropy loss function are arranged to characterize the difference between the decoder feature map (D1, D2, D3) of each layer and the ground truth result, and the parameters of the model are updated by back propagation. In addition, the present application uses the proposed false positive-negative loss function to associate the feature maps of two adjacent layers, and uses the corresponding loss function value to characterize the complementary loss function value of the two, and updates the parameters of the model by back propagation, which promotes the model to produce more accurate output results.

[0081] The image segmentation method based on the complementary information loss function provided in this application makes full use of the complementary information between feature maps of different layers, performs targeted optimization on the feature maps of each layer to improve the segmentation results, and has higher detection accuracy.

[0082] This application has been experimentally verified, and the experimental results show that the image segmentation method based on the complementary information loss function provided by this application can significantly improve the accuracy of the multi-layer deep learning model compared with the existing technology.

[0083] We tested this application using 519 clinical liver tumor ultrasound images with a resolution of 240×240. For quantitative comparison, in addition to frame rate, we also used five other robust performance metrics: Dice, Accuracy, Jaccard, APD, and HD.

[0084] The experimental results are shown below:

[0085] Dice Accuracy Jaccard APD HD Existing methods 81.92±0.17 98.31±0.95 70.54±0.31 4.71±1.11 3.87±0.03 This method 82.80±0.18 98.42±0.09 71.81±0.33 4.42±1.08 3.82±0.05

[0086] As can be seen from the table, the method provided by this application is superior to existing methods in all mainstream detection accuracy.

[0087] It should be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0088] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. An image segmentation method based on a loss function of complementary information, characterized in that: The method comprises: Obtain the image to be segmented; Inputting the image to be segmented into a multi-layer deep learning model to obtain a mask including the lesion area; wherein a loss function is used to perform supervised training on the multi-layer deep learning model, wherein the loss function includes a false positive-negative loss function based on complementary information, and the false positive-negative loss function based on complementary information is used to characterize the complementary information between feature maps of different layers; The false positive-negative loss function based on complementary information includes a false positive loss function and a false negative loss function; The false positive loss function is used to suppress the feature map of deep layer with low spatial resolution, so as to reduce the non-lesion area from being mistaken for the lesion area; The false negative loss function is used to constrain the shallow high spatial resolution feature map to reduce the lesion area from being misclassified as a non-lesion area; The false positive-negative loss function based on complementary information is determined according to the weighted sum of the false positive loss function and the false negative loss function; No. i Layer feature map False positive loss function for: Among them, GT represents the real lesion area corresponding to the current image to be segmented, and It's a reversal for GT. is a feature map from a deeper layer relative to the current layer feature map Get the false positive segmentation mask in; False positive segmentation mask representing the feature map and The Dice loss function value between ; The false positive segmentation mask Through the feature map The 1×1 convolution layer and Sigmoid function on it are calculated as follows: Among them, Conv represents the feature map 1×1 convolution layer on top; Sigmoid() represents the sigmoid activation function; Representation feature map The set difference operation between the real lesion area and the real lesion area; The false negative loss function for: in, For the i Layer feature map The corresponding false negative segmentation template; Representation feature map The segmentation result of express and The Dice loss function value between ; The false negative segmentation template By comparing the real lesion area GT corresponding to the current image to be segmented and the current i Layer feature map Calculation yields: Among them, Sigmoid() represents the sigmoid activation function, and Conv represents the 1×1 convolutional layer.

2. The method according to claim 1, characterized in that The loss function also includes a segmentation loss function.

3. The method according to claim 2, characterized in that The segmentation loss function includes a Dice loss function and a CE loss function; the Dice loss function and the CE loss function are arranged in each layer of the multi-layer deep learning model to constrain the difference between the current layer feature map and the true result.

4. The method according to claim 2 or 3, characterized in that The loss function is determined by weighting the false positive-negative loss function based on complementary information and the segmentation loss function.

Images