Target region segmentation method and device based on target relocation

Abstract

The invention relates to a target region segmentation method and device based on target relocation, and solves the problems that in the prior art, the attention of a classification network focuses on the most discriminative region of a target object, and the false positive is high. An auxiliary data set is introduced to distinguish a foreground and a background, so that false background clues are effectively inhibited, and false positive in background information is reduced; the target relocation attention mechanism enables the pixel-level location mapping in the generated pseudo label to cover the whole target area, so that the finally obtained target area segmentation result is more accurate.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method and device for target region segmentation based on target relocation. Background Art

[0002] Existing marked area segmentation methods can be mainly divided into traditional diagnostic segmentation methods, fully supervised learning segmentation methods, and weakly supervised learning segmentation methods that do not require precise labeling based on differences in technical principles. Traditional diagnostic segmentation methods mainly refer to doctors finding the lesion area through manual segmentation. For some special diseases, such as Crohn's disease, which is an inflammatory bowel disease, the image is blurred and difficult to find in CT slices, so doctors need to observe the patient's entire CT sequence containing hundreds of slices for comparative analysis, which makes diagnosis difficult. In addition, manual segmentation mainly relies on the doctor's clinical experience and subjective consciousness, which leads to certain deviations in diagnosis and poses certain risks to the patient's life safety.

[0003] In recent years, computer vision technology has rapidly emerged and penetrated into all aspects of life. In the medical field, computer vision technology helps doctors analyze medical images more accurately and improve the accuracy and efficiency of disease diagnosis. Among them, the fully supervised learning method has achieved good performance in tumor and organ segmentation, which mainly relies on a large amount of labeled data. However, medical data usually contains sensitive information of patients. In order to protect patient privacy and data security, medical institutions must comply with regulations, which restrict the sharing and use of medical data. In addition, the difficulty of data labeling and the high cost of labeling are also the main problems faced by fully supervised learning. The labeling of medical images requires professional doctors, and the labeling process depends on the doctor's clinical experience and knowledge reserves. Therefore, there may be differences in the labeling between different doctors, which affects the performance of the model. In order to solve the above problems, many scholars have studied weakly supervised learning that does not require precise labeling. Its advantage of easy to obtain labeling occupies an important position in medical image segmentation. However, the accuracy and performance of Crohn's disease segmentation based on the existing weakly supervised segmentation method are low. This is mainly due to some characteristics of Crohn's disease. First, the CT images of Crohn's disease are irregular in shape, the scale of the lesion area in different slices varies greatly, and the number of lesion areas is uncertain. Crohn's disease may appear at any position in the small intestine, and it is difficult to constrain its range by position. Secondly, in CT images, the contrast between the Crohn's disease lesion area and the surrounding background is low, and the boundary is blurred and difficult to distinguish. The classification network has difficulty finding accurate boundary information in the process of generating pseudo-labels, which limits the segmentation performance. In addition, the background of CT images is complex and contains a lot of noise information. The classification network is easily disturbed by similar information in the background, resulting in a high false positive rate of the generated pseudo-labels. Summary of the invention

[0004] To overcome at least one deficiency in the prior art, the present application provides a method and apparatus for target area segmentation based on target relocation.

[0005] In a first aspect, a method for target area segmentation based on target relocation is provided, including:

[0006] Obtain a model training data set, where the target training data set includes a diseased data set, a data set of non-diseased but with small intestine regions, and a data set of non-diseased and without small intestine regions;

[0007] Construct a segmentation model, where the segmentation model includes a feature extraction module, a target relocation attention mechanism, and a classification module;

[0008] Train the segmentation model based on the model training data set to obtain a trained segmentation model; during the training process, the samples in the model training data set are input into the feature extraction module in batches to obtain feature maps corresponding to different batches; the feature maps corresponding to different batches are input into the target relocation attention mechanism for attention operations to obtain updated feature maps; the updated feature maps are input into the classification module to obtain the segmentation result of the target area;

[0009] Input the picture to be segmented into the trained segmentation model to obtain the segmentation result of the target area.

[0010] In one embodiment, the target relocation attention mechanism implements the following functions:

[0011] The feature maps corresponding to different batches are denoted as A1, A2, …, A t , where t is the total number of batches;

[0012] A1, A2, …, A t For any three feature maps A i-2 , A i-1 , A i , 2 < i ≤ t, the feature maps A i-2 , A i-1 perform attention operations to obtain the (i - 2)-th attention result; the attention result is operated with A i to obtain the (i - 1)-th attention result; when i = t, the obtained (i - 1)-th attention result is the updated feature map.

[0013] In one embodiment, the target relocation attention mechanism includes multiple attention mechanisms, and each attention mechanism is used to implement attention operations.

[0014] In one embodiment, the classification module includes a global average pooling layer and a fully connected layer. The global average pooling layer is used to perform a global average pooling operation on the input updated feature map to obtain an average pooled result; the average pooled result is input to the fully connected layer to output the classification result of each pixel in the image.

[0015] In one embodiment, obtaining a model training data set includes:

[0016] Constructing a data candidate pool, the data candidate pool includes diseased data, non-disease-free data with small intestine regions, and non-disease-free data without small intestine regions;

[0017] Filter out diseased data from the data candidate pool;

[0018] A public data set containing small intestine data is used to train a classification network to obtain a trained classification network;

[0019] The remaining data in the data candidate pool are input into the trained classification network to obtain data that are not diseased but have small intestine regions; after removing the data that are not diseased but have small intestine regions, the remaining data in the data candidate pool are data that are not diseased and have no small intestine regions.

[0020] In a second aspect, a target region segmentation device based on target relocation is provided, comprising:

[0021] A data acquisition module is used to acquire a model training data set, wherein the target training data set includes a diseased data set, a non-disease data set with a small intestine region, and a non-disease data set without a small intestine region;

[0022] A model building module is used to build a segmentation model, which includes a feature extraction module, a target relocalization attention mechanism, and a classification module;

[0023] The model training module is used to train the segmentation model based on the model training data set to obtain the trained segmentation model; during the training process, the samples in the model training data set are input into the feature extraction module in batches to obtain feature maps corresponding to different batches; the feature maps corresponding to different batches are input into the target relocalization attention mechanism for attention operation to obtain updated feature maps; the updated feature maps are input into the classification module to obtain the segmentation result of the target area;

[0024] The segmentation module is used to input the image to be segmented into the trained segmentation model to obtain the target area segmentation result.

[0025] In one embodiment, the target relocalization attention mechanism implements the following functions:

[0026] The feature maps corresponding to different batches are denoted as A1, A2,…,A t, where t is the total number of batches;

[0027] A1, A2, …, A t Among any three feature maps A i-2 , A i-1 , A i , 2 < i ≤ t, for the feature maps A i-2 , A i-1 , an attention operation is performed to obtain the (i - 2)-th attention result; the attention result is subjected to an attention operation with A i to obtain the (i - 1)-th attention result; when i = t, the obtained (i - 1)-th attention result is the updated feature map.

[0028] In one embodiment, the target re - localization attention mechanism includes multiple attention mechanisms, and each attention mechanism is used to implement the attention operation.

[0029] In one embodiment, the classification module includes a global average pooling layer and a fully - connected layer. The global average pooling layer is used to perform a global average pooling operation on the input updated feature map to obtain the result after average pooling; the result after average pooling is input into the fully - connected layer to output the classification result of each pixel point in the image.

[0030] In one embodiment, the data acquisition module is further used for:

[0031] Construct a data candidate pool, which includes diseased data, data with small intestine regions but not diseased, and data without small intestine regions and not diseased;

[0032] Screen out the diseased data from the data candidate pool;

[0033] Use a publicly available dataset containing small intestine data to train the classification network to obtain the trained classification network;

[0034] Input the remaining data in the data candidate pool into the trained classification network to obtain the data with small intestine regions but not diseased; excluding the data with small intestine regions but not diseased, the remaining data in the data candidate pool is the data without small intestine regions and not diseased.

[0035] Compared with the prior art, the present application has the following beneficial effects: The method and device for target region segmentation based on target re - localization in the present application solve the problems of the classification network focusing on the most discriminative regions of the target object and high false positives in the prior art. By introducing an auxiliary dataset to distinguish the foreground and background, false background cues are effectively suppressed, thereby reducing false positives in the background information; the target re - localization attention mechanism enables the pixel - level localization mapping in the generated pseudo - labels to cover the entire target region, making the final obtained target region segmentation result more accurate. Brief Description of the Drawings

[0036] The present application may be better understood by referring to the following description given in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification. In the drawings:

[0037] Figure 1 A flowchart of a target region segmentation method based on target relocation according to an embodiment of the present application is shown;

[0038] Figure 2 A schematic diagram of data set division is shown, where (a) is data set D1, (b) is data set D2, and (c) is data set D3;

[0039] Figure 3 A schematic diagram of a segmentation model according to an embodiment of the present application is shown;

[0040] Figure 4 A comparison diagram of the target area segmentation effect is shown;

[0041] Figure 5 A structural block diagram of a target region segmentation device based on target relocation according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present application will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions can be made in the process of developing any such actual embodiments in order to achieve the specific goals of the developer, and these decisions may vary from embodiment to embodiment.

[0043] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, while other details that are not closely related to the present application are omitted.

[0044] It should be understood that the present application is not limited to the described implementation forms due to the following description with reference to the accompanying drawings. In this article, where feasible, the embodiments can be combined with each other, features between different embodiments can be replaced or borrowed, and one or more features can be omitted in one embodiment.

[0045] By comparing the attention maps at different stages, this application observes that at different training stages of the network, the attention maps generated by the classification network will focus on different areas of the target object. If the attention maps at different stages are simply superimposed to generate the final pseudo-label, the classification network will be relatively weak in the initial stage of training, resulting in a large amount of background information in the pseudo-label. The target relocation mechanism can solve this problem. The target relocation attention mechanism is designed to allow the network to pay attention to those edge areas that are no longer noticed as the model is continuously trained, so that the importance of different areas in the image can be better understood and expressed. The final target relocation attention map can be obtained by repeating the update process until the model converges. In the above update process, the target relocation attention mechanism is responsible for preserving the areas where the attention maps at different stages are different from other attention maps to generate a more complete scoring area.

[0046] Based on the above analysis, an embodiment of the present application provides a target region segmentation method based on target relocation. In this embodiment, the target region may be, for example, a lesion region of Crohn's disease. Figure 1 FIG. 1 shows a flow chart of a target region segmentation method based on target relocation according to an embodiment of the present application, see Figure 1 , methods include:

[0047] Step S1, obtaining a model training data set, wherein the target training data set includes a diseased data set, a non-diseased data set with a small intestine region, and a non-diseased data set without a small intestine region.

[0048] Scanning CT images of the abdomen of patients with Crohn's disease. Each patient's CT image contains at least hundreds of slices. Each slice contains rich but different information. Some slices may contain diseased areas, some slices may contain small intestine but no diseased areas, and some slices may not contain small intestine.

[0049] By observing CT images of Crohn's disease, a new data partitioning method is proposed. It provides a distinction between foreground and background cues, and at the same time provides a distinction between target objects and false positive areas. This data partitioning method includes an auxiliary dataset that does not contain any foreground class of interest (Crohn's disease).

[0050] The data set is divided into three categories: a diseased data set D1, a data set D2 that is not diseased but has a small intestine area, and a data set D3 that is not diseased and has no small intestine area. In this embodiment, data sets D2 and D3 are collectively referred to as auxiliary data sets. Compared with image-level labels, obtaining such a data set does not require too much additional annotation work. First, an experienced doctor is required to screen out slices with Crohn's disease from the data candidate pool and classify them into data set D1, and the remaining data is still placed in the data candidate pool. Then, the classification network is trained using the small intestine data in the public data set WORD. After the classification network is trained, the data in the candidate pool is input into the network, and the network is allowed to screen out data with small intestine from the candidate pool and classify them into data set D2. Finally, all the remaining data in the candidate pool are classified as data set D3. The number of auxiliary data sets is set in combination with the number of samples in D1 to prevent excessive or insufficient sample imbalance, so that the classification network tends to focus on categories with more samples. Figure 2 A schematic diagram of data set partitioning is shown.

[0051] Step S2, constructing a segmentation model, which includes a feature extraction module, a target relocation attention mechanism and a classification module. Figure 3 A schematic diagram of a segmentation model according to an embodiment of the present application is shown.

[0052] Here, the feature extraction module can use Resnet-50, and the target relocalization attention mechanism includes multiple attention mechanisms TRA, each of which is used to implement an attention operation. The attention mechanism TRA can use existing structures, such as a self-attention mechanism, a multi-head attention mechanism, and the like.

[0053] The classification module includes a global average pooling layer (GAP) and a fully connected layer (FC). The global average pooling layer is used to perform a global average pooling operation on the updated feature map of the input to obtain the average pooling result; the average pooling result is input to the fully connected layer, and the classification result of each pixel in the image is output. For example, the probability of each pixel belonging to the background area and the target area can be obtained, and the one with a larger probability is selected as the classification result of the pixel.

[0054] Step S3, training the segmentation model based on the model training data set to obtain a trained segmentation model; during the training process, the samples in the model training data set are input into the feature extraction module in batches to obtain feature maps corresponding to different batches; the feature maps corresponding to different batches are input into the target relocalization attention mechanism for attention operation to obtain updated feature maps; the updated feature maps are input into the classification module to obtain the segmentation result of the target area.

[0055] Here, the loss function adopted in the model training process can be the traditional cross-entropy loss. The performance of the model will be determined by comparing the predicted output and the expected output of the model, and then the optimal model can be obtained.

[0056] Specifically, the feature maps corresponding to different batches are denoted as A1, A2, …, A t , where t is the total number of batches;

[0057] A1, A2, …, A t For any three feature maps A i-2 , A i-1 , A i among them, 2 < i ≤ t, the feature maps A i-2 , A i-1 perform an attention operation to obtain the (i - 2)-th attention result; the attention result and A i perform an attention operation to obtain the (i - 1)-th attention result; when i = t, the (i - 1)-th attention result obtained is the updated feature map.

[0058] Step S4: Input the image to be segmented into the trained segmentation model to obtain the target region segmentation result. Here, the target region can be the lesion region of Crohn's disease.

[0059] In this embodiment, the problems of the classification network in the prior art, such as the attention being concentrated on the most discriminative region of the target object and the high false positives, are solved. The introduced auxiliary dataset is used to distinguish the foreground and the background, effectively suppressing the false background clues, thereby reducing the false positives in the background information; the target repositioning attention mechanism enables the pixel-level localization mapping in the generated pseudo-labels to cover the entire target region, making the finally obtained target region segmentation result more accurate.

[0060] To verify the effectiveness of the method of the present application, a comparative experiment was conducted using the method of the present application and the existing method, Figure 4 showing a comparison diagram of the target region segmentation effects. According to Figure 4 it can be seen that the method of the present application has a better segmentation effect.

[0061] Adopting the same inventive concept as the target region segmentation method based on target repositioning, this embodiment also provides a corresponding target region segmentation device based on target repositioning, Figure 5 showing a structural block diagram of the target region segmentation device based on target repositioning according to the embodiment of the present application. Refer to Figure 5 , the device includes:

[0062] A data acquisition module 51, configured to acquire a model training dataset, and the target training dataset includes a diseased dataset, a dataset of non-diseased but having a small intestine region, and a dataset of non-diseased and having no small intestine region;

[0063] A model construction module 52, configured to construct a segmentation model, where the segmentation model includes a feature extraction module, an object relocalization attention mechanism, and a classification module;

[0064] A model training module 53, configured to train the segmentation model based on a model training data set to obtain a trained segmentation model; during the training process, samples in the model training data set are input into the feature extraction module in batches to obtain feature maps corresponding to different batches; the feature maps corresponding to different batches are input into the object relocalization attention mechanism for attention operations to obtain updated feature maps; the updated feature maps are input into the classification module to obtain a segmentation result of the target region;

[0065] A segmentation module 54, configured to input an image to be segmented into the trained segmentation model to obtain a segmentation result of the target region.

[0066] In one embodiment, the object relocalization attention mechanism implements the following functions:

[0067] The feature maps corresponding to different batches are denoted as A1, A2, …, A t , where t is the total number of batches;

[0068] For any three feature maps A t , A i-2 , A i-1 in A1, A2, …, A i , where 2 < i ≤ t, the feature maps A i-2 , A i-1 are subjected to an attention operation to obtain the (i - 2)-th attention result; the attention result and A i are subjected to an attention operation to obtain the (i - 1)-th attention result; when i = t, the obtained (i - 1)-th attention result is the updated feature map.

[0069] In one embodiment, the object relocalization attention mechanism includes multiple attention mechanisms, and each attention mechanism is configured to implement an attention operation.

[0070] In one embodiment, the classification module includes a global average pooling layer and a fully connected layer. The global average pooling layer is configured to perform a global average pooling operation on the input updated feature map to obtain an average pooling result; the average pooling result is input into the fully connected layer to output a classification result of each pixel point in the image.

[0071] In one embodiment, the data acquisition module is further configured to:

[0072] Construct a data candidate pool, where the data candidate pool includes diseased data, data with a small intestine region but not diseased, and data without a small intestine region and not diseased;

[0073] Filter out diseased data from the data candidate pool;

[0074] A public data set containing small intestine data is used to train a classification network to obtain a trained classification network;

[0075] The remaining data in the data candidate pool are input into the trained classification network to obtain data that are not diseased but have small intestine regions; after removing the data that are not diseased but have small intestine regions, the remaining data in the data candidate pool are data that are not diseased and have no small intestine regions.

[0076] The target area segmentation device based on target relocation of this embodiment has the same inventive concept as the target area segmentation method based on target relocation mentioned above. Therefore, the specific implementation method of the device can be seen in the embodiment part of the target area segmentation method based on target relocation mentioned above, and its technical effect corresponds to the technical effect of the above method, which will not be repeated here.

[0077] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A target region segmentation method based on target relocation, characterized in that: include: Acquire a model training data set, wherein the target training data set includes a diseased data set, a non-disease data set with a small intestine region, and a non-disease data set without a small intestine region; Constructing a segmentation model, the segmentation model comprising a feature extraction module, a target relocalization attention mechanism, and a classification module; Training the segmentation model based on the model training data set to obtain a trained segmentation model; During the training process, the samples in the model training data set are input into the feature extraction module in batches to obtain feature maps corresponding to different batches; the feature maps corresponding to different batches are input into the target relocalization attention mechanism for attention operation to obtain updated feature maps; The updated feature map is input into the classification module to obtain a segmentation result of the target area; The image to be segmented is input into the trained segmentation model to obtain the target area segmentation result.

2. The method according to claim 1, characterized in that The target relocalization attention mechanism implements the following functions: The feature maps corresponding to different batches are denoted as A1, A2, ..., A t , where t is the total number of batches; A1, A2, …, A t Among any three feature maps A i-2 , A i-1 , A i , where 2 < i ≤ t, the feature maps A i-2 , A i-1 perform an attention operation to obtain the (i - 2)-th attention result; the attention result is operated with A i to obtain the (i - 1)-th attention result; when i = t, the obtained (i - 1)-th attention result is the updated feature map.

3. The method according to claim 1, characterized in that The target relocalization attention mechanism includes multiple attention mechanisms, each of which is used to implement an attention operation.

4. The method according to claim 1, characterized in that The classification module includes a global average pooling layer and a fully connected layer. The global average pooling layer is used to perform a global average pooling operation on the input updated feature map to obtain an average pooled result; the average pooled result is input to the fully connected layer to output the classification result of each pixel in the image.

5. The method according to claim 1, characterized in that The step of obtaining a model training data set includes: Constructing a data candidate pool, wherein the data candidate pool includes diseased data, non-disease-free data with small intestine regions, and non-disease-free data without small intestine regions; Screening out diseased data from the data candidate pool; A public data set containing small intestine data is used to train a classification network to obtain a trained classification network; The remaining data in the data candidate pool is input into the trained classification network to obtain data that is not diseased but has a small intestine region; after removing the data that is not diseased but has a small intestine region, the remaining data in the data candidate pool is data that is not diseased and has no small intestine region.

6. A target region segmentation device based on target relocation, characterized in that: include: A data acquisition module is used to acquire a model training data set, wherein the target training data set includes a diseased data set, a non-disease data set with a small intestine region, and a non-disease data set without a small intestine region; A model building module, used to build a segmentation model, wherein the segmentation model includes a feature extraction module, a target relocation attention mechanism and a classification module; A model training module is used to train the segmentation model based on the model training data set to obtain a trained segmentation model; during the training process, the samples in the model training data set are input into the feature extraction module in batches to obtain feature maps corresponding to different batches; the feature maps corresponding to different batches are input into the target relocalization attention mechanism to perform attention operation to obtain updated feature maps; The updated feature map is input into the classification module to obtain a segmentation result of the target area; The segmentation module is used to input the image to be segmented into the trained segmentation model to obtain the target area segmentation result.

7. The device according to claim 6, characterized in that The target relocalization attention mechanism implements the following functions: The feature maps corresponding to different batches are denoted as A1, A2, ..., A t , where t is the total number of batches; A1, A2, …, A t Among any three feature maps A i-2 , A i-1 , A i , where 2 < i ≤ t, the feature maps A i-2 , A i-1 perform an attention operation to obtain the (i - 2)-th attention result; the attention result and A i perform an attention operation to obtain the (i - 1)-th attention result; when i = t, the (i - 1)-th attention result obtained is the updated feature map.

8. The device according to claim 6, characterized in that The target relocalization attention mechanism includes multiple attention mechanisms, each of which is used to implement an attention operation.

9. The device according to claim 6, characterized in that The classification module includes a global average pooling layer and a fully connected layer. The global average pooling layer is used to perform a global average pooling operation on the input updated feature map to obtain an average pooled result; the average pooled result is input to the fully connected layer to output the classification result of each pixel in the image.

10. The device according to claim 6, characterized in that The data acquisition module is further used for: Constructing a data candidate pool, wherein the data candidate pool includes diseased data, non-disease-free data with small intestine regions, and non-disease-free data without small intestine regions; Screening out diseased data from the data candidate pool; A public data set containing small intestine data is used to train a classification network to obtain a trained classification network; The remaining data in the data candidate pool is input into the trained classification network to obtain data that is not diseased but has a small intestine region; after removing the data that is not diseased but has a small intestine region, the remaining data in the data candidate pool is data that is not diseased and has no small intestine region.

Images