Medical image weak supervision variation segmentation method based on single-point labeling

Abstract

The invention discloses a medical image weak supervision variation segmentation method based on single-point labeling, and relates to the technical field of tumor segmentation in medical images. The method comprises the following steps: firstly, randomly taking a point in a target tumor area, and constructing a new anisotropic Riemannian metric according to texture direction information of a whole CT image, local direction information of gray scale at a boundary and heterogeneity of the gray scale; secondly, calculating two distances from all points in the whole CT image to the points taken in the tumor area: geodesic distance and Euclidean distance based on new Riemannian metric; then, the two kinds of point-by-point distance information are embedded into a Chan-Vese model; and finally, proving the existence and uniqueness of a model solution, and solving the model solution by adopting a Douglas-Rachford algorithm to obtain a final tumor segmentation result. According to the method, a more accurate, more effective and more robust segmentation result can be obtained, the method can be used for tumor detection, positioning and segmentation, the workload of manual labeling can be greatly reduced, and the method has high clinical application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation, and in particular relates to a medical image weakly supervised variational segmentation method based on single-point annotation. Background Art

[0002] There are two mainstream methods in the field of image segmentation, one is data-driven and the other is model-driven. Data-driven methods mainly refer to methods based on deep learning. In the past decade, semantic segmentation has witnessed the rapid development of semantic segmentation in different fields, such as medical diagnosis, aerial imaging, and autonomous driving. However, excellent segmentation models usually require a large amount of high-quality manual annotations. As we all know, the annotation of images drawn by doctors is very time-consuming and labor-intensive, and the workload is very large. Model-driven methods mainly refer to traditional variational model methods. Although they do not require a large amount of manual annotation, they often show insufficient ability when dealing with complex medical image segmentation problems, and usually require more prior knowledge to assist segmentation tasks. Therefore, in order to alleviate the above problems, weakly supervised segmentation technology has received more and more attention and research. By providing simple weakly supervised information, such as point annotations, border annotations, line annotations, and image-level annotations, information related to the target area can be extracted from the image, which can not only alleviate the problem of difficulty in obtaining manual annotations, but also provide valuable prior information.

[0003] Weakly supervised segmentation based on optimization models uses graph theory or variational models to construct energy functionals, and obtains the final segmentation results by iteratively minimizing the energy functionals, such as graph cuts and selective segmentation methods based on variational models. Weakly supervised segmentation based on deep learning usually uses weak annotations in different semantic environments, such as using weak annotations to obtain pseudo labels before network training to imitate fully supervised learning, and incorporating weak annotations into the network architecture to improve or guide network predictions, as well as placing weakly labeled information in the post-processing stage after inference to repair the prediction results. In fact, whether it is a weakly supervised segmentation method based on optimization models or deep learning, it involves a key step, which is to extract favorable segmentation information through weak annotations to guide subsequent segmentation. In addition, without additional prior knowledge, traditional variational models that only contain simple data terms and regularization terms often do not work well in segmenting complex images, and usually mistake areas that are far or close to the target area and have similar grayscale as the target area. Based on the above facts, the present invention proposes a weakly supervised variational segmentation method for medical images based on single-point annotation, which extracts favorable tumor segmentation information through single-point annotation and obtains higher segmentation accuracy than other weakly supervised methods for medical images based on weak annotation. The present invention can not only be used for tumor detection, positioning and segmentation, but also can greatly reduce the workload of manual annotation, and has high clinical application value. Summary of the invention

[0004] Purpose of the invention: In view of the current situation that fully supervised learning requires a large amount of labeled training data, the present invention proposes a weakly supervised variational segmentation method for medical images based on single-point annotation. By constructing an anisotropic Riemann metric, the tumor grayscale change, texture direction and local edge direction information are embedded, and the two point-by-point distance information is simultaneously embedded in the variational model to guide tumor segmentation, reducing the workload of manual annotation.

[0005] Technical solution: The present invention proposes a weakly supervised variational segmentation method for medical images based only on single-point annotation, which includes the following four steps:

[0006] (a) A point is randomly selected in the target tumor area, and the structure tensor field is decomposed to characterize the texture direction of the entire CT image and the direction of the grayscale at the boundary. At the same time, the local grayscale change is calculated to characterize the grayscale heterogeneity of the tumor.

[0007] (b) A new anisotropic Riemann metric is constructed using the texture direction information of the entire CT image, the local direction information of the grayscale at the boundary, and the heterogeneity of the grayscale.

[0008] (c) Calculate two distances from all points in the entire CT image to the points taken in the tumor area: geodesic distance and Euclidean distance based on the new Riemannian metric to obtain two point-by-point distance information;

[0009] (d) These two point-by-point distance information are embedded into the Chan-Vese model and solved using the Douglas-Rachford algorithm to obtain the final tumor segmentation result.

[0010] Furthermore, the calculation of the structure tensor field in step (a) utilizes a corner detection operator based on an image local autocorrelation function analysis algorithm. The local autocorrelation function represents the local grayscale change when the local image window is slightly translated in different directions, and is calculated as follows:

[0011]

[0012] Where E(Δx1, Δx2) represents the local autocorrelation function, Δx1 and Δx2 represent the translation changes in the horizontal and vertical directions respectively;

[0013]

[0014] Where f(x) represents the grayscale value at the pixel point x = (x1, x2). is a Gaussian function, * indicates convolution operation, f x1 , f x2Represents the horizontal and vertical gradients of the image grayscale respectively. Q(x) can be orthogonally decomposed: Q(x) = λ1(x)v1(x)v1(x) T +λ2(x)v2(x)v2(x) T , where λ1(x), λ2(x) are the eigenvalues ​​of Q(x), v1(x), v2(x) are the corresponding eigenvectors that can describe the texture direction at each pixel on the image and the grayscale direction information at the tumor boundary, and Q(x) is the structure tensor field;

[0015] Furthermore, in step (b), a new anisotropic Riemann metric is constructed, specifically by first introducing a quantity that describes the grayscale non-uniformity of the image: The point p is a point randomly taken in the target area. is the neighborhood centered at point p, I(x) represents the grayscale at pixel x, and I(y) represents the grayscale at pixel y. Next, construct an anisotropic Riemann metric Among them, λ1(x), λ2(x) are the eigenvalues ​​of the structure tensor field Q(x), v1(x), v2(x) are the corresponding eigenvectors that can describe the texture direction at each pixel on the image and the grayscale direction information at the tumor boundary. Finally, the grayscale inhomogeneity As a conformal factor multiplied by Get a new Riemann metric that embeds grayscale variation, local edge information, and texture information

[0016] Furthermore, in step (c), two distances from all points in the entire CT image to the point p in the tumor region are calculated. Specifically, the newly constructed Riemann metric Substitute into the following equation

[0017]

[0018] The geodesic distance is the minimum energy integrated along the path connecting the fixed point p and any point x in Ω, the potential energy function γ(t) is a smooth curve, γ′(t) is the tangent line of the curve γ(t), is the set of all paths connecting point x and point p. In addition, the Euclidean distance The potential energy function Specifically, the Chan-Vese model in step (d) is to transform two distances, namely the geodesic distance and Euclidean distance At the same time, it is embedded into the following Chan-Vese model to guide subsequent tumor segmentation.

[0019]

[0020] Where u(x) represents the probability function that the pixel point x belongs to the target tumor region Ω, and c1 and c2 represent the grayscale means inside and outside the target tumor region Ω, respectively.

[0021] The beneficial effects of the present invention are as follows: the present invention embeds the grayscale variation, texture direction and local edge direction information of the tumor by constructing an anisotropic Riemann metric, and simultaneously embeds the two point-by-point distance information into the variational model to guide tumor segmentation, thereby reducing the workload of manual labeling; the present invention obtains higher segmentation accuracy than other latest weakly supervised methods of medical images based on weak labeling, and can also be used for tumor detection, positioning and segmentation, and has high clinical application value.

[0022] Figure content

[0023] Figure 1 This is a structural diagram of the weakly supervised variational segmentation method for medical images based on single-point annotation proposed in the present invention;

[0024] Figure 2 A visualization diagram of the structure tensor field Q(x) proposed in the present invention;

[0025] Figure 3 This is a comparison chart of the ablation experiments of the two distance embedded variational models proposed in the present invention; (a): original image, (b): tumor segmentation result of the baseline model (neither geodesic distance nor Euclidean distance), (c): tumor segmentation result of only the Euclidean distance model, (d): tumor segmentation result of only the geodesic distance model, (e): tumor segmentation result embedded with two distance models, (f): tumor gold standard;

[0026] Figure 4 This is a graph showing the robustness of the single-point annotation results with respect to position proposed by the present invention; the first, third, fifth and seventh columns respectively represent the different positions of the annotation points in the target tumor area; the second, fourth, sixth and eighth columns respectively represent the corresponding tumor segmentation results; the last column is the gold standard for pancreatic cysts, kidney tumors and lung tumors respectively;

[0027] Figure 5 The comparison chart of the results of the present invention with different weakly supervised segmentation methods on three data sets; (a): original image, (b): tumor gold standard, (c): Graph-cut, (d): Grab-cut, (e): Selective-seg, (f): PA-seg, (g): DTP, (h): SAM, (i): tumor segmentation result of the present invention. DETAILED DESCRIPTION

[0028] The objectives, technical solutions and advantages of the present invention are described in detail below in conjunction with the accompanying drawings and specific embodiments of the present invention.

[0029] The present invention proposes a weakly supervised variational segmentation method for medical images based on single-point annotation. By constructing anisotropic Riemannian metrics, the grayscale variation of tumors, texture direction and local edge direction information are embedded, and the Euclidean distance and geodesic distance are simultaneously embedded in the variational model to obtain the final tumor segmentation result, reducing the workload of manual annotation. The overall flow chart of the present invention is shown in FIG. Figure 1 As shown, the specific steps include the following steps.

[0030] (a) A point is randomly selected in the target tumor area, and a corner detection operator based on the image local autocorrelation function analysis algorithm is used to extract the texture information of the entire CT image and the direction information of the grayscale at the boundary. Note that the local autocorrelation function represents the local grayscale change when the local image window is translated in different directions. It is calculated as follows:

[0031]

[0032] Where E(Δx1, Δx2) represents the local autocorrelation function, Δx1 and Δx2 represent the translation changes in the horizontal and vertical directions respectively;

[0033]

[0034] Here f(x) represents the gray value at the pixel point x=(x1,x2), is a Gaussian function, * represents a convolution operation, Represents the horizontal and vertical gradients of the image grayscale respectively. Q(x) can be orthogonally decomposed: Q(x) = λ1(x)v1(x)v1(x) T +λ2(x)v2(x)v2(x) T , where λ1(x), λ2(x) are the eigenvalues ​​of Q(x), v1(x), v2(x) are the corresponding eigenvectors that can describe the texture direction at each pixel point on the image and the direction information of the grayscale at the boundary. Q(x) is the structure tensor field. The structure tensor field is decomposed to describe the texture direction and the direction of the grayscale at the boundary in the entire CT image, and the local grayscale change is calculated to describe the heterogeneity of the grayscale.

[0035] (b) Using the texture direction information extracted by single point annotation in step (a) and the local direction information of the grayscale at the boundary, a quantity that describes the uneven grayscale of the image is introduced. Point p is a point randomly selected in the target tumor area. is the neighborhood centered at point p, and I(x) represents the grayscale at pixel x. Next, we construct an anisotropic Riemann metric: Among them, λ1(x), λ2(x) are the eigenvalues ​​of the structure tensor field Q(x), v1(x), v2(x) are the corresponding eigenvectors that can describe the texture direction at each pixel point on the image and the direction information of the grayscale at the boundary. Finally, the grayscale inhomogeneity As a conformal factor multiplied by Get a new Riemann metric that embeds grayscale variation, local edge information, and texture information Constructing a new anisotropic Riemannian metric

[0036] (c) Calculate the two distances from all points in the entire CT image to the points in the tumor region: the geodesic distance and the Euclidean distance based on the Riemannian metric constructed in step (b), and convert the newly constructed Riemannian metric Substitute into the following equation

[0037]

[0038] The geodesic distance is the minimum energy integrated along the path connecting the fixed point p and any point x in Ω, the potential energy function γ(t) is a smooth curve, γ′(t) is the tangent line of the curve γ(t), is the set of all paths connecting point x and point p. In addition, the Euclidean distance The potential energy function The fast marching algorithm is used to solve the eikonal equation and obtain two point-by-point distance information;

[0039] (d) The two point-by-point distance information in step (c), i.e., the geodesic distance and Euclidean distance At the same time, it is embedded into the following variational model to guide subsequent tumor segmentation.

[0040]

[0041] Where u(x) represents the probability function that the pixel point x belongs to the target tumor region Ω, and c1 and c2 represent the grayscale mean inside and outside the target tumor region Ω, respectively. The Douglas-Rachford algorithm is used to solve it and obtain the final tumor segmentation result.

[0042] In order to more clearly illustrate the feasibility and superiority of the present invention, the performance of the method of the present invention in the ablation experiment is shown below. In order to analyze the respective contributions of the geodesic distance and the Euclidean distance in the variational model, only one distance is used in the variational model to segment pancreatic cysts, kidney tumors and lung tumors. Figure 3 Examples of these segmentations are shown. (1) Validity of geodesic distance: If there is no embedded geodesic distance Then the pixels close to the target area and with similar grayscale cannot be excluded. Figure 3 (2) Validity of Euclidean distance: If there is no embedded Euclidean distance Then the pixels far from the target area and with similar grayscale cannot be excluded, see Figure 3 (3) Effectiveness of geodesic distance and Euclidean distance: If neither geodesic distance nor Euclidean distance is embedded in the model, the experimental effect of the baseline model is the worst, which shows the importance of the two distances working together, see Figure 3 The second to fifth columns.

[0043] Table 1 Comparison of numerical results of ablation experiments on weakly supervised variational segmentation methods for medical images based on single point annotation

[0044]

[0045]

[0046] In order to illustrate the robustness of the single-point annotation proposed by the present invention with respect to position, the present invention selects different positions in the target tumor area to verify the robustness of the position of the annotation point, see Figure 4 . It can be observed from Table 2 that as long as the annotation point is within the target tumor region, the corresponding tumor segmentation results are not much different. However, if the position of the annotation point is taken on the boundary of the target tumor region, the segmentation result will be disturbed by the surrounding tissue. Therefore, it is concluded that the annotation point can be taken at any position within the target tumor region, but cannot be taken outside the target tumor region or on the boundary, which is also consistent with the theoretical analysis of the present invention.

[0047] Table 2 Comparison of Dice values ​​of segmentation results of single point annotation at four different locations in the target area

[0048] Dataset Result 1 Result 2 Result 3 Result 4 Pancreatic cyst 72.51±10.04 72.83±9.46 72.60±11.62 71.29±10.77 Kidney tumors 81.71±8.86 80.67±9.09 80.84±9.72 79.50±10.39 Lung tumors 77.29±7.85 76.34±5.85 76.23±8.83 74.76±7.07

[0049] In order to illustrate the superiority of the present invention compared with other weakly supervised segmentation methods for medical images, the tumor segmentation effect and numerical experimental comparison of the method of the present invention and other cutting-edge weakly supervised segmentation algorithms for medical images are given here. Figure 5 The segmentation effects of the present invention and other weakly supervised segmentation algorithms on three data sets are shown. Tables 3 to 5 quantitatively give numerical comparisons of the tumor segmentation effects of the present invention and other weakly supervised segmentation algorithms. The numerical results show that the segmentation results proposed by the present invention have the highest accuracy, and only single-label annotation is used, that is, the annotation is easier to obtain.

[0050] Table 3 Comparison of numerical results with other weakly supervised segmentation algorithms on the pancreatic cyst dataset

[0051]

[0052] Table 4 Comparison of numerical results with other weakly supervised segmentation algorithms on the renal tumor dataset

[0053]

[0054]

[0055] Table 5 Comparison of numerical results with other weakly supervised segmentation algorithms on lung tumor dataset

[0056]

Claims

1. A weakly supervised variational segmentation method for medical images based on single-point annotation, characterized in that: The steps include: (a) A point is randomly selected in the target tumor area, and the structure tensor field is decomposed to characterize the texture direction of the entire CT image and the direction of the grayscale at the boundary. At the same time, the local grayscale change is calculated to characterize the grayscale heterogeneity of the tumor. (b) A new anisotropic Riemann metric is constructed using the texture direction information of the entire CT image, the local direction information of the grayscale at the boundary, and the heterogeneity of the grayscale. (c) Calculate two distances from all points in the entire CT image to the points taken in the tumor area: geodesic distance and Euclidean distance based on the new Riemannian metric to obtain two point-by-point distance information; (d) These two point-by-point distance information are embedded into the Chan-Vese model and solved using the Douglas-Rachford algorithm to obtain the final tumor segmentation result.

2. The weakly supervised variational segmentation method for medical images based on single point annotation according to claim 1, characterized in that: The calculation of the structure tensor field in step (a) utilizes a corner detection operator based on an image local autocorrelation function analysis algorithm; the local autocorrelation function represents the local grayscale change when the local image window is translated in small directions, and is calculated as follows: Where E(Δx1, Δx2) represents the local autocorrelation function, Δx1 and Δx2 represent the translation changes in the horizontal and vertical directions respectively; Where f(x) represents the grayscale value at the pixel point x = (x1, x2). is a Gaussian function, * represents a convolution operation, and Represent the horizontal and vertical gradients of the image grayscale respectively; Q(x) is orthogonally decomposed: Q(x)=λ1(x)v1(x)v1(x) T +λ2(x)v2(x)v2(x) T , where λ1(x) and λ2(x) are the eigenvalues ​​of Q(x), v1(x) and v2(x) are the corresponding eigenvectors, which can describe the texture direction at each pixel point on the CT image and the direction information of the grayscale at the boundary. Q(x) is called the structure tensor field.

3. The weakly supervised variational segmentation method for medical images based on single point annotation according to claim 2, characterized in that: In step (b), a new anisotropic Riemann metric is constructed, specifically by first introducing a quantity that describes the grayscale inhomogeneity of the image: Point p is a point randomly selected in the target tumor area. is a neighborhood centered on point p, I(x) represents the grayscale at pixel x, and I(y) represents the grayscale at pixel y; secondly, construct an anisotropic Riemann metric Among them, λ1(x) and λ2(x) are the eigenvalues ​​of the structure tensor field Q(x), v1(x) and v2(x) are the corresponding eigenvectors, which can describe the texture direction of each pixel on the image and the direction information of the grayscale at the boundary; finally, the grayscale inhomogeneity As a conformal factor multiplied by Get a new Riemann metric that embeds grayscale variation, local edge information, and texture information 4. The weakly supervised variational segmentation method for medical images based on single point annotation according to claim 3, characterized in that: In step (c), two distances from all points in the entire CT image to the point p in the tumor region are calculated. Specifically, the newly constructed Riemann metric Substitute into the following equation The geodesic distance is the minimum energy integrated along the path connecting the fixed point p and any point x in Ω, the potential energy function γ(t) is a smooth curve, γ ′ (t) is the tangent line of the curve γ(t), is the set of all paths connecting point x and point p; in addition, the Euclidean distance The potential energy function 5. The weakly supervised variational segmentation method for medical images based on single point annotation according to claim 4, characterized in that: The Chan-Vese model in step (d) is specifically to transform two distances, namely the geodesic distance and Euclidean distance At the same time, it is embedded into the following Chan-Vese model to guide subsequent tumor segmentation. Where u(x) represents the probability function that the pixel point x belongs to the target tumor region Ω, and c1 and c2 represent the grayscale means inside and outside the target tumor region Ω, respectively.

Images