Weakly supervised automatic labeling method and system for non-homogeneous three-dimensional esophageal cancer spectral CT

By combining Hessian matrix response filter, information weight feature extraction, channel attention U-Net network and Poly-RNN algorithm, automatic labeling of esophageal cancer energy spectrum CT images is achieved, solving the problem of low automatic labeling efficiency in the prior art , improves labeling accuracy and speed.

CN112488996BActive Publication Date: 2024-11-01SUN YAT SEN UNIVERSITY CANCER CENTER (CANCER HOSPITAL AFFILIATED TO SUN YAT SEN UNIVERSITY CANCER RESEARCH INSTITUTE OF SUN YAT SEN UNIVERSITY)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011295020.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-11-18
Publication Date
2024-11-01
Estimated Expiration
2040-11-18

AI Technical Summary

Technical Problem

There is a lack of effective automatic labeling methods in the prior art to process esophageal cancer energy spectrum CT data, which makes manual labeling time-consuming and labor-intensive and inefficient, making it difficult to achieve accurate automatic labeling.

Method used

Using multi-structure 3D response filter based on Hessian matrix, feature extraction algorithm for information weights, channel attention and cross-layer fusion U-Net network, Poly-RNN algorithm, and regional center point growth mapping methods to realize non-homogeneous three-dimensional Weak supervision automatic labeling of esophageal cancer energy spectrum CT images.

Benefits of technology

Improve the automatic labeling speed and accuracy of the esophageal cancer energy spectrum CT dataset, reduce the error rate, make full use of the multi-parameter characteristics of energy spectrum CT, realize more efficient automatic labeling, and can effectively reflect the pathology and morphological structure of the lesions. change.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112488996B_ABST
    Figure CN112488996B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for weakly supervised automatic labeling of non-homogeneous three-dimensional esophageal cancer spectral CT. The method comprises the steps of using a multi-structure three-dimensional response filter to enhance other regions in the non-homogeneous spectral CT image, using a feature extraction algorithm based on information weight to extract multiple key faults, using a U-Net network based on channel attention and cross-layer fusion to obtain contour vertices, using a Poly-RNN algorithm to determine labeling information, using regional center point growth mapping, obtaining automatic labeling, and the like. The present invention can effectively reflect the changes in the pathology and morphological structure of the lesion, has a higher speed and a lower error rate than manual labeling, and can more effectively automatically label esophageal cancer spectral CT data samples compared with the existing automatic labeling technology, and has higher labeling accuracy and labeling efficiency. The present invention is widely used in the field of medical image processing technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method and system for weakly supervised automatic labeling of non-homogeneous three-dimensional esophageal cancer spectral CT. Background Art

[0002] With the continuous deepening of informatization and the construction of information platforms in hospitals, computed tomography spectral CT is widely used in clinical diagnosis and disease screening of esophageal cancer in clinical medical diagnosis. In ordinary CT, the only way to evaluate substances is density. Substances with the same density but different properties will be completely judged as the same substance, which may lead to misdiagnosis. Non-homogeneous spectral CT will provide richer feature information on the basis of ordinary CT, such as multiple cross-sections of organs, organ density, lesion size and location, etc. It can also remove the effects of hardening and metal artifacts and make up for the shortcomings of X-rays. For specific esophageal cancer lesions, non-homogeneous spectral CT can observe better results and find some lesions that cannot be found by conventional examinations. However, it is extremely time-consuming and labor-intensive to manually annotate esophageal cancer spectral CT data, and now it is impossible to find a suitable method to accurately and effectively automatically annotate esophageal cancer spectral CT data samples. Therefore, it is very necessary to invent a weakly supervised automatic annotation method for esophageal cancer lesions in non-homogeneous three-dimensional non-homogeneous spectral CT images. This can effectively help doctors diagnose esophageal cancer and is of great significance to the development of medical diagnosis and treatment. Summary of the invention

[0003] In view of at least one of the above technical problems, the object of the present invention is to provide a method and system for weakly supervised automatic labeling of non-homogeneous three-dimensional esophageal cancer spectral CT.

[0004] On the one hand, an embodiment of the present invention includes a non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling method, comprising:

[0005] Using a multi-structure three-dimensional response filter based on a Hessian matrix, enhancing other regions except the esophageal cancer region in the non-homogeneous energy spectrum CT image;

[0006] Extracting a plurality of key faults from the non-homogeneous energy spectrum CT image using a feature extraction algorithm based on information weight;

[0007] Using a U-Net network based on channel attention and cross-layer fusion, segmenting the esophageal cancer region and other regions in the non-homogeneous energy spectrum CT image to obtain at least one set of contour vertices of the esophageal cancer region;

[0008] Using a Poly-RNN algorithm, determining labeling information of the key fault according to the at least one set of contour vertices;

[0009] Using regional center point growing mapping, automatic annotation of the entire sequence of the non-homogeneous energy spectrum CT images is obtained according to the annotation information.

[0010] Furthermore, the use of a multi-structure three-dimensional response filter based on a Hessian matrix to enhance other regions except the esophageal cancer region in the non-homogeneous energy spectrum CT image includes:

[0011] Merging the continuous non-homogeneous energy spectrum CT images into non-homogeneous three-dimensional volume data; the three-dimensional volume data includes a plurality of voxels, and the Hessian matrix of each voxel includes second-order derivatives in different directions;

[0012] Using the linear scale space theory, the elements of the Hessian matrix of the non-homogeneous energy spectrum CT image on the σ scale are determined as Among them, i,j=1,2,3 represent the position of the element, * represents the convolution operation, γ is the parameter, G(x,σ) is the Gaussian function,

[0013] Decomposing the non-homogeneous energy spectrum CT image into continuous two-dimensional images;

[0014] The two-dimensional image decomposed from the non-homogeneous energy spectrum CT image is input into the multi-structure three-dimensional response filter to obtain the output result of the multi-structure three-dimensional response filter; the output result of the multi-structure three-dimensional response filter is to obtain an enhanced non-homogeneous energy spectrum CT image for other areas except the esophageal cancer area.

[0015] Furthermore, the extracting of multiple key faults from the non-homogeneous energy spectrum CT image using a feature extraction algorithm based on information weights includes:

[0016] Reshaping the non-homogeneous energy spectrum CT image;

[0017] Inputting the non-homogeneous energy spectrum CT image into a key fault extraction network; the key fault extraction network comprises a convolutional layer, a first fully connected layer and a second fully connected layer;

[0018] Obtain a key point score of each key fault output by the key fault extraction network;

[0019] The key point score is multiplied by the fault feature map to obtain a corresponding key fault.

[0020] Furthermore, the use of a U-Net network based on channel attention and cross-layer fusion to segment the esophageal cancer region and other regions in the non-homogeneous energy spectrum CT image to obtain at least one set of contour vertices of the esophageal cancer region includes:

[0021] Inputting the key fault into the U-Net network based on channel attention and cross-layer fusion; the U-Net network based on channel attention and cross-layer fusion includes a basic U-Net network and a plurality of cross-layer feature fusion modules, each of which is embedded in the basic U-Net network;

[0022] The output result of the U-Net network based on channel attention and cross-layer fusion is obtained; the output result of the U-Net network based on channel attention and cross-layer fusion is a key fault in which the esophageal cancer area and other areas are separated by the at least one group of contour vertices.

[0023] Furthermore, the using of the Poly-RNN algorithm to determine the labeling information of the key fault according to the at least one group of contour vertices includes:

[0024] Inputting the key fault with at least one set of contour vertices into a convolutional neural network;

[0025] A gated graph neural network is used to perform image sampling on the image features output by the convolutional neural network to obtain the labeling information of the key fault.

[0026] Furthermore, the using of regional center point growth mapping to obtain automatic annotation of the full sequence of the non-homogeneous energy spectrum CT images according to the annotation information includes:

[0027] In each of the key sections, determining the starting section and the ending section of esophageal cancer;

[0028] Determining a starting center on the starting fault according to the marking information;

[0029] According to the order of the key faults, the starting center is mapped to the ending fault through the key faults in sequence.

[0030] Further, determining a starting center on the starting fault according to the annotation information includes:

[0031] According to the marking information, a rectangular area is determined on the starting fault as the starting center; the center of the rectangular area coincides with the center of the starting fault.

[0032] On the other hand, an embodiment of the present invention further includes a non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling system, including:

[0033] The first module is used to enhance other regions except the esophageal cancer region in the non-homogeneous energy spectrum CT image by using a multi-structure three-dimensional response filter based on a Hessian matrix;

[0034] The second module is used to extract multiple key faults from the non-homogeneous energy spectrum CT image using a feature extraction algorithm based on information weight;

[0035] The third module is used to segment the esophageal cancer area and other areas in the non-homogeneous energy spectrum CT image using a U-Net network based on channel attention and cross-layer fusion, and obtain at least one set of contour vertices of the esophageal cancer area;

[0036] A fourth module is used to determine the labeling information of the key fault according to the at least one group of contour vertices using a Poly-RNN algorithm;

[0037] The fifth module is used to use the regional center point growth mapping to obtain automatic annotation of the full sequence of the non-homogeneous energy spectrum CT image according to the annotation information.

[0038] The beneficial effects of the present invention are as follows: the non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling method in the embodiment extracts the key faults in the entire spectral CT esophageal cancer image data set, and uses the channel attention segmentation network to perform foreground and background image segmentation on the key faults to obtain enhanced vertex representation, so as to provide prior knowledge for the subsequent use of the Polygon-RNN automatic labeling algorithm, thereby realizing automatic labeling of the entire non-homogeneous esophageal cancer CT sequence data set, which can effectively reflect the changes in the pathology and morphological structure of the lesion, and has a higher speed and lower error rate compared with manual labeling. Compared with the existing automatic labeling technology, the non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling method in the embodiment can make full use of the multi-parameter and quantitative analysis imaging parameter characteristics of spectral CT, and can more effectively automatically label esophageal cancer spectral CT data samples, with higher labeling accuracy and labeling efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Flow chart of the weakly supervised automatic labeling method for non-homogeneous three-dimensional esophageal cancer spectral CT in the embodiment;

[0040] Figure 2 It is an overall principle diagram of the weakly supervised automatic labeling method of non-homogeneous three-dimensional esophageal cancer spectral CT in the embodiment;

[0041] Figure 3 It is a schematic diagram of key fault extraction based on information weight feature selection algorithm in the embodiment;

[0042] Figure 4 It is a schematic diagram of a segmentation network of U-Net based on channel attention in an embodiment;

[0043] Figure 5 It is a schematic diagram of the Polygon-RNN key fault automatic labeling algorithm in the embodiment;

[0044] Figure 6 It is a schematic diagram of a convolutional neural network encoder in an embodiment;

[0045] Figure 7 It is a schematic diagram of the Polygon-RNN+ evaluation network algorithm in the embodiment;

[0046] Figure 8 It is a schematic diagram of the automatic labeling algorithm of the regional center point mapping method in the embodiment. DETAILED DESCRIPTION

[0047] In the prior art, there is a problem of high cost in labeling a large number of spectral CT esophageal cancer sequences, which restricts the application of existing deep learning algorithms in the field of esophageal cancer CT imaging. Manually labeling esophageal cancer sequence data sets is difficult, the steps are complicated, and the time and energy required are too high. In this embodiment, through the description of the weakly supervised automatic labeling method of non-homogeneous three-dimensional esophageal cancer spectral CT, it is discussed how to effectively improve the efficiency of labeling non-homogeneous spectral CT esophageal cancer sequence data sets, so that researchers in this field have enough labeled data sets for the development of esophageal cancer detection algorithms.

[0048] In this embodiment, refer to Figure 1 The weakly supervised automatic labeling method for non-homogeneous three-dimensional esophageal cancer spectral CT includes the following steps:

[0049] S1. Use a multi-structure three-dimensional response filter based on the Hessian matrix to enhance the areas other than the esophageal cancer area in the non-homogeneous spectral CT image;

[0050] S2. Extract multiple key slices from non-homogeneous spectral CT images using a feature extraction algorithm based on information weight;

[0051] S3. Use a U-Net network based on channel attention and cross-layer fusion to segment the esophageal cancer area and other areas in the non-homogeneous energy spectrum CT image, and obtain at least one set of contour vertices of the esophageal cancer area;

[0052] S4. Using the Poly-RNN algorithm, determining the labeling information of the key fault according to at least one set of contour vertices;

[0053] S5. Use regional center point growing mapping to obtain automatic annotation of the entire sequence of non-homogeneous energy spectrum CT images according to the annotation information.

[0054] In this embodiment, the overall principle of steps S1-S5 is as follows Figure 2 As shown, the principles of each step of steps S1-S5 are described in detail below.

[0055] In this embodiment, step S1, that is, using a multi-structure three-dimensional response filter based on a Hessian matrix to enhance other regions except the esophageal cancer region in the non-homogeneous energy spectrum CT image, specifically includes the following steps:

[0056] S101. Merging continuous non-homogeneous spectral CT images into three-dimensional volume data; wherein the three-dimensional volume data includes a plurality of voxels, and the Hessian matrix of each voxel includes second-order derivatives in different directions;

[0057] S102. Using the linear scale space theory, the elements of the Hessian matrix of the non-homogeneous spectral CT image on the σ scale are determined as Among them, i,j=1,2,3 represent the position of the element, * represents the convolution operation, γ is the parameter, G(x,σ) is the Gaussian function,

[0058] S103. Decomposing the non-homogeneous energy spectrum CT image into continuous two-dimensional images;

[0059] S104. Input the two-dimensional image decomposed from the non-homogeneous energy spectrum CT image into a multi-structure three-dimensional response filter to obtain an output result of the multi-structure three-dimensional response filter; the output result of the multi-structure three-dimensional response filter is to obtain an enhanced non-homogeneous energy spectrum CT image for other areas except the esophageal cancer area.

[0060] The principle of steps S101-S104 is that since the Hessian matrix is ​​related to the local geometric structure, the specific structure can be detected by using the eigenvalues ​​of the Hessian matrix. First, the continuous CT images are merged into a three-dimensional volume data. The Hessian matrix of each voxel in the three-dimensional volume data includes the second-order derivatives in different directions. Let I(x) represent the coordinate x = [x1, x2, x3] on the non-homogeneous energy spectrum CT image T The intensity of the 3D volume data at . To analyze the multi-scale structure, differentiation is performed on the Gaussian scale space. Therefore, using the linear scale space theory, the element of x in the Hessian matrix of σ scale is defined as:

[0061]

[0062] Where i, j = 1, 2, 3 denotes the position of the element, and * denotes the convolution operation. The parameter γ is introduced to rescale the response of the differential operation at multiple scales and is set to 2. The Gaussian function G(x, σ) is defined as:

[0063]

[0064] The second-order derivative of the Gaussian kernel with scale x can be considered as a detection kernel, which can capture the difference between areas within and outside the range (-x, x) in the direction of the derivative. In addition, the half-width of the kernel is set to the integer closest to 3σ. When performing image convolution. Each voxel in the three-dimensional volume data corresponds to a 3×3 Hessian matrix. The enhanced spectral CT three-dimensional volume data is decomposed into continuous two-dimensional images, and the spectral CT sequence is cropped as the input of the segmentation network. This filter can enhance the surrounding tissues and organs to extract prior knowledge from the anatomical structure, which can assist the Polygon-RNN algorithm in detection and enhance the robustness of the network.

[0065] In step S1, after continuous CT images are merged into a three-dimensional volume data, the response of the differential operation is readjusted using the differential of Gaussian scale space and the linear scale space theory. The CT images enhanced with three-dimensional structural information can guide the segmentation network to distinguish esophageal cancer from similar tissues and organs in the CT images.

[0066] In this embodiment, step S2, that is, the step of extracting multiple key faults from the non-homogeneous energy spectrum CT image using a feature extraction algorithm based on information weight, specifically includes the following steps:

[0067] S201. Reshaping the non-homogeneous energy spectrum CT image;

[0068] S202. Inputting the non-homogeneous energy spectrum CT image into a key fault extraction network; the key fault extraction network includes a convolutional layer, a first fully connected layer, and a second fully connected layer;

[0069] S203. Obtain the key point score of each key fault output by the key fault extraction network;

[0070] S204. Multiply the key point score by the fault feature map to obtain the corresponding key fault.

[0071] The principle of steps S201-S204 is: for the non-homogeneous energy spectrum CT image data set, firstly, a key fault is extracted by an information weight-based feature extraction algorithm, and the extraction process is as follows: Figure 3 As shown in the figure. The esophageal cancer spectral CT dataset is first reshaped and then input into the key slice extraction module. The reshaping is to reduce the dimensionality of the convolutional layer and output F* after the maxpooling layer. After expansion, the feature map of each slice is input into the same fully connected layer to calculate the critical score:

[0072] F * =maxpool(W k_c F)

[0073] s i =Wk_fc2 (W k_fc1 F1 * +b k_fc1 )+b k_fc2

[0074]

[0075] Among them, W k_c , W k_fc1 and W k_fc2 are the parameters of the convolutional layer, the first fully connected layer, and the second fully connected layer, respectively. b k_fc1 and b k_fc2 are the bias parameters of the two fully connected layers. i is the critical score of each fault, i∈{1,2,3,…,framenum}, si is performed as a probability operation to obtain S=[s1,s2,…,s framenum ]. In order to multiply the score of each key point in S by the feature map of each fault in F, it is necessary to expand the tensor and change the shape of S. After calculation, the key fault set is obtained:

[0076] F key =reshap(S)F

[0077] The key fault set F key It helps the Polygon-RNN algorithm to enhance the vertex structure representation and improve the labeling performance.

[0078] After step S2 reshapes and expands the non-homogeneous esophageal cancer energy spectrum CT image, the features of each fault are mapped to the same fully connected layer to calculate the critical score. Finally, the key fault set is obtained based on the mapping of each key point score and fault features. The features of the prominent area of ​​esophageal cancer can be weighted by weight to guide the segmentation network to distinguish esophageal cancer from other tissues and organs in the CT image.

[0079] In this embodiment, step S3, that is, using a U-Net network based on channel attention and cross-layer fusion to segment the esophageal cancer area and other areas in the non-homogeneous energy spectrum CT image, and obtaining at least one set of contour vertices of the esophageal cancer area, specifically includes the following steps:

[0080] S301. Input the key fault into the U-Net network based on channel attention and cross-layer fusion; the U-Net network based on channel attention and cross-layer fusion includes a basic U-Net network and multiple cross-layer feature fusion modules, and each cross-layer feature fusion module is embedded in the basic U-Net network;

[0081] S302. Obtain the output result of the U-Net network based on channel attention and cross-layer fusion; the output result of the U-Net network based on channel attention and cross-layer fusion is a key fault in which the esophageal cancer area and other areas are separated by at least one set of contour vertices.

[0082] The principle of steps S301-S302 is: through a new U-Net—channel attention and cross-layer fusion U-Net, it is used to segment other similar surrounding tissue organs and esophageal parts from the background image enhanced by the filter and the foreground image highlighted by the feature extraction algorithm. The network combines the channel attention module and the cross-layer feature fusion module. The former enhances each other through high-level features and low-level and complementary information between features, and the latter uses high-level features to enhance or suppress low-level features to enhance the generalization ability of the network. Because high-level features represent specific tissue information, and low-level features represent detailed information such as edges and contours, the network can learn specific detailed features of specific tissues. Among them, the network structure diagram of the channel attention U-Net is shown in the figure. Figure 4 shown. Figure 4 The network shown is a segmentation network of an encoder-decoder structure built on U-Net. By embedding a cross-layer feature fusion module in the basic U-Net network, the channel attention U-Net in this embodiment is formed. Channel attention U-Net is an end-to-end network architecture consisting of two main parts. The first component is the cross-layer feature fusion module, which consists of several channel attention modules. Cross-layer feature fusion can fuse the features of the top feature map layer by layer into the bottom feature map, and achieve the purpose of guiding the feature fusion of the bottom feature map feature extraction. The second component is U-Net, which is an encoder-decoder structure, the first half of which is used for feature extraction, and the second half is used to upsample the feature map to the original size to obtain the feature map. In particular, there is a jump connection between the encoder and the decoder. The jump connection combines the corresponding downsampled and upsampled feature maps. This process can solve the problem of information loss caused by downsampling and increase the attention to convolution feature extraction. The technology finally obtains an enhanced vertex structure representation with clear boundaries.

[0083] Step S3 combines the channel attention module on the basis of U-Net to guide the selection of high-resolution image information and high-level semantic information between low-level and high-level feature maps, and utilizes the cross-layer feature fusion module to emphasize or suppress low-level features according to high-level features. The channel attention module can be integrated into the cross-layer fusion of the encoder and decoder to fully retain the high-level semantic information and visual feature information of the image, and mark the vertices of the corresponding contours.

[0084] In this embodiment, step S4, that is, using the Poly-RNN algorithm to determine the labeling information of the key fault according to at least one group of contour vertices, specifically includes the following steps:

[0085] S401. Inputting a key fault with at least one set of contour vertices into a convolutional neural network;

[0086] S402. Use a gated graph neural network to perform image sampling on the image features output by the convolutional neural network to obtain labeling information of key faults.

[0087] The principle of steps S401-S402 is: based on the obtained enhanced image vertex representation, use the convolutional neural network to extract image features, introduce the attention mechanism and evaluation network to select the best polygon from the candidate polygons of the recurrent neural network. Finally, use the gated graph neural network to perform image sampling. The overall architecture of the Polygon-RNN algorithm is as follows: Figure 5 As shown. For the convolutional neural network part, refer to Figure 6 , drawing on the approach of ResNet-50, reducing the step size and introducing dilated convolutions, so as to amplify the input feature map without reducing the receptive field of a single neuron. In addition, skip connections are introduced to capture low-level details such as corners and high-level semantic information at the same time. The convolution kernels, group normalization, ReLU, maximum pooling, etc. used can use conventional means. For example, the size of the convolution kernel can be the commonly used 3×3. For the recurrent neural network part, a two-layer ConvLTSM is used to retain spatial information and reduce the number of parameters. The output of the network is a unique hot encoding of (D×D)+1 elements. The first D×D dimensions represent possible vertex positions, and the last dimension marks the end point of the polygon. In order to improve the performance of the RNN part, an attention mechanism is added. Specifically, at time step t, the weighted feature map is calculated:

[0088] α t =softmax(f att (x,f1(h 1,t-1 ), f2(h 2,t-1 )))

[0089]

[0090] In the above formula, x is the jump feature tensor, h is the hidden state tensor, and f1 and f2 use a fully connected layer to transform h 1,t 、h 2,t Mapped to D×D×128. att The inputs are summed and mapped to D×D through a fully connected layer. is the Hadamard product. Intuitively, the attention mechanism uses the previous hidden state of the recurrent neural network to control specific locations in the image feature map, so that the recurrent neural network only pays attention to relevant information in the next time step. In this embodiment, a branch containing two D×D dimensional network layers is added, so that the first layer predicts edges and the second layer predicts vertices. During testing, the first vertex is sampled from the last layer of this branch. In this embodiment, Polygon-RNN++ uses an evaluation network consisting of two 3×3 convolutional layers plus a fully connected layer. Its structure can be referred to Figure 7 . The recurrent neural network outputs a D×D dimensional polygon, with D set to 28. The reason for not taking a larger D is to avoid exceeding the memory limit. In order to increase the final output resolution, Polygon-RNN+ uses a gated graph neural network for upsampling, treating vertices as nodes of the graph and adding nodes between adjacent nodes. The gated graph neural network defines a propagation model that generalizes the recurrent neural network to any graph, which can effectively propagate information before generating output on each node.

[0091]

[0092]

[0093]

[0094] In the above formula, V is the node set of the graph, x v is the initial state of node v, is the hidden state of node v at time step t. Matrix A determines how nodes transmit information to each other, where N represents the number of edge types. In the experiment, a 256-dimensional GRU was used with a propagation step number of T = 5. The output of node v is defined as:

[0095]

[0096] out v =f2(h v )

[0097] f1 and f2 are MLP (Multi-layer Perceptron). The 112×112×256 feature map of the convolutional neural network is passed to the GGNN. Around each node v in the graph (after stretching), an S×S block is extracted to obtain the vector x v , provided to GGNN. After the propagation process, the output of the predicted node v is the position on the D'×D' spatial grid. The grid is based on the original position (v x ,v y) is used as a reference, so the prediction task is actually a relative placement problem and can be regarded as a classification problem and trained based on cross entropy loss. The standard answer of the training is the output of the RNN part. By executing steps S401-S402, the key fault set of the esophageal cancer spectral CT sequence data set that has been automatically labeled is finally obtained.

[0098] After using the convolutional neural network to extract image features in step S4, the attention mechanism and evaluation network are introduced into the recurrent neural network to select the best candidate polygons. Finally, the gated graph neural network is used to perform labeling sampling of key faults, and the automatic labeling of key faults can be obtained based on the segmented CT images.

[0099] In this embodiment, step S5, that is, the step of using the regional center point growth mapping to obtain the automatic annotation of the full sequence of non-homogeneous energy spectrum CT images according to the annotation information, specifically includes the following steps:

[0100] S501. Determine the starting fault and the ending fault of esophageal cancer in each key fault;

[0101] S502. Determine the starting center on the starting fault according to the marked information;

[0102] S503. According to the order of the key faults, the starting center is mapped to the ending fault through the key faults in sequence.

[0103] Among them, step S502, that is, the step of determining the starting center on the starting fault according to the marked information, can be specifically: according to the marked information, a rectangular area is determined on the starting fault as the starting center; the center of the rectangular area coincides with the center of the starting fault.

[0104] The principle of steps S501-S503 is to use the key fault set of the esophageal cancer spectral CT sequence data set that has been annotated obtained by the Polygon-RNN algorithm to perform regional mapping on the entire spectral CT data set. In this embodiment, the above function is achieved by a semi-automatic esophageal or esophageal cancer CT image annotation method, which is used for semi-automatic detection of esophageal and esophageal cancer three-dimensional CT sequences. At the beginning of the semi-automatic detection method, five parameters need to be set manually. They are the starting and ending faults of the esophagus and esophageal cancer, and are the center point of the esophageal area in the first esophageal CT fault. Among them, the starting center point represents the center point of the esophageal area in the first fault in the esophageal CT fault data set. With it as the center, determine an 80×80 rectangular area as the center point of the extraction detection area, map from this center point to the next fault to form the center point of the next fault, and continue to generate an 80×80 rectangular area as input with this center point. The principle of the semi-automatic esophageal or esophageal cancer CT image annotation method used in steps S501-S503 is as follows: Figure 8 As shown in the figure, the premise of this method is that the thickness of the esophagus and esophageal cancer sections is small enough, so the center point of the upper esophagus can correspond to the center point of the lower esophagus in the spectral CT dataset. This situation meets most of the situations in actual diagnosis. After the algorithm is completed, we can get a complete and automatically annotated esophageal cancer spectral CT sequence dataset.

[0105] Step S5 sets the starting and ending faults of the esophagus and esophageal cancer and uses them as the center, and performs regional growth mapping of the next center point step by step based on this center point. According to the mapping of the annotation information of the key faults, the automatic annotation of the entire sequence of esophageal cancer spectral CT images can be obtained.

[0106] In summary, the technical effects of the weakly supervised automatic labeling method for non-homogeneous three-dimensional esophageal cancer spectral CT in this embodiment include:

[0107] The non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling method in this embodiment adopts a weakly supervised automatic labeling method for esophageal cancer spectral CT sequence based on a multi-scale 3D enhancement filter and Polygon-RNN. First, a multi-structure response filter based on a Hessian matrix is ​​used to enhance other tissues and organs except the esophagus. Then, a feature extraction algorithm based on information weight is used to extract key faults. Subsequently, the key faults are input into a channel attention module to achieve high-precision segmentation of esophageal cancer in the key faults to obtain enhanced vertex representations. Then, the Polygon-RNN automatic labeling algorithm is used to automatically label the key faults of esophageal cancer data under weak supervision for the image of the enhanced vertex structure. Finally, the region growing algorithm is used for the key faults that have been automatically labeled to achieve automatic labeling of the entire esophageal cancer spectral CT sequence sample. By extracting the key faults in the entire spectral CT esophageal cancer image dataset and using the channel attention segmentation network to perform foreground and background image segmentation on the key faults to obtain enhanced vertex representation, prior knowledge is provided for the subsequent use of the Polygon-RNN automatic labeling algorithm, thereby realizing automatic labeling of the entire esophageal cancer CT sequence dataset, which can effectively reflect the changes in the pathology and morphological structure of the lesion. Medical staff can make a diagnosis based on the processing results of the non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling method in this embodiment, so as to achieve the purpose of early detection of lesions and diagnosis of diseases. On the other hand, it can realize the complementary advantages of medical image imaging technology, which is of great significance to make up for the defects of medical image presentation technology and the lack of image information.

[0108] In this embodiment, the non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling system includes:

[0109] The first module is used to enhance other regions except the esophageal cancer region in the non-homogeneous energy spectrum CT image using a multi-structure three-dimensional response filter based on the Hessian matrix;

[0110] The second module is used to extract multiple key faults from the non-homogeneous energy spectrum CT image using a feature extraction algorithm based on information weight;

[0111] The third module is used to segment the esophageal cancer area and other areas in the non-homogeneous energy spectrum CT image using a U-Net network based on channel attention and cross-layer fusion, and obtain at least one set of contour vertices of the esophageal cancer area;

[0112] The fourth module is used to determine the labeling information of the key fault according to at least one set of contour vertices using a Poly-RNN algorithm;

[0113] The fifth module is used to use the regional center point growth mapping to obtain automatic annotation of the full sequence of non-homogeneous energy spectrum CT images according to the annotation information.

[0114] In this embodiment, the first module, the second module, the third module, the fourth module and the fifth module can be hardware modules, software modules or a combination of hardware modules and software modules with corresponding functions. By running the corresponding functions of each module, the non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling system can achieve the same technical effect as the non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling method embodiment.

[0115] In this embodiment, a computer device includes a memory and a processor, wherein the memory is used to store at least one program, and the processor is used to load the at least one program to execute the non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling method in the embodiment to achieve the same technical effect as described in the embodiment.

[0116] In this embodiment, a storage medium stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to execute the non-homogeneous three-dimensional esophageal cancer spectral CT weakly supervised automatic labeling method in the embodiment to achieve the same technical effect as described in the embodiment.

[0117] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. In addition, the descriptions of up, down, left, right, etc. used in this disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "said" and "the" used in this disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by technicians in this technical field. The terms used in the specification of this embodiment are only for describing specific embodiments, not for limiting the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0118] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0119] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0120] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed jointly on one or more processors, by hardware or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0121] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or a portion thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0122] The computer program can be applied to input data to perform the functions described in the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0123] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. A weakly supervised automatic labeling method for non-homogeneous three-dimensional esophageal cancer spectral CT, characterized in that: include: A multi-structure three-dimensional response filter based on the Hessian matrix is ​​used to enhance the areas other than the esophageal cancer area in the non-homogeneous energy spectrum CT image; Extracting a plurality of key faults from the non-homogeneous energy spectrum CT image using a feature extraction algorithm based on information weight; Using a U-Net network based on channel attention and cross-layer fusion, segmenting the esophageal cancer region and other regions in the non-homogeneous energy spectrum CT image to obtain at least one set of contour vertices of the esophageal cancer region; Using a Poly-RNN algorithm, determining labeling information of the key fault according to the at least one set of contour vertices; Using regional center point growing mapping, obtaining automatic annotation of the full sequence of the non-homogeneous energy spectrum CT image according to the annotation information; The method of using a multi-structure three-dimensional response filter based on a Hessian matrix to enhance other regions except the esophageal cancer region in the non-homogeneous energy spectrum CT image includes: Merging the continuous non-homogeneous energy spectrum CT images into non-homogeneous three-dimensional volume data; the three-dimensional volume data includes a plurality of voxels, and the Hessian matrix of each voxel includes second-order derivatives in different directions; Using the linear scale space theory, the non-homogeneous spectral CT image is transformed into The elements of the Hessian matrix on the scale are determined as ;in, Indicates the position of the element, * indicates the convolution operation, As parameters, is a Gaussian function, ; Decomposing the non-homogeneous energy spectrum CT image into continuous two-dimensional images; The two-dimensional image decomposed from the non-homogeneous energy spectrum CT image is input into the multi-structure three-dimensional response filter to obtain the output result of the multi-structure three-dimensional response filter; the output result of the multi-structure three-dimensional response filter is to obtain an enhanced non-homogeneous energy spectrum CT image of other areas except the esophageal cancer area; The method of extracting a plurality of key faults from the non-homogeneous energy spectrum CT image using a feature extraction algorithm based on information weight includes: Reshaping the non-homogeneous energy spectrum CT image; Inputting the non-homogeneous energy spectrum CT image into a key fault extraction network; the key fault extraction network comprises a convolutional layer, a first fully connected layer and a second fully connected layer; Obtain a key point score of each key fault output by the key fault extraction network; Multiplying the key point score by the fault feature map to obtain a corresponding key fault; The method of using the regional center point growth mapping to obtain the automatic annotation of the full sequence of the non-homogeneous energy spectrum CT image according to the annotation information includes: In each of the key sections, determining the starting section and the ending section of esophageal cancer; Determining a starting center on the starting fault according to the marking information; According to the order of the key faults, the starting center is mapped to the ending fault through the key faults in sequence.

2. The method for weakly supervised automatic labeling of non-homogeneous three-dimensional esophageal cancer spectral CT according to claim 1 is characterized in that: The method uses a U-Net network based on channel attention and cross-layer fusion to segment the esophageal cancer region and other regions in the non-homogeneous energy spectrum CT image, and obtains at least one set of contour vertices of the esophageal cancer region, including: Inputting the key fault into the U-Net network based on channel attention and cross-layer fusion; the U-Net network based on channel attention and cross-layer fusion includes a basic U-Net network and a plurality of cross-layer feature fusion modules, each of which is embedded in the basic U-Net network; The output result of the U-Net network based on channel attention and cross-layer fusion is obtained; the output result of the U-Net network based on channel attention and cross-layer fusion is a key fault in which the esophageal cancer area and other areas are separated by the at least one group of contour vertices.

3. The method for weakly supervised automatic labeling of non-homogeneous three-dimensional esophageal cancer spectral CT according to claim 1, characterized in that: The using the Poly-RNN algorithm to determine the labeling information of the key fault according to the at least one group of contour vertices includes: Inputting the key fault with at least one set of contour vertices into a convolutional neural network; A gated graph neural network is used to perform image sampling on the image features output by the convolutional neural network to obtain the labeling information of the key fault.

4. The method for weakly supervised automatic labeling of non-homogeneous three-dimensional esophageal cancer spectral CT according to claim 1, characterized in that: Determining a starting center on the starting fault according to the marked information includes: According to the marking information, a rectangular area is determined on the starting fault as the starting center; the center of the rectangular area coincides with the center of the starting fault.

5. A weakly supervised automatic labeling system for non-homogeneous three-dimensional esophageal cancer spectral CT, characterized in that: include: The first module is used to enhance other regions except the esophageal cancer region in the non-homogeneous energy spectrum CT image by using a multi-structure three-dimensional response filter based on a Hessian matrix; The second module is used to extract multiple key faults from the non-homogeneous energy spectrum CT image using a feature extraction algorithm based on information weight; The third module is used to segment the esophageal cancer area and other areas in the non-homogeneous energy spectrum CT image using a U-Net network based on channel attention and cross-layer fusion, and obtain at least one set of contour vertices of the esophageal cancer area; A fourth module is used to determine the labeling information of the key fault according to the at least one group of contour vertices using a Poly-RNN algorithm; A fifth module is used to obtain automatic annotation of the full sequence of the non-homogeneous energy spectrum CT image according to the annotation information by using regional center point growth mapping; The method of using a multi-structure three-dimensional response filter based on a Hessian matrix to enhance other regions except the esophageal cancer region in the non-homogeneous energy spectrum CT image includes: Merging the continuous non-homogeneous energy spectrum CT images into non-homogeneous three-dimensional volume data; the three-dimensional volume data includes a plurality of voxels, and the Hessian matrix of each voxel includes second-order derivatives in different directions; Using the linear scale space theory, the non-homogeneous spectral CT image is transformed into The elements of the Hessian matrix on the scale are determined as ;in, Indicates the position of the element, * indicates the convolution operation, As parameters, is a Gaussian function, ; Decomposing the non-homogeneous energy spectrum CT image into continuous two-dimensional images; The two-dimensional image decomposed from the non-homogeneous energy spectrum CT image is input into the multi-structure three-dimensional response filter to obtain the output result of the multi-structure three-dimensional response filter; the output result of the multi-structure three-dimensional response filter is to obtain an enhanced non-homogeneous energy spectrum CT image of other areas except the esophageal cancer area; The method of extracting a plurality of key faults from the non-homogeneous energy spectrum CT image using a feature extraction algorithm based on information weight includes: Reshaping the non-homogeneous energy spectrum CT image; Inputting the non-homogeneous energy spectrum CT image into a key fault extraction network; the key fault extraction network comprises a convolutional layer, a first fully connected layer and a second fully connected layer; Obtain a key point score of each key fault output by the key fault extraction network; Multiplying the key point score by the fault feature map to obtain a corresponding key fault; The method of using the regional center point growth mapping to obtain the automatic annotation of the full sequence of the non-homogeneous energy spectrum CT image according to the annotation information includes: In each of the key sections, determining the starting section and the ending section of esophageal cancer; Determining a starting center on the starting fault according to the marking information; According to the order of the key faults, the starting center is mapped to the ending fault through the key faults in sequence.