Panoramic interactive tooth segmentation method based on feature enhancement

Abstract

A panoramic interactive tooth segmentation method based on feature enhancement comprises the following steps: reconstructing a two-dimensional panoramic image on the basis of a three-dimensional image of an original cone beam CT, calculating a three-dimensional initial seed point and a single-tooth region of interest, and generating final three-dimensional foreground and background seed points after semantic discrimination and adjustment; generating corresponding foreground and background distance maps, inputting the foreground and background distance maps and a single-tooth cone beam CT image into a Transform model at the same time, calculating and generating intra-layer and inter-layer features, fusing the intra-layer and inter-layer features to generate foreground and background enhanced distance maps, performing image segmentation by using 3DUNet, and outputting a single-tooth initial segmentation result; and evaluating the initial segmentation result, performing secondary interaction on the slices of which the scores are lower than a threshold value by the user to generate secondary seed points, and finally optimizing the initial segmentation result according to the secondary seed points. The method is implemented on the two-dimensional panorama, so that the interaction robustness is improved; the foreground enhanced distance map and the background enhanced distance map serve as two independent sources, and compared with a traditional mode that only foreground seeds serve as interaction sources, the difference characteristics between the background and the foreground can be fully learned.

Description

Technical Field The invention belongs to the field of medical image analysis and processing in computer vision, and in particular relates to a panoramic interactive tooth segmentation method based on feature enhancement. Background Art The tooth segmentation task refers to segmenting all teeth based on one or more given teeth using a segmentation algorithm, which belongs to the instance segmentation task in computer vision. The interactive tooth segmentation method based on cone beam CT (CBCT) images effectively incorporates the individual priors of the target object by designing a convenient interactive strategy. It is less dependent on the scale of training data than fully automatic segmentation, has higher segmentation efficiency and more accurate results, and can extract accurate tooth morphology from 3D CT raw data. The results can be used for quantitative analysis of key clinical indicators and visualization of internal morphology that is actually invisible. At present, the interactive segmentation field mainly attempts to develop from two aspects: interaction strategy and effective interaction propagation. In the prior art, Lin et al. [1] proposed a first click attention network, which combines the guidance information of the first click with the obtained features to reduce the number of interactions and improve the interaction efficiency. Song et al. [2] proposed a bidirectional seed attention network. By adding a bidirectional seed attention module to the backbone segmentation network to enhance the semantic information of the seed, it avoids the weakening of the seed information and obtains better segmentation results. Zhang et al. [3] proposed a method of embedding a gated memory propagation unit ConvRNN network to train patch samples and capture the spatial relationship between adjacent patches to achieve seed point information and image fusion. Sakinis et al. [4] proposed a semi-automatic segmentation method of convolutional neural network, which is trained on a relatively limited training set and can produce fast and accurate segmentation output on CT images. Luo et al. [5] proposed a segmentation MIDeepSeg framework based on minimal interaction deep learning, which uses exponential geodesic distance transform to encode user interactions without the need for additional parameters and thresholds. Zhou et al. [6] proposed a new volumetric memory network called VMN, which enables interaction on two-dimensional slices and can adjust three-dimensional segmentation results. They also proposed an interactive self-evaluation method to assist multiple rounds of interaction.

[0001] Lin Z, Zhang Z, Chen LZ, et al.Interactive image segmentation with first click attention[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition.2020:13339-13348.

[0002] Song G,Lee K M.Bi-directional seed attention network for interactive image segmentation[J].IEEE Signal Processing Letters, 2020,27:1540-1544.

[0003] Zhang J,Shi Y,Sun J,et al.Interactive medical image segmentation via a point-based interaction[J].Artificial Intelligence in Medicine,2021,111:101998.

[0004] Sakinis T,Milletari F,Roth H,et al.Interactive segmentation of medical images through fully convolutional neural networks[J].arxiv preprintarxiv:1903.08205,2019.

[0005] Luo X,Wang G,Song T,et al.MIDeepSeg:Minimally interactive segmentation of unseen objects from medical images using deep learning[J].Medical image analysis,2021,72:102102.

[0006] Zhou T, Li L, Bredell G, et al. Volumetric memory network for interactive medical image segmentation[J]. Medical Image Analysis, 2023,83:102599. At present, the interactive segmentation method faces the following shortcomings and problems: (1) The existing technologies [1, 2, 3] only use a click strategy for interaction. For complex three-dimensional objects, it is necessary to repeatedly observe three standard perspectives to determine the location of the seed point. This is greatly affected by user subjectivity and the interaction lacks robustness. (2) Existing technologies [3, 4] constrain the application range of adjustment by training global or uniform-sized square patches. However, the overall shape of teeth is not uniform and symmetrical. Applying a uniform range will weaken the optimization effect after interaction to a certain extent. (3) The existing technology [5] only focuses on the foreground seed points during the initial interaction, while ignoring the features of the background seed points, and does not fully utilize the semantic associations between multiple types of seed points. (4) In the prior art [6], interactive seed point information is directly used for segmentation. The seed point is used as a local prior and inevitably decays through network learning, resulting in low guidance efficiency. The self-evaluation module has a single input and completely relies on the characteristics of the segmentation network itself, which will lead to inaccurate evaluation results. Biased interactive feedback will eventually lead to segmentation errors. Due to the problems faced by the interactive segmentation methods in the prior art, it is necessary to propose a panoramic interactive tooth segmentation method based on feature enhancement to solve the above problems. It can intelligently identify and correct certain deviation interactions, quantitatively evaluate the quality of interaction, and improve the effectiveness of interaction results in deep model learning. It can be better applied in application scenarios that require precise interaction, such as medical image analysis, human-computer interaction, and augmented reality. Summary of the invention The present invention designs a panoramic interactive tooth segmentation method based on feature enhancement, which can automatically identify and correct user interactions, provide a convenient interactive perspective, and improve the effectiveness of interactive prior propagation by performing pre-feature enhancement before segmentation. This method uses foreground seeds and background seeds as two independent priors, learns them in parallel, and pre-enhances them before the segmentation network to achieve interactive and accurate tooth segmentation results. A panoramic interactive tooth segmentation method based on feature enhancement, comprising: Step 1: Based on the three-dimensional image of the original cone beam CT image, a two-dimensional panoramic image is reconstructed, and a two-dimensional foreground seed point a of the tooth, a two-dimensional background seed point b, and a bounding box c containing a single tooth are marked according to user interaction; a three-dimensional initial seed point and a single tooth region of interest are calculated, and semantic discrimination is performed on the three-dimensional initial seed point. According to the discrimination result, it is determined whether to adjust the position of the three-dimensional initial seed point, and the final three-dimensional foreground seed point a2 and three-dimensional background seed point b2 are generated; Step 2: Combine the three-dimensional foreground seed point a2 and the three-dimensional background seed point b2 with the single tooth cone beam CT image T according to the exponential geodesic distance method to generate the corresponding foreground distance map P F and background distance map P B ; The foreground distance map P F and the background distance map P BThe single tooth cone beam CT image T is input into the Transformer model encoding to calculate the intra-layer feature Fp and inter-layer feature F d , fusion of the intra-layer feature Fp and the inter-layer feature F d , generate the foreground enhancement distance map F(P F ) and background enhancement distance map F(P B ), use 3DUNet to perform image segmentation and output the initial segmentation result R0 of a single tooth; Step 3: Multi-level evaluation and optimization: determine the evaluation network input and construct the evaluation network, evaluate the initial segmentation results, and generate an evaluation score for each slice; filter out slices with scores below the threshold, and the user performs secondary interaction on the slices with scores below the threshold to generate secondary seed points; finally, optimize the initial segmentation results based on the secondary seed points. Furthermore, the reconstruction in step 1 generates a two-dimensional panoramic image according to formula (1): Processing, where X is the tooth target image to be reconstructed, s is a point on the projected dental arch curve, Z is the Zth slice in the CT image, P(S,Z) represents the result of projecting the Zth slice according to the point S on the dental arch curve, r(s) is the coordinate of point S, n(s) is the normal vector of point S, (α,-α) is the projection range along the normal vector, and t is the projection variable. Furthermore, the calculation of the three-dimensional initial seed point and the single tooth region of interest in step 1 includes: back-projecting the two-dimensional foreground seed point a, the two-dimensional background seed point b and the bounding box c according to formula (1) to generate a three-dimensional foreground initial seed point a1, a three-dimensional background initial seed point b1, and a three-dimensional bounding box c1; calculating the circumscribed cuboid c2 of the three-dimensional bounding box c1 to form a three-dimensional single tooth region of interest, and cropping the original cone beam CT image according to the c2 region to generate a single tooth cone beam CT image T composed of multiple slices of uniform size. Furthermore, the semantic discrimination of the three-dimensional initial seed point in step 1 includes discriminating whether the three-dimensional initial seed point is located in the crown area or the root area; then identifying the seed semantics; the identifying the seed semantics further includes, if the three-dimensional initial seed point is in the root, as long as the seed position is in the foreground range of the tooth, it is identified as a foreground point, otherwise it is a background point; if the three-dimensional initial seed point is in the crown, the position coordinates of the three-dimensional initial seed point are vertically projected, and it is discriminated as a foreground point or a background point according to the projection position. Furthermore, the method of determining whether the three-dimensional initial seed is located in the crown area or the root area further includes determining whether the seed point is in the root or crown area by using variance statistics of the tooth foreground area of ​​continuous slices; the method of determining whether it is a foreground point or a background point based on the projection position further includes that if the projection position is located in the foreground with the largest area in the slice, it is a foreground point, otherwise it is a background point. Furthermore, the adjustment of the three-dimensional initial seed point position in step 1 includes, according to the semantic adjustment specified during user interaction, if the user marks it as a foreground point and the semantic judgment is a background point, the seed point is moved to the centroid of the foreground; if the user marks it as a background point and the semantic judgment is a foreground point, the seed point is adjusted to the background area and there is a certain distance from the foreground boundary. Furthermore, the calculation in step 2 generates the intra-layer feature Fp and the inter-layer feature F d Further comprising, according to the foreground distance map P F The encoded K feature and V feature, the encoded Q feature of the image T, and the background distance map P B The encoded K feature and V feature are combined with the encoded Q feature of the image T to calculate the intra-layer pixel relationship to generate the intra-layer feature Fp, and the inter-layer pixel relationship to generate the inter-layer feature F d ; The image segmentation using 3DUNet further includes: F ), background enhancement distance map F(P B ) and the single tooth CBCT image T are used as the input of 3DUNet for image segmentation, and the output result is the initial segmentation result R0. Furthermore, the calculation of the intra-layer pixel relationship to generate the intra-layer feature Fp further includes: the foreground distance map P F The encoded K feature is inner-producted with the Q feature encoded by the image T, and after softmax normalization, it is weightedly summed with the V feature to obtain the foreground distance map P. F The intra-layer feature Fp; the background distance map P B The encoded K feature is subjected to an inner product operation with the Q feature encoded by the image T, and after softmax normalization, it is compared with the background distance map P. B The encoded V features are weighted and summed to obtain the background distance map P B Intra-layer features; The inter-layer pixel relationship is calculated to generate the inter-layer feature F d Further comprising, the foreground distance map P F The encoded K feature, the encoded Q feature of the image T, the background distance map P B After the encoded K feature is permuted by the matrix, the K feature generated by the matrix-converted foreground distance map and the background distance map is respectively inner-producted and softmax-normalized with the Q feature, and then weightedly summed with the V feature generated by the foreground distance map and the background distance map to obtain the inter-layer feature F. d . Furthermore, the step 3 of determining the evaluation network input and constructing the evaluation network further includes: the initial segmentation result R0, the foreground enhancement distance map F(PF), the background enhancement distance map F(PB), and the deepest feature F in the 3DUNet model s After unifying the size, they are connected by channels as evaluation input; the evaluation network is constructed through the convolution layer, the fourth layer structure of ResNet50, the average pooling layer, and the fully connected layer. Furthermore, the fusion of the intra-layer feature Fp and the inter-layer feature F in step 2 d Further including, intra-layer features Fp and inter-layer features F d Add them together to get F0, upsample F0 to get F1 with a scale of 1×D×H×W, and finally convert the foreground distance map P F and background distance map P B , and are connected to F1 by channel to generate the foreground enhancement distance map F(P F ), background enhancement distance map F(P B ), where D is the depth of the original image, H is the height of the original image, and W is the width of the original image. The effects of the present invention are as follows: 1. The interaction of this method is implemented on a two-dimensional panoramic image with a unified perspective. The panoramic image is a reconstructed two-dimensional image. As an object for direct user operation, it is limited to the user interaction stage. However, the object of feature enhancement is three-dimensional single-tooth data, which is extracted by applying it to the original data after intelligent recognition based on the user interaction results. This unified perspective can more comprehensively reflect the spatial position of three-dimensional objects without switching perspectives; at the same time, the seed point generation method based on morphological continuity proposed in this method can identify and correct user interaction points that may have errors, improve the effectiveness of the interaction results, and thus improve the robustness of user interaction. 2. This method proposes an adaptive optimization method, which sets different optimization ranges according to the different tooth locations of the seeds, replacing the traditional setting of a unified optimization range. 3. The interactive information input by this method is richer, including both foreground seed points and background seed points. The foreground enhanced distance map and the background enhanced distance map are taken as two independent sources and input into the segmentation network for processing. Compared with the traditional segmentation method that only takes the foreground seeds as the interactive source, this method ensures that the semantic priors of the seeds themselves are learned while fully learning the difference characteristics between the background and foreground. 4. This method adds feature pre-enhancement processing before segmentation. It takes the foreground distance map and the background distance map as the objects, extracts enhanced features, and forms an enhanced distance map with more significant interaction (including foreground and background). It pre-enhances the traditional distance map and then segmentation, which has a better global guidance effect than directly using the distance map for segmentation. At the same time, the foreground enhanced distance map and the background enhanced distance map will be reused as the input of the subsequent evaluation network. Compared with the traditional method that only uses the deep features of the segmentation network, the added enhanced distance map is directly derived from the user's specification, which has better supervision and therefore more accurate evaluation. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a logic flow chart of the interactive segmentation method of the present invention; Figure 2 It is a schematic diagram of the interactive processing of panoramic image self-identification; Figure 3 It is a single tooth segmentation process based on feature pre-enhancement; Figure 4 It is a feature pre-enhancement processing flow chart; Figure 5 It is a multi-level evaluation and optimization flow chart. DETAILED DESCRIPTION The technical solution of the present invention is specifically described below in conjunction with embodiments. This solution consists of three parts: 1) The interactive processing step of panoramic image self-identification, identifying and generating effective single-tooth interest regions and seed points; 2) The single-tooth segmentation step of feature pre-enhancement, obtaining the initial segmentation result; 3) The multi-level evaluation and optimization step, the segmentation result will be evaluated slice by slice to obtain the evaluation coefficient, and the pictures below the threshold will be selected and interacted again. After the interaction, the seed propagation range is divided into tooth partitions, and different ranges are applied. Multiple rounds of interaction can be performed until a satisfactory result is obtained as the final segmentation result. The three parts are executed in sequence, and the overall process is as follows Figure 1 shown. Step 1, self-identification interactive processing step of the panoramic image. Based on the three-dimensional image of the original cone beam CT image (CBCT), a two-dimensional panoramic image is reconstructed and generated, and the two-dimensional foreground seed point a of the tooth, the two-dimensional background seed point b, and the bounding box c containing the single tooth are marked according to user interaction; the three-dimensional initial seed point and the single tooth interest area are calculated, and the semantic discrimination of the three-dimensional initial seed point is performed, and it is determined whether to adjust the position of the initial seed point according to the discrimination result. A simpler and more intuitive method such as two-dimensional interaction is adopted, and an interactive mode combining points and bounding boxes is utilized. To ensure the objectivity of the interaction, the interaction in this scheme is not performed on the original image slices, but on the reconstructed image under a panoramic perspective, that is, the panoramic image. This interaction separates the original data into a single tooth region of interest, and obtains a number of foreground or background seed points. The processing has two key parts, namely 1) reconstruction of a two-dimensional enhanced panoramic image (corresponding to step 101); 2) a seed generation method based on morphological continuity (corresponding to steps 104-105). Step 1 The interaction processing is as follows Figure 2 As shown, it specifically includes the following steps: Step 101: A set of original cone beam CT images (CBCT) three-dimensional images are used as input. A set of images represents the complete jaw data of a patient, which is usually composed of 300-400 slices. The image is thresholded and segmented. After removing the background area, a two-dimensional panoramic image is reconstructed. The reconstruction formula is shown in formula (1). Where X is the target tooth image to be reconstructed, s is a point on the projected dental arch curve, Z is the Zth slice in the CT image, P(S,Z) represents the result of projecting the Zth slice according to the point S on the dental arch curve, r(s) is the coordinate of point S, n(s) is the normal vector of point S, (α,-α) is the projection range along the normal vector, and t is the projection variable. The present invention provides a unified panoramic viewing angle for the user to interact in step 101, so that the user does not need to switch between three standard viewing angles to determine whether the interaction position complies with his or her intention, thereby making the interaction more convenient. Step 102: The user interacts with the panoramic image P generated in step 101, marking the tooth 2D foreground seed point a, the 2D background seed point b, and the bounding box c containing the single tooth, as shown in FIG. Figure 2 shown. Step 103: Calculate the 3D initial seed point and the single tooth region of interest. The 2D foreground seed point a, the 2D background seed point b and the bounding box c are reversely projected according to formula (1) (general projection method) to form the 3D foreground initial seed point a1, the 3D background initial seed point b1 and the 3D bounding box c1; then the circumscribed cuboid c2 of c1 is calculated to form the 3D single tooth region of interest, i.e., the region of interest that covers a single tooth at the minimum. According to the c2 region, the original CBCT image is cropped to obtain the single tooth CBCT image T, which is composed of multiple slices of uniform size. Step 104: semantic discrimination of the three-dimensional initial seed point. The semantic discrimination is divided into two steps. First, it is determined whether the three-dimensional initial seed (including a1 and b1) is located in the crown area or the root area; then the seed semantics is identified. For seed position area discrimination, the root is first given as the starting point and the crown is given as the end point as the direction of the slice access order in the single-tooth CBCT image T. The area variance of the tooth block of each slice is counted. If the variance is greater than 50 and the slice number is greater than two-thirds of the total number of slices in the single-tooth area of ​​interest, the three-dimensional initial seed is judged to be located in the crown, otherwise it is judged to be in the root. Then identify the semantics of the 3D initial seed point. If the initial seed point is at the root of the tooth, as long as the seed position is within the foreground range of the tooth, it is identified as a foreground point, otherwise it is a background point. If the initial seed point is at the crown of the tooth, take the slice where the initial seed is located as the center, symmetrically take 5 adjacent slices, a total of 11 slices, and vertically project the position coordinates of the 3D seed point to the 11 slices. If there are more than 6 slices, and the projection position is located in the foreground with the largest area in the slice, it is a foreground point, otherwise it is a background point. Through the above method, the 3D initial seed (including a1 and b1) obtains the corresponding semantic discrimination results about the meaning of background or foreground. Step 105: Semantic adjustment of the three-dimensional initial seed point. If the semantic discrimination result of the three-dimensional initial seed point (including a1 and b1) is consistent with the semantics specified during user interaction, no adjustment will be made; if not, the position of the initial seed point will be adjusted. The adjustment methods are divided into the following cases: 1) If the semantic discrimination is a background point, the user specifies the initial seed adjustment as a foreground point. The initial seed position is reset to the center of mass position of the tooth foreground, and the seed semantics after adjustment is the foreground point. 2) If it is identified as a foreground point, the user specifies the seed adjustment as a background point. The seed position is reset to a position within the background range, at least 20 pixels away from the foreground edge, and the seed semantics after adjustment is the background point. The above steps are used to obtain the final foreground seed point a2 and background seed point b2. During the processing of step 1, the user interacts on the panoramic image without having to confirm the interaction position by switching the three perspectives of cross-section, coronal, and sagittal planes on the original slice, thereby ensuring that the user's operation view is unified and comprehensive without the need for additional switching of perspectives, greatly improving the convenience and efficiency of the interaction. At the same time, to ensure a clearer panorama, this solution does not reconstruct on the original image, but reconstructs on the tooth target image after threshold segmentation of the original data and removal of the background area. The present invention proposes a morphologically continuous seed generation method in steps 104-105 to confirm whether the seed point recognition semantics and the user mark semantics are consistent. If they are inconsistent, they are adjusted according to the centroid of the foreground area in accordance with the user semantics. This reduces the accuracy of user interaction, and even if the user interaction is wrong, it can be identified and corrected, thereby improving the robustness of user interaction. The seed point generation method based on morphological continuity is mainly used to identify and generate valid seed points. The two-dimensional interactive points in this application need to be converted into three-dimensional seed points. Through reverse projection using formula (1), a two-dimensional interactive point corresponds to the center point of the three-dimensional region. This type of direct mapping may lead to semantic errors in seed points due to spatial information compression and inaccurate user interaction. For example, the foreground point marked by the user may be the background after direct mapping, and vice versa. For this reason, a seed point generation method based on morphological continuity is proposed to identify the semantic attribution of the initial seed point. If the semantic attribution is wrong, it is adjusted. This method combines the differences between the root and crown of teeth for adaptive recognition, specifically including: 1) using the variance statistics of the tooth foreground area of ​​continuous slices to determine whether the seed point is in the root or crown area, if the variance is less than 50, it is the root, and if the variance is greater than 50, it is the crown; 2) if the root seed point is located in the foreground range, it is a foreground point, otherwise it is a background point; if the crown seed point is in the maximum foreground area, it is a foreground point, otherwise it is a background point; 3) according to the semantic adjustment specified during user interaction, if the user marks it as a foreground point and it is semantically judged as a background point, the seed is moved to the centroid of the foreground; if the user marks it as a background point and it is semantically judged as a foreground point, the seed point is adjusted to the background area and there is a certain distance from the foreground boundary. Step 2, the single tooth segmentation step of feature pre-enhancement, takes the foreground points and background points as two independent inputs to enhance the difference between the foreground and the background. This method mainly consists of three parts: according to the foreground points and the background points, the foreground exponential geodesic distance and the background exponential geodesic distance are calculated respectively; then, the enhanced features are extracted using the feature pre-enhancement module; finally, the single tooth segmentation is realized with the enhanced features as input. In this way, global image enhancement is realized in advance in the early stage of segmentation to prevent the problem that seed points are easily attenuated in the network. An enhanced distance map is generated before the segmentation network, and the background enhancement distance map and the foreground enhancement distance map are output as two independent streams of the feature pre-enhancement module. The specific process is as follows Figure 3 As shown, the specific implementation is as follows. Step 201: Calculate the distance map to obtain the foreground distance map P F and background distance map P B According to the exponential geodesic distance method (classical method), the foreground seed point a2 and the background seed point b2 are combined with the single tooth CBCT image T to generate the corresponding foreground distance map P F and background distance map P B The specific calculation formula is shown in formula (2) (3), where S s represents the final seed point set (including a2 and b2), i is a single voxel in the single tooth CBCT image T, and P i,jrepresents the set of all paths between pixels i and j, p is a feasible path, which is determined by the parameter n∈[0,1], u(n)=p(n)′p(n)′ is a unit vector tangent to the path direction, and EGD(i,Ss,T) represents the path from point i in image T to seed point S s The exponential geodesic distance result, D geo (i,j,T) represents the geodesic distance from point i to point j in image T. Step 202: Feature pre-enhancement processing, including two parallel processes, namely obtaining foreground enhancement features F F and background enhancement feature F B The two processing models have the same structure, the only difference is the input source, specifically obtaining the foreground enhancement feature F F , is the foreground distance map P F and a single tooth CBCT image T as input; and the background enhancement feature F B , is the background distance map P B And a single tooth CBCT image T is taken as input. Each parallel processing is as follows Figure 4 As shown, the input source is first encoded by an encoder (step 2021), and then the intra-layer features and inter-layer features are calculated respectively (step 2022, step 2023), and finally the intra-layer features and inter-layer features are fused (step 2024). The interactive prior information is diffused from local points to global data through the intra-layer and inter-layer. Step 221: Encode the input source. Enhance the feature F(P F ) extraction as an example, the encoding details are as follows. Input foreground distance map P F and a single tooth CBCT image T, both of which have the same scale of 1×D×H×W, are encoded using the Transformer model. After encoding, the image T is Q features; foreground distance map P F After encoding, the size is The K features and size are The V feature of the Transformer model; Q feature, K feature, and V feature are all standard parameter names in the Transformer model. The K feature and V feature have the same scale but different values. C is the number of image channels, D is the image depth, H is the image height, and W is the image width. They are 1 / 8 of the original image size D, H, and W respectively. Centered on a single tooth CBCT image, we search for similarities voxel by voxel in the foreground or background distance to learn the weights such as Figure 4 shown. Step 2022: Calculate the intra-layer pixel relationship and convert the K feature and Q feature into the intra-layer feature F pFirst, the inner product operation is performed on the K feature and the Q feature as shown in formula (4), then the softmax normalization process is used as shown in formula (5), and finally the weighted sum is performed as shown in formula (6) to obtain the intra-layer feature F p : Where M(i,j) represents the foreground distance map P F The relationship between position i on the original image feature map and position j on the original image feature map is then normalized by Softmax to unify the values ​​between 0 and 1, which is expressed as: The weighted sum of the weight value W(i,j) and the V feature is expressed as: Finally, the foreground distance map P is formed F The intra-layer feature Fp of the background distance map P B The calculation of the intra-layer feature Fp is completed by repeating steps 2021 to 2022. When repeating, only the input foreground distance map P F Change to background distance map P B That's it. Step 223: Calculate the inter-layer pixel relationship, that is, convert the K feature and Q feature into inter-layer features F d First, the K and Q features are changed from their original sizes (C, K or Q channels, depth, height, and width respectively), adjusted to This operation can be obtained by simple matrix permutation. After inner product operation, Softmax normalization and weighted summation as shown in formulas (4), (5) and (6), the foreground distance map P is obtained. F Inter-layer feature F d . Background distance map P B The inter-layer feature F d Extraction is completed by repeating steps 221 and 223, and when repeating, only the input foreground distance map P F , replaced by the background distance map P B That's it. Step 2024: Fusion of intra-layer features and inter-layer features. First, the intra-layer features F p and the inter-layer feature F d Add them together to get F0, then upsample F0 to get F1 with a scale of 1×D×H×W, and finally convert the foreground distance map P F and background distance map P B , and are connected with F1 by channel to obtain the foreground enhancement distance map F(P F ), background enhancement distance map F(PB ). The present invention uses feature pre-enhancement processing in step 202 to achieve interactive feature enhancement, specifically by obtaining highly similar parts of the original CBCT image and the distance map, using an addition method to enhance the feature values ​​on the distance map, providing more significant information for the segmentation input, and enhancing before segmentation to prevent the problem that the interactive information represented by the distance map is easily weakened in the deep network. Step 203: Single tooth segmentation. Foreground enhancement distance map F(P F ) and background enhancement distance map F(P B ) and the single tooth CBCT image T are used as the input of 3DUNet for image segmentation, and the output result is the initial segmentation result R0. Step 3, multi-level evaluation and optimization step. This step consists of three parts. First, the score of the initial segmentation result is evaluated (step 301); then the slices with scores below the threshold are screened, and the user interacts with such slices twice to generate secondary seed points (step 302); finally, the initial segmentation result is optimized based on the secondary seed points (step 303). The specific process is shown in 5. Step 301: Evaluating the initial segmentation result includes two aspects: determining the input of the evaluation network and constructing the evaluation network. First, determine the input source of the evaluation network, which consists of four parts: the initial segmentation result R0, the foreground enhancement distance map F(P F ), background enhancement distance map F(P B ), and the deepest feature F in the 3DUNet model s ; Then the sizes of the four parts are unified and connected by channel to form the input of the evaluation network A. The output of the network is the evaluation score Score of each slice, as shown in formula (7). Score=A(R0,F(P F ),F(P F ),F s ) (7) Construct the specific structure of the evaluation network Q. First, a 3×3 convolution layer is used, followed by a ResNet50 fourth layer structure, which consists of two 1×1 convolution layers and a 3×3 convolution layer repeated three times, followed by an average pooling layer, and finally two fully connected layers. The network adopts a lightweight model design and uses fewer network layers to reduce the model size. The present invention enriches the input source of the evaluation network in step 301. In addition to the deep features of the segmentation network, deep features in the feature pre-enhancement module are added. This type of information is generated from seed priors and is direct information after user and semantic recognition. It is more consistent with the real results and more supervisory, thereby improving the accuracy of the evaluation. Step 302: User secondary interaction. The score of each slice is Score(i)∈[0,1], where i is the slice number, N is the total number of slices in the initial segmentation result R0, and i∈[1,N]. The higher the score, the higher the slice segmentation quality, and vice versa. The threshold is set to 0.4, and slices with an evaluation score lower than 0.4 are regarded as secondary interaction slices, thus forming a secondary interaction slice set. The user browses the secondary interaction slice set, performs secondary interaction, and generates a number of secondary seeds X, which include background seeds and foreground seeds. Step 303: Adaptive optimization. This paper uses the IF-GC method proposed by MIDeepSeg. However, the IF-GC method calculates the exponential geodesic distance for the seed point. The calculation range of the geodesic distance is the entire single tooth region of interest. This unified propagation range is obviously not suitable for objects such as teeth with large differences in various parts, such as large crowns and small roots. For this reason, the present invention improves the seed propagation range of IF-GC and obtains an adaptive optimization method. The adaptive optimization is divided into three parts. First, the input source of the IF-GC method is determined: the secondary seed X, the initial segmentation result R0; then it is determined whether the secondary seed X is in the crown or root position (step 104). If the secondary seed is in the crown area, the distance calculation range is limited to 0.3 times the length, width and height of the image T with the seed point as the center. If the secondary seed is in the root area, the distance calculation range is set to 0.1 times the length, width and height of the image T with the seed point as the center; finally, the three input sources are processed by the IF-GC method to obtain the secondary optimization result R1. If the result after optimization is still not satisfactory, steps 301-303 may be repeated for multiple rounds until the segmentation result is adjusted to the best effect. The adaptive optimization of the present invention takes the seed point as the center and creates a rectangular area. If the seed is in the crown part, it is set to 0.3 times the length, width and height of the original image, and if the seed is in the root part, it is set to 0.1 times the length, width and height of the original image. Such a setting ensures the propagation range of seeds applied in different parts, and the placement has an adverse effect on the correct segmentation area far away from the seed point. Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A panoramic interactive tooth segmentation method based on feature enhancement, comprising: Step 1: Based on the three-dimensional image of the original cone beam CT image, a two-dimensional panoramic image is reconstructed, and the two-dimensional foreground seed point a of the tooth, the two-dimensional background seed point b, and the bounding box c containing the single tooth are marked according to user interaction; Calculate the 3D initial seed point and the single tooth interest region, perform semantic discrimination on the 3D initial seed point, determine whether to adjust the position of the 3D initial seed point according to the discrimination result, and generate the final 3D foreground seed point a2 and 3D background seed point b2; Step 2: Combine the three-dimensional foreground seed point a2 and the three-dimensional background seed point b2 with the single tooth cone beam CT image T according to the exponential geodesic distance method to generate the corresponding foreground distance map P F and background distance map P B ; The foreground distance map P F and the background distance map P B The single tooth cone beam CT image T is input into the Transformer model encoding to calculate the intra-layer feature Fp and inter-layer feature F d , fusion of the intra-layer feature Fp and the inter-layer feature F d , generate the foreground enhancement distance map F(P F ) and background enhancement distance map F(P B ), use 3DUNet to perform image segmentation and output the initial segmentation result R0 of a single tooth; Step 3: Multi-level evaluation and optimization: determine the evaluation network input and construct the evaluation network, evaluate the initial segmentation results, and generate an evaluation score for each slice; filter out slices with scores below the threshold, and the user performs secondary interaction on the slices with scores below the threshold to generate secondary seed points; finally, optimize the initial segmentation results based on the secondary seed points.

2. The panoramic interactive tooth segmentation method based on feature enhancement as claimed in claim 1, wherein the reconstruction in step 1 generates a two-dimensional panoramic image according to formula (1): Processing, where X is the tooth target image to be reconstructed, s is a point on the projected dental arch curve, Z is the Zth slice in the CT image, P(S,Z) represents the result of projecting the Zth slice according to the point S on the dental arch curve, r(s) is the coordinate of point S, n(s) is the normal vector of point S, (α,-α) is the projection range along the normal vector, and t is the projection variable.

3. The panoramic interactive tooth segmentation method based on feature enhancement as described in claim 2, wherein the calculation of the three-dimensional initial seed point and the single tooth region of interest in step 1 includes: inversely projecting the two-dimensional foreground seed point a, the two-dimensional background seed point b and the bounding box c according to formula (1) to generate a three-dimensional foreground initial seed point a1, a three-dimensional background initial seed point b1, and a three-dimensional bounding box c1; calculating the circumscribed cuboid c2 of the three-dimensional bounding box c1 to form a three-dimensional single tooth region of interest, and cropping the original cone beam CT image according to the c2 region to generate a single tooth cone beam CT image T composed of multiple slices of uniform size.

4. The panoramic interactive tooth segmentation method based on feature enhancement as claimed in claim 1, wherein the semantic discrimination of the three-dimensional initial seed point in step 1 comprises discriminating whether the three-dimensional initial seed point is located in the crown area or the root area; Then identify the seed semantics; The identifying seed semantics further includes, if the three-dimensional initial seed point is at the tooth root, as long as the seed position is within the tooth foreground range, it is identified as a foreground point, otherwise it is a background point; If the three-dimensional initial seed point is located at the tooth crown, the position coordinates of the three-dimensional initial seed point are vertically projected, and the point is determined to be a foreground point or a background point according to the projection position.

5. According to the feature-enhanced panoramic interactive tooth segmentation method of claim 4, the step of determining whether the three-dimensional initial seed is located in the crown area or the root area further includes determining whether the seed point is in the root or crown area by using variance statistics of the tooth foreground area of ​​continuous slices; the step of determining whether it is a foreground point or a background point based on the projection position further includes that if the projection position is located in the foreground with the largest area in the slice, it is a foreground point, otherwise it is a background point.

6. The panoramic interactive tooth segmentation method based on feature enhancement as described in claim 1, wherein the adjusting of the three-dimensional initial seed point position in step 1 includes, according to the semantic adjustment specified during user interaction, if the user marks it as a foreground point and the semantic judgment is a background point, the seed point is moved to the centroid of the foreground; if the user marks it as a background point and the semantic judgment is a foreground point, the seed point is adjusted to the background area and there is a certain distance from the foreground boundary.

7. The panoramic interactive tooth segmentation method based on feature enhancement as claimed in claim 1, wherein the calculation in step 2 generates the intra-layer feature Fp and the inter-layer feature F d Further comprising, according to the foreground distance map P F The encoded K feature and V feature, the encoded Q feature of the image T, and the background distance map P B The encoded K feature and V feature are combined with the encoded Q feature of the image T to calculate the intra-layer pixel relationship to generate the intra-layer feature Fp, and the inter-layer pixel relationship to generate the inter-layer feature F d ; The image segmentation using 3DUNet further includes: F ), background enhancement distance map F(P B ) and the single tooth CBCT image T are used as the input of 3DUNet for image segmentation, and the output result is the initial segmentation result R0.

8. The panoramic interactive tooth segmentation method based on feature enhancement as claimed in claim 7, The calculation of the intra-layer pixel relationship to generate the intra-layer feature Fp further includes the foreground distance map P F The encoded K feature is inner-producted with the Q feature encoded by the image T, and after softmax normalization, it is weightedly summed with the V feature to obtain the foreground distance map P. F The intra-layer feature Fp; the background distance map P B The encoded K feature is subjected to an inner product operation with the Q feature encoded by the image T, and after softmax normalization, it is compared with the background distance map P. B The encoded V features are weighted and summed to obtain the background distance map P B Intra-layer features; The inter-layer pixel relationship is calculated to generate the inter-layer feature F d Further comprising, the foreground distance map P F The encoded K feature, the encoded Q feature of the image T, the background distance map P B After the encoded K feature is permuted by the matrix, the K feature generated by the matrix-converted foreground distance map and the background distance map is respectively inner-producted and softmax-normalized with the Q feature, and then weightedly summed with the V feature generated by the foreground distance map and the background distance map to obtain the inter-layer feature F. d .

9. The panoramic interactive tooth segmentation method based on feature enhancement as claimed in claim 1, wherein the step 3 of determining the evaluation network input and constructing the evaluation network further comprises: integrating the initial segmentation result R0, the foreground enhancement distance map F(PF), the background enhancement distance map F(PB), and the deepest feature map F in the 3DUNet model s After unifying the size, they are connected by channels as evaluation input; the evaluation network is constructed through the convolution layer, the fourth layer structure of ResNet50, the average pooling layer, and the fully connected layer.

10. The panoramic interactive tooth segmentation method based on feature enhancement according to claim 1, wherein in step 2, the intra-layer feature Fp and the inter-layer feature F are fused. d Further including, intra-layer features Fp and inter-layer features F d Add them together to get F0, upsample F0 to get F1 with a scale of 1×D×H×W, and finally convert the foreground distance map P F and background distance map P B , and are connected to F1 by channel to generate the foreground enhancement distance map F(P F ), background enhancement distance map F(P B ), where D is the depth of the original image, H is the height of the original image, and W is the width of the original image.

Images