Supernumerary tooth automatic detection method, device, storage medium and electronic device

Abstract

The present application discloses a method, device, storage medium and electronic device for automatic detection of supernumerary teeth. The method comprises: acquiring a three-dimensional image, preprocessing the three-dimensional image to obtain multiple image blocks; inputting the image blocks into a trained two-stage deep attention network for identification, and obtaining a probability value of supernumerary teeth contained in the three-dimensional image; wherein the two-stage deep attention network comprises a positioning subnetwork and a classification subnetwork; inputting the image blocks into the trained two-stage deep attention network for identification, including: inputting the image blocks into the positioning subnetwork to obtain a rectangular positioning frame, and inputting the rectangular positioning frame into the classification subnetwork to obtain a probability value of supernumerary teeth contained in the image blocks. The present application can speed up the detection speed of supernumerary teeth and improve the classification accuracy.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a method, device, storage medium and electronic device for automatically detecting supernumerary teeth. Background Art

[0002] Teeth that exceed the normal number of teeth are called supernumerary teeth. They are a common tooth development abnormality, which is more common in the maxillary anterior teeth. Their morphologies include conical, tuberous, supplementary, and odontoma. Supernumerary teeth often cause problems such as abnormal eruption of adjacent teeth and root resorption. Supernumerary teeth in the anterior teeth often cause crowded dentition or abnormal tooth spaces, and severe misalignment and deformity may also lead to caries, periodontal disease, and trauma. Therefore, the diagnosis and treatment of supernumerary teeth needs to be based on early clinical and imaging examinations, and supernumerary tooth positioning is the key to assisting the treatment of supernumerary teeth.

[0003] In recent years, deep learning systems based on convolutional neural networks have been introduced into the field of oral and maxillofacial diagnostic imaging, and have great potential in the application research of supernumerary tooth detection and diagnosis. For example, the method of detecting supernumerary teeth in the maxillary anterior area in curved surface tomographic images using several commonly used deep learning models has achieved high classification accuracy, but this method is limited to the permanent dentition; to compensate for this limitation, the method of using SqueezeNet, ResNet-18, ResNet-101 and Inception-ResNet-V2 models to classify supernumerary teeth in mixed dentition with high accuracy can help clinicians with insufficient clinical experience to make more accurate and faster diagnoses, but this method also has the disadvantage that the maxillary anterior area needs to be manually labeled; in the method of locating the area of ​​interest where supernumerary teeth may exist by using the DeeplabV3plus model, supernumerary teeth are classified by the Inception-resnet-v2 model, realizing a fully automatic supernumerary tooth detection process. The above detection methods are all based on two-dimensional curved surface tomographic images, which may not provide sufficient information for further diagnosis and treatment of some complex anatomical structures or lesions. Summary of the invention

[0004] The embodiments of the present application provide a method, device, storage medium and electronic device for automatic detection of supernumerary teeth, which provide rich 3D spatial information through three-dimensional images, avoid the influence of spatial overlap of oral anatomical structures in two-dimensional curved surface layer images on recognition, can provide more accurate three-dimensional spatial position, and accelerate the detection speed of supernumerary teeth and improve classification accuracy.

[0005] The present application embodiment provides a method for automatically detecting supernumerary teeth, comprising:

[0006] Acquire a three-dimensional image, and preprocess the three-dimensional image to obtain a plurality of image blocks;

[0007] Inputting the image block into a trained two-stage deep attention network for recognition, and obtaining a probability value of containing supernumerary teeth in the three-dimensional image;

[0008] Among them, the two-stage deep attention network includes a positioning subnetwork and a classification subnetwork; the inputting of the image block into the trained two-stage deep attention network for recognition includes: inputting the image block into the positioning subnetwork to obtain a rectangular positioning box, and inputting the rectangular positioning box into the classification subnetwork to obtain the probability value of the image block containing supernumerary teeth.

[0009] Furthermore, in the above-mentioned supernumerary tooth automatic detection method, the positioning sub-network includes an encoder, and the encoder includes a pooling layer, a CGR module, and a plurality of feature extraction branches;

[0010] The step of inputting the image block into the positioning subnetwork to obtain the rectangular positioning frame includes:

[0011] Inputting the image block into two CGR modules to obtain a low-level semantic feature map;

[0012] Inputting the low-level semantic feature map into the first feature extraction branch and the second feature extraction branch to obtain a mid-level semantic feature map;

[0013] Inputting the middle-level semantic feature map into the first feature extraction branch and the second feature extraction branch to obtain a first high-level semantic feature map;

[0014] The first high-level semantic feature map is input into the pooling layer and a CGR module to obtain a low-resolution high-level semantic feature map.

[0015] Furthermore, in the above-mentioned supernumerary tooth automatic detection method, the positioning sub-network further includes a decoder, and the decoder includes an upsampling layer, a CGR module and a three-dimensional convolution layer;

[0016] The step of inputting the image block into the positioning subnetwork to obtain the rectangular positioning frame further includes:

[0017] The low-resolution high-level semantic feature map is passed through a CGR module and an upsampling layer, and then concatenated with the first high-level semantic feature map to obtain a small-size fused feature map;

[0018] The small-size fused feature map is passed through two CGR modules and an upsampling layer, and then concatenated with the middle-layer semantic feature map to obtain a middle-size fused feature map;

[0019] The medium-sized fused feature map is passed through two CGR modules and an upsampling layer, and then concatenated with the low-resolution high-level semantic feature map to obtain a large-sized fused feature map;

[0020] The large-size fused feature map is input into two CGR modules and a three-dimensional convolutional layer to obtain a segmentation result, and the rectangular positioning frame is obtained based on the segmentation result.

[0021] Furthermore, in the above-mentioned supernumerary tooth automatic detection method, the step of obtaining the rectangular positioning frame based on the segmentation result includes:

[0022] Obtaining a probability value of each pixel point in the segmentation result belonging to the maxillary anterior teeth area, and when the probability value is greater than a preset probability threshold, taking the corresponding pixel point as a candidate point of the region of interest;

[0023] Combining adjacent candidate points of the region of interest to obtain multiple three-dimensional connected domains;

[0024] The volume and circumscribed rectangular frame of each connected domain are obtained through the connected domain labeling algorithm;

[0025] The circumscribed rectangular frame is used as a candidate detection frame, and connected domains whose volumes are smaller than a preset volume threshold are screened out to obtain a target candidate detection frame;

[0026] Merge the circumscribed rectangular frame of the maximum connected domain and the candidate detection frames of the neighbors to obtain a rectangular positioning frame;

[0027] The coordinate values ​​of the rectangular positioning frame in the image block are calculated, the coordinate values ​​are used as the boundary range coordinate values, and the distance between the center point of the rectangular positioning frame and the fixed point position is calculated based on the boundary range coordinate values.

[0028] Furthermore, in the above-mentioned supernumerary tooth automatic detection method, the classification subnetwork includes a CBR module, a maximum pooling layer, a plurality of residual modules, an average pooling layer and a fully connected layer;

[0029] The step of inputting the rectangular positioning frame into the classification subnetwork to obtain a probability value of the image block containing supernumerary teeth includes:

[0030] Inputting the rectangular positioning frame into a CBR module, a maximum pooling layer and four residual modules to obtain a high-order feature map;

[0031] The high-order feature map is input into the average pooling layer and the fully connected layer to obtain the probability value of the image block containing supernumerary teeth.

[0032] Furthermore, in the above-mentioned supernumerary tooth automatic detection method, the training process of the two-stage deep attention network includes:

[0033] Acquire a three-dimensional image, obtain a position mark of a maxillary anterior teeth region in the three-dimensional image and information on the type of supernumerary teeth in the maxillary anterior teeth region, and form an image data set;

[0034] Preprocessing the images in the image data set by a three-dimensional sliding block algorithm to obtain a plurality of image blocks;

[0035] Inputting the preprocessed image data set into the positioning subnetwork to obtain a rectangular positioning frame, and training based on the position annotation of the maxillary anterior teeth area, the cross entropy loss function of the segmentation result, and the mixed loss function of the distance between the center point and the vertex position of the rectangular positioning frame;

[0036] The rectangular positioning frame is input into the classification subnetwork to obtain a probability value of containing supernumerary teeth, and training is performed based on supernumerary tooth category information in the maxillary anterior teeth area and a cross entropy loss function.

[0037] Furthermore, in the above-mentioned supernumerary teeth automatic detection method, the preprocessing of the images in the image data set by using a three-dimensional sliding block algorithm comprises:

[0038] For the image dataset, the size is D × W × H For a 3D image, set a size of d × w × h The sliding window has a moving step size of ( sd,sw,sh ), generate the slice range corresponding to each image block in the complete image according to the window and step size:

[0039] The number of slices in the Z direction is

[0040]

[0041] The number of slices in the Y direction is:

[0042]

[0043] The number of slices in the X direction is:

[0044]

[0045] For Slices , calculate the coordinate range of each slice:

[0046]

[0047] Among them, the integer .

[0048] The present application also provides an automatic detection device for supernumerary teeth, comprising:

[0049] A preprocessing module, used for acquiring a three-dimensional image, and preprocessing the three-dimensional image to obtain a plurality of image blocks;

[0050] A prediction module, used for inputting the image block into a trained two-stage deep attention network for recognition, and obtaining a probability value of the three-dimensional image containing supernumerary teeth;

[0051] Among them, the two-stage deep attention network includes a positioning subnetwork and a classification subnetwork; the inputting of the image block into the trained two-stage deep attention network for recognition includes: inputting the image block into the positioning subnetwork to obtain a rectangular positioning box, and inputting the rectangular positioning box into the classification subnetwork to obtain the probability value of the image block containing supernumerary teeth.

[0052] An embodiment of the present application also provides a computer-readable storage medium, in which a plurality of instructions are stored. The instructions are suitable for being loaded by a processor to execute any of the above-mentioned methods for automatically detecting supernumerary teeth.

[0053] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in any of the above-mentioned methods for automatically detecting supernumerary teeth.

[0054] The present application provides a method, device, storage medium and electronic device for automatic detection of supernumerary teeth. The present application obtains the probability value of supernumerary teeth by detecting a three-dimensional image, provides rich 3D spatial information through the three-dimensional image, avoids the influence of spatial overlap of oral anatomical structures in two-dimensional curved body slice images on recognition, and can provide a more accurate three-dimensional spatial position. The two-stage deep attention network of the present invention includes a positioning subnetwork and a classification subnetwork. The positioning subnetwork is used to segment the rectangular positioning frame, and the classification subnetwork is used to identify and calculate the probability value of supernumerary teeth. The positioning subnetwork is based on the segmentation network framework. Compared with the traditional detection network, it does not need to process a large number of anchor points and anchor frames, which speeds up the detection reasoning speed. The classification subnetwork integrates the original input information features, optimizes the feature expression efficiency, enhances the generalization ability of the network, and can improve the classification accuracy. The present application obtains the probability value of supernumerary teeth in the image by identifying the three-dimensional image, which allows users to read the image and the probability value recognized by the image more intuitively, and can assist users in making accurate judgments on supernumerary teeth. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0056] Figure 1 This is a flow chart of the method for automatically detecting supernumerary teeth provided in an embodiment of the present application.

[0057] Figure 2 Another flow chart of the method for automatically detecting supernumerary teeth provided in an embodiment of the present application.

[0058] Figure 3 A schematic diagram of the structure of a two-stage deep attention network provided in an embodiment of the present application.

[0059] Figure 4 A schematic diagram of a process for obtaining a rectangular positioning frame provided in an embodiment of the present application.

[0060] Figure 5 A schematic diagram of a rectangular positioning frame provided in an embodiment of the present application.

[0061] Figure 6 A schematic diagram of the structure of the automatic detection device for supernumerary teeth provided in an embodiment of the present application.

[0062] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0064] The embodiment of the present application provides a method, device, storage medium and electronic device for automatic detection of supernumerary teeth. The embodiment of the present application provides a supernumerary teeth automatic detection device, which can be integrated in an electronic device, which can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0065] See also Figure 1 and Figure 2 , Figure 1 This is a flow chart of the method for automatically detecting supernumerary teeth provided in an embodiment of the present application. Figure 2 Another flow chart of the supernumerary tooth automatic detection method provided in an embodiment of the present application, which is applied to an electronic device, the supernumerary tooth automatic detection method comprises the following steps:

[0066] S1, obtaining a three-dimensional image, and preprocessing the three-dimensional image to obtain a plurality of image blocks.

[0067] Specifically, the three-dimensional image is a three-dimensional cone beam CT image.

[0068] S2, input the image block into the trained two-stage deep attention network for recognition, and obtain the probability value of the supernumerary teeth in the three-dimensional image;

[0069] Among them, the two-stage deep attention network includes a positioning subnetwork and a classification subnetwork; the image block is input into the trained two-stage deep attention network for recognition, including: inputting the image block into the positioning subnetwork to obtain a rectangular positioning box of the image block, and inputting the rectangular positioning box into the classification subnetwork to obtain the probability value of the image block containing supernumerary teeth.

[0070] In one embodiment, Figure 3 The schematic diagram of the structure of the two-stage deep attention network provided in the embodiment of the present application can be found in Figure 3 The positioning sub-network includes an encoder and a decoder. In step S2, the step of inputting the image block into the positioning sub-network to obtain a rectangular positioning frame of the image block specifically includes:

[0071] S211, input the image block into two CGR modules to obtain a low-level semantic feature map;

[0072] S212, inputting the low-level semantic feature map into the first feature extraction branch and the second feature extraction branch to obtain a mid-level semantic feature map;

[0073] S213, inputting the middle-level semantic feature map into the first feature extraction branch and the second feature extraction branch to obtain a first high-level semantic feature map;

[0074] S214, input the first high-level semantic feature map into a maximum pooling layer and a CGR module to obtain a low-resolution high-level semantic feature map.

[0075] Specifically, the encoder first receives an image block of a three-dimensional image as input, and then passes through two three-dimensional convolution, group normalization, and activation function combination module layers, namely, two CGR (Convolution+GroupNorm+ReLU) modules, to output a low-level semantic feature map with the same resolution as the input three-dimensional image. Then, the low-level semantic feature map is input into two feature extraction branches, specifically: the low-level semantic feature map is input into the first feature extraction branch and then jump-connected with the feature map at the same resolution of the decoder, and then the feature map after the jump connection is input into the second feature extraction branch, wherein the second feature extraction branch includes a maximum pooling layer and two CGR modules, and a middle-level semantic feature map with the resolution reduced to half of the original is obtained. The operations of the above two branches are repeated for the middle-level semantic feature map in sequence, and a high-level semantic feature map is output. The high-level semantic feature map passes through the maximum pooling layer and a CGR module again to obtain a low-resolution high-level semantic feature map.

[0076] Furthermore, the step of inputting the image block into the positioning subnetwork to obtain a rectangular positioning frame of the image block in step S2 further includes:

[0077] S215, passing the low-resolution high-level semantic feature map through a CGR module and an upsampling layer, and then splicing it with the first high-level semantic feature map to obtain a small-size fused feature map;

[0078] S216, the small-size fusion feature map is passed through two CGR modules and an upsampling layer, and then concatenated with the middle-layer semantic feature map to obtain a medium-size fusion feature map;

[0079] S217, concatenating the medium-sized fused feature map with the low-resolution high-level semantic feature map through two CGR modules and an upsampling layer to obtain a large-sized fused feature map;

[0080] S218, input the large-size fusion feature map into two CGR modules and a three-dimensional convolution layer to obtain a segmentation result, and obtain a rectangular positioning frame based on the segmentation result.

[0081] Specifically, the low-resolution high-level semantic feature map output by the encoder is first used as the input of the decoder. After passing through a CGR module and an upsampling layer to restore the resolution to the same as that of the high-level semantic feature map, the high-level semantic feature map from the jump connection branch of the encoder is received, and the two are spliced ​​by channel to increase the dimension of the feature to obtain a small-sized fused feature map. The small-sized fused feature map is then passed through two CGR modules and an upsampling layer again, and spliced ​​with the middle-level semantic feature map to output a medium-sized fused feature map. The above operations are repeated for the medium-sized fused feature map in sequence to output a large-sized fused feature map. The large-sized fused feature map passes through two CGR modules and then a 1×1×1 three-dimensional convolutional layer to obtain a segmentation result consistent with the size of the input three-dimensional image. Each value of the result represents the probability value of the pixel point at the corresponding position belonging to the maxillary anterior tooth area.

[0082] Further, step S218 may include the following steps:

[0083] S2181, obtaining a probability value of each pixel point in the segmentation result belonging to the maxillary anterior teeth area, and when the probability value is greater than a preset probability threshold, taking the corresponding pixel point as a candidate point of the region of interest;

[0084] S2182, combining adjacent candidate points of the region of interest to obtain multiple three-dimensional connected domains;

[0085] S2183, obtaining the volume and circumscribed rectangular frame of each connected domain through a connected domain labeling algorithm;

[0086] S2184, taking the circumscribed rectangular frame as a candidate detection frame, filtering out connected domains whose volumes are smaller than a preset volume threshold, and obtaining a target candidate detection frame;

[0087] S2185, merging the circumscribed rectangular frame of the maximum connected domain and the candidate detection frames of the nearest neighbors to obtain a rectangular positioning frame;

[0088] S2186, calculating the coordinate values ​​of the rectangular positioning frame in the image block, and using the coordinate values ​​as the boundary range coordinate values;

[0089] S2187, calculating the distance between the center point and the vertex position of the rectangular positioning frame based on the boundary range coordinate values.

[0090] Specifically, the voxel points at the corresponding positions of the 3D cone beam CT image are obtained through the positioning sub-network p n The probability value of belonging to the maxillary anterior area P n , when the probability value P n Greater than the preset probability threshold PWhen , it is regarded as the foreground, that is, the candidate point of the region of interest, and the adjacent candidate points are combined to form m A three-dimensional connected domain { A1, A2...Am}, use the connected domain labeling algorithm to obtain the volume and circumscribed rectangular frame of each connected domain as a candidate detection frame, filter out the connected domains whose volume is less than the preset volume threshold, and obtain a set of candidate detection frames that meet the requirements { Box1, Box2...Boxm}, then the nearest non-connected domain is merged, based on the maximum connected domain Amax The bounding rectangle of Boxn , the candidate detection box of the nearest neighbor { Boxi...Boxj}and Boxn Merge to get the final predicted rectangular positioning box DetBox , its position coordinates are ( xmin, xmax, ymin, ymax, zmin, zmax ). Finally, according to the scaling ratio Scale calculate DetBox The coordinate values ​​in the original resolution image ( xmin, xmax, ymin, ymax, zmin,zmax ) as the boundary range coordinate value of the rectangular positioning box, and further calculate the center point and vertex coordinates of the rectangular positioning box.

[0091] In one embodiment, Figure 4 The schematic diagram of the process of obtaining the rectangular positioning frame provided in the embodiment of the present application can be referred to Figure 4 , the following combination Figure 4 Describe the specific steps of the nearest neighbor non-connected domain merging algorithm:

[0092] For the maximum connected domain Amax The bounding rectangle of Boxn , whose coordinates are ( xmin n , xmax n , ymin n , ymax n ， zmin n , zmax n ), for the detection boxes of other connected domains { Box1, Box2...Boxm} in a box Boxi , whose coordinates are ( xmin i , xmax i , ymin i , ymax i , zmin i , zmax i ),calculate Boxn and Boxi The minimum distance in different directions in three-dimensional space is as follows:

[0093]

[0094] Similarly:

[0095]

[0096]

[0097] dx i , dy i , dz i for X , Y , Z The minimum distance in the direction, take the maximum value dmax :

[0098]

[0099] When the maximum value does not exceed the preset distance threshold dt When Boxi Put it into the set of neighbor detection boxes, repeat the above operation, and get a series of neighbor detection boxes { Boxi...Boxj}, the neighbor detection box and Boxn Merge into a rectangular positioning box DetBox , where the coordinates of the merged detection box are calculated as follows, Boxk ϵ{ Boxi...Boxj}:

[0100]

[0101]

[0102] By the same process, we get DetBox The coordinate value of ymin , ymax , zmin and zmax .

[0103] See also Figure 3 The classification subnetwork includes a CBR module, a maximum pooling layer, multiple residual modules, an average pooling layer and a fully connected layer. The step of inputting the rectangular positioning frame into the classification subnetwork to obtain the probability value of the image block containing supernumerary teeth in step S2 specifically includes:

[0104] S221, inputting the rectangular positioning frame into a CBR module, a maximum pooling layer and four residual modules to obtain a high-order feature map;

[0105] Among them, the feature extraction part contains 4 residual modules, each of which is composed of 2 identical or similar basic residual modules.

[0106] The first residual module consists of two basic residual modules based on the attention mechanism with the same structure. The basic residual module includes a CBR module, a three-dimensional convolution, a batch normalization, and an SE (Squeeze-and-Excitation) attention module connected in sequence. The rectangular positioning box is sequentially passed through the CBR module, the three-dimensional convolution, and the batch normalization to obtain the first feature map. The first feature is passed through the SE (Squeeze-and-Excitation) attention module to obtain the second feature map. The first feature map and the second feature map are added channel by channel, and then after the activation function, the feature-enhanced fusion feature map is obtained as the output of the first basic residual module; the feature-enhanced fusion feature map is input again into the second basic residual module with the same structure as above to obtain the first fusion feature.

[0107] The second residual module is composed of two basic residual modules with the same structure. The difference from the first residual module is that there is no attention module. In the second residual module, the first fused feature is passed through two CBR modules in sequence to obtain a first intermediate feature map, the first intermediate feature map is fused with the first fused feature map, and the fused feature map is repeatedly operated to obtain a second fused feature map.

[0108] The third residual module is composed of two basic residual modules with similar structures. The two modules differ in whether downsampling is performed, that is, the second fused feature map is sequentially passed through two CBR modules to obtain a second intermediate feature map, and then the second fused feature map is input into the maximum pooling layer to obtain a third intermediate feature map, and the second intermediate feature map and the third intermediate feature map are spliced ​​to obtain a third fused feature map. The third fused feature map is sequentially passed through two CBR modules to obtain a third intermediate feature map, and the third intermediate feature map and the third fused feature map are spliced ​​to obtain a fourth fused feature map.

[0109] The structure of the fourth residual module is consistent with that of the third residual module. After receiving the fourth fusion feature map, the fourth residual module passes through a downsampling basic residual module (containing a maximum pooling layer) and a non-downsampling basic residual module (excluding a maximum pooling layer) in sequence to obtain a high-order feature map of the segmented image block.

[0110] In the attention module, a global average pooling operation is applied to the feature map with an input size of D×H×W×C to obtain a compressed feature map with a size of 1×1×1×C. The compressed feature map passes through two fully connected layers to obtain a weight coefficient with a size of 1×1×1×C. The weight coefficient is used to reflect the importance of C different channels. The weight coefficient is then multiplied by the attention module input feature map channel by channel to obtain a feature map with enhanced channel features.

[0111] S222, input the high-order feature map into the average pooling layer and the fully connected layer to obtain the probability value of the image block containing supernumerary teeth.

[0112] Specifically, the classification subnetwork consists of a feature extraction part and a classification prediction part. The feature extraction part is responsible for extracting the features of the maxillary anterior teeth area, receiving a rectangular positioning frame as input, and sequentially passing through a three-dimensional convolution, batch normalization, and ReLU activation function combination module, namely, a CBR combination module, and then passing through a maximum pooling layer to obtain a feature map with a resolution reduced to half of the input image. The downsampled feature map is then passed through four residual modules in sequence to output a high-order feature map of the maxillary anterior teeth area image block. The classification prediction part uses the high-order feature map obtained by the feature extraction part as input, and sequentially passes through average pooling and fully connected layers, and finally outputs the confidence that the maxillary anterior teeth area contains central supernumerary teeth, that is, the probability value of predicting the presence or absence of supernumerary teeth.

[0113] Figure 5 A schematic diagram of a rectangular positioning frame provided in an embodiment of the present application, such as Figure 5 As shown in the figure, each column from top to bottom is the transverse section, sagittal plane and coronal plane of the same three-dimensional image. Figure 5 The rectangular box in the figure is the maxillary anterior teeth area where the centric supernumerary teeth may be detected in the three-dimensional cone beam CT image by using the algorithm of the present invention. The mark "nst" above the rectangular box represents the absence of centric supernumerary teeth, and the mark "st" represents the presence of centric supernumerary teeth. The numbers represent the confidence level of the absence or presence of supernumerary teeth.

[0114] The above is the application process of the two-stage deep attention network. The following introduces the training process of the two-stage deep attention network:

[0115] SA1, collect three-dimensional images, obtain the position annotation of the maxillary anterior teeth area and the supernumerary tooth category information of the maxillary anterior teeth area in the three-dimensional image, and form an image data set.

[0116] The position marking method is as follows: the maxillary anterior teeth area is used as the region of interest (ROI) during marking, and a three-dimensional rectangular frame is used to mark it; the center point coordinates of the ROI in the three-dimensional image are extracted ( x, y, z ) and the corresponding width and height ( w, h, d ) as the location annotation information of the maxillary anterior teeth area required for training.

[0117] The category labeling method is: using three-dimensional images, the tooth position and morphology in the maxillary anterior area are analyzed from the cross-section and three-dimensional space model, and the situation of whether there are centricular supernumerary teeth in the maxillary anterior area is used as the classification labeling information required for training. If there are centricular supernumerary teeth in the selected area of ​​interest, it is marked as 1, otherwise it is marked as 0.

[0118] SA2, preprocesses the images in the image data set through a three-dimensional sliding block algorithm to obtain multiple image blocks.

[0119] Before the 3D image is input into the localization subnetwork, a segmentation mask of the same size as the 3D image is generated, that is, the pixel value of the ROI area is 1 and that of the non-ROI area is 0. Then, a 3D sliding block algorithm is adopted to divide the cone beam CT image and the segmentation mask into multiple image blocks and mask blocks with partial overlap using a 3D sliding window. These blocks are input into the network respectively. At the same time, according to the input mask block, the 3D coordinates of the center point and 8 vertices of the ROI area in the block are obtained for the subsequent calculation of the block mixing loss.

[0120] Specifically, for an image dataset with size D × W × H For a 3D image, set a size of d × w × h The sliding window moves in steps of ( sd,sw,sh ), generate the slice range corresponding to each image block in the complete image according to the window and step size:

[0121] The number of slices in the Z direction is

[0122]

[0123] The number of slices in the Y direction is:

[0124]

[0125] The number of slices in the X direction is:

[0126]

[0127] For Slices , calculate the coordinate range of each slice:

[0128]

[0129] Among them, the integer .

[0130] According to the calculated slice range, the image blocks of the corresponding range are taken out from the three-dimensional image and input into the positioning sub-network.

[0131] SA3, the preprocessed image dataset is input into the positioning sub-network to obtain a rectangular positioning box, which is trained based on the position annotation of the maxillary anterior area, the cross entropy loss function of the segmentation result, and the mixed loss function of the distance between the center point and the vertex position of the rectangular positioning box.

[0132] Total loss function of the localization sub-network The expression is:

[0133]

[0134] In the formula, is the binary cross entropy loss of the predicted segmentation result of the ROI region in the image block obtained by three-dimensional sliding block, is the segmentation result of the ROI area in the image block, and the distance loss between the center point of the rectangular positioning box and the vertex coordinate position obtained by merging the nearest non-connected domains. is a hyperparameter to balance these two losses.

[0135] It can be expressed by the following formula:

[0136]

[0137] in:

[0138]

[0139] in, Represents the total number of voxel points in the three-dimensional image, represents the true label of each voxel point, Represents the probability value of each voxel point predicted by the localization sub-network to belong to the maxillary anterior area.

[0140] It can be expressed by the following formula:

[0141]

[0142] in, and Represent the set of center points and vertices of the predicted rectangular positioning box and the real ROI area in the image block, respectively. Represents the set of center points and vertices of a 3D image block, represent middle Points and patches The Euclidean distance between points, Represents the true annotation corresponding to the above predicted point middle Points and patches Euclidean distance between points.

[0143] The vertex coordinates and center point of the rectangular positioning box in the image block are obtained through the nearest neighbor non-connected domain merging algorithm, and the loss function is calculated. During the continuous iterative training process, when the loss function value tends to converge, the training is completed.

[0144] SA4, input the rectangular positioning box into the classification sub-network to obtain the probability value of containing supernumerary teeth, and train it based on the supernumerary tooth category information in the maxillary anterior area and the cross entropy loss function.

[0145] The loss function of the classification subnetwork adopts cross entropy loss, which can be expressed by the following formula:

[0146]

[0147] in, represents the number of maxillary anterior teeth area image blocks obtained by clipping the rectangular maxillary anterior teeth area position coordinates; Indicates the number of sample categories. In this embodiment =2, that is, divided into two categories: no supernumerary teeth and supernumerary teeth; Indicates i The true value of the probability of whether the image block contains a central supernumerary tooth; Represents the model for i The probability prediction value of whether an image block contains a central supernumerary tooth. During the continuous iterative training process, when the loss function value tends to converge, the training is completed.

[0148] Here is an example of a two-stage deep attention network training:

[0149] (1) The dataset was divided into a training set and a validation set. The training set and validation set were set to 3D cone-beam CT images of 194 and 55 research cases, respectively.

[0150] (2) The three-dimensional cone beam CT images of the study cases were annotated. The maxillary anterior teeth area was used as the region of interest (ROI) and marked with a three-dimensional rectangular frame. The center point coordinates of the ROI in the three-dimensional cone beam CT images were extracted ( x, y, z ) and the corresponding width and height ( w, h, d) as the regional position annotation information of the maxillary anterior teeth area required for training; the 3D cone beam CT images of the research cases were labeled by category, the tooth position and morphology in the maxillary anterior teeth area were analyzed based on the cross-section of the 3D cone beam CT images and the 3D spatial model, and the situation of whether there were central supernumerary teeth in the maxillary anterior teeth area was used as the classification annotation information required for training. If there were central supernumerary teeth in the selected area of ​​interest, it was labeled as 1, otherwise it was labeled as 0; 137 samples with supernumerary teeth in the training set were labeled, namely positive samples, and 57 samples without supernumerary teeth were labeled, namely negative samples.

[0151] (3) Before the 3D cone-beam CT image is input into the 3D U-shaped localization subnetwork, a segmentation mask of the same size as the image is generated, that is, the pixel value of the ROI area is 1, and the pixel value of the non-ROI area is 0. For a 3D cone-beam CT image of size 305×305×268, a 3D sliding block algorithm is used. A 3D sliding window of size 170×170×80 is used with a step size of 40×40×20 to divide the cone-beam CT image and the segmentation mask into multiple image blocks and mask blocks with partial overlap. These blocks are input into the network respectively. At the same time, according to the input mask block, the 3D coordinates of the center point and 8 vertices of the ROI area in the block are obtained. According to the actual position of the annotation box, that is, the above-mentioned ROI area, the complete 3D cone-beam image is cropped to obtain the segmentation result containing only the maxillary anterior area, which is used as the training set and input into the classification subnetwork.

[0152] (4) Set the initial values ​​of the hyperparameters of the positioning subnetwork model. The initial learning rate is set to 0.0001, the number of training rounds is set to 80, the Adam optimizer is used, and the hyperparameters of the balanced coordinate distance loss in the loss function are Set to 0.01; set the initial values ​​of the hyperparameters of the three-dimensional classification sub-network model based on the attention mechanism, set the initial learning rate to 0.00001, set the number of training rounds to 150, and use the Adam optimizer.

[0153] (5) Repeat steps (3) to (4) as the training stage of the two-stage 3D deep attention network model to obtain the final model weights.

[0154] (6) The two-stage three-dimensional deep attention network model trained in step (5) is used to test the newly acquired three-dimensional cone beam CT image. The trained two-stage three-dimensional deep attention network model is used as input to obtain the image block obtained by three-dimensional sliding segmentation of the newly acquired three-dimensional cone beam CT image. The segmentation result of the image block is output through the positioning subnetwork, and the image segmentation block is merged and restored to the original position and the segmented area is merged through the nearest non-connected domain to obtain the predicted coordinates of the rectangular maxillary anterior tooth area position. The maxillary anterior tooth area is cropped out according to the coordinates of the rectangular maxillary anterior tooth area position. The predicted category confidence is output through the classification subnetwork, and finally the predicted maxillary anterior tooth area position in the newly acquired three-dimensional cone beam CT image and the confidence of the presence of the central supernumerary tooth at this position are obtained.

[0155] The present invention uses the three-dimensional image and the position and category of the annotated central supernumerary teeth as model experimental data, provides rich 3D spatial information, avoids the influence of spatial overlap of oral anatomical structures in two-dimensional curved body slice images on recognition, and can provide a more accurate three-dimensional spatial position for subsequent treatment. The positioning subnetwork of the present invention is based on the segmentation network framework. Compared with the traditional detection network, it does not need to process a large number of anchor points and anchor frames, which speeds up the detection reasoning speed; the present invention uses cross entropy loss and prediction frame coordinate distance loss as loss functions in the training of the positioning subnetwork, and the prediction frame coordinate distance loss can effectively improve the detection accuracy of the segmentation boundary box. The classification subnetwork of the present invention combines the attention mechanism with the residual module. The attention mechanism can strengthen the channel representation of the useful information features in the feature map, and the residual module can fuse the original input information features, optimize the feature expression efficiency, enhance the network generalization ability, and finally achieve the purpose of improving the classification accuracy. The present invention divides the entire three-dimensional image into several image blocks with partial regional overlap through a three-dimensional sliding block algorithm, and then inputs them into the positioning subnetwork for training and reasoning, which can effectively organize the local features of the image, reduce the computational consumption, and improve the segmentation accuracy and stability. The present invention converts the segmentation result from several non-connected neighboring regions into a target detection frame through a neighbor non-connected domain merging algorithm, which fully considers that when segmentation replaces the traditional detection algorithm, the foreground may be divided into multiple small regions, which affects the detection accuracy, and can further improve the detection accuracy. The present invention inputs the tested three-dimensional image into a trained two-stage three-dimensional deep attention network model for testing. No manual operation is required in the test stage. Only one scan of the entire three-dimensional image is required to quickly and automatically identify whether there are centric supernumerary teeth in the image, thereby helping to improve the clinical diagnosis efficiency of centric supernumerary teeth.

[0156] According to the method described in the above embodiment, this embodiment will be further described from the perspective of an automatic detection device for supernumerary teeth. The automatic detection device for supernumerary teeth can be implemented as an independent entity or integrated in an electronic device. The electronic device can be a terminal, a server, etc., wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0157] See also Figure 6 , Figure 6 The supernumerary teeth automatic detection device provided in the embodiment of the present application is specifically described, and is applied to an electronic device. The supernumerary teeth automatic detection device may include:

[0158] A preprocessing module, used to obtain a three-dimensional image and preprocess the three-dimensional image to obtain a plurality of image blocks;

[0159] The prediction module is used to input the image block into the trained two-stage deep attention network for recognition and obtain the probability value of the supernumerary teeth in the three-dimensional image;

[0160] Among them, the two-stage deep attention network includes a positioning subnetwork and a classification subnetwork; the image block is input into the trained two-stage deep attention network for recognition, including: inputting the image block into the positioning subnetwork to obtain a rectangular positioning box, and inputting the rectangular positioning box into the classification subnetwork to obtain the probability value of the image block containing supernumerary teeth.

[0161] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0162] In addition, an embodiment of the present application also provides an electronic device, which may be a computer, a tablet computer, or other device. Figure 7 As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0163] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or loading applications stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device as a whole.

[0164] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402 to implement various functions:

[0165] Acquire a three-dimensional image, and pre-process the three-dimensional image to obtain a plurality of image blocks;

[0166] The image block is input into the trained two-stage deep attention network for recognition, and the probability value of supernumerary teeth in the three-dimensional image is obtained;

[0167] Among them, the two-stage deep attention network includes a positioning subnetwork and a classification subnetwork; the image block is input into the trained two-stage deep attention network for recognition, including: inputting the image block into the positioning subnetwork to obtain a rectangular positioning box, and inputting the rectangular positioning box into the classification subnetwork to obtain the probability value of the image block containing supernumerary teeth.

[0168] The electronic device can implement the steps in any embodiment of the supernumerary teeth automatic detection method provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any supernumerary teeth automatic detection method provided in the embodiments of the present invention. Please refer to the previous embodiments for details and will not be repeated here.

[0169] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores a plurality of instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the supernumerary tooth automatic detection method provided in the embodiment of the present invention.

[0170] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0171] Since the instructions stored in the storage medium can execute the steps in any embodiment of the supernumerary teeth automatic detection method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any supernumerary teeth automatic detection method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0172] The above is a detailed introduction to the supernumerary teeth automatic detection method, device, storage medium and electronic device provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for automatically detecting supernumerary teeth, characterized in that: The method comprises: Acquire a three-dimensional image, and preprocess the three-dimensional image to obtain a plurality of image blocks; Inputting the image block into a trained two-stage deep attention network for recognition, and obtaining a probability value of containing supernumerary teeth in the three-dimensional image; The two-stage deep attention network includes a positioning subnetwork and a classification subnetwork; the step of inputting the image block into the trained two-stage deep attention network for recognition includes: inputting the image block into the positioning subnetwork to obtain a rectangular positioning frame, and inputting the rectangular positioning frame into the classification subnetwork to obtain a probability value of the image block containing supernumerary teeth; The positioning sub-network includes an encoder, which includes a pooling layer, a CGR module and a plurality of feature extraction branches; the positioning sub-network also includes a decoder, which includes an upsampling layer, a CGR module and a three-dimensional convolution layer; The step of inputting the image block into the positioning subnetwork to obtain the rectangular positioning frame includes: inputting the image block into two CGR modules to obtain a low-level semantic feature map; inputting the low-level semantic feature map into a first feature extraction branch and a second feature extraction branch to obtain a middle-level semantic feature map; inputting the middle-level semantic feature map into the first feature extraction branch and the second feature extraction branch to obtain a first high-level semantic feature map; inputting the first high-level semantic feature map into the pooling layer and a CGR module to obtain a low-resolution high-level semantic feature map; The CGR module includes three-dimensional convolution, group normalization and activation function combination module layers connected in sequence; the step of inputting the image block into two CGR modules to obtain a low-level semantic feature map includes: passing the image block through two CGR modules in sequence to obtain a low-level semantic feature map with the same image resolution as the image block; The first feature extraction branch includes two CGR modules, and the second feature extraction branch includes a maximum pooling layer and two CGR modules; the inputting the low-level semantic feature map into the first feature extraction branch and the second feature extraction branch to obtain a middle-level semantic feature map includes: inputting the low-level semantic feature map into the first feature extraction branch to obtain a first branch feature map, performing a jump connection between the first branch feature map and a feature map at the same resolution output by the decoder, and inputting the jump-connected feature map into the second feature extraction branch to obtain a middle-level semantic feature map; The step of inputting the middle-level semantic feature map into the first feature extraction branch and the second feature extraction branch to obtain a first high-level semantic feature map comprises: inputting the middle-level semantic feature map into the first feature extraction branch to obtain a second branch feature map, performing a jump connection between the second branch feature map and a feature map at the same resolution output by a decoder, and inputting the jump-connected feature map into the second feature extraction branch to obtain a first high-level semantic feature map; The step of inputting the image block into the positioning subnetwork to obtain the rectangular positioning frame further includes: passing the low-resolution high-level semantic feature map through a CGR module and an upsampling layer, and then splicing it with the first high-level semantic feature map to obtain a small-size fused feature map; passing the small-size fused feature map through two CGR modules and an upsampling layer, and then splicing it with the middle-level semantic feature map to obtain a middle-size fused feature map; passing the middle-size fused feature map through two CGR modules and an upsampling layer, and then splicing it with the low-resolution high-level semantic feature map to obtain a large-size fused feature map; inputting the large-size fused feature map into two CGR modules and a three-dimensional convolutional layer to obtain a segmentation result, and obtaining the rectangular positioning frame based on the segmentation result; Wherein, obtaining the rectangular positioning frame based on the segmentation result includes: obtaining the probability value of each pixel point in the segmentation result belonging to the maxillary anterior teeth area, and when the probability value is greater than a preset probability threshold, using the corresponding pixel point as a candidate point of the region of interest; combining adjacent candidate points of the region of interest to obtain multiple three-dimensional connected domains; obtaining the volume and circumscribed rectangular frame of each connected domain through a connected domain marking algorithm; using the circumscribed rectangular frame as a candidate detection frame, screening out connected domains with a volume less than a preset volume threshold, and obtaining a target candidate detection frame; merging the circumscribed rectangular frame of the largest connected domain and the candidate detection frames of the neighbors to obtain a rectangular positioning frame; calculating the coordinate values ​​of the rectangular positioning frame in the image block, using the coordinate values ​​as boundary range coordinate values, and calculating the distance between the center point of the rectangular positioning frame and the fixed point position based on the boundary range coordinate values; The classification subnetwork includes a CBR module, a maximum pooling layer, multiple residual modules, an average pooling layer and a fully connected layer; wherein the CBR module includes a three-dimensional convolutional layer, a group normalization and an activation function combination module layer connected in sequence; the multiple residual modules include a first residual module, a second residual module, a third residual module and a fourth residual module; The step of inputting the rectangular positioning frame into the classification subnetwork to obtain the probability value of the image block containing supernumerary teeth comprises: inputting the rectangular positioning frame into a CBR module, a maximum pooling layer, a first residual module, a second residual module, a third residual module and a fourth residual module in sequence to obtain a high-order feature map; inputting the high-order feature map into an average pooling layer and a fully connected layer to obtain the probability value of the image block containing supernumerary teeth; Among them, the first residual module includes a first basic residual module and a second basic residual module, and the first basic residual module or the second basic residual module includes a CBR module, a three-dimensional convolution, a batch normalization, and an SE attention module connected in sequence; in the first residual module, the feature map output by the maximum pooling layer is input into the first basic residual module to obtain a feature-enhanced fusion feature map, and the feature-enhanced fusion feature map is input into the second basic residual module to obtain a first fusion feature map; in the first basic residual module, the feature map output by the maximum pooling layer is sequentially passed through the CBR module, the three-dimensional convolution, and the batch normalization to obtain a first feature map, the first feature map is passed through the SE attention module to obtain a second feature map, and the first feature map and the second feature map are added item by item according to the channel and then passed through the activation function to obtain a feature-enhanced fusion feature map; The second residual module includes four CBR modules; in the second residual module, the first fused feature map is sequentially input into two CBR modules to obtain a first intermediate feature map, the first intermediate feature map is fused with the first fused feature map to obtain a fused feature map, and the fused feature map is sequentially input into two CBR modules to obtain a second fused feature map; The third residual module includes four CBR modules and a maximum pooling layer; in the third residual module, the second fused feature map is sequentially input into two CBR modules to obtain a second intermediate feature map, the second fused feature map is input into the maximum pooling layer to obtain a third intermediate feature map, the second intermediate feature map and the third intermediate feature map are spliced ​​to obtain a third fused feature map, and the third fused feature map is sequentially input into two CBR modules to obtain a fourth fused feature map; The fourth residual module has the same structure as the third residual module, and the fourth fused feature map is input into the fourth residual module to obtain a high-order feature map.

2. The method for automatically detecting supernumerary teeth according to claim 1, characterized in that: The training process of the two-stage deep attention network includes: Acquire a three-dimensional image, obtain a position mark of a maxillary anterior teeth region in the three-dimensional image and information on the type of supernumerary teeth in the maxillary anterior teeth region, and form an image data set; Preprocessing the images in the image data set by a three-dimensional sliding block algorithm to obtain a plurality of image blocks; Inputting the preprocessed image data set into the positioning subnetwork to obtain a rectangular positioning frame, and training based on the position annotation of the maxillary anterior teeth area, the cross entropy loss function of the segmentation result, and the mixed loss function of the distance between the center point and the vertex position of the rectangular positioning frame; The rectangular positioning frame is input into the classification subnetwork to obtain a probability value of containing supernumerary teeth, and training is performed based on supernumerary tooth category information in the maxillary anterior teeth area and a cross entropy loss function.

3. The method for automatically detecting supernumerary teeth according to claim 2, characterized in that: The preprocessing of the images in the image data set by using a three-dimensional sliding block algorithm includes: For the image dataset, the size is D × W × H For a 3D image, set a size of d × w × h The sliding window has a moving step size of ( sd,sw,sh ), generate the slice range corresponding to each image block in the complete image according to the window and step size: The number of slices in the Z direction is: ； The number of slices in the Y direction is: ； The number of slices in the X direction is: ； For Slices , calculate the coordinate range of each slice: ； Among them, the integer , round The () function is used to round a number to a given number of digits. When the last significant digit is 5 or greater, it is rounded up; when the last significant digit is less than 5, it is rounded down.

4. A supernumerary tooth automatic detection device, the supernumerary tooth automatic detection device is used to implement the supernumerary tooth automatic detection method of claim 1, characterized in that: include: A preprocessing module, used for acquiring a three-dimensional image, and preprocessing the three-dimensional image to obtain a plurality of image blocks; A prediction module, used for inputting the image block into a trained two-stage deep attention network for recognition, and obtaining a probability value of the three-dimensional image containing supernumerary teeth; Among them, the two-stage deep attention network includes a positioning subnetwork and a classification subnetwork; the inputting of the image block into the trained two-stage deep attention network for recognition includes: inputting the image block into the positioning subnetwork to obtain a rectangular positioning box, and inputting the rectangular positioning box into the classification subnetwork to obtain the probability value of the image block containing supernumerary teeth.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the method for automatically detecting supernumerary teeth as described in any one of claims 1 to 3.

6. An electronic device, characterized in that: It comprises a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the automatic detection method of supernumerary teeth as described in any one of claims 1 to 3.

Images