Tooth abnormal region detection method

Abstract

The embodiment of the invention provides a tooth abnormal region detection method. The method comprises the following steps: acquiring a tooth diagnosis and treatment image comprising at least one tooth; inputting the tooth diagnosis and treatment image into a tooth anomaly detection model to obtain an abnormal region mask image corresponding to a suspected abnormal region in the tooth diagnosis and treatment image, abnormal region bounding box information, an abnormal region classification result and a tooth position classification result to which the abnormal region belongs, the tooth anomaly detection model comprises a feature extraction network, a region-of-interest extraction network, a region feature alignment network for determining region feature information of regions of interest, and an abnormal region evaluation network for determining an abnormal region detection result according to the region feature information of each region of interest. The mask image corresponding to the abnormal region, the bounding box, the abnormal region classification result and the tooth position information of the abnormal region can be identified, and the detection accuracy, reliability and efficiency of the tooth abnormal region are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for detecting abnormal tooth regions. Background Art

[0002] In dental diagnosis and treatment, medical imaging is a commonly used method. For example, the roots of teeth and the surrounding bone can be observed through small dental radiographs (periapical films / bitewing films). Periapical radiograph: This type of X-ray film shows the complete length of a single or multiple teeth, including the crown, root, and surrounding tissues, and is used to detect periapical diseases, pulp diseases, etc. Bitewing radiograph: This type of X-ray film shows the occlusal surfaces of the upper and lower teeth and is mainly used to detect periodontal diseases and dental caries. Low-density shadows (such as dental caries) and high-density shadows (such as dentin hyperplasia or calcification deposits) are common abnormal manifestations on small dental radiographs and may be related to periodontal diseases, dental caries, or other oral diseases.

[0003] Currently, during dental diagnosis and treatment, doctors usually detect abnormal tooth regions by observing low / high-density shadows in medical images. However, this method highly depends on the subjective judgment of doctors, is not only time-consuming but also easily affected by factors such as doctors' experience and fatigue, with a very high missed detection rate, and there are technical problems of low detection efficiency, poor accuracy, and poor reliability in detecting tooth abnormalities. Summary of the Invention

[0004] Embodiments of the present invention provide a method for detecting abnormal tooth regions to identify the mask image, bounding box, abnormal region classification result, and tooth position information where the abnormal region is located corresponding to the abnormal region, and improve the detection accuracy, reliability, and efficiency of abnormal tooth regions.

[0005] In a first aspect, embodiments of the present invention provide a method for detecting abnormal tooth regions, and the method includes:

[0006] Obtain a dental diagnosis and treatment image to be detected; wherein, at least one tooth is included in the dental diagnosis and treatment image;

[0007] Input the dental diagnosis and treatment image into a tooth abnormality detection model to obtain an abnormal region detection result corresponding to a suspected abnormal region in the dental diagnosis and treatment image; wherein, the abnormal region detection result includes an abnormal region mask image, abnormal region bounding box information, abnormal region classification result, and tooth position classification result to which the abnormal region belongs;

[0008] Among them, the dental abnormality detection model includes: a feature extraction network for extracting a dental diagnosis feature map from the dental diagnosis image, a region of interest extraction network for extracting a region of interest from the dental diagnosis feature map, a region feature alignment network for determining the region feature information of the region of interest, and an abnormal region evaluation network for determining the abnormal region detection result according to the region feature information of each region of interest.

[0009] In a second aspect, an embodiment of the present invention further provides a dental abnormal region detection device, which includes:

[0010] A dental diagnosis image acquisition module, configured to acquire a dental diagnosis image to be detected; wherein, at least one tooth is included in the dental diagnosis image;

[0011] An abnormal result determination module, configured to input the dental diagnosis image into a dental abnormality detection model to obtain an abnormal region detection result corresponding to a suspected abnormal region in the dental diagnosis image; wherein, the abnormal region detection result includes an abnormal region mask image, abnormal region bounding box information, an abnormal region classification result, and a tooth position classification result to which the abnormal region belongs;

[0012] Among them, the dental abnormality detection model includes: a feature extraction network for extracting a dental diagnosis feature map from the dental diagnosis image, a region of interest extraction network for extracting a region of interest from the dental diagnosis feature map, a region feature alignment network for determining the region feature information of the region of interest, and an abnormal region evaluation network for determining the abnormal region detection result according to the region feature information of each region of interest.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, which includes:

[0014] One or more processors;

[0015] A storage device, configured to store one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the dental abnormal region detection method according to any one of the embodiments of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute the dental abnormal region detection method according to any one of the embodiments of the present invention when executed by a computer processor.

[0018] The technical solution of the embodiment of the present invention is to obtain a dental diagnosis and treatment image including at least one tooth, and then input the dental diagnosis and treatment image into a dental abnormality detection model to obtain an abnormal region mask image corresponding to the suspected abnormal region in the dental diagnosis and treatment image, abnormal region bounding box information, abnormal region classification results, and tooth position classification results to which the abnormal region belongs. Among them, the dental abnormality detection model includes: a feature extraction network for extracting a dental diagnosis and treatment feature map from the dental diagnosis and treatment image, a region of interest extraction network for extracting a region of interest from the dental diagnosis and treatment feature map, a region feature alignment network for determining the region feature information of the region of interest, and an abnormal region evaluation network for determining the abnormal region detection result according to the region feature information of each region of interest. The technical solution provided by the embodiment of the present invention can automatically detect the abnormal region in the dental diagnosis and treatment image through the dental abnormality detection model, and can not only identify the mask image, bounding box and abnormal region classification result corresponding to the abnormal region, but also identify the tooth position information where the abnormal region is located, improving the detection accuracy, reliability and efficiency of the dental abnormal region. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the introduced drawings are only the drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a schematic flowchart of a method for detecting a dental abnormal region provided by an embodiment of the present invention;

[0021] Figure 2 It is a schematic diagram of the arrangement of deciduous teeth and permanent teeth involved in an embodiment of the present invention;

[0022] Figure 3 It is a schematic diagram of a dental abnormality detection model involved in an embodiment of the present invention;

[0023] Figure 4 It is a schematic flowchart of another method for detecting a dental abnormal region provided by an embodiment of the present invention;

[0024] Figure 5 It is a schematic structural diagram of a device for detecting a dental abnormal region provided by an embodiment of the present invention;

[0025] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for ease of description, only parts related to the present invention rather than all structures are shown in the drawings.

[0027] Figure 1 The following is a schematic flowchart of a method for detecting abnormal tooth regions provided by an embodiment of the present invention. This embodiment is applicable to the situation where automated detection of abnormal regions on teeth is required. This method can be executed by a device for detecting abnormal tooth regions, and this device can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC terminal, or a server, etc.

[0028] As Figure 1 shown, the method for detecting abnormal tooth regions includes:

[0029] S110. Obtain a dental diagnostic image to be detected.

[0030] Among them, the dental diagnostic image is an X-ray image generated during the dental diagnosis process of the patient. The dental diagnostic image includes at least one tooth. For example, the dental diagnostic image can be a small dental film. A small dental film is an image of teeth and their surrounding tissues obtained through X-ray projection technology, which can show the shape, size, position of the teeth, and their relationship with adjacent tissues. When there are lesions such as dental caries, pulpitis, and periapical periodontitis, these abnormal conditions may be manifested in the image.

[0031] Specifically, when the patient is seeing a doctor, an image of the teeth and their surrounding tissues can be obtained through X-ray projection technology to obtain the dental diagnostic image to be detected; in addition, it can also be obtaining the dental diagnostic image taken by a certain patient in a historical period from a database.

[0032] In this embodiment, for a certain tooth of a certain patient that may have an abnormal region, by obtaining the dental diagnostic image and then automatically identifying the position and imaging manifestations of the abnormal region in this dental diagnostic image, the abnormal region on the tooth can be quickly and accurately identified. Finally, a dental detection report can be automatically generated to assist the doctor in further analysis and decision-making.

[0033] Specifically, in order to improve the detection accuracy of dental diagnosis images, after obtaining the dental diagnosis images, preprocessing can be performed on the dental diagnosis images. Among them, the specific implementation methods of preprocessing can include but are not limited to the following methods: Grayscale processing: Convert the dental diagnosis image (because the dental diagnosis image can be an image with RGB three channels) into a grayscale image. For example, it can be achieved by the method of cv2.cvtColor(image, cv2.COLOR_BGR2GRAY). Denoising processing: Use median filtering to perform denoising processing on the dental diagnosis image. For example, the cv2.medianBlur(image, ksize) function can be used, where ksize is set to 3. Normalization processing: Standardize the size of the dental diagnosis image. For example, adjust the dental diagnosis image to a fixed size (for example, 800x800 pixels); and the min-max normalization method can be used to normalize the pixel values of the dental diagnosis image to the range of [0, 1] or [-1, 1]. For example, it can be achieved by the formula normalized_image = (image - min_val) / (max_val - min_val).

[0034] S120. Input the dental diagnosis image into the dental anomaly detection model to obtain the anomaly region detection result corresponding to the suspected anomaly region in the dental diagnosis image. Among them, the anomaly region detection result includes the anomaly region mask image, the anomaly region bounding box information, the anomaly region classification result, and the tooth position classification result to which the anomaly region belongs.

[0035] Among them, the dental anomaly detection model is pre-trained. The dental anomaly detection model is used to detect whether there are anomaly regions in the dental diagnosis image and determine the refined information corresponding to the anomaly regions. These refined information are the anomaly region mask image, the anomaly region bounding box information, the anomaly region classification result, and the tooth position classification result to which the anomaly region belongs.

[0036] Specifically, the anomaly region mask image refers to using mask technology to identify and highlight the anomaly regions in the image. The mask is usually a binary image, where the anomaly regions are marked as white (or higher gray values), while the normal regions are marked as black (or lower gray values). In this way, the anomaly parts in the image can be clearly identified and separated.

[0037] The anomaly region bounding box refers to a rectangular box used to represent the anomaly region on the tooth. The anomaly region bounding box information refers to the position and size information of this rectangular box in the dental diagnosis image. For example, the anomaly region bounding box information can be expressed as (x, y, h, w), where x and y are the center coordinates of the bounding box, and w and h are the width and height of the bounding box.

[0038] The classification result of the abnormal area is used to characterize the abnormal category corresponding to the abnormal area. The classification result of the abnormal area may be at least one of the prosthetic area category, the root filling area category, the high-density lesion area category, the low-density lesion area category, the supernumerary tooth area category, the fracture line area category, the dental caries area category, and the vacant area category. The prosthetic area category means that a certain area on the tooth may be a prosthesis, such as a crown restoration, etc.; the root filling area category means that a certain area on the tooth may be a root filling; the high-density lesion area category means that a certain area in the dental diagnostic image of the tooth presents a high-density form, and there may be pathological changes such as pulp calcification, pulp stones, and diffuse calcification in this area; the low-density lesion area category means that a certain area in the dental diagnostic image of the tooth presents a low-density form, and there may be pathological changes such as abnormal root canal morphology, external root resorption, irregular resorption, inflammation, apical resorption, periapical periodontitis, periapical abscess, and periapical granuloma in this area; the supernumerary tooth area category means that a certain area on the tooth may be a supernumerary tooth, that is, a tooth with a number of human teeth exceeding the normal range. The fracture line area category means that a certain area on the tooth may be an alveolar process fracture, root fracture, fracture line, etc. The dental caries area category means that a certain area on the tooth may be a carious site. The vacant area category means that a certain area on the tooth may be an abnormal blank, such as tooth defect, tooth dislocation, etc.

[0039] Next, the basic concept of tooth position is introduced. Tooth position refers to the position of teeth in the oral cavity. Clinically, a specific representation method is used to mark the position of each tooth, and each tooth can be represented by two Arabic numerals. A person's life has a total of two dentitions. The first dentition is called deciduous teeth, and the second dentition is called permanent teeth. Permanent teeth are the teeth that grow after the deciduous teeth fall off. Therefore, deciduous teeth are the first set of teeth in the human body, and permanent teeth are the second set of teeth. For the arrangement diagrams of deciduous teeth and permanent teeth, see Figure 2 Deciduous teeth usually have 20 teeth, which are divided into four quadrants, with 5 teeth in each quadrant. For the arrangement diagram of deciduous tooth positions, see Figure 2 Figure (a), Figure 2 The specific arrangement of the deciduous tooth positions shown in Figure (a) is as follows: Left upper jaw: 55, 54, 53, 52, 51; Right upper jaw: 61, 62, 63, 64, 65; Left lower jaw: 85, 84, 83, 82, 81; Right lower jaw: 71, 72, 73, 74, 75. Permanent teeth usually have 32 teeth, which are divided into four quadrants, with 8 teeth in each quadrant. For the arrangement diagram of permanent tooth positions, see Figure 2 Figure (b), Figure 2(b) The specific arrangement of permanent teeth positions is as follows: On the left side of the upper jaw: 18, 17, 16, 15, 14, 13, 12, 11; On the right side of the upper jaw: 21, 22, 23, 24, 25, 26, 27, 28; On the left side of the lower jaw: 48, 47, 46, 45, 44, 43, 42, 41; On the right side of the lower jaw: 31, 32, 33, 34, 35, 36, 37, 38. In this embodiment, the tooth position classification result to which the abnormal area belongs refers to the tooth position information of the tooth where the abnormal area is located. For example, if the tooth position classification result to which a certain abnormal area belongs is "75", it means that the abnormal area is located at the 75 tooth position.

[0040] In this embodiment, for the schematic diagram of the tooth abnormality detection model, see Figure 3 , as Figure 3 shown, the tooth abnormality detection model includes a feature extraction network, a region of interest extraction network, a region feature alignment network, and an abnormal area evaluation network. Among them, the feature extraction network is used to extract a tooth diagnosis feature map according to the tooth diagnosis image, the region of interest extraction network is used to extract the region of interest from the tooth diagnosis feature map, the region feature alignment network is used to determine the region feature information of the region of interest, and the abnormal area evaluation network is used to determine the abnormal area detection result according to the region feature information of each region of interest. More specifically, the feature extraction network may further include a backbone network and a feature pyramid network, the region of interest extraction network includes a candidate region extraction network and a candidate region screening network, and the abnormal area evaluation network includes a tooth position recognition network and an abnormal area recognition network.

[0041] In the actual application process, the tooth diagnosis image is input into the tooth abnormality detection model, and each network unit in the tooth abnormality detection model processes and analyzes the tooth diagnosis image to obtain an abnormal area mask image, abnormal area bounding box information, abnormal area classification result, and tooth position classification result to which the abnormal area belongs corresponding to the suspected abnormal area in the tooth diagnosis image.

[0042] More specifically, the specific implementation method of inputting the tooth diagnosis image into the tooth abnormality detection model to obtain the abnormal area detection result corresponding to the suspected abnormal area in the tooth diagnosis image includes:

[0043] S1201. Input the tooth diagnosis image into the feature extraction network to obtain at least one tooth diagnosis feature map corresponding to the tooth diagnosis image.

[0044] Among them, the tooth diagnosis feature map refers to the feature image extracted from the tooth diagnosis image through convolution and feature extraction operations.

[0045] In this embodiment, the feature extraction network may include a backbone network and a Feature Pyramid Network. Among them, the backbone network may be a convolutional neural network. The Feature Pyramid Network (FPN) is used to generate feature maps of multiple scales to better process targets of different sizes. The FPN fuses feature maps of different levels through a top-down path and lateral connections.

[0046] Specifically, the specific implementation manner of inputting a dental diagnosis and treatment image into the feature extraction network to obtain at least one dental diagnosis and treatment feature map corresponding to the dental diagnosis and treatment image may include: inputting the dental diagnosis and treatment image into the backbone network to obtain at least one first feature map corresponding to the dental diagnosis and treatment image; inputting each first feature map into the Feature Pyramid Network to obtain at least one dental diagnosis and treatment feature map corresponding to the dental diagnosis and treatment image.

[0047] In this embodiment, by inputting the dental diagnosis and treatment image into the backbone network, the backbone network extracts features from the dental diagnosis and treatment image to obtain one or more first feature maps; further, each first feature map can be input into the Feature Pyramid Network, and the Feature Pyramid Network processes each first feature map to obtain feature maps of multiple scales, that is, multiple dental diagnosis and treatment feature maps.

[0048] Exemplarily, ResNet-50 in Caffe style can be used as the backbone network, which contains 50 convolutional layers and is divided into four stages. Each stage generates feature maps of different scales. Stage 1: It contains a 7x7 convolutional layer with a stride of 2, followed by a 3x3 max-pooling layer with a stride of 2. Stage 2-4: Each stage contains multiple Residual Blocks, and each Residual Block consists of two or three convolutional layers, and the dimension of the feature map is maintained through skip connections. A feature pyramid is constructed based on the output feature map of ResNet-50, which includes a bottom-up path and a top-down path. Bottom-up path: Extract the first feature maps (C2, C3, C4, C5) from each stage of ResNet-50, where C2, C3, C4, C5 correspond to feature maps of different scales respectively. Top-down path: The high-level semantic information is transmitted to the low-level through upsampling and lateral connections to generate new feature maps (P2, P3, P4, P5), that is, the dental diagnosis and treatment feature maps. 1x1 convolutional layer: A 1x1 convolutional layer is used in each lateral connection to adjust the number of channels and keep the dimension of the feature map consistent.

[0049] S1202. For each dental diagnosis feature map, input the current dental diagnosis feature map into the region of interest extraction network to obtain at least one region of interest corresponding to the current dental diagnosis feature map.

[0050] Among them, the region of interest (ROI) refers to the area that needs special attention outlined from the dental diagnosis feature map. These areas are usually marked by means of rectangles, circles, ellipses, irregular polygons, etc. for further processing and analysis. In particular, the regions of interest in this embodiment include each tooth and the abnormal areas on the teeth.

[0051] In this embodiment, the region of interest extraction network includes a candidate region extraction network and a candidate region screening network. Among them, the candidate region extraction network can be a Region Proposal Network (RPN). RPN is a key component in the Faster R-CNN network. Its main function is to extract candidate regions (Region Proposals) from the input image. These candidate regions are then used for object detection and localization. RPN generates a series of candidate regions by sliding a small network (called an anchor box) on the feature map and scores these regions to determine whether they contain an object. Apply the region proposal network on the feature map generated by FPN to generate candidate regions (Region Proposals). The candidate region screening network can use the non-maximum suppression method to screen multiple candidate regions to obtain one or more regions of interest. In this embodiment, the candidate region screening network can use a Non-Maximum Suppression (NMS) network to remove overlapping candidate regions through NMS. The purpose of NMS is to retain the most likely candidate regions and remove those regions that overlap significantly with high-confidence candidate regions.

[0052] Specifically, for each dental diagnosis feature map, the specific implementation method of inputting the current dental diagnosis feature map into the region of interest extraction network to obtain at least one region of interest corresponding to the current dental diagnosis feature map may include: for each dental diagnosis feature map, input the current dental diagnosis feature map into the candidate region extraction network to obtain at least one candidate region corresponding to the current dental diagnosis feature map; based on the candidate region screening network, perform region screening on the at least one candidate region corresponding to the current dental diagnosis feature map to obtain at least one region of interest corresponding to the current dental diagnosis feature map.

[0053] In this embodiment, the processing procedure for each dental diagnosis feature map is the same. Here, any one of the dental diagnosis feature maps is used as the current dental diagnosis feature map, and an exemplary description is given taking the current dental diagnosis feature map as an example. Among them, the current dental diagnosis feature map refers to the dental diagnosis feature map being currently processed. By inputting the current dental diagnosis feature map into the candidate region extraction network, the candidate region extraction network can generate multiple anchor boxes with different scales and aspect ratios on the current dental diagnosis feature map. Furthermore, binary classification (foreground or background) and bounding box regression can be performed on each anchor point (i.e., each pixel point within the anchor box) to generate candidate regions and determine the confidence value corresponding to each candidate region. Further, the confidence values corresponding to the respective candidate regions and the corresponding candidate region bounding box coordinates are input into the candidate region screening network. After the candidate region screening network performs region screening on these candidate regions, one or more candidate regions obtained after screening are the regions of interest corresponding to the current dental diagnosis feature map.

[0054] S1203. Input the current dental diagnosis feature map and at least one region of interest corresponding to the current dental diagnosis feature map into the region feature alignment network to obtain the region feature information corresponding to each region of interest.

[0055] Among them, the region feature information refers to the feature data corresponding to the region of interest.

[0056] In this embodiment, the region feature alignment network can be a Region of Interest Align (RoI Align). RoI Align is used to extract features in the region of interest and perform alignment operations to more accurately capture the feature information of the target.

[0057] Specifically, the current dental diagnosis feature map (i.e., P2, P3, P4, or P5 in the above example) and each region of interest corresponding to the current dental diagnosis feature map can be input into the region feature alignment network. The region feature alignment network performs refined processing on the region of interest through the method of bilinear interpolation to obtain the region feature information corresponding to each region of interest. Specifically, RoI Align performs interpolation operations on the floating-point coordinate positions within the region of interest, retaining more accurate position information, enabling this layer to better capture the accurate contour and features of the target in object detection and instance segmentation tasks.

[0058] S1204. Input the region feature information of at least one region of interest of each dental diagnosis feature map into the abnormal region evaluation network to obtain the abnormal region mask image, abnormal region bounding box information, abnormal region classification result, and tooth position classification result to which the abnormal region belongs in the dental diagnosis image.

[0059] In this embodiment, the abnormal region evaluation network includes a tooth position recognition network and an abnormal region recognition network. The tooth position recognition network is used to determine the tooth position information in the dental treatment image according to the region feature information of each region of interest in each dental treatment feature map. The purpose of determining the tooth position information is that by providing the doctor with the tooth position information where the abnormal region is located, the doctor can quickly locate the specific position of the abnormal part in the patient's oral cavity, which is beneficial to improving the doctor's diagnosis and treatment efficiency. The abnormal region recognition network is used to determine the specific information corresponding to the suspected abnormal part in the dental treatment image according to the region feature information of each region of interest in each dental treatment feature map.

[0060] Specifically, the specific implementation manner of this step may include: inputting the region feature information of at least one region of interest in each dental treatment feature map into the tooth position recognition network to obtain the tooth position bounding box information and tooth position classification result corresponding to each tooth in the dental treatment image; inputting the region feature information of at least one region of interest in each dental treatment feature map into the abnormal region recognition network to obtain the abnormal region mask image, abnormal region bounding box information, and abnormal region classification result corresponding to the suspected abnormal region in the dental treatment image; and determining the tooth position classification result to which the abnormal region belongs based on the abnormal region bounding box information and the tooth position bounding box information and tooth position classification result corresponding to each tooth.

[0061] Among them, the tooth position bounding box information represents a rectangular box of a certain tooth.

[0062] In this embodiment, the tooth position recognition network may specifically include a tooth bounding box regression sub-network and a tooth position classification sub-network. After inputting the region feature information of each region of interest in each dental treatment feature map into the tooth position evaluation network, the tooth bounding box regression sub-network can output the tooth position bounding box information corresponding to each tooth in the dental treatment image after processing and analyzing these region feature information; the tooth position classification sub-network can output the tooth position classification result corresponding to each tooth in the dental treatment image after processing and analyzing these region feature information. It should be particularly noted that the tooth position evaluation network may further include a tooth mask generation sub-network, which can output the tooth mask image of the tooth position corresponding to each tooth in the dental treatment image after processing and analyzing these region feature information.

[0063] The abnormal region recognition network may specifically include an abnormal region bounding box regression sub-network, an abnormal region classification sub-network, and an abnormal region mask generation sub-network. After inputting the region feature information of each region of interest in each dental diagnosis and treatment feature map into the abnormal region evaluation network, the abnormal region bounding box regression sub-network can output the abnormal region bounding box information of each abnormal region in the recognized dental diagnosis and treatment image after processing and analyzing these region feature information; after the abnormal region classification sub-network processes and analyzes these region feature information, it can output the abnormal region classification results of each abnormal region in the dental diagnosis and treatment image; after the abnormal region mask generation sub-network processes and analyzes these region feature information, it can output the abnormal region mask image of each abnormal region in the dental diagnosis and treatment image. For example, the abnormal region mask generation sub-network can be a fully convolutional network, and a pixel-level segmentation mask can be generated through the fully convolutional network. The size of the abnormal region mask image can be the same as that of the dental diagnosis and treatment image.

[0064] Based on this, the target center coordinates corresponding to the abnormal region can be determined according to the abnormal region bounding box information. Through the target center coordinates and the tooth position bounding box information, it can be determined which tooth position bounding box the target center coordinates are located in, and this tooth position bounding box can be determined as the target tooth position bounding box to which the suspected abnormal region belongs. Further, the tooth position classification result corresponding to the target tooth position bounding box can be determined as the tooth position classification result to which the abnormal region belongs.

[0065] In the technical solution of the embodiment of the present invention, by obtaining a dental diagnosis and treatment image including at least one tooth, and then inputting the dental diagnosis and treatment image into a tooth abnormality detection model, an abnormal region mask image, abnormal region bounding box information, abnormal region classification results, and the tooth position classification result to which the abnormal region belongs corresponding to the suspected abnormal region in the dental diagnosis and treatment image are obtained. Among them, the tooth abnormality detection model includes: a feature extraction network for extracting a dental diagnosis and treatment feature map from the dental diagnosis and treatment image, a region of interest extraction network for extracting regions of interest from the dental diagnosis and treatment feature map, a region feature alignment network for determining the region feature information of the regions of interest, and an abnormal region evaluation network for determining the abnormal region detection result according to the region feature information of each region of interest. The technical solution provided by the embodiment of the present invention can automatically detect the abnormal regions in the dental diagnosis and treatment image through the tooth abnormality detection model, and can not only identify the mask image, bounding box, and abnormal region classification results corresponding to the abnormal regions, but also identify the tooth position information where the abnormal regions are located, improving the detection accuracy, reliability, and efficiency of the tooth abnormal regions.

[0066] Embodiment 2

[0067] Figure 4It is a schematic flowchart of a method for detecting tooth abnormal regions provided in the second embodiment of the present invention. On the basis of the foregoing embodiment, the training method of the tooth abnormality detection model is further refined. Among them, the same or corresponding technical terms as those in the above embodiments will not be described in detail here.

[0068] As Figure 4 shown, the method includes:

[0069] S210. Obtain a plurality of sample images containing annotation information.

[0070] Among them, the annotation information includes tooth annotation information corresponding to each tooth in the sample image and abnormal region annotation information corresponding to the abnormal regions on the teeth. The tooth annotation information includes a first annotation bounding box, a first annotation mask image, and a tooth position classification label corresponding to each tooth. The abnormal region annotation information includes a second annotation bounding box, a second annotation mask image, and an abnormal classification label corresponding to the tooth abnormal region.

[0071] In this embodiment, historical tooth diagnosis and treatment images generated in a historical period can be obtained as sample images. The historical tooth diagnosis and treatment images are tooth X-ray images corresponding to patients with abnormal regions on their teeth.

[0072] A labeling tool can be used to label the historical tooth diagnosis and treatment images to generate a labeling file in JSON format. For each sample image, the labeling file includes a first annotation bounding box, a first annotation mask image, and a tooth position classification label corresponding to each tooth; and, a second annotation bounding box, a second annotation mask image, and an abnormal classification label corresponding to each tooth abnormal region.

[0073] S220. For each sample image, input the current sample image into the tooth abnormality detection model to be trained for image processing, and obtain tooth prediction information corresponding to each tooth in the current sample image and abnormal region prediction information corresponding to the abnormal regions on the teeth.

[0074] Among them, the tooth prediction information includes a first prediction bounding box, a first prediction mask image, and a predicted tooth position classification corresponding to each tooth. The abnormal region prediction information includes a second prediction bounding box, a second prediction mask image, and a predicted abnormal classification corresponding to the tooth abnormal region.

[0075] In this embodiment, the model parameters of the tooth abnormality detection model to be trained are initial values, and the model parameters need to be corrected during the model training process to finally obtain the tooth abnormality detection model. The model structure of the tooth abnormality detection model to be trained is Figure 2 the model structure shown.

[0076] Specifically, the processing process for each sample image is the same. Here, any one of the sample images is used as the current sample image, and an exemplary description is given taking the current sample image as an example. The current sample image is the sample image currently being processed. The current sample image can be input into the to-be-trained dental anomaly detection model. After the feature extraction network, region of interest extraction network, and region feature alignment network in the to-be-trained dental anomaly detection model process the current sample image, the sample region feature information corresponding to each region of interest in the current sample image can be obtained. Further, after the tooth bounding box regression sub-network, tooth position classification sub-network, and tooth mask generation sub-network in the tooth position recognition network respectively process and analyze these sample region feature information, the first predicted bounding box, predicted tooth position classification, and first predicted mask image corresponding to each tooth in the current sample can be output. After the abnormal region bounding box regression sub-network, abnormal region classification sub-network, and abnormal region mask generation sub-network in the abnormal region recognition network respectively process and analyze these sample region feature information, the second predicted bounding box, predicted abnormal classification, and second predicted mask image corresponding to each suspected abnormal region in the current sample can be output.

[0077] S230. Based on the tooth annotation information and tooth prediction information, determine the tooth result loss value corresponding to the current sample image.

[0078] Among them, the tooth result loss value is used to characterize the difference between the tooth prediction information and the tooth annotation information.

[0079] Specifically, the specific implementation manner of determining the tooth result loss value based on the tooth annotation information and tooth prediction information may include the following steps:

[0080] S1. Based on the offsets of the first annotation bounding boxes corresponding to each tooth and the offsets of the first predicted bounding boxes, determine the tooth bounding box loss value corresponding to the current sample image. Based on the first annotation mask images and the first predicted mask images corresponding to each tooth, determine the tooth mask loss value corresponding to the current sample image. Based on the tooth position classification labels and the predicted tooth position classifications corresponding to each tooth, determine the tooth position classification loss value corresponding to the current sample image.

[0081] First, the determination method of the tooth bounding box loss value mainly includes:

[0082] In this embodiment, for each tooth in the current sample image, the bounding box loss value of each tooth is determined respectively, and then the sum operation is performed on the bounding box loss values to obtain the tooth bounding box loss value corresponding to the current sample image. The formula for calculating the tooth bounding box loss value corresponding to the current sample image is as follows:

[0083] (1)

[0084] In the formula, represents the tooth bounding box loss value, represents the offset of the first annotated bounding box corresponding to the th tooth in the current sample image, represents the offset of the first predicted bounding box corresponding to the th tooth in the current sample image; is the smooth L1 loss function, defined as:

[0085] (2)

[0086] Among them, the offset of the first annotated bounding box is obtained from the annotation data. Suppose we have a true annotated bounding box , is the center coordinate of the bounding box, and are the width and height of the bounding box. For each region of interest, the offset between it and the true bounding box can be calculated. Suppose the bounding box of the region of interest is , then the true bounding box offset can be expressed as:

[0087] (3)

[0088] In the formula: and are the offsets of the center coordinates of the first annotated bounding box. and are the width offset and height offset of the first annotated bounding box.

[0089] The offset of the first predicted bounding box is predicted by the model through the tooth bounding box regression sub-network. Suppose the bounding box predicted by the model is , then the offset of the first predicted bounding box can be expressed as:

[0090] (4)

[0091] In the formula: and are the offsets of the center coordinates of the first predicted bounding box. and are the width offset and height offset of the first predicted bounding box.

[0092] Secondly, the determination method of the tooth mask loss value mainly includes:

[0093] In this embodiment, for each tooth in the current sample image, the mask loss value of each tooth is determined respectively, and then the mask loss values are summed to obtain the tooth mask loss value corresponding to the current sample image. The formula for the mask loss value corresponding to a tooth is as follows:

[0094] (5)

[0095] In the formula, represents the mask pixel value (0 or 1) corresponding to the th pixel in the first labeled mask image, represents the mask pixel value (0 or 1) corresponding to the th pixel in the first predicted mask image, is the resolution of the mask. Based on this, the mask loss value corresponding to each tooth can be calculated in the same way, and these mask loss values are summed to obtain the tooth mask loss value.

[0096] Again, the determination method of the tooth position classification loss value mainly includes:

[0097] In this embodiment, for each tooth in the current sample image, the tooth position loss value of each tooth is determined respectively, and then the tooth position loss values are summed to obtain the tooth position classification loss value corresponding to the current sample image. For each tooth in the current sample image, the predicted tooth position classification corresponding to the tooth can be represented as a 1×52-dimensional vector, and each element in the vector represents the probability value that the current tooth is in this tooth position; similarly, the tooth position classification label is also a 1×52-dimensional vector. The calculation formula for the tooth position classification loss value corresponding to a tooth is as follows:

[0098] (6)

[0099] In the formula, represents the total number of tooth positions (i.e., 52), represents the tooth position classification label of the th tooth position, represents the predicted tooth position classification of the th tooth position.

[0100] S2. Based on the predicted tooth position classification corresponding to each tooth, the first predicted bounding box, and the preset tooth position constraint function, determine the tooth position constraint loss value corresponding to the current sample image.

[0101] In this embodiment, considering the spatial distribution characteristics of the tooth positions, a tooth position constraint loss value is introduced to ensure that the tooth positions predicted by the model conform to the actual arrangement rules of the teeth.

[0102] Among them, the preset tooth position constraint function can be expressed as:

[0103] (7)

[0104] Wherein, represents the tooth position constraint loss value;

[0105] represents a preset position constraint matrix. Considering the adjacent relationship between permanent teeth and deciduous teeth and their mutual exclusion relationship, the preset position constraint matrix is predefined , the dimension of the preset position constraint matrix is 52x52, and the value at each element position in this matrix is used to characterize the relationship between the corresponding two tooth positions. Exemplarily, continue to refer to Figure 2 , if tooth position 11 and tooth position 12 are adjacent, then ; if 11 and 52 are adjacent, then ; if 11 and 51 are mutually exclusive, then ; if 11 and 14 are not adjacent, then ; In particular, the deciduous teeth 55, 54, 53, 52, 51 on the left side of the upper jaw and the permanent teeth 15, 14, 13, 12, 11 are in a mutually exclusive relationship, that is, permanent teeth and deciduous teeth cannot coexist in the same position; Similarly, the deciduous teeth 61, 62, 63, 64, 65 on the right side of the upper jaw and the permanent teeth 21, 22, 23, 24, 25 are in a mutually exclusive relationship; the deciduous teeth 85, 84, 83, 82, 81 on the left side of the lower jaw and the permanent teeth 45, 44, 43, 42, 41 are in a mutually exclusive relationship; the deciduous teeth 71, 72, 73, 74, 75 on the right side of the lower jaw and the permanent teeth 31, 32, 33, 34, 35 are in a mutually exclusive relationship.

[0106] represents the distance between the center coordinates of the first predicted bounding box of the th tooth and the center coordinates of the first predicted bounding box of the th tooth.

[0107] In this embodiment, based on the predicted tooth position classifications corresponding to each tooth, these information are brought into the preset tooth position constraint function, and the tooth position constraint loss value can be obtained.

[0108] In this embodiment, the tooth position constraint loss value penalizes the predictions that do not conform to the rules by calculating the distance between the predicted positions and combining the relative position relationship in the position constraint matrix, promotes the model to learn the tooth arrangement rules, and improves the robustness of the model.

[0109] S3. Based on the bounding box loss value, the tooth mask loss value, the tooth position classification loss value, and the tooth position constraint loss value, determine the tooth result loss value corresponding to the current sample image.

[0110] In this embodiment, the weighted sum of the bounding box loss value, tooth mask loss value, tooth position classification loss value, and tooth position constraint loss value corresponding to each tooth can be determined as the tooth result loss value.

[0111] S240. Based on the abnormal region annotation information and the abnormal region prediction information, determine the abnormal region result loss value corresponding to the current sample image.

[0112] Among them, the abnormal region result loss value is used to characterize the difference between the abnormal region prediction information and the abnormal region annotation information.

[0113] Specifically, based on the abnormal region annotation information and the abnormal region prediction information, the specific implementation manner of determining the abnormal region result loss value may include the following steps:

[0114] S1. For each abnormal region, based on the offset of the second annotation bounding box corresponding to the current abnormal region and the offset of the second prediction bounding box, determine the abnormal region bounding box loss value corresponding to the sample image.

[0115] In this embodiment, for each abnormal region in the current sample image, the bounding box loss value of each abnormal region is determined respectively, and then the sum of the bounding box loss values is calculated to obtain the abnormal region bounding box loss value corresponding to the current sample image. The formula for calculating the abnormal region bounding box loss value corresponding to the sample image can be seen in formula (1). The difference from the S1 step in S230 is that in this step, represents the abnormal region bounding box loss value, represents the offset of the second annotation bounding box corresponding to the th abnormal region in the sample image, represents the offset of the second prediction bounding box corresponding to the th abnormal region in the sample image. It should be particularly noted that the determination method of the offset of the second annotation bounding box can refer to the specific processing process of the offset of the first annotation bounding box, and the determination method of the offset of the second prediction bounding box can refer to the specific processing process of the offset of the first prediction bounding box, which will not be elaborated here.

[0116] S2. Based on the second annotation mask image and the second prediction mask image, determine the abnormal region mask loss value corresponding to the sample image.

[0117] In this embodiment, for each abnormal region in the current sample image, the mask loss value of each abnormal region is determined respectively, and then the sum of the mask loss values is calculated to obtain the abnormal region mask loss value corresponding to the current sample image. The formula for the mask loss value corresponding to an abnormal region can be seen in formula (5). The difference from the S1 step in S230 is that in this step, Indicates the mask pixel value (0 or 1) corresponding to the th pixel in the second labeled mask image, Indicates the mask pixel value (0 or 1) corresponding to the th pixel in the second predicted mask image, is the resolution of the mask. Based on this, the mask loss value corresponding to each abnormal region can be calculated in the same way, and the sum operation is performed on these mask loss values to obtain the mask loss value of the abnormal region.

[0118] S3. Determine the abnormal region classification loss value corresponding to the sample image based on the abnormal classification label and the predicted abnormal classification.

[0119] In this embodiment, for each abnormal region in the current sample image, the abnormal region loss value corresponding to the abnormal region can be represented as a 1×N-dimensional vector, and each element in the vector represents the probability value that the current abnormal region is classified as this type of abnormal region; similarly, the abnormal classification label is also a 1×N-dimensional vector. For example, if there are a total of 8 abnormal region classification results (i.e., at least one of the prosthesis region category, root filling region category, high-density lesion region category, low-density lesion region category, supernumerary tooth region category, fracture line region category, caries region category, and vacancy region category), the abnormal region loss value can be represented as a 1×8-dimensional vector.

[0120] The calculation formula for the tooth position classification loss value corresponding to the abnormal region is as follows:

[0121] (8)

[0122] In the formula, represents the total number of abnormal region classifications (i.e., 8), represents the abnormal classification label corresponding to the th abnormal region, represents the predicted abnormal classification corresponding to the th abnormal region.

[0123] S4. Determine the abnormal region result loss value corresponding to the current sample image based on the abnormal region bounding box loss value, abnormal region mask loss value, and abnormal region classification loss value corresponding to each abnormal region.

[0124] In this embodiment, the weighted sum of the abnormal region bounding box loss value, abnormal region mask loss value, and abnormal region classification loss value corresponding to each abnormal region can be determined as the abnormal region result loss value.

[0125] S250. Train the to-be-trained dental anomaly detection model based on the dental result loss value and the abnormal area result loss value of each sample image until the training ends when a preset condition is met, and obtain the dental anomaly detection model.

[0126] Among them, the preset condition is a condition set in advance. When training the to-be-trained dental anomaly detection model, when the training result meets the preset condition, it indicates that the to-be-trained dental anomaly detection model has been trained.

[0127] Optionally, the preset condition includes at least one of the following situations: the number of training iterations reaches a preset iteration threshold; the dental result loss value and the abnormal area result loss value are respectively less than their corresponding loss thresholds; a comprehensive loss value is determined from the dental result loss value and the abnormal area result loss value, and the comprehensive loss value is less than the comprehensive loss threshold.

[0128] In the embodiment of the present invention, the dental result loss value and the abnormal area result loss value jointly optimize the parameters of the to-be-trained dental anomaly detection model. Each time a sample image is input into the preset image processing model, a set of dental result loss values and abnormal area result loss values can be determined. The dental result loss values and abnormal area result loss values determined each time can be respectively fed back into the to-be-trained dental anomaly detection model to optimize the parameters in the to-be-trained dental anomaly detection model. In addition, a comprehensive loss value can be determined from the dental result loss value and the abnormal area result loss value, and the comprehensive loss value is fed back into the to-be-trained dental anomaly detection model, thereby optimizing the parameters in the to-be-trained dental anomaly detection model. Subsequently, the to-be-trained dental anomaly detection model after model optimization processes the next set of sample images. An iteration threshold can be preset. When the number of times the to-be-trained dental anomaly detection model is trained based on the sample image reaches the iteration threshold, and the dental result loss value and the abnormal area result loss value are respectively less than their corresponding loss thresholds, it can be proved that the dental anomaly detection model training is completed. In another embodiment, when the number of times the to-be-trained dental anomaly detection model is trained based on the sample image reaches the iteration threshold, and the comprehensive loss value is less than the loss threshold, it can be proved that the dental anomaly detection model training is completed.

[0129] Specifically, before training the to-be-trained dental abnormality detection model, training parameters can be preset. For example, the learning rate can be set to 0.02 (8 GPUs), the batch size can be set to 16, the optimizer can be set to the Stochastic Gradient Descent (SGD) optimizer, the momentum can be set to 0.9, and the weight decay can be set to 0.0001. In addition, during the model training process, data augmentation methods such as random horizontal flipping, random scaling (scale range from 0.8 to 1.2), and random color jittering can be applied to the sample images for data augmentation, so as to increase the diversity and quantity of the training data, thereby improving the generalization ability and robustness of the model and reducing the overfitting phenomenon.

[0130] S260. Obtain a dental diagnosis and treatment image to be detected; wherein, the dental diagnosis and treatment image includes at least one tooth.

[0131] S270. Input the dental diagnosis and treatment image into the dental abnormality detection model to obtain an abnormality area detection result corresponding to the suspected abnormality area in the dental diagnosis and treatment image; wherein, the abnormality area detection result includes an abnormality area mask image, abnormality area bounding box information, an abnormality area classification result, and a tooth position classification result to which the abnormality area belongs.

[0132] Among them, the dental abnormality detection model includes: a feature extraction network for extracting a dental diagnosis and treatment feature map from the dental diagnosis and treatment image, a region of interest extraction network for extracting a region of interest from the dental diagnosis and treatment feature map, a region feature alignment network for determining the region feature information of the region of interest, and an abnormality area evaluation network for determining the abnormality area detection result according to the region feature information of each region of interest.

[0133] In the technical solution of this embodiment, during the process of training the dental anomaly detection model, multiple sample images containing annotation information are obtained. The annotation information includes dental annotation information corresponding to each tooth in the sample image and anomaly region annotation information corresponding to the abnormal regions on the teeth. The dental annotation information includes a first annotation bounding box, a first annotation mask image, and a dental position classification label corresponding to each tooth. The anomaly region annotation information includes a second annotation bounding box, a second annotation mask image, and an anomaly classification label corresponding to the dental abnormal region. Furthermore, for each sample image, the current sample image is input into the to-be-trained dental anomaly detection model for image processing to obtain dental prediction information corresponding to each tooth in the current sample image and anomaly region prediction information corresponding to the abnormal regions on the teeth. The dental prediction information includes a first prediction bounding box, a first prediction mask image, and a predicted dental position classification corresponding to each tooth. The anomaly region prediction information includes a second prediction bounding box, a second prediction mask image, and a predicted anomaly classification corresponding to the dental abnormal region. Further, based on the dental annotation information and the dental prediction information, the dental result loss value corresponding to the current sample image is determined. Based on the anomaly region annotation information and the anomaly region prediction information, the anomaly region result loss value corresponding to the current sample image is determined. Thus, based on the dental result loss values and the anomaly region result loss values of each sample image, the to-be-trained dental anomaly detection model is trained until the preset conditions are met and the training ends, obtaining the dental anomaly detection model. In this embodiment, the to-be-trained dental anomaly detection model is trained with multi-dimensional loss values, improving the performance of the dental anomaly detection model. In particular, a dental position constraint loss value is designed. The dental position constraint loss value penalizes irregular predictions by calculating the distance between predicted positions and combining the relative position relationships in the position constraint matrix, promoting the model to learn the dental arrangement rules and improving the robustness of the model. When applying the trained dental anomaly detection model subsequently, more accurate dental position information can be provided. By providing the doctor with the dental position information of the abnormal region, the doctor can quickly locate the specific position of the abnormal part in the patient's oral cavity, which is beneficial to improving the doctor's diagnosis and treatment efficiency.

[0134] Embodiment III

[0135] Figure 5 FIG. is a schematic structural diagram of a dental abnormal region detection device provided by an embodiment of the present invention, as Figure 5 shown, the device includes: a dental diagnosis and treatment image acquisition module 310 and an anomaly result determination module 320.

[0136] Among them, the dental diagnosis and treatment image acquisition module 310 is configured to acquire a dental diagnosis and treatment image to be detected; among them, the dental diagnosis and treatment image includes at least one tooth;

[0137] An abnormal result determination module 320 is configured to input a dental diagnosis image into a dental abnormality detection model to obtain an abnormal area detection result corresponding to a suspected abnormal area in the dental diagnosis image; wherein, the abnormal area detection result includes an abnormal area mask image, abnormal area bounding box information, an abnormal area classification result, and a dental position classification result to which the abnormal area belongs.

[0138] Among them, the dental abnormality detection model includes: a feature extraction network for extracting a dental diagnosis feature map from the dental diagnosis image, a region of interest extraction network for extracting a region of interest from the dental diagnosis feature map, a region feature alignment network for determining region feature information of the region of interest, and an abnormal area evaluation network for determining an abnormal area detection result according to the region feature information of each region of interest.

[0139] Based on the above embodiments, optionally, the abnormal result determination module 320 includes:

[0140] A feature map determination unit is configured to input the dental diagnosis image into the feature extraction network to obtain at least one dental diagnosis feature map corresponding to the dental diagnosis image;

[0141] A region of interest determination unit is configured to input the current dental diagnosis feature map into the region of interest extraction network for each dental diagnosis feature map to obtain at least one region of interest corresponding to the current dental diagnosis feature map;

[0142] A region feature determination unit is configured to input the current dental diagnosis feature map and at least one region of interest corresponding to the current dental diagnosis feature map into the region feature alignment network to obtain region feature information corresponding to each region of interest;

[0143] An abnormal result determination unit is configured to input the region feature information of at least one region of interest of each dental diagnosis feature map into the abnormal area evaluation network to obtain an abnormal area mask image, abnormal area bounding box information, an abnormal area classification result, and a dental position classification result to which the abnormal area belongs corresponding to the suspected abnormal area in the dental diagnosis image.

[0144] Based on the above embodiments, optionally, the feature extraction network includes a backbone network and a feature pyramid network, and the feature map determination unit includes:

[0145] A first feature map determination subunit is configured to input the dental diagnosis image into the backbone network to obtain at least one first feature map corresponding to the dental diagnosis image;

[0146] A feature map determination subunit is configured to input each first feature map into the feature pyramid network to obtain at least one dental diagnosis feature map corresponding to the dental diagnosis image.

[0147] Based on the above embodiments, optionally, the region of interest extraction network includes a candidate region extraction network and a candidate region screening network. The region of interest determination unit includes:

[0148] A candidate region determination subunit, configured to input the current dental diagnosis and treatment feature map into the candidate region extraction network for each dental diagnosis and treatment feature map, to obtain at least one candidate region corresponding to the current dental diagnosis and treatment feature map;

[0149] An interested region determination subunit, configured to perform region screening on at least one candidate region corresponding to the current dental diagnosis and treatment feature map based on the candidate region screening network, to obtain at least one interested region corresponding to the current dental diagnosis and treatment feature map.

[0150] Based on the above embodiments, optionally, the abnormal region evaluation network includes a tooth position recognition network and an abnormal region recognition network. The abnormal result determination unit includes:

[0151] A tooth position result determination subunit, configured to input the region feature information of at least one interested region of each of the dental diagnosis and treatment feature maps into the tooth position recognition network, to obtain the tooth position bounding box information and tooth position classification result corresponding to each tooth in the dental diagnosis and treatment image;

[0152] An abnormal region result determination subunit, configured to input the region feature information of at least one interested region of each of the dental diagnosis and treatment feature maps into the abnormal region recognition network, to obtain the abnormal region mask image, abnormal region bounding box information, and abnormal region classification result corresponding to the suspected abnormal region in the dental diagnosis and treatment image;

[0153] An abnormal result determination subunit, configured to determine the tooth position classification result to which the abnormal region belongs based on the abnormal region bounding box information, the tooth position bounding box information and tooth position classification result corresponding to each tooth.

[0154] Based on the above embodiments, optionally, the dental abnormal region detection device further includes: a model training module; the model training module includes:

[0155] A sample image acquisition unit, configured to acquire a plurality of sample images containing annotation information; wherein, the annotation information includes the tooth annotation information corresponding to each tooth in the sample image and the abnormal region annotation information corresponding to the abnormal region on the tooth. The tooth annotation information includes the first annotation bounding box, the first annotation mask image, and the tooth position classification label corresponding to each tooth, and the abnormal region annotation information includes the second annotation bounding box, the second annotation mask image, and the abnormal classification label corresponding to the tooth abnormal region;

[0156] A prediction information determination unit for each of the sample images, inputting the current sample image into the to-be-trained dental anomaly detection model for image processing, to obtain dental prediction information corresponding to each tooth in the current sample image and anomaly region prediction information corresponding to the abnormal regions on the teeth; wherein, the dental prediction information includes a first predicted bounding box, a first predicted mask image, and a predicted tooth position classification corresponding to each tooth, and the anomaly region prediction information includes a second predicted bounding box, a second predicted mask image, and a predicted anomaly classification corresponding to the dental anomaly region;

[0157] A tooth loss determination unit for determining a tooth result loss value corresponding to the current sample image based on the tooth annotation information and the dental prediction information;

[0158] An anomaly region loss determination unit for determining an anomaly region result loss value corresponding to the current sample image based on the anomaly region annotation information and the anomaly region prediction information;

[0159] A detection model training unit for training the to-be-trained dental anomaly detection model based on the tooth result loss value and the anomaly region result loss value of each of the sample images, until the training ends when a preset condition is satisfied, to obtain the dental anomaly detection model.

[0160] Based on the above embodiments, optionally, the tooth loss determination unit includes:

[0161] A first loss determination subunit for determining a tooth bounding box loss value corresponding to the sample image based on the offset between the first annotated bounding box and the first predicted bounding box corresponding to each tooth, determining a tooth mask loss value corresponding to the sample image based on the first annotated mask image and the first predicted mask image corresponding to each tooth, and determining a tooth position classification loss value corresponding to the sample image based on the tooth position classification label and the predicted tooth position classification corresponding to each tooth;

[0162] A tooth position constraint loss determination subunit for determining a tooth position constraint loss value corresponding to the sample image based on the predicted tooth position classification, the first predicted bounding box corresponding to each tooth, and a preset tooth position constraint function;

[0163] A tooth loss determination subunit for determining a tooth result loss value corresponding to the current sample image based on the bounding box loss value, the tooth mask loss value, the tooth position classification loss value, and the tooth position constraint loss value.

[0164] Based on the above embodiments, optionally, the preset tooth position constraint function is:

[0165]

[0166] In the formula, represents the tooth position constraint loss value, represents the preset position constraint matrix, represents the distance between the central coordinates of the first predicted bounding box of the th tooth and the central coordinates of the first predicted bounding box of the

[0167] Based on the above embodiments, optionally, the abnormal region loss determination unit includes:

[0168] The bounding box loss determination subunit is configured to determine, for each of the abnormal regions, an abnormal region bounding box loss value corresponding to the sample image based on the offsets of the second labeled bounding box and the second predicted bounding box corresponding to the current abnormal region;

[0169] The mask loss determination subunit is configured to determine an abnormal region mask loss value corresponding to the sample image based on the second labeled mask image and the second predicted mask image;

[0170] The region classification loss determination subunit is configured to determine an abnormal region classification loss value corresponding to the sample image based on the abnormal classification label and the predicted abnormal classification.

[0171] The region result loss determination subunit is configured to determine an abnormal region result loss value corresponding to the current sample image based on the abnormal region bounding box loss value, the abnormal region mask loss value, and the abnormal region classification loss value corresponding to each of the abnormal regions.

[0172] Based on the above embodiments, optionally, the abnormal region classification result includes at least one of a prosthesis region category, a root filling region category, a high-density lesion region category, a low-density lesion region category, a supernumerary tooth region category, a fracture line region category, a dental caries region category, and a vacancy region category.

[0173] The technical solution of the embodiment of the present invention obtains a dental diagnosis and treatment image including at least one tooth, and then inputs the dental diagnosis and treatment image into a dental abnormality detection model to obtain an abnormal region mask image corresponding to the suspected abnormal region in the dental diagnosis and treatment image, abnormal region bounding box information, an abnormal region classification result, and a tooth position classification result to which the abnormal region belongs. The dental abnormality detection model includes: a feature extraction network for extracting a dental diagnosis and treatment feature map from the dental diagnosis and treatment image, a region of interest extraction network for extracting a region of interest from the dental diagnosis and treatment feature map, a region feature alignment network for determining region feature information of the region of interest, and an abnormal region evaluation network for determining an abnormal region detection result based on the region feature information of each region of interest. The technical solution provided by the embodiment of the present invention can automatically detect abnormal regions in a dental diagnosis and treatment image through the dental abnormality detection model, and can not only identify the mask image, bounding box, and abnormal region classification result corresponding to the abnormal region, but also identify the tooth position information where the abnormal region is located, improving the detection accuracy, reliability, and efficiency of the dental abnormal region.

[0174] The dental abnormal region detection device provided by the embodiment of the present invention can execute the dental abnormal region detection method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0175] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiment of the present invention.

[0176] Embodiment 4

[0177] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Figure 6 It shows a block diagram of an exemplary electronic device 40 suitable for implementing the embodiment mode of the embodiment of the present invention. Figure 6 The shown electronic device 40 is only an example and should not bring any limitation to the functions and usage scope of the embodiment of the present invention.

[0178] As Figure 6 shown, the electronic device 40 is presented in the form of a general-purpose computing device. The components of the electronic device 40 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).

[0179] The bus 403 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, a processor bus, or a local bus using any of the various bus architectures. By way of example, and without limitation, these architectures include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0180] The electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 40, including both volatile and nonvolatile media, removable and non-removable media.

[0181] The system memory 402 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. The electronic device 40 can further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, a storage system 406 can be provided for reading from and writing to non-removable, nonvolatile magnetic media ( Figure 6 not shown and typically called a "hard disk drive"). Although Figure 6 not shown in the figures, a disk drive for reading from and writing to a removable nonvolatile disk (e.g., a "floppy disk"), and an optical disk drive for reading from and writing to a removable nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) can be provided. In these instances, each drive can be connected to the bus 403 by one or more data media interfaces. The system memory 402 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of the embodiments of the present invention.

[0182] A program / utility 408 having a set (at least one) of program modules 407 can be stored, for example, in the system memory 402. Such program modules 407 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a network environment. The program modules 407 generally carry out the functions and / or methods of the embodiments described herein.

[0183] The electronic device 40 can also communicate with one or more external devices 409 (such as a keyboard, a pointing device, a display 410, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 40, and / or communicate with any device that enables the electronic device 40 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 411. Moreover, the electronic device 40 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 412. As shown in the figure, the network adapter 412 communicates with other modules of the electronic device 40 through a bus 403. It should be understood that although Figure 6 not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0184] The processing unit 401 executes various functional applications and page processing by running programs stored in the system memory 402, for example, implementing the tooth anomaly region detection method provided by the embodiments of the present invention.

[0185] In particular, according to the embodiments of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present invention include a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program codes for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the I / O interface 411, or installed from a storage system 406. When the computer program is executed by the processing unit 401, the above functions defined in the method of the embodiments of the present invention are executed.

[0186] Embodiment Five

[0187] The embodiments of the present invention also provide a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute the tooth anomaly region detection method when executed by a computer processor. The method includes:

[0188] Obtain a dental diagnosis and treatment image to be detected; wherein, at least one tooth is included in the dental diagnosis and treatment image;

[0189] Input the dental diagnosis image into a dental anomaly detection model to obtain an anomaly region detection result corresponding to a suspected anomaly region in the dental diagnosis image; wherein, the anomaly region detection result includes an anomaly region mask image, anomaly region bounding box information, an anomaly region classification result, and a dental position classification result to which the anomaly region belongs.

[0190] Wherein, the dental anomaly detection model includes: a feature extraction network for extracting a dental diagnosis feature map from the dental diagnosis image, a region of interest extraction network for extracting a region of interest from the dental diagnosis feature map, a region feature alignment network for determining region feature information of the region of interest, and an anomaly region evaluation network for determining the anomaly region detection result according to the region feature information of each region of interest.

[0191] The computer storage medium of the embodiments of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device.

[0192] The computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0193] The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0194] Computer program code for performing the operations of the embodiments of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or, may be connected to an external computer (e.g., connected through the Internet using an Internet service provider).

[0195] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments here. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for detecting abnormal tooth areas, characterized in that: include: Acquire a tooth diagnosis image to be detected; wherein the tooth diagnosis image includes at least one tooth; The dental diagnosis and treatment image is input into a dental abnormality detection model to obtain an abnormal area detection result corresponding to a suspected abnormal area in the dental diagnosis and treatment image; wherein the abnormal area detection result includes an abnormal area mask image, abnormal area bounding box information, an abnormal area classification result, and a tooth position classification result to which the abnormal area belongs; Among them, the dental abnormality detection model includes: a feature extraction network for extracting a dental diagnosis feature map based on the dental diagnosis image, a region of interest extraction network for extracting a region of interest from the dental diagnosis feature map, a region feature alignment network for determining the region feature information of the region of interest, and an abnormal region evaluation network for determining the abnormal region detection result based on the region feature information of each region of interest.

2. The method according to claim 1, characterized in that The step of inputting the dental treatment image into a dental abnormality detection model to obtain an abnormal area detection result corresponding to a suspected abnormal area in the dental treatment image includes: Inputting the dental diagnosis and treatment image into the feature extraction network to obtain at least one dental diagnosis and treatment feature map corresponding to the dental diagnosis and treatment image; For each of the tooth diagnosis and treatment feature maps, inputting the current tooth diagnosis and treatment feature map into the region of interest extraction network to obtain at least one region of interest corresponding to the current tooth diagnosis and treatment feature map; Inputting the current tooth diagnosis and treatment feature map and at least one region of interest corresponding to the current tooth diagnosis and treatment feature map into a regional feature alignment network to obtain regional feature information corresponding to each region of interest; The regional feature information of at least one region of interest of each of the dental diagnosis and treatment feature images is input into the abnormal region evaluation network to obtain the abnormal region mask image, abnormal region boundary box information, abnormal region classification result and tooth position classification result to which the abnormal region belongs corresponding to the suspected abnormal region in the dental diagnosis and treatment image.

3. The method according to claim 2, characterized in that The feature extraction network includes a backbone network and a feature pyramid network. The step of inputting the dental diagnosis image into the feature extraction network to obtain at least one dental diagnosis feature map corresponding to the dental diagnosis image includes: Inputting the dental diagnosis and treatment image into the backbone network to obtain at least one first feature map corresponding to the dental diagnosis and treatment image; Each of the first feature maps is input into the feature pyramid network to obtain at least one tooth diagnosis feature map corresponding to the tooth diagnosis image.

4. The method according to claim 2, characterized in that: The region of interest extraction network includes a candidate region extraction network and a candidate region screening network. For each of the tooth diagnosis and treatment feature maps, the current tooth diagnosis and treatment feature map is input into the region of interest extraction network to obtain at least one region of interest corresponding to the current tooth diagnosis and treatment feature map, including: For each of the tooth diagnosis and treatment feature maps, inputting the current tooth diagnosis and treatment feature map into the candidate region extraction network to obtain at least one candidate region corresponding to the current tooth diagnosis and treatment feature map; At least one candidate region corresponding to the current dental diagnosis and treatment feature map is region screened based on the candidate region screening network to obtain at least one region of interest corresponding to the current dental diagnosis and treatment feature map.

5. The method according to claim 2, characterized in that: The abnormal area assessment network includes a tooth position recognition network and an abnormal area recognition network. The region feature information of at least one region of interest of each of the tooth diagnosis and treatment feature images is input into the abnormal area assessment network to obtain an abnormal area mask image corresponding to the suspected abnormal area in the tooth diagnosis and treatment image, abnormal area boundary box information, abnormal area classification results, and tooth position classification results to which the abnormal area belongs, including: Inputting the regional feature information of at least one region of interest of each of the dental diagnosis and treatment feature images into the tooth position recognition network to obtain the tooth position bounding box information and tooth position classification result corresponding to each tooth in the dental diagnosis and treatment image; Inputting the regional feature information of at least one region of interest of each of the dental diagnosis and treatment feature images into the abnormal region recognition network, obtaining an abnormal region mask image, abnormal region bounding box information and abnormal region classification result corresponding to the suspected abnormal region in the dental diagnosis and treatment image; The tooth position classification result to which the abnormal region belongs is determined based on the abnormal region boundary box information, the tooth position boundary box information corresponding to each of the teeth, and the tooth position classification result.

6. The method according to claim 1, characterized in that Also includes: Acquire multiple sample images containing annotation information; wherein the annotation information includes tooth annotation information corresponding to each tooth in the sample image and abnormal region annotation information corresponding to the abnormal region on the tooth, the tooth annotation information includes a first annotation bounding box corresponding to each tooth, a first annotation mask image, and a tooth position classification label, and the abnormal region annotation information includes a second annotation bounding box corresponding to the abnormal region of the tooth, a second annotation mask image, and an abnormal classification label; For each of the sample images, the current sample image is input into the tooth abnormality detection model to be trained for image processing, and the tooth prediction information corresponding to each tooth in the current sample image and the abnormal area prediction information corresponding to the abnormal area on the tooth are obtained; wherein the tooth prediction information includes a first prediction bounding box, a first prediction mask image, and a predicted tooth position classification corresponding to each tooth, and the abnormal area prediction information includes a second prediction bounding box, a second prediction mask image, and a predicted abnormality classification corresponding to the tooth abnormal area; Determine a tooth result loss value corresponding to the current sample image based on the tooth annotation information and the tooth prediction information; Based on the abnormal region annotation information and the abnormal region prediction information, determining the abnormal region result loss value corresponding to the current sample image; Based on the tooth result loss value and the abnormal area result loss value of each sample image, the tooth abnormality detection model to be trained is trained until the training is completed when a preset condition is met, thereby obtaining the tooth abnormality detection model.

7. The method according to claim 6, characterized in that The determining, based on the tooth annotation information and the tooth prediction information, a tooth result loss value corresponding to the current sample image comprises: Determine a tooth bounding box loss value corresponding to the sample image based on the offset of the first annotated bounding box corresponding to each tooth and the offset of the first predicted bounding box; determine a tooth mask loss value corresponding to the sample image based on the first annotated mask image corresponding to each tooth and the first predicted mask image; determine a tooth position classification loss value corresponding to the sample image based on the tooth position classification label corresponding to each tooth and the predicted tooth position classification; Determine a tooth position constraint loss value corresponding to the sample image based on the predicted tooth position classification corresponding to each of the teeth, the first predicted bounding box, and a preset tooth position constraint function; Based on the bounding box loss value, the tooth mask loss value, the tooth position classification loss value and the tooth position constraint loss value, a tooth result loss value corresponding to the current sample image is determined.

8. The method according to claim 7, characterized in that The preset tooth position constraint function is: In the formula, represents the tooth position constraint loss value, represents the preset position constraint matrix, Indicates The center coordinates of the first predicted bounding box of the tooth are The distance from the center coordinates of the first predicted bounding box of the tooth.

9. The method according to claim 6, characterized in that The determining, based on the abnormal region labeling information and the abnormal region prediction information, the abnormal region result loss value corresponding to the current sample image includes: For each of the abnormal regions, based on an offset of the second annotated bounding box corresponding to the current abnormal region and an offset of the second predicted bounding box, determining a loss value of a bounding box of the abnormal region corresponding to the sample image; Determining a mask loss value of an abnormal area corresponding to the sample image based on the second annotated mask image and the second predicted mask image; Based on the abnormal classification label and the predicted abnormal classification, determining the abnormal region classification loss value corresponding to the sample image; Based on the abnormal region bounding box loss value, the abnormal region mask loss value, and the abnormal region classification loss value corresponding to each of the abnormal regions, the abnormal region result loss value corresponding to the current sample image is determined.

10. The method according to claim 1, characterized in that The abnormal area classification results include: at least one of a restoration area category, a root filling area category, a high-density lesion area category, a low-density lesion area category, an extra tooth area category, a fracture line area category, a caries area category and a vacant area category.

Images