Abnormal tooth area identification method, training method, device, equipment and medium

Abstract

The present application provides a method, training method, device, equipment and medium for identifying abnormal tooth areas. The method for identifying abnormal tooth areas includes: obtaining the texture in the first tooth model to be identified and the geometric information of the three-dimensional points in the first tooth model; projecting the texture in the first tooth model and the geometric information of the three-dimensional points in the first tooth model respectively to obtain the texture map and geometric information map of the first tooth model; superimposing the texture map and geometric information map of the first tooth model into a first target image; using a pre-trained abnormal tooth area recognition model to perform abnormal tooth area recognition processing on the first target image to obtain an abnormal area recognition result corresponding to the first target image. The efficiency and accuracy of abnormal tooth area recognition are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for identifying a tooth abnormality region, a training method for a tooth abnormality region identification model, a tooth abnormality region identification device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the digital oral scanning scenario, the doctor will compile and generate the corresponding oral examination report based on the patient's 3D dental model in order to analyze and summarize the patient's oral problems. This process usually requires the doctor to identify and judge oral diseases based on his or her own experience, and manually take screenshots and feed back to the patient. The operation is relatively cumbersome and requires a certain amount of time and manpower costs. In addition, the 3D dental model usually requires the doctor to manually rotate and observe it, and abnormal areas are identified during this process, which may result in false detection or missed detection of abnormal areas. Summary of the invention

[0003] In view of this, the present application provides a method for identifying a tooth abnormal area, a training method for a tooth abnormal area identification model, a tooth abnormal area identification device, an electronic device and a computer-readable storage medium.

[0004] Specifically, the present application is implemented through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present application, a method for identifying an abnormal tooth region is provided, comprising:

[0006] Acquire the texture in the first tooth model to be identified and the geometric information of the three-dimensional points in the first tooth model;

[0007] Projecting the texture in the first tooth model and the geometric information of the three-dimensional points in the first tooth model respectively to obtain a texture map and a geometric information map of the first tooth model;

[0008] superimposing the texture map and the geometric information map of the first tooth model into a first target image;

[0009] Using a pre-trained tooth abnormality region recognition model to perform tooth abnormality region recognition processing on the first target image, to obtain an abnormal region recognition result corresponding to the first target image;

[0010] Among them, the tooth abnormality area recognition model is obtained through supervised training based on several training samples, and the training samples include a second target image of a second tooth model and an abnormal area label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second tooth model.

[0011] According to a second aspect of an embodiment of the present application, a method for training a tooth abnormality region recognition model is provided, comprising:

[0012] Acquire a plurality of training samples, wherein the training samples include a second target image of a second tooth model and an abnormal region label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second tooth model; different training samples correspond to different second tooth models;

[0013] Inputting the training sample into the tooth abnormality region recognition model to be trained, so that the tooth abnormality region recognition model to be trained performs tooth abnormality region recognition based on the second target image to obtain an abnormal region recognition result corresponding to the second target image, and training the tooth abnormality region recognition model to be trained with minimizing the difference between the abnormal region recognition result and the label in the training sample as the optimization goal;

[0014] Among them, the trained tooth abnormality area recognition model is applied to the tooth abnormality area recognition method mentioned in the first aspect.

[0015] According to a third aspect of an embodiment of the present application, a device for identifying an abnormal tooth region is provided, comprising:

[0016] A module for acquiring information of a model to be identified, used for acquiring texture in a first tooth model to be identified and geometric information of three-dimensional points in the first tooth model;

[0017] A projection module, used to project the texture in the first tooth model and the geometric information of the three-dimensional points in the first tooth model respectively to obtain a texture map and a geometric information map of the first tooth model;

[0018] A superposition module, used for superimposing the texture map and the geometric information map of the first tooth model into a first target image;

[0019] An abnormal region recognition module is used to perform tooth abnormal region recognition processing on the first target image using a pre-trained tooth abnormal region recognition model to obtain an abnormal region recognition result corresponding to the first target image;

[0020] Among them, the tooth abnormality area recognition model is obtained through supervised training based on several training samples, and the training samples include a second target image of a second tooth model and an abnormal area label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second tooth model.

[0021] According to a fourth aspect of an embodiment of the present application, a training device for a tooth abnormality region recognition model is provided, comprising:

[0022] A training sample acquisition module is used to acquire a number of training samples, wherein the training samples include a second target image of a second tooth model and an abnormal region label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second tooth model; different training samples correspond to different second tooth models;

[0023] A model training module, used for inputting the training samples into the tooth abnormal area recognition model to be trained, so that the tooth abnormal area recognition model to be trained can recognize the tooth abnormal area based on the second target image, obtain the abnormal area recognition result corresponding to the second target image, and train the tooth abnormal area recognition model to be trained with minimizing the difference between the abnormal area recognition result and the label in the training samples as the optimization goal;

[0024] Among them, the trained tooth abnormality area recognition model is applied to the tooth abnormality area recognition method mentioned in the first aspect.

[0025] According to a fifth aspect of an embodiment of the present application, there is provided an electronic device, including a memory, a processor, and executable instructions stored in the memory and executable on the processor;

[0026] Wherein, when the processor executes the executable instructions, the steps of the method mentioned in the first aspect are implemented.

[0027] According to a sixth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the steps of the method mentioned in the first aspect are implemented.

[0028] According to a seventh aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method mentioned in the first aspect when executed by a processor.

[0029] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0030] The training method of the tooth abnormality area recognition model provided in the embodiment of the present application provides a rich training data set for model training by obtaining a second target image obtained by superimposing texture maps and geometric information maps of different second tooth models, and abnormal area labels corresponding to the second target image, so that the model can undergo sufficient training and learn to recognize various tooth abnormalities on tooth models with different characteristics, which is conducive to improving the training and recognition effects of the model.

[0031] The method for identifying abnormal tooth areas provided in the embodiment of the present application comprehensively considers the texture of the teeth and the geometric information of the three-dimensional points, and uses the abnormal tooth area recognition model to automatically identify the abnormal tooth areas, thereby significantly improving the efficiency and accuracy of abnormal tooth area recognition and providing strong auxiliary support for doctors.

[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in the specification and constitute a part of this application, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0034] Figure 1 It is a flowchart of a method for training a tooth abnormal area recognition model shown in an exemplary embodiment of the present application.

[0035] Figure 2 It is a schematic diagram of a second tooth model and a texture map and a geometric information map obtained by a three-dimensional mesh texture expansion algorithm shown in an exemplary embodiment of the present application.

[0036] Figure 3 It is a schematic diagram of a second image corresponding to the curvature, depth and normal vector of a three-dimensional point and a fused geometric information graph shown in an exemplary embodiment of the present application.

[0037] Figure 4 It is a schematic diagram of the labeling process shown in an exemplary embodiment of the present application.

[0038] Figure 5 It is a schematic diagram of deformation of abnormal area of ​​teeth when texture is unfolded, shown in an exemplary embodiment of the present application.

[0039] Figure 6a It is a schematic diagram of a first switching operation switching process shown in an exemplary embodiment of the present application.

[0040] Figure 6b 1 is a schematic diagram of a switching process of a second switching operation shown in an exemplary embodiment of the present application.

[0041] Figure 7 It is a schematic diagram showing an exemplary embodiment of the present application, which shows back-projecting a two-dimensional tooth image superimposed with abnormal area indication information into a three-dimensional space.

[0042] Figure 8 It is a flowchart of a method for identifying abnormal tooth areas according to an exemplary embodiment of the present application.

[0043] Fig. 9It is a two-dimensional screenshot of multiple three-dimensional points belonging to abnormal areas generated according to an exemplary embodiment of the present application.

[0044] Fig.10 It is a schematic diagram of the structure of a device for identifying abnormal tooth areas shown in an exemplary embodiment of the present application.

[0045] Fig.11 It is a structural schematic diagram of a training device for a tooth abnormal area recognition model shown in an exemplary embodiment of the present application.

[0046] Fig.12 It is a structural schematic diagram of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0049] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0050] In digital oral scanning scenarios, for example, in the detection and identification process of common oral diseases such as caries, tartar, and pigmentation, doctors will compile and generate corresponding oral inspection reports based on the patient's 3D tooth model in order to analyze and summarize the patient's oral problems. This process usually requires doctors to manually rotate the 3D tooth model, identify and judge oral diseases based on their own experience, and manually take screenshots and feed back to the patient. The operation is cumbersome and requires a certain amount of time and manpower costs, and there is a possibility of abnormal areas being misdetected or missed.

[0051] Here, the process of generating a 3D tooth model is described by way of example. An oral scanner can be used to obtain a 3D tooth model of a patient. A dentist operates the oral scanner and slowly moves the scanning head of the oral scanner in the patient's mouth to capture the image of the teeth. The scanning head emits a low-power light beam to illuminate the teeth. After the light beam is reflected back, it is captured by the scanner's sensor and the precise position of each point on the tooth surface is calculated through the triangulation principle. A 3D tooth model can be generated based on the data collected by the scanner. However, this is not limited to the above, and other methods can also be used to obtain a 3D tooth model.

[0052] Based on this, the embodiment of the present application provides a training method for a tooth abnormality area recognition model and a tooth abnormality area recognition method. By comprehensively considering the texture of the teeth and the geometric information of the three-dimensional points, and using the tooth abnormality area recognition model to automatically identify the tooth abnormality area, the efficiency and accuracy of tooth abnormality area recognition are significantly improved, providing strong auxiliary support for doctors.

[0053] Next, the training process of the tooth abnormality area recognition model is illustrated as follows:

[0054] The training method of the dental abnormality area recognition model provided in the embodiment of the present application can be executed by electronic devices, including but not limited to physical servers, server machines, cloud servers, smart phones / mobile phones, tablet computers, personal digital assistants (PDAs), laptop computers, and desktop computers.

[0055] See also Figure 1 , shows a flow chart of a method for training a tooth abnormality region recognition model, the method comprising:

[0056] In S101, a number of training samples are obtained, the training samples including a second target image of a second tooth model and an abnormal area label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second tooth model; different training samples correspond to different second tooth models.

[0057] In this embodiment, the model training process comprehensively considers the texture of the tooth model and the geometric information of the three-dimensional points. By superimposing the texture map and the geometric information map, a more comprehensive description of the tooth features is provided for the second tooth model, which significantly improves the accuracy of the model in identifying abnormal areas. Specifically, the texture map can reflect the visual features of the tooth surface, such as color, texture pattern, brightness, etc. These features are crucial for distinguishing normal tooth tissue from abnormal areas. For example, abnormal areas may appear as color changes, irregular textures, or abnormal brightness. The geometric information map can provide the spatial structure and shape characteristics of the teeth, which helps the model understand the structural and morphological changes of the teeth and perform deeper feature extraction, so as to accurately locate and identify abnormal areas. For example, abnormal areas may appear as uneven teeth or abnormal shapes. Through this comprehensive consideration, the false detection of abnormal areas (for example, only considering the tooth texture to misidentify black pigmentation as caries) and missed detection are effectively avoided, the overall performance of the model is improved, and strong technical support is provided for clinical applications.

[0058] In S102, the training samples are input into the tooth abnormality area recognition model to be trained, so that the tooth abnormality area recognition model to be trained performs tooth abnormality area recognition based on the second target image to obtain the abnormal area recognition result corresponding to the second target image, and the tooth abnormality area recognition model to be trained is trained with minimizing the difference between the abnormal area recognition result and the label in the training samples as the optimization goal; wherein the trained tooth abnormality area recognition model is used to perform tooth abnormality area recognition on the first target image of the first tooth model to obtain the abnormal area recognition result corresponding to the first target image.

[0059] In this embodiment, by obtaining the second target image obtained by superimposing the texture map and geometric information map of different second tooth models, and the abnormal area label corresponding to the second target image, a rich training data set is provided for model training, so that the model can be fully trained and learn to identify various tooth abnormalities on tooth models with different characteristics, which is conducive to improving the training and recognition effect of the model. The trained tooth abnormal area recognition model can quickly process the target image obtained by superimposing the texture map and geometric information map of the tooth model, automatically identify potential tooth abnormal areas, provide strong auxiliary support for doctors, and greatly improve the efficiency and accuracy of tooth abnormal area recognition.

[0060] It can be understood that, for the sake of convenience, in the embodiment of the present application, the tooth model used as a training sample in the model training process is called the second tooth model, and the tooth model to be identified in the model application process is called the first tooth model.

[0061] Exemplarily, the electronic device can obtain the texture in the second dental model to be identified and the geometric information of the three-dimensional points in the second dental model, and then project the texture in the second dental model and the geometric information of the three-dimensional points in the second dental model from the three-dimensional space to the two-dimensional plane, respectively, to obtain a texture map and a geometric information map of the second dental model.

[0062] However, in the above process, due to improper selection of observation or projection angle, the second tooth model may have visual blind spots, making the texture and geometric information of the three-dimensional points obtained therefrom incomplete and unable to be projected onto the two-dimensional plane. As a result, it is impossible to obtain a complete texture map and geometric information map, resulting in the omission of abnormal tooth areas within a small range such as adjacent caries, increasing the risk of missed detection of abnormal areas.

[0063] Therefore, in some embodiments, in order to avoid the problem of missed detection of tooth abnormality areas and improve the accuracy and reliability of tooth abnormality area identification, the second tooth model may include a number of three-dimensional grids, and the electronic device determines the coordinate mapping relationship from the three-dimensional space to the two-dimensional space based on a preset three-dimensional grid texture expansion algorithm. Then, the electronic device projects the texture in the second tooth model into the two-dimensional space based on the coordinate mapping relationship to obtain a texture map of the second tooth model; and, based on the coordinate mapping relationship, projects the geometric information of the three-dimensional points in the second tooth model into the two-dimensional space to obtain a geometric information map of the second tooth model.

[0064] Among them, the 3D mesh texture expansion algorithm can project all the texture and geometric information of the 3D points on the second tooth model into the 2D space through the precise coordinate mapping relationship, and lay it on the 2D plane, so as to ensure that the texture and geometric information of the entire tooth model can be fully observed in the 2D plane, thereby avoiding the problem of missing detection of abnormal tooth areas. The 3D mesh texture expansion algorithm can include but is not limited to the LSCM algorithm, ABF algorithm, discrete conformal mapping algorithm, global parameterization algorithm, etc.

[0065] See for example Figure 2 , showing a second tooth model and a schematic diagram of a texture map and a geometric information map obtained by a three-dimensional mesh texture unfolding algorithm.

[0066] Exemplarily, the three-dimensional points in the second tooth model include at least two types of geometric information; the electronic device can obtain the geometric information map of the second tooth model in the following manner: for each type of geometric information of the three-dimensional points, the electronic device can project the geometric information of this type into a two-dimensional space based on a coordinate mapping relationship and map it to one of the color channels of a specified color mode, thereby obtaining a color map corresponding to the geometric information of this type; wherein different types of geometric information of the three-dimensional points correspond to different color channels. Then, the electronic device fuses the color maps corresponding to all types of geometric information of the three-dimensional points in the second tooth model, thereby obtaining the geometric information map of the second tooth model.

[0067] The specified color mode includes but is not limited to RGB (red, green, blue) mode, CMYK (cyan, magenta, yellow, black) mode, and HSV / HSB (hue, saturation, brightness) mode, etc., and can be selected based on the actual application scenario.

[0068] Exemplarily, the geometric information of the 3D point includes at least one of the following types: the curvature, depth and normal vector of the 3D point. The curvature of the 3D point is usually the average of the maximum and minimum curvatures of the 3D point, the depth of the 3D point is the depth relative to a reference object (such as a camera), and the normal vector of the 3D point is the normal vector relative to the gum base.

[0069] The curvature, depth, and normal vector of a 3D point can be mapped to different color channels of the RGB model respectively.

[0070] For example, after acquiring the curvature, depth and normal vector of the three-dimensional point in the second tooth model, the electronic device can project the curvature, depth and normal vector of the three-dimensional point into the two-dimensional space based on the coordinate mapping relationship, and obtain a first image corresponding to the curvature of the three-dimensional point, a first image corresponding to the depth of the three-dimensional point, and a first image corresponding to the normal vector of the three-dimensional point.

[0071] Then, the electronic device maps the first image corresponding to the curvature of the three-dimensional point to the green channel of the RGB mode to obtain the second image corresponding to the curvature of the three-dimensional point; maps the first image corresponding to the depth of the three-dimensional point to the red channel of the RGB mode to obtain the second image corresponding to the depth of the three-dimensional point; and maps the first image corresponding to the normal vector of the three-dimensional point to the blue channel of the RGB mode to obtain the second image corresponding to the normal vector of the three-dimensional point. It can be understood that all the second images are single-channel images.

[0072] After obtaining the second images corresponding to the curvature, depth, and normal vector of the three-dimensional point, the second images are fused to obtain a geometric information map of the second tooth model. For example, the second image corresponding to the curvature of the three-dimensional point is a1, the second image corresponding to the depth of the three-dimensional point is b1, and the second image corresponding to the normal vector of the three-dimensional point is c1. A1, b1, and c1 are fused into a geometric information map.

[0073] See for example Figure 3 , showing a second image corresponding to the curvature, depth and normal vector of a three-dimensional point respectively and a schematic diagram of a fused geometric information graph.

[0074] For each training sample, the electronic device can superimpose the texture map in the training sample and the geometric information map corresponding to the texture map into a second target image, and the second target image is represented by at least four channels, of which three channels are used to describe the texture map, and the remaining at least one channel is used to describe the geometric information map corresponding to the texture map. The number of channels of the second target image is positively correlated with the type of geometric information of the three-dimensional point. The more types of geometric information, the more channels of the second target image.

[0075] In one example, the geometric information of the 3D point includes the following three types: the curvature, depth and normal vector of the 3D point. Then the second target image is represented by six channels, of which three channels are used to describe the texture map, and the remaining three channels are used to describe the geometric information map corresponding to the texture map.

[0076] In some embodiments, the process of obtaining labels for indicating abnormal regions in training samples is exemplarily described as follows:

[0077] The electronic device can display the second target image in the training sample in the display interface, so that it can intuitively and clearly display the tooth image in a two-dimensional perspective, which is convenient for users to confirm and mark the abnormal area and avoid missing the abnormal area. Furthermore, the user can determine the location and abnormal type of the abnormal area in the second target image based on professional experience and mark it. The electronic device responds to the marking operation of the second target image and obtains the label in the training sample; wherein the marking operation includes the abnormal area drawing operation and the abnormal type selection operation; the label includes the location of the abnormal area and the abnormal type to which the abnormal area belongs.

[0078] Exemplarily, the user can draw the abnormal area and select the lesion type by means of a mouse, keyboard, stylus or skin touch. The area drawn by the user is the location of the marked abnormal area, and the selected abnormal type is the abnormal type to which the marked abnormal area belongs. The marking method can be to draw the area on the second target image using shapes such as points or bounding boxes, and the marking tool can be software such as labelme. The marking method provided in this embodiment is simple and direct, which is conducive to simplifying the user's marking steps and improving marking efficiency.

[0079] See for example Figure 4 , which shows a schematic diagram of the labeling process.

[0080] In the process of obtaining labels using the above-mentioned annotation method, it is found that when the texture of the three-dimensional tooth model is expanded to obtain a texture map, some abnormal areas of the teeth will be deformed, and distortion will appear on the second target image. For example, see Figure 5 , shows a schematic diagram of the deformation of the abnormal area of ​​the tooth when the texture is expanded. This phenomenon may affect the user's judgment of the abnormal area when marking, resulting in mislabeling or missing of the abnormal area.

[0081] Therefore, in some embodiments, in order to improve the accuracy and efficiency of user labeling, the electronic device may also respond to the first switching operation before responding to the labeling operation on the second target image, and switch the second target image displayed in the display interface to a second dental model corresponding to the second target image, so that the user can first preliminarily determine the possible dental abnormality area based on the second target image, and then switch to the three-dimensional perspective of the dental model to make an accurate abnormal area judgment, thereby avoiding misjudgment or omission of abnormal areas.

[0082] Then, the electronic device responds to the second switching operation and switches the second tooth model displayed in the display interface to the second target image, wherein the observation perspective of the second target image in the display interface is the same as the observation perspective of the second tooth model in the display interface, so that the user can accurately determine the abnormal area in the three-dimensional perspective, and can promptly switch back to the corresponding two-dimensional second target image for labeling, avoiding the tedious steps of constantly switching perspectives when the user is performing labeling work in the three-dimensional perspective throughout the process, effectively improving the labeling efficiency, and saving labeling time and labor costs.

[0083] Exemplarily, the user can perform the first switching operation and the second switching operation by mouse click, keyboard shortcut, stylus or skin touch, etc. In the first switching operation and the second switching operation, the corresponding second tooth model and the center of the field of view of the second target image can be specifically switched to facilitate abnormal area determination and marking.

[0084] See for example Figure 6a and Figure 6b , respectively show the switching process schematic diagrams of the first switching operation and the second switching operation.

[0085] After obtaining the training samples based on the above method, illustratively, the electronic device can input the training samples into the tooth abnormality region recognition model to be trained, and execute the following process in a loop until the optimization target is met: using the current tooth abnormality region recognition model to recognize the tooth abnormality region based on the second target image of the second tooth model, and obtaining the abnormal region recognition result corresponding to the second target image; judging whether the optimization target is met based on the difference between the abnormal region recognition result and the corresponding label; if the optimization target is not met, adjusting the model parameters of the current tooth abnormality region recognition model based on the difference between the abnormal region recognition result and the corresponding label, and using the adjusted tooth abnormality region recognition model as the current tooth abnormality region recognition model in the next cycle. Among them, to determine whether the optimization target is met, the loss function of the model can be used for judgment. The loss function can include but is not limited to the cross entropy loss function, the Dice loss function, the mean square error loss function, the IoU loss function (Intersection over Union Loss) or the bounding box regression loss, etc. This embodiment does not limit the loss function used in the training process.

[0086] In addition, in some embodiments, in order to more intuitively display the abnormal area recognition results obtained through the tooth abnormal area recognition model, the abnormal area recognition results may include a second target image superimposed with abnormal area indication information; wherein the abnormal area indication information may be a two-dimensional bounding box superimposed and displayed on the abnormal area; after obtaining the abnormal area recognition results, the electronic device may also back-project the second target image superimposed with the abnormal area indication information from the two-dimensional space to the three-dimensional space to determine the three-dimensional points in the second tooth model belonging to the abnormal area; then, when the second tooth model is displayed on the display interface, the visual information used to indicate the abnormal area is superimposed and displayed for the three-dimensional points belonging to the abnormal area.

[0087] Exemplarily, the abnormal region recognition result also includes the abnormal type to which each abnormal region belongs; the abnormal type includes at least one of wedge-shaped defect, dental calculus, pigmentation, dental caries, gingival redness and swelling, and gingival atrophy; wherein the visual information used to indicate abnormal regions belonging to different abnormal types is different. The visual information can take a variety of forms, such as different colors, shapes, textures, or marks, etc., and this embodiment does not limit the specific form of the visual information.

[0088] See for example Figure 7, showing a schematic diagram of back-projecting the second target image superimposed with abnormal area indication information into a three-dimensional space, wherein the colors indicating abnormal areas belonging to different abnormal types are different, such as red for caries, green for tartar, and blue for pigmentation.

[0089] It is understandable that the electronic device used to train the tooth abnormality region recognition model and the electronic device used to perform the tooth abnormality region recognition method can be the same device or different devices, and this embodiment does not impose any restrictions on this. For example, the tooth abnormality region recognition model can be trained by a certain electronic device, and then the trained tooth abnormality region recognition model can be transplanted to another electronic device used to perform the tooth abnormality region recognition method.

[0090] The following is an exemplary description of the method for identifying abnormal tooth areas provided in the embodiment of the present application:

[0091] See also Figure 8 , shows a flow chart of a method for identifying tooth abnormality areas, which can be performed by an electronic device, including but not limited to a physical server, a server machine, a cloud server, a smart phone / mobile phone, a tablet computer, a personal digital assistant (PDA), a laptop computer, and a desktop computer. The method includes:

[0092] In S801, the texture in the first tooth model to be identified and the geometric information of the three-dimensional points in the first tooth model are obtained.

[0093] In S802, the texture in the first tooth model and the geometric information of the three-dimensional points in the first tooth model are projected respectively to obtain a texture map and a geometric information map of the first tooth model.

[0094] In S803, the texture map and the geometric information map of the first tooth model are superimposed to form a first target image.

[0095] By obtaining the texture of the first tooth model and the geometric information of the three-dimensional points, and superimposing the corresponding texture map and geometric information map, a more comprehensive tooth feature description is provided for the first tooth model, effectively avoiding false detection and missed detection of abnormal areas caused by single tooth information, and significantly improving the recognition accuracy of abnormal areas.

[0096] In S804, a pre-trained dental abnormality area recognition model is used to perform dental abnormality area recognition processing on the first target image to obtain an abnormal area recognition result corresponding to the first target image; wherein the dental abnormality area recognition model is obtained through supervised training based on a number of training samples, and the training samples include a second target image of a second dental model, and an abnormal area label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second dental model.

[0097] Using the trained tooth abnormality area recognition model to identify tooth abnormality areas can effectively avoid the tedious manual operations in traditional manual recognition methods, reduce time and labor costs, and achieve fast and accurate automatic recognition and processing of tooth abnormality areas. At the same time, automated processing can also ensure the consistency and repeatability of the recognition process, providing doctors with fast and accurate diagnostic support, thereby improving overall clinical work efficiency and patient satisfaction.

[0098] The method for identifying abnormal tooth areas provided in this embodiment comprehensively considers the texture of the teeth and the geometric information of the three-dimensional points, and uses the abnormal tooth area recognition model to automatically identify abnormal tooth areas, thereby significantly improving the efficiency and accuracy of abnormal tooth area recognition and providing strong auxiliary support for doctors.

[0099] In some embodiments, with respect to S802, similar to the training process, the first tooth model may include a plurality of three-dimensional grids, and the electronic device determines the coordinate mapping relationship from the three-dimensional space to the two-dimensional space based on a preset three-dimensional grid texture expansion algorithm. Then, the electronic device projects the texture in the first tooth model to the two-dimensional space based on the coordinate mapping relationship to obtain a texture map of the first tooth model; and, based on the coordinate mapping relationship, projects the geometric information of the three-dimensional points in the first tooth model to the two-dimensional space to obtain a geometric information map of the first tooth model.

[0100] In this embodiment, the three-dimensional grid texture unfolding algorithm can project all the texture and geometric information of the three-dimensional points on the first tooth model into two-dimensional space through precise coordinate mapping relationships, and flatten it on a two-dimensional plane, ensuring that the texture and geometric information of the entire tooth model can be fully observed in a two-dimensional plane, thereby avoiding the problem of missed detection of abnormal tooth areas and effectively improving the accuracy and reliability of identifying abnormal tooth areas.

[0101] Exemplarily, the three-dimensional points in the first tooth model include at least two types of geometric information. In the process of obtaining the geometric information map of the first tooth model, similar to the training process, for each type of geometric information of the three-dimensional points, the electronic device can project the geometric information of this type into the two-dimensional space based on the coordinate mapping relationship and map it to one of the color channels of the specified color mode to obtain a color map corresponding to the type of geometric information; wherein different types of geometric information of the three-dimensional points correspond to different color channels. Then, the electronic device fuses the color maps corresponding to all types of geometric information of the three-dimensional points in the first tooth model, thereby obtaining the geometric information map of the first tooth model.

[0102] Exemplarily, the geometric information of the 3D point includes at least one of the following types: curvature, depth and normal vector of the 3D point; the specified color mode includes RGB mode, and the curvature, depth and normal vector of the 3D point can be mapped to different color channels of the RGB mode respectively.

[0103] In this embodiment, the electronic device corresponds each type of geometric information of the three-dimensional point to a different color channel. This color coding method helps to separate different types of geometric information so that effective data analysis and recognition can be performed in subsequent steps. Finally, the color maps corresponding to all types of geometric information are fused to obtain a geometric information map, so that the geometric information map can contain different types of geometric information of the same tooth area, and each type of geometric information is encoded with a different color channel. This comprehensive image can provide a comprehensive data perspective to help the tooth abnormal area recognition model accurately locate and analyze potential problems and lesions on the teeth.

[0104] In step S803, the texture map and the geometric information map of the first tooth model are superimposed into a first target image, wherein the first target image can be represented by at least four channels, three channels are used to describe the texture map, and the remaining at least one channel is used to describe the geometric information map corresponding to the texture map. The number of channels of the first target image is positively correlated with the type of geometric information of the three-dimensional point. The more types of geometric information, the more channels of the first target image.

[0105] In one example, the geometric information of the three-dimensional points in the first tooth model includes the following three types: curvature, depth and normal vector of the three-dimensional points. The first target image is represented by six channels, three of which are used to describe the texture map, and the remaining three channels are used to describe the geometric information map corresponding to the texture map.

[0106] In some embodiments, the process of performing dental abnormality area recognition processing on the first target image using a pre-trained dental abnormality area recognition model in step S804 to obtain an abnormal area recognition result corresponding to the first target image is exemplified: the electronic device inputs the first target image into the dental abnormality area recognition model, so that the dental abnormality area recognition model extracts color texture features and geometric features from the first target image, determines various parts in the oral environment by analyzing the geometric features, and classifies them based on the color texture features and geometric features corresponding to various parts in the oral environment to obtain an abnormal area recognition result corresponding to the first target image; wherein the various parts in the oral environment include at least one of the occlusal surface of the tooth, the non-occlusal surface of the tooth, the junction of the tooth and the gum, the gum, the labial side and the lingual side.

[0107] In this embodiment, the model identifies various parts of the oral environment and can comprehensively analyze dental health from multiple angles. Combining color texture features and geometric features, it can more accurately identify abnormal parts of teeth and the oral cavity, thereby improving the accuracy of diagnosis.

[0108] Exemplarily, the geometric features extracted from the first target image by the tooth abnormality region recognition model include at least one of depth features, curvature features, and normal vector features.

[0109] Depth features can help distinguish the occlusal and non-occlusal surfaces of teeth. The occlusal surface is the highest point of the tooth and the area in contact with the opposing teeth, while the non-occlusal surface is the side of the tooth or the area not directly involved in the occlusion; the occlusal surface usually has a lower depth value, while the non-occlusal surface is relatively high. Depth features can also be used to identify the junction between teeth and gums. The gums usually surround the base of the teeth and have a higher depth value, while the surface depth of the teeth themselves is lower. By analyzing the changes in the depth gradient, the boundary between the teeth and gums can be determined.

[0110] The occlusal surface usually shows a high curvature change, especially in the cusp and pit and fissure areas, which have high curvature values ​​due to the high occlusal pressure and complex morphology. The non-occlusal surface of the tooth has a small curvature change and is relatively smooth, especially on the labial and lingual sides of the tooth. These relatively smooth areas can be distinguished by analyzing the curvature values. The gingival margin usually shows an area with a sharp change in curvature, which reflects the transition from the tooth surface to the gingival surface.

[0111] The normal vector on the occlusal surface usually points in the occlusal direction of the tooth, that is, perpendicular to the occlusal surface of the tooth; on the non-occlusal surface, the normal vector is tilted or points in other directions. On the labial and lingual sides, the normal vector generally points to the outside or inside of the tooth. These features are significantly different from the normal vector on the occlusal surface and help to accurately distinguish these areas. At the junction of the tooth and the gum, the normal vector may undergo a significant change in direction. This change is caused by the different geometric shapes when the tooth surface transitions to the gum surface.

[0112] Therefore, various parts in the oral environment can be determined by analyzing at least one of the depth features, curvature features, and normal vector features. Furthermore, the above three geometric features can be combined to effectively identify and distinguish various parts in the oral environment. The depth feature provides basic information on height and position, the curvature feature reveals the complexity of the surface morphology, and the normal vector feature provides detailed information on the surface orientation. Combining these geometric features, the occlusal surface, non-occlusal surface, the junction between the teeth and gums, the gums, the labial side and the lingual side can be accurately determined, thereby providing a basis for further abnormality detection and diagnosis.

[0113] For example, after determining various parts in the oral environment, the dental abnormality region recognition model may be based on classifying various parts in the oral environment:

[0114] (1) If the part is the occlusal surface of the tooth, the color texture characteristics corresponding to the part indicate that it belongs to the dark range, and the geometric characteristics corresponding to the part show a changing trend, it is determined that there is a wedge-shaped defect in the part.

[0115] Specifically, wedge-shaped defects usually manifest as a decrease in depth in certain areas of the tooth surface, that is, the surface is eroded or missing, resulting in a low local depth value. Wedge-shaped defects are usually accompanied by a sharp change in curvature, especially in the damaged edge area, where the curvature value increases significantly. The formation of a wedge-shaped defect usually causes a change in the direction of the normal vector, especially at the edge of the defect, where the normal vector may no longer point to the normal occlusal direction of the tooth. Due to surface damage or exposure of inner tissues such as dentin, the wedge-shaped defect area is usually dark. Therefore, when the occlusal surface is dark and the geometric features (such as reduced depth, increased curvature, and normal vector change) are abnormal, the model can determine that a wedge-shaped defect exists in this area.

[0116] (2) If the location is the junction of the teeth and gums, the color texture characteristics corresponding to the location indicate that it belongs to the light color range, and the geometric characteristics corresponding to the location show a changing trend, it is determined that dental calculus exists at the location.

[0117] Specifically, tartar buildup at the tooth-gingival interface will cause the depth value in that area to increase because the tartar forms an additional covering layer. Tartar often causes changes in the surface curvature at the tooth-gingival interface, especially in areas with more buildup, where the curvature value increases. Because tartar covers the tooth surface, the normal vector may slightly deviate from the direction of the normal tooth surface. Tartar is often light or white, especially when newly formed. Therefore, when the color of the tooth-gingival interface is light and the geometric features (such as increased depth, increased curvature, and slightly deviated normal vector) are abnormal, the model can determine that tartar is present in that area.

[0118] (3) If the site is the non-occlusal surface, labial side or lingual side of the tooth, the color and texture characteristics corresponding to the site indicate that it belongs to the dark range and the geometric characteristics corresponding to the site show a stable trend, it is determined that pigmentation exists in the site.

[0119] Specifically, pigmentation does not usually cause significant changes in the depth of the tooth surface, so the depth feature remains stable. Pigmentation also does not significantly change the morphology of the tooth surface, so the curvature feature should also remain stable. Pigmentation has little effect on the geometric morphology of the tooth surface, and the direction of the normal vector generally does not change. Pigmentation usually manifests as a darker color on the tooth surface, which is common on the labial or lingual side. Therefore, when the color of the non-occlusal surface, labial or lingual area is dark but the geometric features (such as depth, curvature, normal vector) remain stable, the model can determine that pigmentation exists in that area.

[0120] (4) If the site is the non-occlusal surface, labial side or lingual side of the tooth, the color texture characteristics corresponding to the site indicate that it belongs to the dark range and the geometric characteristics corresponding to the site show a changing trend, it is determined that caries exists in the site.

[0121] Specifically, caries usually leads to a decrease in the depth of the tooth surface, that is, the tooth surface is corroded and pits are formed. Caries areas are usually accompanied by obvious changes in curvature, especially at the edge of the caries, where the curvature value may increase. Due to the pits formed by caries, the direction of the normal vector in this area will change, pointing to the inside of the pit. Caries areas are usually dark, which is caused by the loss of material on the tooth surface and the exposure of diseased tissue. Therefore, when the color of the non-occlusal surface, labial or lingual area is dark and the geometric features (such as reduced depth, increased curvature, and normal vector change) are abnormal, the model can determine that caries exists in this area.

[0122] (5) If the site is the gums, the color texture feature corresponding to the site indicates that it belongs to the red range, and the geometric feature corresponding to the site shows a changing trend, it is determined that the site has red and swollen gums or gum atrophy.

[0123] Specifically, gingival redness usually results in increased depth values ​​for the gingival area, while gingival recession results in decreased depth values. Gum redness may result in increased local curvature, while gingival recession results in decreased or smoothed curvature. The normal vector of the gingival area may slightly deviate from the normal direction due to redness or recession. Gum redness areas are usually red, while recession gums may become pale or lighter in color. Therefore, when the color of the gingival area is red and the geometric features (such as increased or decreased depth, curvature changes, normal vector deviation) are abnormal, the model can determine that gingival redness or recession exists in that area.

[0124] By jointly analyzing these geometric features and color and texture features, the tooth abnormality area recognition model can effectively identify and classify abnormalities in different areas of the oral cavity, thereby improving the accuracy and reliability of diagnosis.

[0125] Regarding the process of performing dental abnormality area recognition processing by the model in step S804, in addition to directly inputting the first target image into the model for recognition processing, in some embodiments, the electronic device may also first segment the first target image to obtain multiple segmented images, and then input each segmented image into the dental abnormality area recognition model respectively, so that the dental abnormality area recognition model performs dental abnormality area recognition processing on each segmented image to obtain abnormal area recognition results corresponding to each segmented image, and finally obtains abnormal area recognition results corresponding to the first target image based on the abnormal area recognition results corresponding to each segmented image.

[0126] In this embodiment, segmenting the first target image into multiple segmented images can help reduce the amount of data that needs to be processed in the subsequent model recognition process, thereby reducing the demand for computing resources such as the GPU of the electronic device, and improving the efficiency and accuracy of the model in identifying and processing abnormal tooth areas. Among them, the image segmentation method used in the first target image segmentation process may include but is not limited to a threshold segmentation method, an edge detection method, a pixel segmentation method, a deep learning method, or an attention mechanism model, etc., which can be specifically set according to the actual application scenario, and this embodiment does not impose any restrictions.

[0127] On the basis of the above embodiment, in the process of segmenting the first target image, several irregular segmented images may be generated, resulting in overlapping areas between the segmented images. These overlapping areas are often not the actual content of the image, but the result of noise interference in the image segmentation process. In this case, the recognition accuracy of the model in these areas will be seriously affected, resulting in misjudgment. Therefore, in order to improve the reliability of the overall recognition result, in some embodiments, the electronic device can first identify and count the overlapping areas between the segmented images. Subsequently, for the identified overlapping areas, if the model determines that the number of times the area is identified as an abnormal area in all the segmented images to which it belongs exceeds a preset threshold, the model determines that the overlapping area is an abnormal area, otherwise, the model regards the overlapping area as noise rather than an actual abnormal area.

[0128] In some embodiments, the abnormal area recognition result may include a first target image superimposed with abnormal area indication information. After obtaining the abnormal area recognition result, the electronic device may also back-project the first target image superimposed with abnormal area indication information from two-dimensional space to three-dimensional space to determine three-dimensional points belonging to the abnormal area in the first tooth model; then, when displaying the first tooth model on the display interface, visual information indicating the abnormal area is superimposed on the three-dimensional points belonging to the abnormal area.

[0129] Exemplarily, the abnormal area identification result also includes the abnormal type to which each abnormal area belongs; the abnormal type includes at least one of wedge-shaped defects, tartar, pigmentation, caries, gingival redness and swelling, and gingival atrophy; wherein the visual information used to indicate abnormal areas belonging to different abnormal types is different.

[0130] In this embodiment, the abnormal area identification results are back-projected into three-dimensional space so that the abnormal area on the first tooth model can be superimposed to display visual information indicating the abnormal area, ensuring that the abnormal area can be presented intuitively and clearly on the three-dimensional tooth model, which is beneficial for doctors or patients to understand the condition of dental lesions and assist in subsequent treatment.

[0131] In some embodiments, in order to facilitate the doctor to quickly locate and count the abnormal area of ​​the tooth, the electronic device can also determine the non-blocked viewing angle of the three-dimensional points belonging to the abnormal area on the first tooth model; wherein the non-blocked viewing angle is the viewing angle at which the three-dimensional points belonging to the abnormal area can be observed on the first tooth model. Specifically, the non-blocked viewing angle refers to the viewing angle at which the three-dimensional points belonging to the abnormal area are not blocked by other teeth or oral structures when observed from a certain angle of the three-dimensional tooth model, and can be clearly seen. This viewing angle can ensure that the details of the abnormal area can be fully displayed during the observation and diagnosis process, and avoid information loss or misjudgment due to viewing angle problems. Then, when the electronic device displays the first tooth model on the display interface, the non-blocked viewing angle is used as the screenshot viewing angle of the display interface, and the display interface is screenshoted to generate a two-dimensional screenshot containing the three-dimensional points belonging to the abnormal area. After determining the screenshot viewing angle, the electronic device screenshots the display interface to generate a two-dimensional screenshot containing the three-dimensional points in the abnormal area. The screenshot not only retains the realism of the three-dimensional model, but can also be easily shared or used for further analysis. For example, the two-dimensional screenshot can be embedded in a report, used for patient communication, or saved as part of a medical record. Since the screenshot perspective is a carefully selected non-occluded perspective, these screenshots can clearly show the target abnormal area in any situation, making it easier for doctors and patients to understand the problem.

[0132] Exemplarily, the electronic device can also associate the screenshot perspective with the observation perspective of the first tooth model, that is, establish a correspondence between the two-dimensional screenshot of the determined abnormal area and the corresponding position in the three-dimensional model, so that when the doctor views the two-dimensional screenshot, he can clearly know which specific position in the three-dimensional tooth model the screenshot corresponds to, thereby quickly locating the abnormal area. At the same time, the electronic device can also provide an area of ​​interest for a specific abnormal position, that is, mark the abnormal area that the doctor needs to pay special attention to in the two-dimensional screenshot, so that the doctor can more intuitively identify and analyze the abnormal area, locate and count the lesions, and effectively avoid omissions.

[0133] See for example Fig. 9, shows a plurality of generated two-dimensional screenshots containing three-dimensional points belonging to the abnormal area, and the area framed by the green box in the figure is the three-dimensional point belonging to the abnormal area.

[0134] It is not difficult to understand that the schemes described in the above embodiments can be combined when there is no conflict, and they are not listed one by one in the embodiments of the present application.

[0135] In some embodiments, see Fig.10 The present application also provides a device for identifying abnormal tooth areas, including:

[0136] The module 1001 for acquiring information of the model to be identified is used to acquire the texture of the first tooth model to be identified and the geometric information of the three-dimensional points in the first tooth model;

[0137] A projection module 1002 is used to project the texture in the first tooth model and the geometric information of the three-dimensional points in the first tooth model respectively to obtain a texture map and a geometric information map of the first tooth model;

[0138] A superposition module 1003, used for superimposing the texture map and the geometric information map of the first tooth model into a first target image;

[0139] The abnormal region recognition module 1004 is used to perform tooth abnormal region recognition processing on the first target image using a pre-trained tooth abnormal region recognition model to obtain an abnormal region recognition result corresponding to the first target image;

[0140] Among them, the tooth abnormality area recognition model is obtained through supervised training based on several training samples, and the training samples include a second target image of a second tooth model and an abnormal area label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second tooth model.

[0141] In one implementation, the first tooth model includes several three-dimensional grids; the projection module 1002 is specifically used to: determine the coordinate mapping relationship from three-dimensional space to two-dimensional space based on a preset three-dimensional grid texture expansion algorithm; project the texture in the first tooth model based on the coordinate mapping relationship to obtain a texture map of the first tooth model; and, project the geometric information of the three-dimensional points in the first tooth model based on the coordinate mapping relationship to obtain a geometric information map of the first tooth model.

[0142] In one implementation, the three-dimensional points in the first tooth model include at least two types of geometric information; the projection module 1002 is specifically used to: for each type of geometric information of the three-dimensional points, project the type of geometric information into a two-dimensional space based on a coordinate mapping relationship and map it to one of the color channels of a specified color mode to obtain a color map corresponding to the type of geometric information; wherein different types of geometric information of the three-dimensional points correspond to different color channels; and merge the color maps corresponding to all types of geometric information of the three-dimensional points in the first tooth model to obtain a geometric information map of the first tooth model.

[0143] In one implementation, the geometric information of the 3D point includes at least one of the following types: curvature, depth and normal vector of the 3D point. The first target image is represented by at least four channels, three of which are used to describe the first tooth image, and the remaining at least one channel is used to describe the geometric information map.

[0144] In one implementation, the abnormal area identification module 1004 is specifically used to: input the first target image into the tooth abnormal area identification model, so that the tooth abnormal area identification model extracts color texture features and geometric features from the first target image, determines various parts in the oral environment by analyzing the geometric features, and classifies them based on the color texture features and geometric features corresponding to each part in the oral environment to obtain an abnormal area identification result corresponding to the first target image; wherein the various parts in the oral environment include at least one of the occlusal surface of the tooth, the non-occlusal surface of the tooth, the junction of the tooth and the gum, the gum, the labial side and the lingual side.

[0145] In one implementation, the abnormal area identification module 1004 is specifically used for: if the part is the occlusal surface of the tooth, the color texture feature indication corresponding to the part belongs to the preset dark range and the geometric features corresponding to the part show a changing trend, it is determined that there is a wedge-shaped defect at the part; if the part is the junction of the tooth and the gum, the color texture feature indication corresponding to the part belongs to the preset light range and the geometric features corresponding to the part show a changing trend, it is determined that there is tartar at the part; if the part is the non-occlusal surface, labial side or lingual side of the tooth, the color texture feature indication corresponding to the part belongs to the preset dark range and the geometric features corresponding to the part show a stable trend, it is determined that there is pigmentation at the part; if the part is the non-occlusal surface, labial side or lingual side of the tooth, the color texture feature indication corresponding to the part belongs to the preset dark range and the geometric features corresponding to the part show a changing trend, it is determined that there is caries at the part; if the part is the gum, the color texture feature indication corresponding to the part belongs to the preset red range and the geometric features corresponding to the part show a changing trend, it is determined that there is redness and swelling of the gums or gum atrophy at the part.

[0146] In one implementation, the abnormal area identification module 1004 is specifically used to: segment the first target image to obtain multiple segmented images; input each of the segmented images into a dental abnormal area identification model, so that the dental abnormal area identification model performs dental abnormal area identification processing on each segmented image to obtain abnormal area identification results corresponding to each segmented image; based on the abnormal area identification results corresponding to each segmented image, obtain the abnormal area identification results corresponding to the first target image.

[0147] In one implementation, the abnormal area identification result includes a first target image superimposed with abnormal area indication information; and also includes an abnormal area indication module, which is used to back-project the first target image superimposed with abnormal area indication information to determine the three-dimensional points belonging to the abnormal area in the first tooth model; when the first tooth model is displayed on the display interface, the three-dimensional points belonging to the abnormal area are superimposed with visual information used to indicate the abnormal area.

[0148] In one implementation, the abnormal area identification result also includes the abnormal type to which each abnormal area belongs; the abnormal type includes at least one of wedge-shaped defects, tartar, pigmentation, caries, gingival redness and swelling, and gingival atrophy; wherein the visual information used to indicate abnormal areas belonging to different abnormal types is different.

[0149] In one implementation, it also includes a screenshot module, which is used to determine the non-occluded viewing angle of the three-dimensional points belonging to the abnormal area on the first tooth model; wherein the non-occluded viewing angle is a viewing angle at which the three-dimensional points belonging to the abnormal area can be observed on the first tooth model; when the first tooth model is displayed on the display interface, the non-occluded viewing angle is used as the screenshot viewing angle of the display interface, and the display interface is screenshoted to generate a two-dimensional screenshot containing the three-dimensional points belonging to the abnormal area.

[0150] In some embodiments, see Fig.11 The present application also provides a training device for a tooth abnormality region recognition model, including:

[0151] The training sample acquisition module 1101 is used to acquire a number of training samples, wherein the training samples include a second target image of a second tooth model and an abnormal region label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second tooth model; different training samples correspond to different second tooth models;

[0152] The model training module 1102 is used to input the training samples into the tooth abnormality area recognition model to be trained, so that the tooth abnormality area recognition model to be trained can recognize the tooth abnormality area based on the second target image, obtain the abnormal area recognition result corresponding to the second target image, and train the tooth abnormality area recognition model to be trained with minimizing the difference between the abnormal area recognition result and the label in the training samples as the optimization goal; wherein the trained tooth abnormality area recognition model is applied to the above-mentioned tooth abnormality area recognition method.

[0153] In one implementation, the training sample acquisition module 1101 is specifically used to: display the second target image in the training sample in the display interface; in response to a first switching operation, switch the second target image displayed in the display interface to a second tooth model corresponding to the second target image; and in response to a second switching operation, switch the second tooth model displayed in the display interface to the second target image; wherein the observation angle of the second target image in the display interface is the same as the observation angle of the second tooth model in the display interface; in response to a labeling operation on the second target image, obtain a label in the training sample; wherein the labeling operation includes an abnormal area drawing operation and an abnormal type selection operation; the label includes the location of the abnormal area and the abnormal type to which the abnormal area belongs.

[0154] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units.

[0155] In some embodiments, the embodiments of the present application further provide an electronic device, including a memory, a processor, and executable instructions stored in the memory and executable on the processor;

[0156] Wherein, the processor implements the steps in the above method when executing the executable instructions.

[0157] Exemplarily, the processor includes but is not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0158] Exemplarily, the memory may include at least one type of storage medium, the storage medium including flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc.

[0159] For example, Fig.12 is a schematic structural diagram of a device provided by an exemplary embodiment. Fig.12 At the hardware level, the device includes a processor 1202, an internal bus 1204, a network interface 1206, a memory 1208, and a non-volatile memory 1210, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 1202 reading the corresponding computer program from the non-volatile memory 1210 into the memory 1208 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0160] In some embodiments, the embodiments of the present application further provide a computer program product, including a computer program, which is used to implement the above method when executed by a processor.

[0161] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0162] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a terminal, enables the terminal to execute the above method.

[0163] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0164] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0165] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for identifying abnormal tooth areas, characterized in that: include: Acquire at least two types of geometric information of a texture in a first tooth model to be identified and a three-dimensional point in the first tooth model; Projecting the texture in the first tooth model to obtain a texture map of the first tooth model, and projecting the geometric information of the three-dimensional points in the first tooth model based on a coordinate mapping relationship to obtain a geometric information map of the first tooth model; superimposing the texture map and the geometric information map of the first tooth model into a first target image; Using a pre-trained tooth abnormality region recognition model to perform tooth abnormality region recognition processing on the first target image to obtain an abnormal region recognition result corresponding to the first target image; The tooth abnormal area recognition model is obtained by supervised training based on a number of training samples, wherein the training samples include a second target image of a second tooth model and an abnormal area label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second tooth model; The step of projecting the geometric information of the three-dimensional points in the first tooth model based on the coordinate mapping relationship to obtain a geometric information graph of the first tooth model includes: For each type of geometric information of the three-dimensional point, project and map the geometric information of this type to one of the color channels of the specified color mode based on the coordinate mapping relationship to obtain a color map corresponding to the geometric information of this type; wherein different types of geometric information of the three-dimensional point correspond to different color channels; fusing the color maps corresponding to all types of geometric information of the three-dimensional points in the first tooth model to obtain a geometric information map of the first tooth model; The method of performing tooth abnormality region recognition processing on the first target image using the pre-trained tooth abnormality region recognition model to obtain an abnormal region recognition result corresponding to the first target image includes: The first target image is input into the tooth abnormal area recognition model so that the tooth abnormal area recognition model extracts color texture features and geometric features from the first target image, determines various parts in the oral environment by analyzing the geometric features, and classifies them based on the color texture features and geometric features corresponding to the various parts in the oral environment to obtain abnormal area recognition results corresponding to the first target image; wherein the various parts in the oral environment include at least one of the occlusal surface of the tooth, the non-occlusal surface of the tooth, the junction between the tooth and the gum, the gum, the labial side and the lingual side.

2. The method according to claim 1, characterized in that The first tooth model includes a plurality of three-dimensional grids; the coordinate mapping relationship is a coordinate mapping relationship from three-dimensional space to two-dimensional space determined based on a preset three-dimensional grid texture expansion algorithm; Projecting the texture in the first tooth model to obtain a texture map of the first tooth model includes: The texture in the first tooth model is projected based on the coordinate mapping relationship to obtain a texture map of the first tooth model.

3. The method according to claim 1, characterized in that The geometric information of the three-dimensional point includes at least one of the following types: curvature, depth and normal vector of the three-dimensional point; and / or The first target image is represented by at least four channels, three of which are used to describe the texture map, and the remaining at least one channel is used to describe the geometric information map.

4. The method according to claim 1, characterized in that: The classifying the parts based on the color texture features and geometric features corresponding to the parts at least includes: If the part is the occlusal surface of the tooth, the color texture feature corresponding to the part indicates that it belongs to a preset dark range, and the geometric feature corresponding to the part shows a changing trend, it is determined that there is a wedge-shaped defect at the part; If the part is the junction of the teeth and gums, the color texture feature indication corresponding to the part belongs to the preset light color range, and the geometric feature corresponding to the part shows a changing trend, it is determined that tartar exists at the part; If the part is the non-occlusal surface, labial side or lingual side of the tooth, the color texture feature indication corresponding to the part belongs to the preset dark range and the geometric feature corresponding to the part shows a stable trend, it is determined that pigmentation exists in the part; If the part is the non-occlusal surface, labial side or lingual side of the tooth, the color texture feature corresponding to the part indicates that it belongs to the preset dark range and the geometric feature corresponding to the part shows a changing trend, it is determined that caries exists at the part; If the part is the gums, the color texture feature indication corresponding to the part belongs to a preset red range, and the geometric feature corresponding to the part shows a changing trend, it is determined that red and swollen gums or gum atrophy exist in the part.

5. The method according to claim 1, characterized in that The method of performing tooth abnormality region recognition processing on the first target image using the pre-trained tooth abnormality region recognition model to obtain an abnormal region recognition result corresponding to the first target image includes: Segmenting the first target image to obtain a plurality of segmented images; Inputting each of the segmented images into the tooth abnormality region recognition model respectively, so that the tooth abnormality region recognition model performs tooth abnormality region recognition processing on each of the segmented images to obtain abnormal region recognition results corresponding to each of the segmented images; Based on the abnormal region recognition results corresponding to each of the segmented images, an abnormal region recognition result corresponding to the first target image is obtained.

6. The method according to claim 1, characterized in that The abnormal area recognition result includes a first target image superimposed with abnormal area indication information; The method further comprises: Back-projecting the first target image superimposed with the abnormal area indication information to determine the three-dimensional points belonging to the abnormal area in the first tooth model; When the first tooth model is displayed on the display interface, the three-dimensional points belonging to the abnormal area are superimposed and displayed with visual information indicating the abnormal area.

7. The method according to claim 6, characterized in that The abnormal area recognition result also includes the abnormal type to which each abnormal area belongs; the abnormal type includes at least one of wedge-shaped defect, dental calculus, pigmentation, dental caries, gingival redness and swelling, and gingival atrophy; Therein, the visual information used to indicate abnormal regions belonging to different abnormality types is different.

8. The method according to claim 6 or 7, characterized in that: Also includes: Determine a non-blocked viewing angle of the three-dimensional point belonging to the abnormal area on the first tooth model; wherein the non-blocked viewing angle is a viewing angle at which the three-dimensional point belonging to the abnormal area can be observed on the first tooth model; When the first tooth model is displayed on the display interface, the non-blocked viewing angle is used as the screenshot viewing angle of the display interface, and the display interface is screenshot processed to generate a two-dimensional screenshot including the three-dimensional points belonging to the abnormal area.

9. A method for training a tooth abnormality region recognition model, characterized in that: include: Acquire a plurality of training samples, wherein the training samples include a second target image of a second tooth model and an abnormal region label corresponding to the second target image; The second target image is obtained by superimposing the texture map and the geometric information map of the second tooth model; Different training samples correspond to different second tooth models; the geometric information graph of the second tooth model is obtained by projecting at least two types of geometric information of three-dimensional points in the second tooth model based on a coordinate mapping relationship; Inputting the training sample into the tooth abnormality region recognition model to be trained, so that the tooth abnormality region recognition model to be trained performs tooth abnormality region recognition based on the second target image to obtain an abnormal region recognition result corresponding to the second target image, and training the tooth abnormality region recognition model to be trained with minimizing the difference between the abnormal region recognition result and the label in the training sample as the optimization goal; Wherein, the trained tooth abnormality region recognition model is applied to the tooth abnormality region recognition method as claimed in any one of claims 1 to 8; The step of projecting at least two types of geometric information of three-dimensional points in the second tooth model based on the coordinate mapping relationship to obtain a geometric information graph of the second tooth model includes: For each type of geometric information of the three-dimensional point, project and map the geometric information of this type to one of the color channels of the specified color mode based on the coordinate mapping relationship to obtain a color map corresponding to the geometric information of this type; wherein different types of geometric information of the three-dimensional point correspond to different color channels; fusing the color maps corresponding to all types of geometric information of the three-dimensional points in the second tooth model to obtain a geometric information map of the second tooth model; The tooth abnormal area recognition model to be trained performs tooth abnormal area recognition based on the second target image to obtain an abnormal area recognition result corresponding to the second target image, including: The second target image is input into the tooth abnormal area recognition model to be trained, so that the tooth abnormal area recognition model to be trained extracts color texture features and geometric features from the second target image, determines various parts in the oral environment by analyzing the geometric features, and classifies them based on the color texture features and geometric features corresponding to the various parts in the oral environment to obtain abnormal area recognition results corresponding to the second target image; wherein the various parts in the oral environment include at least one of the occlusal surface of the tooth, the non-occlusal surface of the tooth, the junction between the tooth and the gum, the gum, the labial side and the lingual side.

10. The method according to claim 9, characterized in that Acquiring an abnormal region label corresponding to the second target image in the training sample includes: Displaying a second target image in the training sample in a display interface; In response to the first switching operation, switching the second target image displayed in the display interface to display a second tooth model corresponding to the second target image; and In response to a second switching operation, the second tooth model displayed in the display interface is switched to be displayed as the second target image; wherein the observation angle of the second target image in the display interface is the same as the observation angle of the second tooth model in the display interface; In response to a labeling operation on the second target image, the label in the training sample is obtained; wherein the labeling operation includes an abnormal region drawing operation and an abnormal type selection operation; the label includes a location of the abnormal region and an abnormal type to which the abnormal region belongs.

11. A device for identifying abnormal tooth areas, characterized in that: include: A module for acquiring information of a model to be identified, used for acquiring at least two types of geometric information of texture in a first tooth model to be identified and three-dimensional points in the first tooth model; A projection module, used for projecting the texture in the first tooth model to obtain a texture map of the first tooth model, and projecting the geometric information of the three-dimensional points in the first tooth model based on a coordinate mapping relationship to obtain a geometric information map of the first tooth model; A superposition module, used for superimposing the texture map and the geometric information map of the first tooth model into a first target image; an abnormal region recognition module, configured to perform tooth abnormal region recognition processing on the first target image using a pre-trained tooth abnormal region recognition model, and obtain an abnormal region recognition result corresponding to the first target image; The tooth abnormal area recognition model is obtained by supervised training based on a number of training samples, wherein the training samples include a second target image of a second tooth model and an abnormal area label corresponding to the second target image; the second target image is obtained by superimposing a texture map and a geometric information map of the second tooth model; The step of projecting the geometric information of the three-dimensional points in the first tooth model based on the coordinate mapping relationship to obtain a geometric information graph of the first tooth model includes: For each type of geometric information of the three-dimensional point, project and map the geometric information of this type to one of the color channels of the specified color mode based on the coordinate mapping relationship to obtain a color map corresponding to the geometric information of this type; wherein different types of geometric information of the three-dimensional point correspond to different color channels; fusing the color maps corresponding to all types of geometric information of the three-dimensional points in the first tooth model to obtain a geometric information map of the first tooth model; The method of performing tooth abnormality region recognition processing on the first target image using the pre-trained tooth abnormality region recognition model to obtain an abnormal region recognition result corresponding to the first target image includes: The first target image is input into the tooth abnormal area recognition model so that the tooth abnormal area recognition model extracts color texture features and geometric features from the first target image, determines various parts in the oral environment by analyzing the geometric features, and classifies them based on the color texture features and geometric features corresponding to the various parts in the oral environment to obtain abnormal area recognition results corresponding to the first target image; wherein the various parts in the oral environment include at least one of the occlusal surface of the tooth, the non-occlusal surface of the tooth, the junction between the tooth and the gum, the gum, the labial side and the lingual side.

12. An electronic device, characterized in that: including a memory, a processor, and executable instructions stored in the memory and executable on the processor; Wherein, when the processor executes the executable instructions, the steps in the method as described in any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images