A digital oral model landmark point identification method, device and electronic equipment

Abstract

The application provides a digital oral model mark point recognition method and device and electronic equipment, and the method comprises the following steps: step 1, inputting dental arch model image data, segmenting the dental crown of the dental arch model, and recognizing the tooth position number of the dental crown, wherein the dental arch model comprises a gum and dental crowns of different tooth positions. Step 2, based on automatic recognition of the dental crown mark point of a single dental crown, for each segmented dental crown, the multi-view graph network is used to automatically recognize the dental model mark point, and the mark point on each dental crown is obtained. The method of the application does not predict the mark point on the three-dimensional point cloud, but maps the three-dimensional data into two-dimensional projection image data, and calculates by using a two-dimensional network, so that the calculation amount and the dependence on the computing power can be reduced compared with a three-dimensional network.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of images, and in particular to a digital oral model landmark point recognition method and device based on a deep neural network and electronic equipment. BACKGROUND

[0002] Three-dimensional digital dental models have been widely used in orthodontic clinics due to their safety and efficiency. Dental models record information about the upper and lower jaws, dental arches, and basal bones of patients. Accurate measurement and analysis of dental models are crucial for doctors to assist in diagnosis and treatment. In measuring and analyzing dental models, a basic and key step is the identification of tooth feature points, such as the cuspid points on premolars and molars, and the incisal points on incisors, canines, and fangs. These inherent feature points on the tooth surface can help doctors develop treatment plans and monitor treatment effectiveness. Since each patient needs to be labeled with dozens of markers, relying entirely on manual point setting by doctors is time-consuming and labor-intensive. Automatic positioning of landmark points on three-dimensional digital models is a challenging problem. On the one hand, high positioning accuracy is required, typically to an accuracy of 0.5 mm. On the other hand, due to normal tooth wear or tooth repair treatments, the shapes of teeth vary from patient to patient. Moreover, digital models contain hundreds of thousands or even millions of grid elements, and processing and computing these high-resolution digital models require a tremendous amount of computer power.

[0003] Traditional researches are mostly based on geometric methods to identify feature points. Kumar et al. proposed a feature point recognition algorithm based on region segmentation, using a curvature and height mixed height function for watershed algorithm, and taking the local maximum points identified as tooth feature points. However, this method is sensitive to model noise, prone to over-division and under-division, leading to inaccurate feature point recognition. Chu Yuwei et al. proposed a tooth feature automatic recognition algorithm based on DBSCAN and K-Means hybrid clustering, which is more robust and accurate in recognition. However, both of the above methods are based on the priori of single tooth, and only realize the identification of molar cuspid points with significant features, which is less in number and cannot meet the needs of clinical diagnosis. SUMMARY

[0004] The present application proposes a digital oral model landmark point recognition method, device and electronic equipment based on a deep neural network, which automatically locates landmark points by performing heat map regression prediction on 2D renderings of tooth crown models taken at any angle of view.

[0005] The technical solution of the present application is as follows: a digital oral model landmark point recognition method based on a deep neural network, comprising the following steps:

[0006] Step 1, inputting dental arch model image data, segmenting the dental crown of the dental arch model, and identifying the tooth position number of the dental crown, wherein the dental arch model comprises gingiva and dental crowns of different tooth positions.

[0007] Step 2, automatic identification of dental crown landmarks based on single dental crown, for each segmented dental crown, automatic identification of dental model landmarks is performed by using a multi-view graph network to obtain the landmarks on each dental crown.

[0008] Further, the dental crown of the dental arch model is segmented by extracting point cloud attribute information from the dental arch model as the input of the neural network model, and a point cloud instance segmentation network is used to perform instance segmentation on the input point cloud, thereby realizing the segmentation of each dental crown.

[0009] Further, the step 2 specifically comprises:

[0010] The dental crown file segmented in the first step is photographed at a random view angle to obtain multiple view graphs at different view angles; the view graph is used as the input of the multi-view network to perform landmark prediction of the two-dimensional view graph; and the predicted two-dimensional landmarks of the multiple view graphs are mapped back to the three-dimensional space to obtain the three-dimensional landmark coordinates corresponding to the dental crown.

[0011] According to another aspect, a digital oral model landmark identification device based on a deep neural network is also provided, comprising:

[0012] The segmentation and identification module segments the dental crown of the dental arch model by using a point cloud segmentation network, and identifies the tooth position number of the dental crown, wherein the dental arch model comprises gingiva and dental crowns of different tooth positions;

[0013] The dental model landmark identification module performs automatic identification of dental crown landmarks based on a single dental crown, and for each segmented dental crown, automatic identification of dental model landmarks is performed by using a multi-view graph network to obtain the landmarks on each dental crown.

[0014] According to another aspect, an electronic device is also provided, comprising at least one processor and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the digital oral model landmark identification method based on a deep neural network as described above.

[0015] Compared with the prior art:

[0016] The method of the present application does not perform landmark prediction on a three-dimensional point cloud, but maps three-dimensional data to a two-dimensional projection image data, and performs calculation by using a two-dimensional network, which can reduce the calculation amount and dependence on computing power compared with a three-dimensional network.

[0017] The method of the present application does not limit the projection angle, and any group of images can be projected at any angle of the dental crown model according to the calculation requirement. For relatively simple parts such as anterior teeth, a smaller number of multi-angle projection view images can be used; for relatively complex molars, the projection angle can be increased to extract more image features at different angles, and the number and angle of the projection images can be flexibly set according to the specific use requirement. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 : The overall block diagram of the present application;

[0019] Figure 2 : Network structure diagram of the implementation example based on DeepGCN;

[0020] Figure 3 : The automatic identification process of the dental model landmark points using the multi-view image network of the present application;

[0021] Figure 4 : Schematic diagram of the present application assuming that a plurality of virtual cameras take pictures of the dental crown model at any number of angles;

[0022] Figure 5 : Multi-view image network structure;

[0023] Figure 6 : Device principle block diagram of one embodiment of the present application;

[0024] Figure 7 : Electronic device principle block diagram of one embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0026] According to the embodiments of the present application, a digital oral model landmark point recognition method based on a deep neural network is proposed, as shown in Figure 1 , 3 , including the following steps:

[0027] Step 1, input the dental model image data, segment the dental crown of the dental model, and identify the tooth number of the dental crown, wherein the dental model includes the gums and the dental crowns of different tooth positions.

[0028] Step 2, automatic identification of dental crown landmark points based on single dental crown, for each segmented dental crown, the multi-view image network is used to automatically identify the dental model landmark points to obtain the landmark points on each dental crown.

[0029] Specifically, as described above, the method adopts a two-stage manner, first performs tooth segmentation, and then performs feature point recognition.

[0030] As described in step 1, the first stage of processing is performed, the dental model image data is input, and the dental crown of the dental model is segmented. In an embodiment of the present application, a variety of image segmentation methods can be used to segment the dental crown of the dental model, for example, a point cloud segmentation network is used for segmentation, a level set method is used for segmentation, or a clustering-based method is used for segmentation, etc. can be used; preferably, for the input point cloud data, a point cloud segmentation network is used for segmentation; the dental model in the present application includes gingiva and dental crowns of different tooth positions, the main purpose of this step is to segment each dental crown and identify the tooth position number of the dental crown, for the second stage of automatic identification of dental crown landmark points based on single dental crown. The specific way is to extract the point cloud attribute information from the dental model as the input of the neural network model, such as vertex coordinate information, face normal information, curvature information, face center coordinate information, etc.; the point cloud instance segmentation network is used to do instance segmentation on the input point cloud, so as to realize the segmentation of each dental crown. Compared with the shallow graph convolutional network used in the prior art, the network deepens the graph convolutional network by learning from the idea of residual connection, and avoids the problem of gradient disappearance, which is particularly important for dental mesh models, because deeper graph convolutional networks correspond to larger receptive fields. Only by capturing the receptive field of all teeth can the local and global information of the teeth that are beneficial to tooth segmentation be completely obtained. In addition, compared with the segmentation network MeshSegNet, the network does not need to perform large-scale matrix calculation in advance, and the calculation amount is significantly reduced.

[0031] In one embodiment, as shown in Figure 2 , a DeepGCN deep network is used to perform instance segmentation on the dental model (part of the dental model contains the third molar) to obtain each dental crown. Extract three vertex coordinates, face center coordinates, and face normal vectors of each face, etc. Multiple dimensional attribute information as input of the model. ResGCN structure is used as the back propagation layer. The ResGCN structure deepens the graph convolutional network by learning from the idea of residual connection, and avoids the problem of gradient disappearance, which is particularly important for dental mesh models, because deeper graph convolutional networks correspond to larger receptive fields. Only by capturing the receptive field of all teeth can the local and global information of the teeth that are beneficial to tooth segmentation be completely obtained. The feature fusion layer uses a 1x1 convolution to link the global max pooling layer to fuse the features extracted by the ResGCN structure.

[0032] The prediction layer adopts four 1x1 convolutions, and the output of the last layer is defined as 17, i.e., gingiva + 16 tooth positions, a total of 17 categories (for the upper dental arch model, gingiva, T11-T18, T21-T28, a total of 17 categories; for the lower dental arch model, gingiva, T31-T38, T41-T48, a total of 17 categories).

[0033] As shown in step 2, in the second stage, according to the tooth segmentation result obtained in the first stage, a multi-view convolutional neural network is used to identify the dental model landmark points. As shown in Figure 3 , it is a flow chart for automatically identifying the dental crown landmark points by using the multi-view image network. The dental crown file segmented in the first step is photographed at a random view to obtain multiple view images at different views; the view image is used as the input of the multi-view network, and the landmark points of the two-dimensional view image are predicted; the predicted two-dimensional landmark points of the multiple view images are mapped back to the three-dimensional space to obtain the three-dimensional landmark point coordinates of the dental crown.

[0034] It is assumed that multiple virtual cameras are distributed around the dental crown model at different views, so that different view "photographed" images can be obtained, i.e., the 2D rendering view images of the dental crown at the corresponding view. The generated two-dimensional view images are used as the training data of the two-dimensional landmark point identification network, and the two-dimensional landmark point identification network is used to automatically identify the landmark points of the view images at different views. Then the points identified on the two-dimensional image are mapped back to the dental crown according to the view parameters when the view image is generated to calculate the corresponding three-dimensional landmark point coordinates on the dental crown. As shown in Figure 4 , it is assumed that multiple virtual cameras photograph the dental crown model at any number of angles, i.e., the view images rendered at the corresponding view can be obtained;

[0035] According to one embodiment of the present application, a three-level hourglassnet stacked network is used as the main framework for automatic identification of landmark points on a two-dimensional image. As shown in Figure 5 , it is a network structure diagram of the embodiment, and N is the number of landmark points on the dental crown.

[0036] 1) Set 60 view images for each dental crown, i.e., assume that 60 cameras at random views are distributed around the dental crown, and each camera can obtain a photographed image of the dental crown, i.e., the view image of the dental crown at the corresponding view; at the same time, record the view parameters of the camera relative to the dental crown file, which are used to convert the two-dimensional landmark point coordinates back to the three-dimensional landmark point coordinates in the subsequent steps.

[0037] 2) Input each image into the two-dimensional image landmark point automatic identification network. The image first calculates the features through a general residual module, then connects three stacked hourglass modules, and finally outputs N heat maps of the same size as the input image through upsampling.

[0038] 3) Take the peak value of each heat map as the two-dimensional coordinates (x, y) of the corresponding landmark point of the viewgraph under the corresponding view angle.

[0039] In order to calculate the three-dimensional landmark point coordinates on the dental crown model, the two-dimensional coordinates (x, y) can be mapped back to the three-dimensional space according to the view angle parameters of the generated viewgraph to calculate the corresponding three-dimensional landmark point coordinates (x, y, z) of the dental crown. Since multiple view angles (for example, 60 in this case) are used, for each three-dimensional landmark point on the dental crown, the corresponding two-dimensional landmark point coordinates can be obtained from the viewgraph under each view angle. For a certain landmark point on the dental crown, the three-dimensional landmark point coordinates calculated by mapping the two-dimensional coordinates of the viewgraph under multiple view angles back to the three-dimensional space may not be at the same position, and a final unique three-dimensional landmark point coordinate needs to be fitted.

[0040] In summary, the specific practice of the present application at this stage is to first perform random view two-dimensional projection on the three-dimensional mesh model of the teeth, then send the obtained series of two-dimensional projection images under different view angles into the network respectively, estimate the feature point positions for each two-dimensional projection image under the view angle, and finally use the least squares method and further adopt the random sample consensus algorithm to fit the feature point positions obtained under different view angles into the final feature point positions. The method proposed by the present application achieves the best performance in the identification of feature points of dental arch models, with an average accuracy of 0.31 mm according to the tooth position statistics and an average accuracy of 0.33 mm according to the feature point statistics.

[0041] According to another embodiment of the present application, a digital oral model landmark point identification device based on a deep neural network is also proposed, as shown in Figure 6 , which comprises:

[0042] The segmentation and identification module 201 segments the dental crowns of the dental arch model using a point cloud segmentation network and identifies the tooth position numbers of the dental crowns, wherein the dental arch model comprises gums and dental crowns of different tooth positions;

[0043] The dental model landmark point identification module 202 automatically identifies the dental model landmark points based on the dental crown landmark points of a single dental crown, automatically identifies the dental model landmark points of each segmented dental crown using a multi-viewgraph network, and obtains the landmark points on each dental crown.

[0044] The segmentation and identification module 201 adopts a mode of extracting point cloud attribute information from the dental arch model as the input of the neural network model, performs instance segmentation on the input point cloud using a point cloud instance segmentation network, and thereby realizes the segmentation of each dental crown. In the embodiments of the present application, a variety of image segmentation methods can be used to segment the dental crowns of the dental arch model, such as using a point cloud segmentation network for segmentation, using a level set method for segmentation, or using a clustering-based method for segmentation, etc.

[0045] Further, the dental mold landmark point recognition module 202 photographs the dental crown file segmented in the first step from a random perspective to obtain multiple perspective images from different perspectives; takes the perspective image as the input of the multi-perspective network to perform landmark point prediction on the two-dimensional perspective image; maps the predicted two-dimensional landmark points of the multiple perspective images back to the three-dimensional space to obtain the three-dimensional landmark point coordinates corresponding to the dental crown;

[0046] The embodiment of the present application also provides an electronic device, which is shown in the figure. Figure 7 As shown in the figure, the electronic device can include a processor 301 and a memory 302, wherein the processor 301 and the memory 302 can be connected through a bus or other means, Figure 7 The bus connection is taken as an example.

[0047] The processor 301 can be a central processing unit (CPU). The processor 301 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. chips, or combinations of the above various chips.

[0048] The memory 302 as a kind of non-transient computer readable storage medium can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules of the facial nerve unfolding method based on surface reconstruction in the embodiment of the present application. The processor 301 executes the various functions of the processor and data processing by running the non-transient software programs, instructions and modules stored in the memory 302, that is, the facial nerve unfolding method based on surface reconstruction in the above method embodiment is realized.

[0049] The memory 302 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; the data storage area can store data created by the processor 401 and the like. In addition, the memory 302 can include a high-speed random access memory, and can also include a non-transient memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transient solid-state memory device. In some embodiments, the memory 302 can optionally include a memory remotely arranged with respect to the processor 401, and these remote memories can be connected to the processor 301 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0050] The one or more modules are stored in the memory 302 and, when executed by the processor 301, perform the methods as described above. Figure 1 One of the embodiments shown is a digital oral model landmark recognition method based on a deep neural network.

[0051] The above electronic device specific details can be understood by referring to the corresponding description and effects of the previous embodiments, which will not be repeated here.

[0052] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. Among them, the storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0053] Although the above describes the specific embodiments of the present application in order to facilitate the understanding of the present application by those skilled in the art, it should be clear that the present application is not limited to the scope of the specific embodiments. For those skilled in the art, any changes within the spirit and scope of the present application as defined in the appended claims are obvious, and all the inventions using the concept of the present application are within the scope of protection.

Claims

1. A deep neural network-based digital dental model landmark identification method, characterized in that, The method comprises the following steps: Step 1: inputting dental arch model image data, segmenting the dental arch of the dental arch model, and identifying the tooth position number of the dental arch, wherein the dental arch model comprises gingiva and dental arches of different tooth positions; the segmentation of the dental arch of the dental arch model is achieved by extracting point cloud attribute information from the dental arch model as the input of a neural network model, performing instance segmentation on the input point cloud by using a point cloud instance segmentation network, and thus realizing the segmentation of each dental arch; Step 2: automatic identification of dental arch landmark points based on a single dental arch, wherein for each segmented dental arch, a multi-view image network is used to automatically identify dental model landmark points, and landmark points on each dental arch are obtained; The step 2 specifically comprises: The dental arch file segmented in the first step is photographed under a random view to obtain multiple view images under different views; the view images are used as the input of a multi-view network to predict landmark points in the two-dimensional view images; and the two-dimensional landmark points corresponding to the multiple view images are mapped back to three-dimensional space to obtain three-dimensional landmark point coordinates corresponding to the dental arch. Specifically, multiple virtual cameras are assumed to be distributed around the dental arch model at different views, so that different images obtained by "shooting" under different views are 2D rendering view images of the dental arch under the corresponding views; the generated two-dimensional view images are used as training data of a two-dimensional landmark point recognition network, the two-dimensional landmark point recognition network is used to automatically identify landmark points in the view images under different views, and then the points identified in the two-dimensional images are mapped back to the dental arch according to the view parameters when the view images are generated to calculate corresponding three-dimensional landmark point coordinates on the dental arch.

2. The method of claim 1, wherein, Multiple view images are set for each dental arch, that is, multiple cameras at random views are assumed to be distributed around the dental arch, and each camera can obtain a photographed image of the dental arch, which is a view image of the dental arch under the corresponding view; meanwhile, the view parameters of the camera relative to the dental arch file are recorded, which are used to convert two-dimensional landmark point coordinates back to three-dimensional landmark point coordinates in the subsequent step.

3. A digital dental model landmark recognition apparatus based on deep neural network, using the method of claim 1, characterized in that, The method comprises: A segmentation and identification module is configured to input dental arch model image data, segment the dental arch of the dental arch model, and identify the tooth position number of the dental arch, wherein the dental arch model comprises gingiva and dental arches of different tooth positions; A dental model landmark point identification module is configured to automatically identify dental arch landmark points based on a single dental arch, wherein for each segmented dental arch, a multi-view image network is used to automatically identify dental model landmark points, and landmark points on each dental arch are obtained.

4. An electronic device, comprising: The method comprises: At least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the deep neural network-based digital oral model landmark point identification method according to any one of claims 1-2.

Images