Oral modeling method, system and storage medium based on monocular vision

Abstract

The present invention discloses an oral modeling method, system and storage medium based on monocular vision, wherein a calibration object is attached to the front surface of the gums, and then a monocular camera is used to obtain an oral image with the calibration object to generate first point cloud data containing features of the oral cavity and the calibration object, and then the first point cloud data is separated after removing outliers and interference information to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data, and the calibration object point cloud data is compared with the pre-stored actual model of the calibration object to determine the calibration object scaling factor, and the tooth point cloud data is reconstructed according to the scaling factor to obtain a tooth three-dimensional model. The entire intraoral scanning process can be completed by relying on a single-sided camera, overcoming the problem that the existing intraoral scanner that relies on binocular reconstruction or RGBD to perform three-dimensional reconstruction of teeth in the oral cavity has high equipment cost and is difficult to miniaturize.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and in particular to an oral cavity modeling method based on monocular vision, a system and a storage medium. Background Art

[0002] In the process of clinical oral diagnosis and restoration, the methods of obtaining a three-dimensional digital model of the inside of the oral cavity through three-dimensional scanning are mainly divided into two types: extraoral scanning and intraoral scanning. The extraoral scanning method requires the doctor to first make an impression of the oral cavity to obtain a plaster model of the teeth, and then use a three-dimensional scanning device to scan the plaster dental mold to obtain its three-dimensional digital model; the intraoral scanning method is to insert the oral scanner into the oral cavity to directly scan the teeth to obtain a three-dimensional digital model of the teeth. This method is simple to operate, efficient and has a fast measurement speed, thus saving the doctor's chairside operation time. Because there is no need for manual impressions, it also avoids errors caused by mold making and mold turning, and has high measurement accuracy. At present, most intraoral scanning technologies use binocular reconstruction or RGBD to perform three-dimensional reconstruction of teeth in the oral cavity to obtain a digital three-dimensional model of the oral cavity. These binocular cameras and RGBD sensors can measure the distance through disparity maps and structured light technologies to obtain the scale. However, this type of oral scan that uses binocular cameras or RGBD sensors for three-dimensional reconstruction is expensive and difficult to miniaturize. If 3D reconstruction can be performed using a monocular camera, the cost and size of the scanner can be effectively reduced. However, due to the limitations of monocular 3D reconstruction itself, it is impossible to determine the actual scale of the model based on images alone. Summary of the invention

[0003] The purpose of the present invention is to solve the above problems and disclose a method, system and storage medium for oral modeling based on monocular vision, comprising the following steps:

[0004] S1, acquiring an oral cavity image with a calibration object through a monocular camera, wherein the calibration object is attached to the front surface of the gums, and generating first point cloud data including features of the oral cavity and the calibration object;

[0005] S2, classifying the first point cloud data into first-category points and second-category points by a first neural network classifier, wherein the first-category points are point clouds attached to the surface of teeth, gums or calibration objects, and the second-category points are discrete point clouds;

[0006] S3, traversing each frame of the oral cavity image, deleting the feature points and constraint relationships in each frame of the image as the second type of points, taking the feature points in all the remaining frame images as candidate feature points of the first type of points, and deleting the entire frame of the image that does not contain the first type of points, to obtain the second point cloud data composed of the first type of points;

[0007] S4, screening and classifying the second point cloud data by a second neural network classifier to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data respectively;

[0008] S5, comparing the calibration object point cloud data with the pre-stored calibration object actual model to determine the calibration object scaling factor, and reconstructing the tooth point cloud data according to the scaling factor to obtain a tooth three-dimensional model.

[0009] Preferably, step S2 includes: obtaining first point cloud data as a sample and annotating it, annotating the point cloud attached to the surface of the teeth, gums or calibration object as first-category points, annotating the discrete point cloud as second-category points, and sending the annotated first-category points and second-category points to a first neural network classifier for training for screening the first-category points.

[0010] Preferably, the step S2 further comprises:

[0011] Each feature point in the first point cloud data as a sample is judged separately. If the feature point is larger than the surrounding area, the feature point is retained. Otherwise, the feature point adjacent to the right is used as a maximum value candidate point.

[0012] Search to the right in a monotonically increasing manner until a feature point is found that satisfies I[i]>I[i+1]. If i<=W-1, keep the point.

[0013] Calculate the quality of the feature point If the quality of the point is greater than a certain threshold, the point is retained and multiplied by the contribution weight of the surrounding points to the point, otherwise the quality of the point is set to zero;

[0014] Determine whether there is redundancy between the point and the feature points at adjacent positions in the previous and next frames. If there is redundancy, delete the feature point; otherwise, retain the feature point and mark it as the first type point.

[0015] Preferably, step S4 includes: the calibration object is a rectangular calibration object, obtaining second point cloud data as a sample, and differently annotating the point clouds belonging to the rectangular calibration object and the teeth in the second point cloud data, respectively, and sending the annotated second point cloud data to a second neural network classifier for training for screening the calibration object point cloud data and the tooth point cloud data.

[0016] Preferably, step S5 comprises:

[0017] Uniformly sampling the three-dimensional model of the calibration object separately to obtain first sampling data;

[0018] Performing plane extraction on the first sampling data to obtain point data of four planes of the calibration object that are in contact with each other as adjacent surface point cloud data;

[0019] Adding a plane regularization constraint to the first sampling data to obtain the second sampling data, wherein the plane regularization constraint is that the distance from each associated point of the adjacent surface point cloud data to the current front is minimized, and the angle between the two surfaces with less point cloud data is close to zero, and the angle between the two surfaces with less point cloud data and the two surfaces with more point cloud data is close to 90 degrees;

[0020] The second sampling data is compared with the calibration object point cloud data to determine the calibration object scaling factor.

[0021] The present invention also discloses an oral scanning system based on monocular vision, comprising a single-sided camera and a controller connected to the monocular camera, wherein the controller is configured to: obtain an oral image with a calibration object through the monocular camera, wherein the calibration object is attached to the front surface of the gums, and generate first point cloud data containing features of the oral cavity and the calibration object; classify the first point cloud data through a first neural network classifier to form first-class points and second-class points, wherein the first-class points are point clouds attached to the teeth, gums or the surface of the calibration object, and the second-class points are discrete point clouds; traverse each frame of the oral image, delete each frame, and then generate a first point cloud data containing features of the oral cavity and the calibration object; classify the first point cloud data through a first neural network classifier to form first-class points and second-class points, wherein the first-class points are point clouds attached to the surface of the teeth, gums or the calibration object, and the second-class points are discrete point clouds; traverse each frame of the oral image, delete each frame, and then generate a first point cloud data containing features of the oral cavity and the calibration object; classify the first point cloud data into first-class points and second-class points, wherein the first-class points are point clouds attached to the surface of the teeth, gums or the calibration object, and the second-class points are discrete point clouds ... The feature points and constraint relationships of the second category points in the frame image are used as candidate feature points of the first category points, and the entire frame image that does not contain the first category points is deleted to obtain the second point cloud data composed of the first category points; the second point cloud data is screened and classified by the second neural network classifier to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data respectively; the calibration object point cloud data is compared with the pre-stored calibration object actual model to determine the calibration object scaling factor, and the tooth point cloud data is reconstructed according to the scaling factor to obtain a tooth three-dimensional model.

[0022] Preferably, the controller is also configured to annotate the first point cloud data acquired as a sample, annotate the point cloud attached to the teeth, gums or the surface of the calibration object as first-category points, annotate the discrete point cloud as second-category points, and send the annotated first-category points and second-category points to the first neural network classifier for training for screening the first-category points.

[0023] Preferably, the controller is also configured to obtain second point cloud data as a sample, respectively label the point clouds belonging to the rectangular calibration object and the teeth in the second point cloud data differently, and send the labeled second point cloud data to a second neural network classifier for training for screening the calibration object point cloud data and the tooth point cloud data.

[0024] The present invention also discloses an oral modeling device based on monocular vision, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0025] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0026] The oral modeling method, system and storage medium based on monocular vision disclosed in the present invention attach the calibration object to the front surface of the gums, then use a monocular camera to obtain an oral image with the calibration object, and generate first point cloud data containing oral and calibration object features, then remove outliers and interference information from the first point cloud data, separate it to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data, compare the calibration object point cloud data with the pre-stored calibration object actual model to determine the calibration object scaling factor, and reconstruct the tooth point cloud data according to the scaling factor to obtain a tooth three-dimensional model. The entire intraoral scanning process can be completed by relying on a single-sided camera, overcoming the problem that the existing intraoral scanner that relies on binocular reconstruction or RGBD to perform three-dimensional reconstruction of teeth in the oral cavity has high equipment cost and is difficult to miniaturize. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of the steps of an oral cavity modeling method based on monocular vision disclosed in one embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of a specific flow chart of step S5 disclosed in one embodiment of the present invention.

[0029] Figure 3 A schematic diagram of sampling the true value of a calibration object disclosed in an embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram of a plane after regularization constraint according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0033] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0034] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0035] Intraoral scanning 3D reconstruction technology can perform 3D reconstruction of teeth in the oral cavity through monocular reconstruction to obtain a digital 3D model of the oral cavity. However, due to the limitations of monocular 3D reconstruction itself, it is often difficult to determine the actual scale of the model based on only one picture. At the same time, in order to reduce costs, the use of additional sensors to complete the recovery of scale factors should be avoided. In order to achieve this requirement, the present invention uses a method of measuring the 3D model of the calibration object and obtains the scale scaling factor of the entire model by comparing it with the physical size of the actual calibration object. As shown in the attached figure Figure 1 As shown, the oral cavity modeling method based on monocular vision may specifically include the following steps:

[0036] Step S1, acquiring an oral cavity image with a calibration object through a monocular camera, wherein the calibration object is attached to the front surface of the gums, and generating first point cloud data including features of the oral cavity and the calibration object.

[0037] The calibration object can be any geometric body whose three-dimensional scale data is known in advance, such as a cuboid, an ellipsoid, and other irregular objects. Before scanning, the calibration object is attached to the front surface of the gums without blocking the tooth features, and then a monocular camera is used to scan and obtain an oral image containing the calibration object. In this embodiment, a cuboid is preferably used as the calibration object, and the cuboid calibration object is attached to the front surface of the gums to participate in the entire subsequent modeling process.

[0038] Step S2, classifying the first point cloud data by a first neural network classifier to form first-category points and second-category points, wherein the first-category points are point clouds attached to the surface of teeth, gums or calibration objects, and the second-category points are discrete point clouds.

[0039] In this embodiment, the system obtains the first point cloud data as a sample and labels it, labels the point cloud attached to the teeth, gums or the surface of the calibration object as the first type of points, labels the discrete point cloud as the second type of points, and sends the labeled first type of points and second type of points to the first neural network classifier for training for screening the first type of points.

[0040] Specifically, since it is difficult to ensure that only teeth exist in the scene during the tooth scanning process, part of the inner wall of the mouth will also be included in the model. However, the muscles of the inner wall of the mouth have irregular movements, and the point cloud formed by this part of the content is usually wrong, and will cause serious interference to the subsequent dense reconstruction. Therefore, it is necessary to first perform a round of cleaning on the sparse point cloud so that the correct points are retained and the wrong or useless points are removed. Therefore, after the sparse reconstruction is completed, the point cloud is first manually annotated, and the point cloud of the teeth attached to the tooth surface is regarded as Class A, that is, the first type of points, and the discrete point cloud is regarded as Class B, that is, the second type of points. These two types of data are sent to a three-dimensional point cloud classification model, namely the first neural network classifier, for training. After training, the model has the ability to perceive outliers.

[0041] In this embodiment, the step S2 further includes:

[0042] Step S21, each feature point in the first point cloud data as a sample is judged respectively, if the feature point is larger than the surrounding area, the feature point is retained, otherwise the feature point adjacent to the right is used as a maximum value candidate point.

[0043] Step S22, search rightward in a monotonically increasing manner until a feature point is found that satisfies I[i]>I[i+1] of the i-th point, and if i<=W-1, the point is retained, where w is the total number of elements, and i must not cross the boundary when traversing.

[0044] Step S23, calculate the quality of the feature point If the quality of the point is greater than a certain threshold, the point is retained and multiplied by the contribution weight of the surrounding points to the point, otherwise the quality of the point is set to zero. The contribution weight of the surrounding points to the point can be given by neural network regression.

[0045] Step S24, determine whether there is redundancy between the point and the feature points at adjacent positions of the previous and next frames. If there is redundancy, delete the feature point; otherwise, keep the feature point and mark it as a first-category point.

[0046] In this embodiment, the screening rules may include: the number of feature points contained in the image, the quality of the feature points contained in the image, and the redundancy with the surrounding neighborhood images, and the judgment order is executed from front to back. Specifically, firstly, feature extraction is performed on an image. Considering the reflective and light-absorbing characteristics of the teeth themselves, the usual feature extraction methods such as SIFT cannot be used directly, and high-dimensional features of the surface and contour should be extracted. Non-maximum suppression is performed on it. In a preferred embodiment, the specific algorithm is as follows:

[0047]

[0048] First, according to the third row, determine whether the current feature point I[i] is larger than its surrounding neighborhood. If it meets the conditions, it is retained. If the feature I[i] does not meet the judgment conditions of the third row, its right neighbor I[i+1] is used as a maximum candidate, corresponding to the seventh row. Use a monotonically increasing method to search to the right until a feature point that satisfies I[i]>I[i+1] is found. If i<=W-1, the point is retained.

[0049] Then judge the quality of the point: If the point is greater than a certain threshold, the point is retained and multiplied by the contribution weight of the surrounding points to the point, otherwise it is directly set to 0. Finally, it is determined whether the point is redundant with the feature points of the adjacent positions of the previous and next frames to avoid the same features due to too long a stay time.

[0050] Step S3, traverse each frame of the oral image, delete the feature points and constraints of the second type of points in each frame, use the feature points in all the remaining frame images as candidate feature points of the first type of points, and delete the entire frame image that does not contain the first type of points, and obtain the second point cloud data composed of the first type of points. That is, when a new point cloud enters, first use the three-dimensional point cloud classification model that has been trained in step S2, that is, the first neural network classifier to classify it, so that the A type points pass through and prevent the B type points from participating in subsequent calculations. At the same time, traverse all images and delete the feature points and constraints with the B type points. Only the A type points are candidate feature points in all the remaining frames. If the image does not contain the A type points, delete the frame image.

[0051] Step S4, screening and classifying the second point cloud data by a second neural network classifier, and obtaining tooth point cloud data, calibration object point cloud data and gum point cloud data respectively.

[0052] The step S4 includes: the calibration object is a rectangular calibration object, obtaining second point cloud data as a sample, labeling the point clouds belonging to the rectangular calibration object and teeth in the second point cloud data differently, and sending the labeled second point cloud data to the second neural network classifier for training to screen the calibration object point cloud data and the tooth point cloud data. Specifically, after obtaining the complete model, the point cloud in the rectangular calibration block is regarded as a target for training through manual labeling, so that it has the ability to distinguish the calibration object.

[0053] Step S5, comparing the calibration object point cloud data with the pre-stored calibration object actual model to determine the calibration object scaling factor, and reconstructing the tooth point cloud data according to the scaling factor to obtain a tooth three-dimensional model.

[0054] Due to the existence of modeling errors and segmentation errors, the point cloud obtained at this time cannot be directly used for scale calculation, and a series of processing is required to improve this process. Figure 2 As shown, step S5 includes:

[0055] Step S51, uniformly sampling the three-dimensional model of the calibration object to obtain first sampling data, as shown in the attached Figure 3 The point cloud obtained after uniform sampling of the calibration object is shown. In this embodiment, the original calibration object STL model generated by Rhino modeling and the calibration object 3D model generated by monocular 3D reconstruction can be uniformly sampled. The uniform sampling step mainly includes the following contents:

[0056] Step S511, setting an expected number of sampling points N which is much larger than the number of triangles in the 3D model file.

[0057] Step S512, take out one triangle face at a time in order and record it as Triangle, where the coordinates of the three vertices are recorded as v 1 、v 2 and v 3 .

[0058] Step S513, generating a random number r according to a continuous uniform distribution 1 With r 2 , and the weights a, b, c of each vertex can be calculated based on the above random numbers. Sampling generates the coordinates v' of the new point, where v' = a*v 1 +b*v 2 +c*v 3 .

[0059] Step S514: when the number of sampled points is less than the required number of samples N, repeat steps S512-S513.

[0060] Step S52, plane extraction is performed on the first sampled data, and point data of four planes of the calibration object that are in contact with each other are obtained as adjacent surface point cloud data. Specifically, plane extraction is performed on the first sampled data obtained in step S51, and it is divided into four surfaces A, B, C, and D. This step may specifically include the following contents:

[0061] Step S521: randomly select three non-collinear points from all points in the point cloud to construct a plane equation Let these three points be represented by p 1 、p 2 、p 3 ,in Among them, d can be the intercept, and p1 can be a point randomly selected from p1 p2 p3.

[0062] Step S522: Calculate the distance between the remaining points and the plane equation. Points with distances less than the threshold can be used as the inner points of the plane model. i Indicates other points in the point cloud except the above three points. The distance formula can be expressed as:

[0063] Step S523, repeating the above steps S521 and S522 for a first predetermined number of times, where the first predetermined number of times may be 1000 times, and the plane equation having the largest number of internal points is the surface extracted this time.

[0064] Step S524, repeat step S523 for a second predetermined number of times, where the second predetermined number of times may be 4 times, and the four faces A, B, C, and D may be extracted respectively. Then, based on the relationship between the two parallel and coplanar faces, the ab face and the cd face may be distinguished. Thus, the equations of the fitted planes and the interior points of the planes may be obtained.

[0065] Step S53, adding a plane regularization constraint to the first sampling data to obtain the second sampling data, wherein the plane regularization constraint is that the distance from each associated point of the adjacent surface point cloud data to the current front is minimized, and the angle between the two surfaces with less point cloud data is close to zero, and the angle between the two surfaces with less point cloud data and the two surfaces with more point cloud data is close to 90 degrees.

[0066] Specifically, the plane regularization constraint is added to the point cloud extracted in step S53. There are two error terms here, one is the distance from the point to the plane, and the other is the angle between the planes. It is necessary to ensure that the distance from the point associated with each plane to the current front is minimized, and at the same time, it is also necessary to ensure that the angle between plane A and plane B is close to zero, and the angle between plane A and plane C, and between plane C and plane D is close to 90 degrees, as shown in the attached figure. Figure 4 shown.

[0067] Step S54: compare the second sampled data with the calibration object point cloud data to determine the calibration object scaling factor.

[0068] Specifically, based on the results of the previous steps, the scaling factor s of the calibration object obtained by monocular reconstruction can be estimated: 0 ,s 0 is the ratio of the actual length of the calibration object to the estimated length after reconstruction. Then [0.8s 0 ,1.2s 0 ] to perform multi-step search within the range, the specific steps are as follows:

[0069] Set the step size of the first round of search to 0.1s 0 The standard point cloud is cube gt , the reconstructed point cloud is cube i . Traverse the cube i For each point in the cube, find its gt The nearest point in the cube is set as the corresponding point. Use the iterative method to solve the translation and rotation so that the cube i The sum of the squares of the distances from the upper point to the corresponding point on the tangent plane is the smallest, and the overall error at this time is recorded as error i .

[0070] Repeat the above steps for a predetermined number of times, in this case 20 times, until the search interval is exhausted, at which point s′ with the minimum error can be obtained. 0 Reduce the step size by 0.1 times and iterate the above operation 3 times to get a more accurate scale factor.

[0071] In this embodiment, the calibration object is attached to the front surface of the gums, and then a monocular camera is used to obtain an oral image with the calibration object, and a first point cloud data containing oral and calibration object features is generated. Then, after removing outliers and interference information from the first point cloud data, the first point cloud data is separated to obtain tooth point cloud data, calibration object point cloud data, and gum point cloud data. The calibration object point cloud data is compared with the pre-stored actual model of the calibration object to determine the calibration object scaling factor, and the tooth point cloud data is reconstructed according to the scaling factor to obtain a tooth three-dimensional model. The entire intraoral scanning process can be completed by relying on a single-sided camera, overcoming the problem that the existing intraoral scanners that rely on binocular reconstruction or RGBD to perform three-dimensional reconstruction of teeth in the oral cavity are high in equipment cost and difficult to miniaturize.

[0072] In other embodiments, an oral scanning system based on monocular vision is also disclosed, comprising a single-sided camera and a controller connected to the monocular camera, wherein the controller is configured to: acquire an oral image with a calibration object through the monocular camera, wherein the calibration object is attached to the front surface of the gums, and generate first point cloud data containing oral and calibration object features; classify the first point cloud data through a first neural network classifier to form first-category points and second-category points, wherein the first-category points are point clouds attached to teeth, gums or the surface of the calibration object, and the second-category points are discrete point clouds; traverse each frame of the oral image, Delete the feature points and constraint relationships of the second category points in each frame image, use the feature points in all the remaining frame images as candidate feature points of the first category points, and delete the entire frame image that does not contain the first category points to obtain the second point cloud data composed of the first category points; screen and classify the second point cloud data through the second neural network classifier to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data respectively; compare the calibration object point cloud data with the pre-stored calibration object actual model to determine the calibration object scaling factor, and reconstruct the tooth point cloud data according to the scaling factor to obtain a tooth three-dimensional model.

[0073] In this embodiment, the controller is also configured to annotate the first point cloud data obtained as a sample, annotate the point cloud attached to the teeth, gums or the surface of the calibration object as the first type of points, annotate the discrete point cloud as the second type of points, and send the annotated first type of points and second type of points to the first neural network classifier for training for screening the first type of points.

[0074] In this embodiment, the controller is also configured to obtain second point cloud data as a sample, and to perform different annotations on the point clouds belonging to the rectangular calibration object and the teeth in the second point cloud data, and to send the labeled second point cloud data to the second neural network classifier for training for screening the calibration object point cloud data and the tooth point cloud data.

[0075] The specific functions of the above-mentioned oral scanning system based on monocular vision correspond to the oral scanning method based on monocular vision disclosed in the previous embodiment, so it will not be described in detail here. For details, please refer to the various embodiments of the oral scanning method based on monocular vision disclosed in the previous embodiment. It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0076] In other embodiments, an oral scanning device based on monocular vision is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the oral scanning method based on monocular vision as described in the above embodiments is implemented.

[0077] The oral scanning device based on monocular vision may include, but is not limited to, a processor and a memory. A person skilled in the art may understand that the schematic diagram is only an example of an oral scanning device based on monocular vision, and does not constitute a limitation on the oral scanning device based on monocular vision. The device may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the oral scanning device based on monocular vision may also include input and output devices, network access devices, buses, etc.

[0078] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the oral scanning device based on monocular vision, and uses various interfaces and lines to connect various parts of the entire oral scanning device based on monocular vision.

[0079] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the device for oral scanning based on monocular vision by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0080] If the oral scanning device based on monocular vision is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the above-mentioned steps of each embodiment of the oral scanning method based on monocular vision. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0082] In conclusion, the above is only a preferred embodiment of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the patent of the present invention.

Claims

1. An oral cavity modeling method based on monocular vision, characterized in that: The steps include: S1, acquiring an oral cavity image with a calibration object through a monocular camera, wherein the calibration object is attached to the front surface of the gums, and generating first point cloud data including features of the oral cavity and the calibration object; S2, classifying the first point cloud data into first-category points and second-category points by a first neural network classifier, wherein the first-category points are point clouds attached to the surface of teeth, gums or calibration objects, and the second-category points are discrete point clouds; S3, traversing each frame of the oral cavity image, deleting the feature points and constraint relationships in each frame of the image as the second type of points, taking the feature points in all the remaining frame images as candidate feature points of the first type of points, and deleting the entire frame of the image that does not contain the first type of points, to obtain the second point cloud data composed of the first type of points; S4, screening and classifying the second point cloud data by a second neural network classifier to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data respectively; S5, comparing the calibration object point cloud data with the pre-stored calibration object actual model to determine the calibration object scaling factor, and reconstructing the tooth point cloud data according to the scaling factor to obtain a tooth three-dimensional model.

2. The oral cavity modeling method based on monocular vision according to claim 1, characterized in that: The step S2 comprises: The first point cloud data as a sample is obtained and annotated, the point cloud attached to the teeth, gums or the surface of the calibration object is annotated as the first type of points, and the discrete point cloud is annotated as the second type of points. The annotated first type of points and second type of points are sent to the first neural network classifier for training to screen the first type of points.

3. The oral cavity modeling method based on monocular vision according to claim 2, characterized in that: The step S2 further comprises: Each feature point in the first point cloud data as a sample is judged separately. If the feature point is larger than the surrounding area, the feature point is retained. Otherwise, the feature point adjacent to the right is used as a maximum value candidate point. Search to the right in a monotonically increasing manner until a feature point is found that satisfies I[i]>I[i+1]. If i<=W-1, keep the point. Calculate the quality of the feature point If the quality of the point is greater than a certain threshold, the point is retained and multiplied by the contribution weight of the surrounding points to the point, otherwise the quality of the point is set to zero; Determine whether there is redundancy between the point and the feature points at adjacent positions in the previous and next frames. If there is redundancy, delete the feature point; otherwise, retain the feature point and mark it as the first type point.

4. The oral cavity modeling method based on monocular vision according to claim 3, characterized in that: The step S4 comprises: The calibration object is a rectangular calibration object, and second point cloud data is obtained as a sample. The point clouds belonging to the rectangular calibration object and the teeth in the second point cloud data are differently labeled, and the labeled second point cloud data is sent to the second neural network classifier for training for screening the calibration object point cloud data and the tooth point cloud data.

5. The oral cavity modeling method based on monocular vision according to claim 4, characterized in that: The step S5 comprises: Uniformly sampling the three-dimensional model of the calibration object separately to obtain first sampling data; Performing plane extraction on the first sampling data to obtain point data of four planes of the calibration object that are in contact with each other as adjacent surface point cloud data; Adding a plane regularization constraint to the first sampling data to obtain the second sampling data, wherein the plane regularization constraint is that the distance from each associated point of the adjacent surface point cloud data to the current front is minimized, and the angle between the two surfaces with less point cloud data is close to zero, and the angle between the two surfaces with less point cloud data and the two surfaces with more point cloud data is close to 90 degrees; The second sampling data is compared with the calibration object point cloud data to determine the calibration object scaling factor.

6. An oral scanning system based on monocular vision, characterized in that: The invention comprises a monocular camera and a controller connected to the monocular camera, wherein the controller is configured as follows: Acquire an oral cavity image with a calibration object through a monocular camera, wherein the calibration object is attached to the front surface of the gums, and generate first point cloud data including features of the oral cavity and the calibration object; Classifying the first point cloud data by a first neural network classifier to form first-category points and second-category points, wherein the first-category points are point clouds attached to the surface of teeth, gums or calibration objects, and the second-category points are discrete point clouds; Traversing each frame of the oral cavity image, deleting the feature points and constraint relationships in each frame of the image as the second type of points, taking the feature points in all the remaining frame images as candidate feature points of the first type of points, and deleting the entire frame of the image that does not contain the first type of points, to obtain the second point cloud data composed of the first type of points; The second point cloud data is screened and classified by a second neural network classifier to obtain tooth point cloud data, calibration object point cloud data and gum point cloud data respectively; The calibration object point cloud data is compared with the pre-stored calibration object actual model to determine the calibration object scaling factor, and the tooth point cloud data is reconstructed according to the scaling factor to obtain a tooth three-dimensional model.

7. The oral cavity scanning system based on monocular vision according to claim 6, characterized in that: The controller is also configured to annotate the first point cloud data obtained as a sample, annotate the point cloud attached to the teeth, gums or the surface of the calibration object as the first type of points, annotate the discrete point cloud as the second type of points, and send the annotated first type of points and second type of points to the first neural network classifier for training for screening the first type of points.

8. The oral cavity scanning system based on monocular vision according to claim 7, characterized in that: The controller is also configured to obtain second point cloud data as a sample, and to perform different annotations on the point clouds belonging to the rectangular calibration object and the teeth in the second point cloud data, and to send the annotated second point cloud data to a second neural network classifier for training to screen the calibration object point cloud data and the tooth point cloud data.

9. An oral modeling device based on monocular vision, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images