A method, system and storage medium for oral cavity three-dimensional reconstruction based on smart phone

Abstract

The present invention discloses a smart phone-based oral cavity 3D reconstruction method, system and storage medium, which uses a smart phone lens as an oral cavity scanning device to perform 3D construction of teeth in the oral cavity, obtains a series of images with different depths of field in the patient's mouth by changing the offset of the mobile phone lens while taking high-speed photos, processes the acquired images to form a single-view depth map, converts the depth maps of multiple viewpoints into point cloud data, splices and grids them to form a final 3D model, and transmits the real image of the patient's oral cavity teeth to the smart phone to restore it into an oral cavity model, thereby greatly reducing the cost of oral cavity scanning services and improving efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional modeling, and in particular relates to a method, system and storage medium for oral three-dimensional reconstruction based on a smartphone. Background Art

[0002] In the process of clinical oral diagnosis and treatment and restoration, the methods of obtaining a three-dimensional digital model of the inside of the oral cavity through three-dimensional scanning are mainly divided into two types: extraoral scanning and intraoral scanning. The extraoral scanning method requires the doctor to first make an impression of the oral cavity to obtain a plaster model of the teeth, and then use a three-dimensional scanning device to scan the plaster dental mold to obtain its three-dimensional digital model; the intraoral scanning method is to insert the oral scanner into the oral cavity to directly scan the teeth to obtain a three-dimensional digital model of the teeth. This method is simple to operate, efficient and fast in measurement, thus saving the doctor's chairside operation time. Since there is no need for manual impressions, it also avoids errors caused by mold making and mold turning, and has high measurement accuracy. However, the existing oral scanners are composed of an oral scanner and a huge host station connected to the oral scanner. During operation, the doctor holds the scanner and puts it into the patient's mouth. The scanner camera continuously shoots and transmits it to the host station, and the host station software makes real-time 3D modeling. During the operation, there are upper and lower teeth and tongue in the oral cavity, which are complex in shape. When the patient opens his mouth wide, the mouth and tongue are uncomfortable and constantly moving, requiring the doctor to observe and adjust at any time and communicate with the patient. At the same time, in order to allow the scanner to capture images at the required position without often hitting or bumping into other parts of the mouth, the doctor's line of sight needs to constantly move between the mouth and the host station screen nearby, resulting in reduced scanning efficiency. In addition, the huge host station causes the entire oral scanner system to occupy a large amount of limited diagnosis and treatment space and is expensive. Summary of the invention

[0003] The purpose of the present invention is to solve the above problems and disclose a method for oral 3D reconstruction based on a smartphone, comprising the following steps:

[0004] S1, calibrating the focus module of the smartphone to obtain the depth value of each focal plane when the smartphone is focusing;

[0005] S2, controlling the lens module of the smartphone to zoom and simultaneously obtaining different depth-of-field images of the interior of the oral cavity through an external probe as a first image group, and recording the corresponding focal lengths;

[0006] S3, sequentially selecting the first and second frames of the first image group and searching them using the SIFT algorithm to obtain their respective feature points, using the RANSAC algorithm to solve the transformation relationship between the two sets of feature points, and solving the corresponding homography matrix, and transforming the second image using the solved matrix to obtain a second image group composed of pictures that eliminate the near-big-far-small effect;

[0007] S4, superimposing the frames of images in the second image group together to perform Laplace transform, selecting the image with the largest response value at the same pixel coordinate as the focal plane corresponding to the pixel, and obtaining the depth corresponding to each pixel according to the focal plane depth value;

[0008] S5, sequentially obtaining the pixel coordinates of each pixel at which the contrast is the largest from front to back and recording them to form the viewing angle depth map, and simultaneously obtaining the position and posture information corresponding to the smartphone when collecting the group of depth maps;

[0009] S6, repeat the above steps to obtain multi-view images and convert them into multi-view point cloud data, splice and grid the multi-view point cloud data to form a digital model of the oral cavity.

[0010] Preferably, the step S2 specifically includes:

[0011] S21, when the lens module of the smartphone can obtain a tooth image through the external probe, control the lens module to continuously adjust the focal length to a focal length at which the current tooth can be clearly presented on the smartphone display screen, and scale the focal length to obtain a focal range that can cover the current tooth surface;

[0012] S22, controlling the lens module to capture a continuous initial image group of the interior of the oral cavity at a high rate, and capturing a plurality of images in the continuous initial image group according to the focal length range as a first image group.

[0013] Preferably, the step S3 specifically includes:

[0014] S31, searching the previous and next two frames of images in the first image group using the SIFT algorithm to obtain their respective feature points;

[0015] S32, randomly selecting multiple groups of matching feature points to calculate the homography matrix, calculating the errors of the remaining matching feature point pairs after the homography matrix transformation, and obtaining the number of feature point pairs with errors less than a threshold, and obtaining the homography matrix with the largest number of feature point pairs with errors less than the threshold;

[0016] S33, transforming the second image using the solved homography matrix to obtain a transformed image that eliminates the near-big-far-small effect;

[0017] S34, continuously adopting the aforementioned steps S31 to S33 to sequentially transform the two frames of images in the first image group to obtain a transformed image that eliminates the near-big-far-small effect, thereby forming a second image group.

[0018] Preferably, the step S6 specifically includes:

[0019] According to the overlapping parts of the two frames, the target tooth is superimposed on the corresponding tooth position; the point cloud sets A and B of the two frames are downsampled as shown below:

[0020]

[0021]

[0022]

[0023] Among them A l , B l Respectively represent the point cloud sets A and B at resolution level 1, f sample↓ Indicates downsampling operation, L is the maximum resolution level set, represents a single point cloud a with sequence number i in the point cloud set A at resolution level 1, The number of point clouds in the point cloud set A is a positive integer, and there are n in total; represents a single point cloud b with sequence number j in the point cloud set B at resolution level l, The number of point clouds in the point cloud set B is a positive integer, and there are m in total;

[0024] Use the nearest neighbor search to get two point cloud sets A l With B l Bidirectional transformation relationship at resolution l (k-1) th , as shown below:

[0025]

[0026]

[0027] Where K represents the number of iterations, They represent respectively that at resolution 1, in (k-1) iterations, The values ​​of variables i and j with the smallest error; They respectively represent the scaling matrix, rotation matrix, and translation matrix required for the transformation in (k-1) iterations at resolution level l. represents a single point cloud a with sequence number i in the point cloud set A at resolution level 1, Represents a single point cloud b with serial number j in the point cloud set B at resolution level 1; calculate the new transformation matrix between the two point clouds based on the transformation matrix obtained in the previous step, and update and Repeat the above steps until the error accuracy reaches resolution l=1.

[0028] The present invention also discloses an oral 3D reconstruction system based on a smart phone, comprising: a calibration module, used to calibrate the focus module of the smart phone, and obtain the depth value of each focal plane when the smart phone is focused; a first image group acquisition module, used to control the lens module of the smart phone to zoom and simultaneously obtain different depth of field images of the inside of the oral cavity as the first image group through an external probe, and record the corresponding focal length; a second image group acquisition module, used to sequentially select the front and rear two frames of the first image group, use the SIFT algorithm to search to obtain their respective feature points, use the RANSAC algorithm to solve the transformation relationship between the two groups of feature points, and solve the corresponding homography matrix, and use the solved matrix to transform the second image to eliminate the near big and far small The image of the effect forms a transformed second image group; a depth acquisition module is used to overlap the frames of the second image group and perform Laplace transform, select the image with the largest response value under the same pixel coordinate as the focal plane corresponding to the pixel, and obtain the depth corresponding to each pixel according to the focal plane depth value; a single-view depth map module is used to obtain the pixel coordinates of each pixel at the maximum contrast in order from front to back and record them to form the perspective depth map, and at the same time obtain the posture information corresponding to the smartphone when collecting the group of depth maps; a model generation module is used to obtain a multi-view image based on the obtained multiple single-view depth maps and convert it into multi-view point cloud data, splice and grid the multi-view point cloud data to form a digital model of the oral cavity.

[0029] Preferably, the first image group acquisition module specifically includes: a focal range acquisition module, which is used to control the lens module to continuously adjust the focal length to a focal length at which the current teeth can be clearly presented on the smartphone display screen when the lens module of the smartphone can acquire a tooth image through an external probe, and to scale the focal length to acquire a focal range that can cover the current tooth surface; an image capture module, which is used to control the lens module to capture a continuous initial image group of the inside of the oral cavity at a high rate, and to capture multiple pictures in the continuous initial image group according to the focal range as the first image group.

[0030] Preferably, the second image group acquisition module specifically includes: a feature point acquisition module, which is used to search the front and rear two frames of images in the first image group using the SIFT algorithm to obtain their respective feature points; a homography matrix acquisition module, which is used to randomly select multiple groups of matching feature points to calculate the homography matrix, calculate the errors of the remaining matching feature point pairs after the homography matrix transformation, and obtain the number of feature point pairs whose errors are less than a threshold, and obtain the homography matrix with the largest number of feature point pairs whose errors are less than the threshold; an image transformation module, which is used to transform the second image using the solved homography matrix to obtain a transformed image that eliminates the near-big-far-small effect; a second image group generation module, which is used to sequentially transform the front and rear two frames of images in the first image group to obtain a transformed image that eliminates the near-big-far-small effect to form a second image group.

[0031] Preferably, the model generation module specifically includes: performing registration according to the overlapping parts of the two frames of images, superimposing the target tooth on the corresponding tooth position, and downsampling the point cloud sets A and B of the two frames of images, as shown below:

[0032]

[0033]

[0034]

[0035] Among them A l , B l Respectively represent the point cloud sets A and B at resolution level l, f sample↓ Indicates downsampling operation, L is the maximum resolution level set, represents a single point cloud a with sequence number i in the point cloud set A at resolution level 1, The number of point clouds in the point cloud set A is a positive integer, and there are n in total; represents a single point cloud b with sequence number j in the point cloud set B at resolution level l, The number of point clouds in the point cloud set B is a positive integer, and there are m in total;

[0036] Use the nearest neighbor search to get two point cloud sets A l With B l Bidirectional transformation relationship at resolution l (k-1) th , as shown below:

[0037]

[0038]

[0039] Where K represents the number of iterations, They represent respectively that at resolution 1, in (k-1) iterations, The values ​​of variables i and j that minimize the error. They respectively represent the scaling matrix, rotation matrix, and translation matrix required for the transformation in (k-1) iterations at resolution level l. represents a single point cloud a with sequence number i in the point cloud set A at resolution level 1, Represents a single point cloud b with serial number j in the point cloud set B at resolution level l. Based on the obtained transformation matrix, a new transformation matrix between the two point clouds is calculated and updated and Until the error accuracy reaches resolution l=1.

[0040] The present invention also discloses a smart phone-based oral three-dimensional reconstruction device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-described methods when executing the computer program.

[0041] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0042] The present invention discloses a method, system and storage medium for oral cavity three-dimensional reconstruction based on a smart phone. The lens module of the smart phone is used as an oral cavity scanning device to perform three-dimensional reconstruction of teeth in the oral cavity. By changing the offset of the mobile phone lens, that is, moving the position of the focal plane, a series of images with different depths of field in the patient's mouth are obtained by high-speed photography. The collected data is filtered by deblurring and a series of images are aligned. The last image is used as a calibration reference, each deblurred image is matched with the calibration, the zoom ratio between the two images before and after is calculated, and then the image is scaled according to the zoom ratio to eliminate the near-large-far-small effect. This group of images is Laplace transformed and filtered, and then superimposed together to form a single-view depth map. The depth maps of multiple perspectives are formed into a point cloud, which is aligned, spliced ​​and gridded to form a final three-dimensional model. Thus, the real image in the patient's oral cavity environment is transmitted to the smart phone to restore its real model, which greatly reduces the cost of the oral cavity scanning business and improves efficiency. The problem that current oral scanners are composed of an oral scanner and a huge host station connected to the oral scanner, and the doctor's line of sight needs to constantly move between the oral cavity and the host station screen beside him during operation, resulting in reduced scanning efficiency. In addition, the huge host station causes the entire oral scanner system to occupy a large amount of limited diagnosis and treatment space and is expensive. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a method for oral three-dimensional reconstruction based on a smartphone according to an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of connecting a smart phone to an external probe according to an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a specific flow chart of step S2 disclosed in one embodiment of the present invention;

[0046] Figure 4 This is a feature matching effect diagram disclosed in an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of a specific flow chart of step S3 disclosed in one embodiment of the present invention;

[0048] Figure 6 A schematic diagram of a single-view point cloud disclosed in an embodiment of the present invention;

[0049] Figure 7 This is a digital model of the oral cavity after multi-view stitching disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0052] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0053] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0054] Attached Figure 1 An oral cavity three-dimensional reconstruction method based on a smart phone disclosed in an embodiment of the present invention comprises the following steps:

[0055] Step S1, calibrate the focus module of the smartphone to obtain the depth value of each focal plane when the smartphone is focused. By shooting the calibration plate at different distances, calibrate the moving distance of the focal plane between every two pictures during the focusing process, where the calibration plate can be a circular array. The specific steps are as follows:

[0056] Step S11, using the Hough Circle Transform algorithm to measure the radius of the point spread function. When the point spread function is applied to the image, a blur radius is generated, where the area close to the center of the circle is clearer and the radius edge is blurrier. For each pattern, the part with a clarity between 5% and 95% is located in the effective area, and the other parts that are too blurry are discarded, so that the range and blur radius generated by the blur can be obtained. On this basis, by setting the thresholds of 5% and 95% of the image intensity, the blur radius of the circular boundary area is obtained.

[0057] Step S12, calculate the change of the distance between each circle center, and obtain the change curve of the magnification during the focusing process. By extracting feature points from adjacent frame images, estimating the change relationship of these feature points, and using the difference between these feature points to estimate the change of the image, the change curve of the image during the focusing process is calculated, and this method alleviates the problem of near large and far small.

[0058] Step S2, controlling the lens module of the smartphone to zoom and simultaneously obtaining different depth of field images of the inside of the oral cavity through an external probe as a first image group, and recording the corresponding focal lengths. Figure 2 As shown in the figure, the inside of the oral cavity is photographed, and the lens module is controlled to zoom and record the current focal length during shooting, while recording the images with different depths of field inside the corresponding oral cavity. Figure 3 As shown, the step S2 specifically includes:

[0059] Step S21, when the lens module of the smartphone can obtain a tooth image through an external probe, the lens module is controlled to continuously adjust the focal length to a focal length at which the current teeth can be clearly presented on the smartphone display screen, and the focal length is scaled to obtain a focal range that can cover the current tooth surface.

[0060] Step S22, controlling the lens module to capture a continuous initial image group of the interior of the oral cavity at a high rate, and capturing a plurality of images in the continuous initial image group according to the focal length range as a first image group.

[0061] When photographing the inside of the oral cavity, the portable optical lens device needs to be installed on the front end of the mobile phone. The dentist then inserts the device into the patient's mouth and scans the patient's teeth in a certain order. While scanning, the mobile phone camera takes pictures at high speed to avoid the phenomenon of mismatch between consecutive frames due to irregular muscle shaking. The focal length of the lens is changed during the shooting process to obtain a set of continuous images of the inside of the oral cavity. In this embodiment, the camera is first turned on and aimed at a tooth, and the focal length that makes the current tooth clearest is found through automatic focus. Figure 4 As shown, the focal length is scaled to find the focal range that can cover the current tooth surface. Then start taking photos at high speed and quickly zoom in the focal range to avoid the phenomenon of mismatch between consecutive frames due to irregular muscle shaking, thereby obtaining a set of continuous initial images inside the mouth. For example, if the mobile phone shoots at a rate of 120fps, 120 photos can be obtained in 1 second, and then a subset of the images is intercepted for subsequent processing based on the focal range calculated by autofocus.

[0062] Step S3, sequentially select the two frames before and after the first image group and use the SIFT algorithm to search to obtain their respective feature points, use the RANSAC algorithm to solve the transformation relationship between the two sets of feature points, and solve the corresponding homography matrix, and use the solved matrix to transform the second image to obtain a picture that eliminates the near big far small effect, forming a transformed second image group. Specifically, since the movement of the focal plane will cause this group of pictures to present a process from clear to blurred, and will also produce the near big far small effect, it is necessary to align and correct this group of images. During alignment correction, first use the SIFT algorithm to search the two frames before and after to obtain their respective feature points; then use the RANSAC algorithm to solve the transformation relationship between the two sets of feature points and solve the corresponding homography matrix; finally, use the solved matrix to transform the second image to eliminate the near big far small effect. Use median filtering to smooth this group of images to eliminate the interference caused by the noise generated by jitter.

[0063] In this embodiment, as shown in the attached Figure 5 As shown, step S3 may also specifically include the following contents.

[0064] Step S31, the SIFT algorithm is used to search the two frames of images in the first image group to obtain their respective feature points. Since the handheld shooting method has poor stability, it is necessary to eliminate the image shake and the near small and far big effect caused by the movement of the lens. The SIFT algorithm is first used to search the two frames of images to obtain their respective feature points.

[0065] Step S32, randomly select multiple groups of matching feature points to calculate the homography matrix, calculate the errors of the remaining matching feature points after the homography matrix transformation, and obtain the number of feature point pairs with errors less than the threshold, and obtain the homography matrix with the largest number of feature point pairs with errors less than the threshold. Specifically, use the RANSAC algorithm to solve the transformation relationship between the two groups of feature points, that is, solve the corresponding homography matrix. The specific steps of RANSAC are: first select 4 groups of matching feature points and calculate their homography matrices. Then calculate the errors of the remaining matching feature points after the homography transformation, and record the number of feature point pairs with errors less than the threshold. After the final iteration is completed, the homography matrix with the largest number of feature point pairs with errors less than the threshold is what is required.

[0066] In this embodiment, the Hamming distance between descriptors is used as a metric, and the matching relationship between the two sets of feature points is obtained by combining the approximate nearest neighbor search, and the homography matrix is ​​calculated by randomly selecting four pairs of matching feature points. The Hamming distance formula is as follows: Where a and b represent the binary descriptors of each group of feature points.

[0067] In this embodiment, the homography matrix is ​​calculated as follows:

[0068] Suppose there is a pair of matching feature points (u1, v1), (u2, v2), which satisfy a homography transformation relationship:

[0069] Where (u1, v1) and (u2, v2) represent the pixel coordinates (x, y) of a pair of feature points on the two images. The matrix formed by h1-h9 is the homography matrix H for a pair of points.

[0070] The cases expanded to four pairs of matching points can be:

[0071]

[0072] The vector H here is the required homography transformation matrix H.

[0073] Step S33, transform the second image using the solved homography matrix to obtain a transformed image that eliminates the near-big-far-small effect. Specifically, transform the second image I using the solved matrix H to obtain an image I' that eliminates the near-big-far-small effect, where I'=HI.

[0074] Step S34, continue to use the above steps S31 to S33 to transform the two frames of images in the first image group in turn to obtain a transformed image that eliminates the near big and far small effect, forming a second image group. In addition, the group of images is smoothed using a median filter to eliminate the interference caused by noise caused by jitter. The pixel value of each pixel point output after the group of images is smoothed by the median filter is the median value of the pixel value in the window of the filter kernel size.

[0075] Step S4, superimpose the frames in the second image group and perform Laplace transform, select the image with the largest response value under the same pixel coordinate as the focal plane corresponding to the pixel, and obtain the depth corresponding to each pixel according to the focal plane depth value. Specifically, Laplace transform is performed on the image to find the focal plane corresponding to each pixel, and the focal length is measured at the same time. According to the camera shooting principle, when a pixel is located on the focal plane, the pixel is in a clear or high sharpness state, otherwise it is blurred or the sharpness value is low. The group of photos is superimposed together, Laplace transform is performed, and the focal length value corresponding to the moment when a pixel presents the maximum sharpness is found, and the depth estimation can be performed. First, LoG filtering is performed on each picture to extract the pixel position with the largest gradient change in each picture, that is, LoG filtering can extract the pixel coordinates corresponding to the focal plane when each photo is taken. For the case where each picture has multiple response values ​​at the same pixel coordinate, we select the image with the largest response value as the focal plane corresponding to that pixel. At the same time, since the focal plane has spatiotemporal continuity when moving, the focal plane has a relative depth relationship. The focal plane corresponding to the previous frame of the photo will be deeper (shallower) than the focal plane of the next frame. Based on the focal plane corresponding to each pixel when it is taken, and the motion model of the focal plane movement obtained from the previous steps, that is, the image change curve during the focusing process, the distance of each pixel to the camera, that is, the depth, can be obtained.

[0076] Step S5, sequentially obtaining the pixel coordinates of each pixel at the maximum contrast from front to back and recording them to form the viewing angle depth map, and simultaneously obtaining the position and posture information corresponding to the smartphone when collecting the group of depth maps.

[0077] A single-view depth map can be obtained from the focal length and the original image. At the same time, the smartphone inertial device module is called to record and save the position of the smartphone when the depth map is obtained. Specifically, the pixel coordinates of each pixel with the maximum contrast are found from front to back and recorded to form a depth map of the view, which can be converted into point cloud data as shown in the attached figure. Figure 6 At the same time, the inertial device module of the smart phone is called to record and save the posture information of the mobile phone when the group of depth images is collected.

[0078] Step S6, repeat the above steps to obtain multi-view images and convert them into multi-view point cloud data, splice and grid the multi-view point cloud data to form a digital model of the oral cavity, as shown in the attached figure. Figure 7 Specifically, the above process is repeated to obtain multi-view images, and the images corresponding to the multiple viewpoints are converted into point cloud data, which are sent to the memory together with the pose transformation matrix, and then the multi-view point clouds are spliced, and the spliced ​​point clouds are meshed to form a complete oral digital model.

[0079] Among them, the images of multiple perspectives are aligned using the improved ICP algorithm. When stitching, the translation and rotation matrices of the pose changes between the two frames and the scale information estimated according to the calibration results are used as initial values ​​for iteration to minimize the alignment error of the two point clouds and finally obtain a three-dimensional model of oral teeth.

[0080] In this embodiment, step S6 specifically includes:

[0081] According to the overlapped parts of the two frames, the target tooth is superimposed on the corresponding tooth position. Specifically, the point cloud sets A and B of the two frames are downsampled as shown below:

[0082]

[0083]

[0084]

[0085] Among them A l , B l Respectively represent the point cloud sets A and B at resolution level l, f sample↓ Indicates downsampling operation, L is the maximum resolution level set, represents a single point cloud a with sequence number i in the point cloud set A at resolution level 1, The number of point clouds in the point cloud set A is a positive integer, and there are n in total; represents a single point cloud b with sequence number j in the point cloud set B at resolution level l, The number of point clouds in the point cloud set B is a positive integer, and there are m point clouds in total.

[0086] Use the nearest neighbor search to get two point cloud sets A l With B l Bidirectional transformation relationship at resolution l (k-1) th , as shown below:

[0087]

[0088]

[0089] Where K represents the number of iterations, They represent respectively that at resolution 1, in (k-1) iterations, The values ​​of variables i and j that minimize the error. They represent the scaling matrix, rotation matrix, and translation matrix required for the transformation in (k-1) iterations at resolution level 1. represents a single point cloud a with sequence number i in the point cloud set A at resolution level 1, Represents a single point cloud b with sequence number j in the point cloud set B at resolution level 1.

[0090] Based on the transformation matrix H obtained in the previous step, the new transformation matrix between the two point clouds is calculated and updated and in and They respectively represent the scaling matrix, rotation matrix, and translation matrix required for transformation in k iterations at resolution level 1.

[0091] Repeat the above two steps until the error accuracy reaches the resolution l.

[0092] Repeat the previous step until the resolution l=1.

[0093] After the splicing is completed, a three-dimensional model of the entire mouth of teeth can be formed. Dentists can easily prepare teeth based on this model and then carry out a series of subsequent diagnosis and treatment plans.

[0094] The oral 3D reconstruction method based on a smartphone disclosed in the above embodiment uses the lens module of a smartphone as an oral scanning device to perform 3D reconstruction of the teeth in the oral cavity. By changing the offset of the mobile phone lens, that is, moving the position of the focal plane, a series of images with different depths of field in the patient's mouth are obtained by high-speed photography. The collected data is filtered by deblurring and a series of images are aligned. The last image is used as a calibration reference, each deblurred image is matched with the calibration, the zoom ratio between the two images before and after is calculated, and then the image is scaled according to the zoom ratio to eliminate the near-large-far-small effect. This group of images is Laplace transformed and filtered and then superimposed together to form a single-view depth map. The depth maps of multiple perspectives are formed into a point cloud, which is aligned, spliced ​​and gridded to form the final 3D model. Thus, the real image of the patient's internal oral environment is transmitted to the smartphone to restore its real model, which greatly reduces the cost of the oral scanning service and improves efficiency. The problem that current oral scanners are composed of an oral scanner and a huge host station connected to the oral scanner, and the doctor's line of sight needs to constantly move between the oral cavity and the host station screen beside him during operation, resulting in reduced scanning efficiency. In addition, the huge host station causes the entire oral scanner system to occupy a large amount of limited diagnosis and treatment space and is expensive.

[0095] In other embodiments, a smart phone-based oral 3D reconstruction system is also disclosed, including: a calibration module, used to calibrate the focus module of the smart phone to obtain the depth value of each focal plane when the smart phone is focused; a first image group acquisition module, used to control the lens module of the smart phone to zoom and simultaneously obtain different depth of field images of the inside of the oral cavity as the first image group through an external probe, and record the corresponding focal length; a second image group acquisition module, used to sequentially select the front and rear two frames of the first image group and use the SIFT algorithm to search to obtain their respective feature points, use the RANSAC algorithm to solve the transformation relationship between the two groups of feature points, and solve the corresponding homography matrix, and use the solved matrix to transform the second image to obtain the near-field image. The pictures with the large-far-small effect form a transformed second image group; the depth acquisition module is used to overlap the frames of the second image group and perform Laplace transform, select the image with the largest response value under the same pixel coordinate as the focal plane corresponding to the pixel, and obtain the depth corresponding to each pixel according to the focal plane depth value; the single-view depth map module is used to obtain the pixel coordinates of each pixel at the maximum contrast in order from front to back and record them to form the perspective depth map, and at the same time obtain the posture information corresponding to the smartphone when collecting the group of depth maps; the model generation module is used to obtain multi-view images according to the obtained multiple single-view depth maps and convert them into multi-view point cloud data, splice and grid the multi-view point cloud data to form a digital model of the oral cavity.

[0096] In this embodiment, the first image group acquisition module specifically includes: a focal range acquisition module, which is used to control the lens module to continuously adjust the focal length to a focal length at which the current teeth can be clearly presented on the smartphone display screen when the lens module of the smartphone can acquire a tooth image through an external probe, and to scale the focal length to acquire a focal range that can cover the current tooth surface; an image capture module, which is used to control the lens module to capture a continuous initial image group of the inside of the oral cavity at a high rate, and to capture multiple pictures in the continuous initial image group as the first image group according to the focal range.

[0097] Preferably, the second image group acquisition module specifically includes: a feature point acquisition module, which is used to search the front and rear two frames of images in the first image group using the SIFT algorithm to obtain their respective feature points; a homography matrix acquisition module, which is used to randomly select multiple groups of matching feature points to calculate the homography matrix, calculate the errors of the remaining matching feature point pairs after the homography matrix transformation, and obtain the number of feature point pairs whose errors are less than a threshold, and obtain the homography matrix with the largest number of feature point pairs whose errors are less than the threshold; an image transformation module, which is used to transform the second image using the solved homography matrix to obtain a transformed image that eliminates the near-big-far-small effect; a second image group generation module, which is used to sequentially transform the front and rear two frames of images in the first image group to obtain a transformed image that eliminates the near-big-far-small effect to form a second image group.

[0098] Preferably, the model generation module specifically includes: performing registration according to the overlapping parts of the two frames of images, superimposing the target tooth on the corresponding tooth position, and downsampling the point cloud sets A and B of the two frames of images, as shown below:

[0099]

[0100]

[0101]

[0102] Among them A l , B l Respectively represent the point cloud sets A and B at resolution level l, f sample↓ Indicates downsampling operation, L is the maximum resolution level set, represents a single point cloud a with sequence number i in the point cloud set A at resolution level 1, The number of point clouds in the point cloud set A is a positive integer, and there are n in total; represents a single point cloud b with sequence number j in the point cloud set B at resolution level l, The number of point clouds in the point cloud set B is a positive integer, and there are m in total.

[0103] Use the nearest neighbor search to get two point cloud sets A l With B l Bidirectional transformation relationship at resolution l (k-1) th , as shown below:

[0104]

[0105]

[0106] Where K represents the number of iterations, They represent respectively that at resolution l, in (k-1) iterations, The values ​​of variables i and j that minimize the error. They respectively represent the scaling matrix, rotation matrix, and translation matrix required for the transformation in (k-1) iterations at resolution level l. represents a single point cloud a with sequence number i in the point cloud set A at resolution level l, Represents a single point cloud b with sequence number j in the point cloud set B at resolution level l. and They respectively represent the scaling matrix, rotation matrix, and translation matrix required for transformation in k iterations at resolution level 1.

[0107] Based on the obtained transformation matrix, calculate the new transformation matrix between the two point clouds and update and Until the error accuracy reaches resolution l=1.

[0108] The specific functions of the above-mentioned oral 3D reconstruction system based on a smartphone correspond one to one with the oral 3D reconstruction method based on a smartphone disclosed in the previous embodiments, so they will not be described in detail here. For details, please refer to the various embodiments of the oral 3D reconstruction method based on a smartphone disclosed in the previous embodiments. It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0109] In other embodiments, a smartphone-based oral three-dimensional reconstruction device is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the smartphone-based oral three-dimensional reconstruction method described in the above embodiments is implemented.

[0110] The oral 3D reconstruction device based on a smartphone may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the schematic diagram is merely an example of an oral 3D reconstruction device based on a smartphone and does not constitute a limitation on the oral 3D reconstruction device based on a smartphone. The device may include more or fewer components than shown in the diagram, or may combine certain components, or different components. For example, the oral 3D reconstruction device based on a smartphone may also include input and output devices, network access devices, buses, etc.

[0111] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the device for oral 3D reconstruction based on a smartphone, and uses various interfaces and lines to connect various parts of the device for oral 3D reconstruction based on a smartphone.

[0112] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the device for three-dimensional reconstruction of oral cavity based on smart phone by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0113] If the oral 3D reconstruction device based on a smartphone is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the above-mentioned steps of each embodiment of the oral 3D reconstruction method based on a smartphone. Among them, the computer program includes a computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0115] In conclusion, the above is only a preferred embodiment of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the patent of the present invention.

Claims

1. A method for oral 3D reconstruction based on a smartphone, characterized in that: The steps include: S1, calibrating the focus module of the smartphone to obtain the depth value of each focal plane when the smartphone is focusing; S2, controlling the lens module of the smartphone to zoom and simultaneously obtaining different depth-of-field images of the interior of the oral cavity through an external probe as a first image group, and recording the corresponding focal lengths; S3, sequentially selecting the first and second frames of the first image group and searching them using the SIFT algorithm to obtain their respective feature points, using the RANSAC algorithm to solve the transformation relationship between the two sets of feature points, and solving the corresponding homography matrix, and transforming the second image using the solved matrix to obtain a second image group composed of pictures that eliminate the near-big-far-small effect; S4, superimposing the frames of images in the second image group together to perform Laplace transform, selecting the image with the largest response value at the same pixel coordinate as the focal plane corresponding to the pixel, and obtaining the depth corresponding to each pixel according to the focal plane depth value; S5, sequentially obtaining the pixel coordinates of each pixel at which the contrast is the largest from front to back and recording the depth map formed by the current viewing angle, and obtaining the corresponding position information of the smart phone when collecting the depth map; S6, repeating the above steps to obtain a multi-view image and converting it into multi-view point cloud data, splicing and gridding the multi-view point cloud data to form a digital model of the oral cavity; the step S6 specifically includes: According to the overlapping parts of the two frames, the target tooth is superimposed on the corresponding tooth position; the point cloud sets A and B of the two frames are downsampled as shown below: Among them A l , B l Respectively represent the point cloud sets A and B at resolution level l, f sample↓ Indicates downsampling operation, L is the maximum resolution level set, represents a single point cloud a with sequence number i in the point cloud set A at resolution level l, The number of point clouds in the point cloud set A is a positive integer, and there are n in total; represents a single point cloud b with sequence number j in the point cloud set B at resolution level l, The number of point clouds in the point cloud set B is a positive integer, and there are m in total; Use the nearest neighbor search to get two point cloud sets A l With B l Bidirectional transformation relationship at resolution l (k-1) th , as shown below: Where K represents the number of iterations, They represent respectively that at resolution l, in (k-1) iterations, The values ​​of variables i and j with the smallest error; They represent the scaling matrix, rotation matrix and translation matrix required for transformation in (k-1) iterations at resolution level l respectively; represents a single point cloud a with sequence number i in the point cloud set A at resolution level l, Represents a single point cloud b with serial number j in the point cloud set B at resolution level l. Based on the obtained transformation matrix, a new transformation matrix between the two point clouds is calculated and updated. and Repeat the above steps until the error accuracy reaches resolution l=1.

2. The oral 3D reconstruction method based on a smartphone according to claim 1, characterized in that: The step S2 specifically includes: S21, when the lens module of the smartphone acquires a tooth image through an external probe, controlling the lens module to continuously adjust the focal length to a focal length at which the current tooth can be clearly presented on the display screen of the smartphone, and scaling the focal length to obtain a focal range that can cover the current tooth surface; S22, controlling the lens module to capture a continuous initial image group of the interior of the oral cavity at a high rate, and capturing a plurality of images in the continuous initial image group according to the focal length range as a first image group.

3. The oral 3D reconstruction method based on a smartphone according to claim 2, characterized in that: The step S3 specifically includes: S31, searching the previous and next two frames of images in the first image group using the SIFT algorithm to obtain their respective feature points; S32, randomly selecting multiple groups of matching feature points to calculate the homography matrix, calculating the errors of the remaining matching feature point pairs after the homography matrix transformation, and obtaining the number of feature point pairs with errors less than a threshold, and obtaining the homography matrix with the largest number of feature point pairs with errors less than the threshold; S33, transforming the second image using the solved homography matrix to obtain a transformed image that eliminates the near-big-far-small effect; S34, continuously adopting the aforementioned steps S31 to S33 to sequentially transform the two frames of images in the first image group to obtain a transformed image that eliminates the near-big-far-small effect, thereby forming a second image group.

4. An oral 3D reconstruction system based on a smartphone, characterized in that: include: A calibration module is used to calibrate the focus module of the smartphone to obtain the depth value of each focal plane when the smartphone is focusing; A first image group acquisition module, used to control the lens module of the smartphone to zoom and simultaneously acquire different depth-of-field images of the interior of the oral cavity through an external probe as a first image group, and record the corresponding focal length; The second image group acquisition module is used to sequentially select the two frames before and after the first image group and use the SIFT algorithm to search to obtain their respective feature points, use the RANSAC algorithm to solve the transformation relationship between the two groups of feature points, and solve the corresponding homography matrix, and use the solved matrix to transform the second image to obtain a picture that eliminates the near big and far small effect to form a transformed second image group; A depth acquisition module is used to overlap the frames in the second image group and perform Laplace transform, select the image with the largest response value under the same pixel coordinate as the focal plane corresponding to the pixel, and obtain the depth corresponding to each pixel according to the focal plane depth value; The single-view depth map module is used to obtain the pixel coordinates of each pixel at the maximum contrast from front to back and record the depth map of the current view, and obtain the corresponding posture information of the smartphone when collecting the depth map; The model generation module is used to obtain a multi-view image based on the obtained multiple single-view depth maps and convert it into multi-view point cloud data, splice and grid the multi-view point cloud data to form a digital model of the oral cavity, specifically including: aligning the overlapping parts of the previous and next two frames of images, superimposing the target teeth on the corresponding tooth positions; downsampling the point cloud sets A and B of the previous and next two frames, as shown below: Among them A l , B l Respectively represent the point cloud sets A and B at resolution level l, f sample↓ Indicates downsampling operation, L is the maximum resolution level set, represents a single point cloud a with sequence number i in the point cloud set A at resolution level l, The number of point clouds in the point cloud set A is a positive integer, and there are n in total; represents a single point cloud b with sequence number j in the point cloud set B at resolution level l, The number of point clouds in the point cloud set B is a positive integer, and there are m in total; Use the nearest neighbor search to get two point cloud sets A l With B l Bidirectional transformation relationship at resolution l (k-1) th , as shown below: Where K represents the number of iterations, They represent respectively that at resolution l, in (k-1) iterations, The values ​​of variables i and j with the smallest error; They represent the scaling matrix, rotation matrix and translation matrix required for transformation in (k-1) iterations at resolution level l respectively; represents a single point cloud a with sequence number i in the point cloud set A at resolution level l, Represents a single point cloud b with serial number j in the point cloud set B at resolution level l. Based on the obtained transformation matrix, a new transformation matrix between the two point clouds is calculated and updated. and Repeat the above steps until the error accuracy reaches resolution l=1.

5. The oral 3D reconstruction system based on a smartphone according to claim 4, characterized in that: The first image group acquisition module specifically includes: A focal range acquisition module, which is used to control the lens module to continuously adjust the focal length to a focal length at which the current teeth can be clearly presented on the display screen of the smartphone when the lens module of the smartphone acquires a tooth image through an external probe, and to scale the focal length to obtain a focal range that can cover the current tooth surface; The image capture module is used to control the lens module to capture a continuous initial image group of the interior of the oral cavity at a high rate, and to capture multiple pictures in the continuous initial image group as the first image group according to the focal length range.

6. The oral 3D reconstruction system based on a smartphone according to claim 5, characterized in that: The second image group acquisition module specifically includes: A feature point acquisition module is used to search the previous and next two frames of images in the first image group using the SIFT algorithm to obtain their respective feature points; A homography matrix acquisition module is used to randomly select multiple groups of matching feature points to calculate the homography matrix, calculate the errors of the remaining matching feature point pairs after the homography matrix transformation, and obtain the number of feature point pairs whose errors are less than a threshold, and obtain the homography matrix with the largest number of feature point pairs whose errors are less than the threshold; An image transformation module is used to transform the second image using the solved homography matrix to obtain a transformed image that eliminates the near-big-far-small effect; The second image group generating module is used to sequentially transform the two frames of images in the first image group to obtain transformed images that eliminate the near-big-far-small effect to form a second image group.

7. An oral 3D reconstruction device based on a smart phone, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images