Oral cavity three-dimensional scanning system and control method thereof, electronic equipment and storage medium

Abstract

The invention discloses an oral cavity three-dimensional scanning system and a control method thereof, electronic equipment and a storage medium, and relates to the technical field of oral cavity three-dimensional scanning. The method comprises the following steps: controlling a projection module to alternately project white light and projection fringes generated based on Debrueine coding on a preset chessboard graph in different time periods towards a target; controlling the target to move, and controlling the imaging module to shoot a plurality of target images towards the target; the target image comprises a coding stripe image collected at the same position and a checkerboard image collected under white light; the multiple checkerboard images are calibrated to obtain imaging calibration parameters of the imaging module, meanwhile, after the multiple coded stripe images are calibrated to obtain projection calibration parameters of the projection module, a system calibration result is determined, and calibration is completed; and performing three-dimensional reconstruction processing based on the system calibration result on the to-be-reconstructed object to obtain a three-dimensional reconstruction result. The calibration efficiency and the calibration accuracy when the oral cavity three-dimensional scanning system is calibrated can be improved.

Description

Technical Field

[0001] This application relates to the technical field of three-dimensional intraoral scanning, and in particular to an intraoral three-dimensional scanning system, its control method, an electronic device, and a storage medium. Background Art

[0002] Three-dimensional reconstruction is an interdisciplinary field of computer vision, graphics, and sensor technology, aiming to restore the three-dimensional geometric structure of an object through the acquired two-dimensional data (such as images and depth information). With the development of computer vision technology and computer hardware and software, three-dimensional reconstruction technology has developed more and more rapidly and is widely applied in industries such as autonomous driving, medicine, visual guidance, measurement and detection. Traditional oral medical technology mainly relies on manual operations by doctors, which are often inefficient and cause discomfort to patients. Nowadays, many dental clinics have introduced advanced three-dimensional scanners, and the calibration of the scanner is a very crucial step. A fast and accurate calibration method is the prerequisite for achieving high-precision three-dimensional reconstruction, and the accuracy of its calibration results and the stability of the algorithm directly affect the accuracy of the results generated by the camera's work. The calibration process of traditional intraoral three-dimensional scanners is complex. Usually, a calibration instrument with high precision requirements, large volume, high cost, and difficult operation and control is required to cooperate with the oral scanning system for calibration, and the calibration efficiency is relatively low. Summary of the Invention

[0003] This application aims to solve at least one of the technical problems existing in the prior art. For this reason, this application provides an intraoral three-dimensional scanning system, its control method, an electronic device, and a storage medium, which can improve the calibration efficiency and calibration accuracy when calibrating the intraoral three-dimensional scanning system.

[0004] In a first aspect, an embodiment of this application provides a control method for an intraoral three-dimensional scanning system, which is applied to the host computer of the intraoral three-dimensional scanning system. The system further includes: a calibrator and a scanning device respectively electrically connected to the host computer; the calibrator includes a sliding table and a target that can move along the sliding table; there is a preset checkerboard pattern on the target; the scanning device includes a projection module and an imaging module;

[0005] The method includes:

[0006] Controlling the projection module to project towards the target, and alternately projecting white light and projection stripes generated based on De Bruijn coding on the preset checkerboard pattern in different time periods;

[0007] Controlling the movement of the target and controlling the imaging module to capture multiple target images towards the target; the target images include: encoded stripe images collected at the same position and checkerboard images collected under white light; different target images are obtained by photographing the target at different positions;

[0008] Performing imaging module calibration processing on multiple checkerboard images to obtain the imaging calibration parameters of the imaging module; simultaneously, performing projection module calibration processing on multiple encoded stripe images to obtain the projection calibration parameters of the projection module;

[0009] Determining the system calibration result according to the imaging calibration parameters and the projection calibration parameters to complete the calibration;

[0010] Controlling the projection module and the imaging module to perform three-dimensional reconstruction processing on the object to be reconstructed based on the system calibration result to obtain a three-dimensional reconstruction result.

[0011] In a second aspect, an embodiment of the present application provides an oral three-dimensional scanning system, including: a host computer, a calibrator and a scanning device respectively electrically connected to the host computer; the calibrator includes a slide table and a target capable of moving along the slide table; there is a preset checkerboard pattern on the target; the scanning device includes a projection module and an imaging module; the host computer is configured to execute the control method of the oral three-dimensional scanning system according to any one of the embodiments in the first aspect.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including at least one processor and a memory for communicatively connecting with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the control method of the oral three-dimensional scanning system according to any one of the embodiments in the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions for causing a computer to execute the control method of the oral three-dimensional scanning system according to any one of the embodiments in the first aspect.

[0014] Embodiments of the present application include: An oral three-dimensional scanning system includes: a host computer, a calibrator, and a scanning device, which are electrically connected to the host computer respectively; the calibrator includes a sliding table and a target capable of moving along the sliding table; a preset checkerboard pattern is provided on the target; the scanning device includes a projection module and an imaging module. By using the oral three-dimensional scanning system, first, control the projection module to project towards the target, and alternately project white light and projection stripes generated based on De Bruijn coding on the preset checkerboard pattern in different time periods; then, control the movement of the target, and control the imaging module to capture multiple target images towards the target; the target images include: encoded stripe images collected at the same position and checkerboard images collected under white light; different target images are obtained by photographing targets at different positions; then, perform imaging module calibration processing on multiple checkerboard images to obtain the imaging calibration parameters of the imaging module; at the same time, perform projection module calibration processing on multiple encoded stripe images to obtain the projection calibration parameters of the projection module; calibrate the imaging module and the projection module in the oral three-dimensional scanning system at the same time, improving the calibration efficiency and calibration accuracy; then, determine the system calibration result according to the imaging calibration parameters and the projection calibration parameters to complete the calibration; thus laying a foundation for three-dimensional reconstruction and being beneficial to improving the accuracy of three-dimensional reconstruction performed by the oral three-dimensional scanning system; finally, control the projection module and the imaging module to perform three-dimensional reconstruction processing on the object to be reconstructed based on the system calibration result to obtain a three-dimensional reconstruction result. That is to say, the embodiments of the present application can improve the calibration efficiency and calibration accuracy when calibrating the oral three-dimensional scanning system. Description of the Drawings

[0015] Figure 1 is a schematic structural diagram of an oral three-dimensional scanning system provided by an embodiment of the present application;

[0016] Figure 2 is a schematic flowchart of a control method for an oral three-dimensional scanning system provided by an embodiment of the present application;

[0017] Figure 3 is a schematic diagram of a captured target image provided by an embodiment of the present application;

[0018] Figure 4 is a schematic diagram of the light plane result of projection module calibration processing provided by an embodiment of the present application;

[0019] Figure 5 is a schematic diagram of the decoded phase information provided by an embodiment of the present application;

[0020] Figure 6A is a schematic diagram of a point cloud image scanned by a calibrated oral three-dimensional scanning system not based on the present application;

[0021] Figure 6B It is a schematic diagram of a point cloud image obtained by scanning with an oral three-dimensional scanning system calibrated based on this application;

[0022] Figure 7 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of this application. Detailed implementation manners

[0023] In order to make the objectives, technical solutions and advantages of this application clearer, the following further elaborates on this application in conjunction with the accompanying drawings and embodiments.

[0024] It should be understood that in the description of this application, the orientation descriptions involved, such as up, down, front, back, left, right, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. This is only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to this application.

[0025] It should be noted that although the logical order is shown in the flowchart in the description of this application, in some cases, the steps shown or described may be executed in a different order than that in the flowchart. In the description of this application, the meaning of "several" is one or more, and the meaning of "multiple" is two or more. The descriptions of "first" and "second" are only for the purpose of distinguishing technical features, and should not be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or the sequence of the indicated technical features.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0027] The present application provides a control method for an oral three-dimensional scanning system, an oral three-dimensional scanning system, an electronic device, and a computer-readable storage medium, which relates to the technical field of three-dimensional scanning within the oral cavity. The method includes: controlling a projection module to alternately project white light and projection stripes generated based on De Bruijn coding onto a preset checkerboard pattern in segments towards a target; controlling the target to move, and controlling an imaging module to capture a plurality of target images towards the target; the target images include: a coded stripe image collected at the same position and a checkerboard image collected under white light; calibrating a plurality of checkerboard images to obtain imaging calibration parameters of the imaging module, and at the same time, calibrating a plurality of coded stripe images to obtain projection calibration parameters of the projection module, and then determining a system calibration result to complete calibration; performing three-dimensional reconstruction processing on an object to be reconstructed based on the system calibration result to obtain a three-dimensional reconstruction result. It can improve the calibration efficiency and calibration accuracy when calibrating an oral three-dimensional scanning system.

[0028] The following further elaborates on the embodiments of the present application in conjunction with the accompanying drawings.

[0029] As Figure 1 shown, the oral three-dimensional scanning system 100 includes: a host computer 110, a calibrator 120 and a scanning device 130 that are electrically connected to the host computer 110 respectively; the calibrator 120 includes a sliding table 121 and a target 122 that can move along the sliding table; there is a preset checkerboard pattern on the target 122; the scanning device 130 includes a projection module 131 and an imaging module 132.

[0030] Specifically, the sliding table 121 is used to limit the moving range of the target 122. The calibrator further includes: a motor, the control input end of the motor is electrically connected to the host computer 110, and the driving output end of the motor is connected to the target 122. The motor is used to drive the target 122 to move along the front-back direction of the sliding table 121 under the control of the host computer.

[0031] Specifically, in the scanning device 130, the projection module 131 is used to perform projection processing towards the target 122, and alternately project white light and projection stripes generated based on De Bruijn coding onto the preset checkerboard pattern in segments; the imaging module 132 captures a plurality of target images towards the target; it lays a data foundation for subsequent system calibration.

[0032] The host computer 110 is used to execute the control method for the oral three-dimensional scanning system provided by the embodiments of the present application. In the oral three-dimensional scanning system of the embodiments of the present application, the host computer 110, the calibrator 120, and the scanning device 130 cooperate with each other to implement the control method provided by the embodiments of the present application, complete system calibration, and can improve the calibration efficiency and calibration accuracy when calibrating an oral three-dimensional scanning system.

[0033] Those skilled in the art can understand that the system structure shown in the figure does not limit the embodiments of the present application. It may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0034] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0035] Those skilled in the art can understand that the system architecture and application scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not limit the technical solutions provided by the embodiments of the present application. As those skilled in the art know, with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0036] Based on the above system structure, various embodiments of the control method of the oral three-dimensional scanning system of the present application are proposed below.

[0037] As Figure 2 shown, the control method of the oral three-dimensional scanning system can be applied to a host computer as Figure 1 shown. The oral three-dimensional scanning system further includes: a calibrator and a scanning device respectively electrically connected to the host computer; the calibrator includes a slide table and a target capable of moving along the slide table; there is a preset checkerboard pattern on the target; the scanning device includes a projection module and an imaging module; the control method of the oral three-dimensional scanning system may include but is not limited to steps S110 to S150.

[0038] Step S110: Control the projection module to project towards the target, and alternately project white light and projection stripes generated based on De Bruijn coding onto the preset checkerboard pattern in different time periods.

[0039] Step S120: Control the target to move, and control the imaging module to capture multiple target images towards the target; the target images include: encoded stripe images collected at the same position and checkerboard images collected under white light; different target images are obtained by capturing the target at different positions.

[0040] Step S130: Perform imaging module calibration processing on multiple checkerboard images to obtain the imaging calibration parameters of the imaging module; at the same time, perform projection module calibration processing on multiple encoded stripe images to obtain the projection calibration parameters of the projection module.

[0041] Step S140: Determine the system calibration result according to the imaging calibration parameters and the projection calibration parameters to complete the calibration.

[0042] Step S150: Control the projection module and the imaging module to perform three-dimensional reconstruction processing on the object to be reconstructed based on the system calibration result, and obtain a three-dimensional reconstruction result.

[0043] In the embodiment of the present application, through steps S110 to S150, by using an oral three-dimensional scanning system, first, control the projection module to project towards the target, and alternately project white light and projection stripes generated based on De Bruijn coding onto a preset checkerboard pattern in different time periods; then, control the target to move, and control the imaging module to capture multiple target images towards the target; the target images include: the coded stripe images collected at the same position and the checkerboard images collected under white light; different target images are obtained by photographing the targets at different positions; then, perform imaging module calibration processing on multiple checkerboard images to obtain the imaging calibration parameters of the imaging module; at the same time, perform projection module calibration processing on multiple coded stripe images to obtain the projection calibration parameters of the projection module; calibrate the imaging module and the projection module in the oral three-dimensional scanning system at the same time, improving the calibration efficiency and calibration accuracy; then, determine the system calibration result according to the imaging calibration parameters and the projection calibration parameters to complete the calibration; thus laying a foundation for three-dimensional reconstruction, which is beneficial to improving the accuracy of the three-dimensional reconstruction performed by the oral three-dimensional scanning system; finally, control the projection module and the imaging module to perform three-dimensional reconstruction processing on the object to be reconstructed based on the system calibration result, and obtain a three-dimensional reconstruction result. Therefore, the embodiment of the present application can improve the calibration efficiency and calibration accuracy when calibrating the oral three-dimensional scanning system.

[0044] Further illustrate step S110. The projection stripes in the embodiment of the present application are generated based on De Bruijn coding. The De Bruijn coding is solidified onto the projection stripe grating in the form of color coding. When the light source irradiates on the projection stripe grating, the light passes through the projection stripe grating to form stable and bright projection stripes. These projection stripes are irradiated on the target or the object to be reconstructed, and the required coded stripe images can be collected through the imaging module. Through a specially designed optical path, it is ensured that enough image frames can be collected to cover the entire stripe period. De Bruijn coding is a technology used for decoding structured light stripes and is commonly found in three-dimensional reconstruction and surface detection. The De Bruijn sequence is a special sequence that can generate all possible subsequences on a finite alphabet. Its main advantage is its good periodicity and coverage, enabling the coded stripes to be decoded efficiently.

[0045] It can be understood that in terms of hardware, in the present application, the projection stripes and white light are projected onto the preset checkerboard pattern of the target in different time periods through the projection module, so that it is more convenient to obtain the coded stripe images, and the coded stripe images show the coded stripes based on De Bruijn coding.

[0046] Further describe step S120. The calibration data (i.e., multiple target images) is collected through step S120, laying a data foundation for subsequent imaging module calibration processing, projection module calibration processing, and system calibration processing. The specific process of collecting target images in step S120 is as follows: Control the target to move along the front-back direction within the moving range defined by the sliding table. At the same time, control the imaging module to face the target and collect a series of target images. The collected target images are as Figure 3 shown.

[0047] It can be understood that the difference between the encoded fringe image collected at the same position and the checkerboard image collected under white light is that the projection fringes are not displayed in the checkerboard image collected under white light.

[0048] It can be understood that different target images are obtained by photographing the target at different positions. When an initial light plane position is determined within the moving range defined by the sliding table and the positions other than the initial light plane position are defined as other light plane positions, the target images can be divided into: the reference image collected from the target at the initial light plane position and the other reference images collected from the targets at other light plane positions. Thus, a data foundation is laid for subsequent projection module calibration processing.

[0049] It can be understood that in the embodiments of the present application, the preset checkerboard pattern is photographed from multiple positions (angles and distances), thereby ensuring a wide coverage of the field of view; and ensuring uniform illumination, reducing the influence of strong light and shadows on the quality of the collected target images.

[0050] In the embodiments of the present application, data preparation is carried out through steps S110 to S120, laying a data foundation for subsequent processing.

[0051] The control process of the three-dimensional oral scanning system in the embodiments of the present application mainly includes: the data preparation stage implemented through steps S110 to S120, the calibration stage implemented through steps S130 to S140, and the three-dimensional reconstruction stage through step S150. Specifically, the calibration stage mainly involves: imaging calibration processing, projection calibration processing, and system calibration processing. Further, the imaging module calibration processing, projection module calibration processing, and system calibration processing are specifically described in sequence.

[0052] According to some embodiments of the present application, further describe step S130, wherein the imaging calibration parameters of the imaging module are obtained by performing imaging module calibration processing on multiple checkerboard images, including but not limited to steps S210 to S230.

[0053] Step S210: Perform corner extraction processing on multiple checkerboard images through a corner detection algorithm, and correspondingly obtain multiple corner pixel coordinates; each corner pixel coordinate is obtained based on one frame of checkerboard image.

[0054] Step S220: Match the corner pixel coordinates with the actual corner coordinates to obtain the matrix correspondence between the corner pixel coordinates and the actual corner coordinates.

[0055] Step S230: Solve the matrix correspondence through the least squares method to obtain imaging calibration parameters; the imaging calibration parameters include: imaging internal parameters and imaging external parameters.

[0056] It should be noted that the processing object of the imaging module calibration process in this application embodiment is: checkerboard images; based on the checkerboard calibration method, identifying checkerboard images can provide feature points that are easy to identify, which is beneficial to improving the calibration accuracy and calibration efficiency.

[0057] It should be noted that the traditional Zhang's calibration method is used for imaging module calibration, and Zhang's calibration method completes calibration by identifying the centers of black dots in the target image. This application combines the checkerboard calibration method to identify corners to complete calibration.

[0058] Specifically, step S210 is: after capturing the checkerboard image, apply a corner detection algorithm in an image processing library such as OpenCV to perform corner extraction processing on the checkerboard image, extract the checkerboard corners and their corner pixel coordinates; save the corner pixel coordinates of the obtained multiple checkerboard images. This lays a data foundation for the matching process.

[0059] It can be understood that the corner detection algorithm is an algorithm used to identify corners in images in the fields of computer vision and image processing. Corners usually refer to the intersections of two or more edges, points with sufficient brightness changes in all directions on the image, and points with maximum curvature on the edge curve. Corner detection has wide applications in computer vision fields such as 3D scene reconstruction, motion estimation, visual tracking, and image registration and matching. The corner detection algorithm adopted in this application embodiment can be a corner detection algorithm based on gray intensity, a corner detection algorithm based on edge contours, etc., and this application does not make specific restrictions on the type of corner detection algorithm adopted.

[0060] Specifically, steps S220 to step 230 are implemented for calibration based on Zhang Zhengyou's calibration method, and imaging calibration parameters including imaging internal parameters and imaging external parameters are solved. Specifically, the imaging internal parameters include: focal length, principal point, distortion coefficient; the imaging external parameters include: rotation matrix and translation vector.

[0061] Specifically, the Zhang Zhengyou calibration method is a widely used camera calibration method. The Zhang Zhengyou calibration method employed in the calibration algorithm of this application first inputs the extracted corner pixel coordinates. Next, the processing procedure involves matching the corner pixel coordinates with the actual corner coordinates to obtain matching results. It is understood that the actual corner coordinates are based on the physical dimensions of a preset chessboard pattern in the actual 3D world coordinates. Finally, based on the matching results, the least squares method is used to solve for the camera's imaging intrinsic and imaging extrinsic parameters.

[0062] Through steps S210 to S230, the imaging module calibration process is implemented by combining the Zhang Zhengyou calibration method with the checkerboard calibration method, effectively improving the accuracy and efficiency of the imaging module calibration. Furthermore, the imaging calibration parameters obtained in the imaging model are beneficial for improving the accuracy of image acquisition in actual 3D reconstruction.

[0063] According to some embodiments of the present application, step S130 is further described, wherein a projection module calibration process is performed on a plurality of coded fringe images to obtain projection calibration parameters of the projection module, including but not limited to steps S310 to S330.

[0064] Step S310: performing image preprocessing on the plurality of coded stripe images to obtain a plurality of preprocessed coded stripe images.

[0065] Step S320: performing feature extraction processing on the pre-processed multiple coded stripe images based on the Canny edge detection algorithm to obtain the light band contour feature information of each coded stripe image.

[0066] Step S330: performing light plane fitting processing and light plane optimization processing based on the least square method in sequence according to the profile feature information of each light band to obtain a target light plane equation.

[0067] Step S340: determining projection calibration parameters according to the light plane parameters in the target light plane equation, where the projection calibration parameters include: projection intrinsic parameters and projection extrinsic parameters.

[0068] According to some embodiments of the present application, the image preprocessing in step S310 is further described. Specifically, image preprocessing includes image denoising, threshold segmentation, and image enhancement. Image preprocessing includes performing image denoising and threshold segmentation on the initial image to be processed to obtain a denoised initial image; and performing image enhancement on the denoised initial image to obtain a preprocessed target image. In the projection module calibration process, the initial image to be processed is a coded stripe image. After image preprocessing, the target image obtained is: multiple preprocessed coded stripe images.

[0069] Specifically, in the embodiments of the present application, through image denoising processing, a filter is used to remove background noise in the encoded fringe image, improving the quality of the encoded fringe image. In the embodiments of the present application, through image enhancement processing, the contrast of the fringes in the encoded fringe image is improved, making the fringes more obvious.

[0070] In the embodiments of the present application, through step S310, when performing the calibration processing of the projection module, first, it is necessary to perform image preprocessing on the encoded fringe image to obtain a good-quality encoded fringe image, laying a data foundation for subsequent feature extraction processing, light plane fitting processing, and light plane optimization processing based on the least squares method.

[0071] It can be understood that different target images are obtained by photographing the targets at different positions. When within the moving range defined by the sliding table, an initial light plane position is determined, and the positions other than the initial light plane position are defined as other light plane positions, then the target images can be divided into: a reference image collected from the target at the initial light plane position, and other reference images collected from the targets at other light plane positions. Thus, it lays a data foundation for subsequent projection module calibration processing. By selecting the clearest position within the moving range of the target as the initial light plane position, it can ensure that the projected fringe of the projection module is stable and easy to identify; in the embodiments of the present application, by moving the target of the calibrator back and forth, the imaging module is used to capture target images including the initial light plane position and other different light plane positions, obtaining all the data within the depth of field of the scanning system as calibration data.

[0072] Further illustrate step S320. Specifically, step S320 includes: extracting features from the encoded fringe image collected from the target at the initial light plane position, and at the same time, extracting features from the encoded fringe image collected from the targets at other light plane positions, and identifying the light band contour feature information based on the Canny edge detection algorithm.

[0073] Specifically, based on the Canny edge detection algorithm: it can provide an accurate and robust edge detection method. The main objective of Canny edge detection is to detect the places where the brightness changes significantly in the image, which usually correspond to the edges in the image. The Canny algorithm can effectively detect the edges in the image. The specific process of the present application based on the Canny edge detection algorithm will not be elaborated.

[0074] Further illustrate step S330. Specifically, in the embodiments of the present application, through step S330, an explicit calibration algorithm based on the light plane is implemented, and the schematic of the obtained light plane result is as Figure 4 shown, which can provide better flexibility and adaptability for color-encoded fringes with different rules and intervals.

[0075] In step S330, first, a light plane fitting process is performed based on the profile feature information of each light band, the light plane parameters are calculated, and the light plane equation is obtained; then, a light plane optimization process is performed: the extracted feature points are fitted based on the least squares method to obtain the optimal target light plane equation.

[0076] It's important to note that the light plane-based explicit calibration algorithm is a calibration method used in 3D measurement systems, particularly those using line-structured light. The primary purpose of this algorithm is to improve the accuracy and efficiency of the measurement system by precisely calibrating the position and orientation of the light plane.

[0077] The key steps of the explicit calibration algorithm based on the light plane are as follows: (1) Calculate the plane equation of the camera coordinate system of the checkerboard plane at the corresponding position through the external parameter matrix of the imaging module. (2) Calculate the homogeneous coordinates of the center line of the laser stripe in the camera coordinate system using the camera internal parameter matrix and the pixel coordinates of the center line of the laser stripe. (3) Combine the camera homogeneous coordinates of the center line of the laser stripe with the checkerboard plane equation in the camera coordinate system, and convert the camera homogeneous coordinates of the center line of the laser stripe into three-dimensional coordinates in the camera coordinate system. (4) Determine the diffraction laser line order corresponding to each laser stripe on the laser stripe image collected under low exposure parameters, and substitute the camera coordinates of the center line of the corresponding diffraction laser stripe into the laser stripe light plane equation of the corresponding order to obtain the initial value of the coefficient of the light plane equation of each diffraction laser stripe. (4) Select the square of the distance from the center point of the laser stripe to the light plane equation as the objective function, and the diffraction angle relationship of the grating equation as the constraint. The optimized value of each plane normal vector is obtained by the constrained least squares optimization algorithm. According to the fact that each diffracted light plane intersects a straight line in space, the optimization of the position parameters of each light plane is completed by expressing it through the spatial straight line equation and substituting it into the optimized value of the normal vector of each light plane.

[0078] Further explaining step S340, in the embodiment of the present application, through step S340, the light plane parameters in the target light plane equation determine the projection calibration parameters and establish the imaging model of the projection module.

[0079] Specifically, the projection intrinsic parameters include but are not limited to: focal length, principal point position, rotation matrix and translation vector. The projection extrinsic parameters include but are not limited to: position and direction.

[0080] Through steps S310 to S330, the projection module calibration process is performed based on the collected coded fringe image, which can effectively improve the calibration accuracy and calibration efficiency of the projection module.

[0081] To further illustrate step S140, step S140: Determine the system calibration result based on the imaging calibration parameters and the projection calibration parameters to complete the calibration. That is, determine the spatial geometric relationship between the two optical path systems of the imaging module and the projection module, and the spatial geometric relationship can be represented by two sets of internal and external parameter matrices and vectors. Thus, it lays a foundation for subsequent 3D reconstruction. The calibrated parameters will be applied to tasks such as oral 3D scene reconstruction and object recognition.

[0082] According to some embodiments of the present application, step S140 is further illustrated. Before completing the calibration, the control method further includes steps S410 to S420.

[0083] Step S410: When the current system calibration result is obtained, calibrate the imaging module and the projection module based on the current system calibration result.

[0084] Step S420: Perform error analysis and optimization processing on the calibrated imaging module and projection module to obtain an optimized system calibration result.

[0085] It can be understood that based on the system calibration result for testing, new checkerboard images and encoded stripe images are taken to verify the calibration effect.

[0086] It can be understood that the error analysis and optimization processing include: First, perform error analysis to analyze the gap between the reprojection error and the actual measurement value, and evaluate the reliability of the calibration. Specifically, after the calibration is completed, calculate the reprojection error. When the calibration accuracy is greater than the error threshold, it is evaluated that the calibration accuracy is unqualified. When the reprojection error is less than or equal to the error threshold, it is evaluated that the calibration accuracy is qualified. In the case where the calibration accuracy is greater than the error threshold, a non-linear optimization algorithm (such as Levenberg-Marquardt) can be used to further optimize the parameters. Until the reprojection error is less than or equal to the error threshold required by the oral 3D scanning system. Specifically, through multiple experimental data, fit and obtain the minimized reprojection error. By comparing the positions of the reprojected feature points and the actual feature points, error analysis can be performed, combined with multiple experiments and iterative calculations until the intraoral 3D scanning accuracy requirements are met.

[0087] Through steps S410 to S420, a parameter optimization mechanism is realized, so that the obtained system calibration result is more reliable.

[0088] According to some embodiments of the present application, step S150 is further illustrated. Step S150: Control the projection module and the imaging module to perform 3D reconstruction processing on the object to be reconstructed based on the system calibration result to obtain a 3D reconstruction result, including but not limited to steps S510 to S530.

[0089] Step S510: Control the calibrated projection module to perform projection processing, and alternately project white light and projection fringes generated based on De Bruijn coding on the surface of the object to be reconstructed in different time periods.

[0090] Step S520: Control the imaging module to collect a surface image facing the surface of the object to be reconstructed; the coded fringe image is displayed in the surface image.

[0091] Step S530: Perform surface reconstruction processing on the surface image based on the system calibration result to obtain a three-dimensional reconstruction result.

[0092] Prepare for data acquisition through step S510, and realize data acquisition through step S520.

[0093] Realize three-dimensional reconstruction more efficiently and reliably based on the system calibration result through steps S510 to S530.

[0094] According to some embodiments of the present application, step S530 is further described. Step S530: Perform surface reconstruction processing on the surface image based on the system calibration result to obtain a three-dimensional reconstruction result, including but not limited to steps S531 to S534.

[0095] Step S531: Perform image preprocessing on the surface image to obtain a preprocessed surface image.

[0096] Step S532: Perform fringe detection processing on the preprocessed surface image to obtain coded fringe information.

[0097] Step S533: Perform phase decoding processing and phase reconstruction processing on the coded fringe information to obtain reconstructed phase information.

[0098] Step S534: Perform three-dimensional information reconstruction processing according to the reconstructed phase information and the system calibration result to obtain a three-dimensional reconstruction result.

[0099] Specifically, the surface image also includes: a coded fringe image and a checkerboard image collected under white light.

[0100] According to some embodiments of the present application, the image preprocessing in step S531 is further described. Specifically, the image preprocessing includes: image denoising processing, threshold segmentation processing, and image enhancement processing. The image preprocessing includes: performing image denoising processing and threshold segmentation processing on the initial image to be processed to obtain a denoised initial image; performing image enhancement processing on the denoised initial image to obtain a preprocessed target image. In the surface reconstruction processing, the initial image to be processed is a coded fringe image, and after image preprocessing, the obtained target image refers to: multiple preprocessed coded fringe images.

[0101] Specifically, in the embodiments of the present application, through image denoising processing, a filter is used to remove background noise in the surface image, improving the quality of the surface image. In the embodiments of the present application, through image enhancement processing, the contrast of the stripes in the surface image is increased, making the stripes more obvious.

[0102] In the embodiments of the present application, through step S531, when performing surface reconstruction processing, it is first necessary to perform image preprocessing on the surface image to obtain a surface image with good quality, laying a data foundation for subsequent processing.

[0103] The preprocessed image can continue with stripe analysis. The main steps include: stripe detection, detecting the position and direction of the stripes; and phase extraction, extracting the phase information of each stripe through a phase decoding algorithm.

[0104] It can be understood that conventional algorithms require a large number of frames to be captured (such as the four-step phase-shifting method), which cannot meet the scanning speed requirements of a handheld 3D scanner. Based on this, the present application does not adopt the traditional phase extraction scheme. In the present application, the captured encoded stripe image includes different optical stripe color information that conforms to De Bruijn coding. By identifying the encoded stripe image, the corresponding phase information can be obtained. It should be noted that for occluded or shadowed areas where the De Bruijn-encoded stripes cannot be recognized, the phase information cannot be obtained, and stripe decoding and reconstruction cannot be performed.

[0105] Further elaborating on step S532, step S532: Perform stripe detection processing on the preprocessed surface image to obtain encoded stripe information. It can be understood that the original stripe image collected by the imaging module must undergo image preprocessing, including denoising and threshold segmentation, to obtain a clean stripe image. In step S532, after image processing, the stripes in the encoded stripe image in the surface image are still relatively thick. Therefore, centerline extraction based on a peak-seeking algorithm is performed. After centerline extraction, a color-coded image (i.e., encoded stripe information) with blue, green, and cyan colors attached can be obtained.

[0106] Further elaborating on step S533.

[0107] Specifically, phase decoding processing refers to: restoring the encoded stripe information to phase information. In the present application, De Bruijn coding conforms to a certain periodic law. Therefore, as long as the coding of the points on the stripes in the image is determined, the period and position to which the points belong can be determined. Subsequently, through the calculation of matrices and vectors in the parameter information of the system calibration result, the three-dimensional point cloud information can be reconstructed.

[0108] Specifically, the specific process of phase decoding processing is as follows: Decode the encoded stripe information in combination with the known De Bruijn sequence encoding (3x3 encoding), and extract the phase information corresponding to the scanned stripes. After corresponding all the coordinates extracted from the centerlines, a reconstructed phase map can be obtained. This phase map consists of the decoding results, rather than RGB information. At this time, the coordinate information is calculated with the system calibration results of the imaging module and the projection module obtained through the calibration process (i.e., including: imaging calibration parameters, projection calibration parameters), and the three-dimensional point information at this decoding position can be obtained.

[0109] Specifically, the specific process of decoding phase information and performing phase reconstruction processing using the characteristics of the De Bruijn sequence is as follows: Construct a De Bruijn graph, construct a graph according to the known De Bruijn sequence, and the nodes represent different states (such as the phase of the stripes). Matching process: Match the extracted phase information with the De Bruijn graph to find the most likely path. Reconstruct the phase map, and reconstruct the entire phase map according to the matching result.

[0110] It can be understood that no formula calculation is required in the decoding process of this application. This application scans the original stripe image from top to bottom, that is, scans each row once, scans the gray values of each pixel, determines the peak position, then determines the RGB color, performs decoding extraction, and combines the three-dimensional calibration parameters and the calibration vector result to determine the three-dimensional depth information.

[0111] Further illustrate step S534. Step S534: Perform three-dimensional information reconstruction processing according to the reconstructed phase information and the system calibration result to obtain a three-dimensional reconstruction result.

[0112] The three-dimensional information reconstruction processing of the embodiment of this application includes: Through a three-dimensional reconstruction algorithm, as shown in Figure 5 the three-dimensional reconstruction can be performed in combination with the internal and external parameters of the camera. The three-dimensional reconstruction algorithm adopted by the present invention is the triangulation method. The triangulation method refers to calculating the three-dimensional coordinates of the object surface through the known camera position and angle. Finally, a surface reconstruction algorithm is used, and algorithms such as Poisson reconstruction are used to generate the complete surface of the object; the three-dimensional reconstruction is completed.

[0113] Specifically, the three-dimensional information reconstruction processing of the embodiment of this application further includes: Model optimization processing; Model optimization processing includes: post-processing and meshing processing of the reconstruction result. It can be understood that since there will be certain decoding and reconstruction errors after the three-dimensional points are proposed, post-processing of the three-dimensional information is required. Post-processing of the reconstruction result includes smoothing processing to reduce surface noise and improve the model quality. And meshing, converting the point cloud data into a mesh model for subsequent visualization and analysis.

[0114] Provide a complete description of the control method provided by the embodiment of this application.

[0115] Starting from three-dimensional calibration, by moving the target back and forth, the preset chessboard image of the target is photographed to obtain the chessboard image under white light, and at the same time, the fringe image is photographed; using the Zhang's calibration method, a set of chessboard images under white light can obtain the three-dimensional calibration vector. For the encoded fringe image photographed simultaneously with the chessboard image under white light, because De Bruijn coding is used and the fringe order is fixed, each fringe can also correspond to the pixel points of the white light image. Therefore, accurate three-dimensional depth information can be obtained by converting through the three-dimensional calibration vector result, and the light plane result as shown in Figure 3 is obtained. After calibration, the acquisition of three-dimensional original images can be carried out. In one cycle, the system will acquire a chessboard image under white light and an encoded fringe image, and these two images are photographed at the same position. The encoded fringe image can be decoded, and then based on the light plane calibration result, it can be converted into depth information. Since the encoded fringe image is composed of one pixel, after adding the depth information, it is the point cloud result in three-dimensional space; obtaining the point cloud result can complete three-dimensional reconstruction based on the point cloud result.

[0116] For the calibration and three-dimensional reconstruction algorithm of the embodiments of the present application, without a scanning rod and positioning assistance, the root mean square error of single-chip three-dimensional point cloud reconstruction is as low as 57.9um, approaching the imaging resolution limit and the industry standard of 50um. The explicit calibration algorithm of the oral scanning system based on the light plane can effectively improve the measurement accuracy of the system. The overall plane error is as low as 0.06mm, and the plane error in the field of view area is reduced to 0.015mm. It can complete an automated calibration process and has more flexible adaptability to color-encoded fringes with different rules and intervals. At the same time, an optimized mechanism is proposed for the preprocessing, extraction, and decoding of fringe images. The algorithm has high robustness, and the decoding algorithm can better adapt to the scanning measurement of concave, high-brightness, truncated, etc. on the object surface, as shown in Figure 6A shown, Figure 6A is the scanned point cloud image without using the above optimization mechanism, Figure 6B is the scanned point cloud image when using the optimization mechanism for analysis and reconstruction. Comparing Figure 6A at point a and Figure 6B at point b, it can be seen that the single-frame scanned point cloud at point b is more complete, and irregular and error points can be eliminated to the greatest extent, greatly improving the reconstruction accuracy and meeting the high-precision requirements.

[0117] As shown in Figure 7 shown, the present invention also provides an electronic device, including:

[0118] A processor 701, which can be implemented in ways such as a general-purpose central processing unit, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0119] The memory 702 can be implemented in the form of a read-only memory, a static storage device, a dynamic storage device, or a random access memory, etc. The memory 702 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 702, and the processor 701 is used to call and execute the control method of the oral three-dimensional scanning system in the embodiments of this application;

[0120] The input / output interface 703 is used to implement information input and output;

[0121] The communication interface 704 is used to implement the communication interaction between this device and other devices. It can communicate through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0122] The bus 705 transmits information between various components of the device (such as the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0123] Among them, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are communicatively connected to each other inside the device through the bus 705.

[0124] The embodiments of this application also provide a storage medium. The storage medium is a computer-readable storage medium. This storage medium stores a computer program, and when the computer program is executed by a processor, it implements the control method of the above-mentioned oral three-dimensional scanning system.

[0125] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and may be located in one place, or may also be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] Those of ordinary skill in the art will understand that all or some of the steps and systems disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include a computer storage medium (or non-transitory medium) and a communication medium (or transitory medium). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes but is not limited to RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and can include any information delivery medium.

[0127] The above is a specific description of the preferred embodiment of the present application. However, the present application is not limited to the above embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present application, and these equivalent deformations or substitutions are all included within the scope defined by the present application.

Claims

1. A control method for an oral three-dimensional scanning system, characterized in that A host computer applied to the oral three-dimensional scanning system, the system further comprising: a calibrator and a scanning device respectively electrically connected to the host computer; the calibrator includes a sliding table and a target capable of moving along the sliding table; there is a preset checkerboard pattern on the target; the scanning device includes a projection module and an imaging module; The method includes: Controlling the projection module to project onto the target, and alternately projecting white light and projection stripes generated based on De Bruijn coding onto the preset checkerboard pattern in time segments; Controlling the movement of the target and controlling the imaging module to capture a plurality of target images towards the target; the target images include: coded stripe images collected at the same position and checkerboard images collected under white light; different target images are obtained by photographing the target at different positions; Performing imaging module calibration processing on a plurality of the checkerboard images to obtain imaging calibration parameters of the imaging module; at the same time, performing projection module calibration processing on a plurality of the coded stripe images to obtain projection calibration parameters of the projection module; Determining a system calibration result according to the imaging calibration parameters and the projection calibration parameters to complete the calibration; Controlling the projection module and the imaging module to perform three-dimensional reconstruction processing on the object to be reconstructed based on the system calibration result to obtain a three-dimensional reconstruction result.

2. The control method of the oral three-dimensional scanning system according to claim 1, characterized in that The performing imaging module calibration processing on a plurality of the checkerboard images to obtain imaging calibration parameters of the imaging module includes: Performing corner extraction processing on a plurality of the checkerboard images through a corner detection algorithm to correspondingly obtain a plurality of corner pixel coordinates; each corner pixel coordinate is obtained based on one frame of the checkerboard image; Matching the corner pixel coordinates with the actual corner coordinates to obtain a matrix correspondence relationship between the corner pixel coordinates and the actual corner coordinates; Solving the matrix correspondence relationship through the least squares method to obtain the imaging calibration parameters; the imaging calibration parameters include: imaging internal parameters and imaging external parameters.

3. The control method of the oral three-dimensional scanning system according to claim 1, wherein The performing projection module calibration processing on a plurality of the coded stripe images to obtain projection calibration parameters of the projection module includes: Performing image preprocessing on the plurality of coded stripe images to obtain a plurality of preprocessed coded stripe images; Performing feature extraction processing on the plurality of preprocessed coded stripe images based on the Canny edge detection algorithm to correspondingly obtain light band contour feature information of each of the coded stripe images; Performing light plane fitting processing and light plane optimization processing based on the least squares method in sequence according to the light band contour feature information of each to obtain a target light plane equation; Determining the projection calibration parameters according to the light plane parameters in the target light plane equation, the projection calibration parameters include: projection internal parameters and projection external parameters.

4. The control method of the oral three-dimensional scanning system according to claim 1, characterized in that, Before completing the calibration, the control method further includes: In the case of obtaining the current system calibration result, calibrating the imaging module and the projection module based on the current system calibration result; Performing error analysis and optimization processing on the calibrated imaging module and projection module to obtain an optimized system calibration result.

5. The control method of the oral three-dimensional scanning system according to claim 1, characterized in that, Controlling the projection module and the imaging module to perform three-dimensional reconstruction processing on the object to be reconstructed based on the system calibration result to obtain a three-dimensional reconstruction result, including: Controlling the calibrated projection module to perform projection processing, and alternately projecting white light and projection stripes generated based on De Bruijn coding on the surface of the object to be reconstructed in different time periods; Controlling the imaging module to collect a surface image facing the surface of the object to be reconstructed; the coded stripe image is displayed in the surface image; Performing surface reconstruction processing on the surface image based on the system calibration result to obtain the three-dimensional reconstruction result.

6. The control method of the oral three-dimensional scanning system according to claim 5, characterized in that, The performing surface reconstruction processing on the surface image based on the system calibration result to obtain the three-dimensional reconstruction result includes: Performing image preprocessing on the surface image to obtain a preprocessed surface image; Performing stripe detection processing on the preprocessed surface image to obtain coded stripe information; Performing phase decoding processing and phase reconstruction processing on the coded stripe information to obtain reconstructed phase information; Performing three-dimensional information reconstruction processing based on the reconstructed phase information and the system calibration result to obtain the three-dimensional reconstruction result.

7. The control method of the oral three-dimensional scanning system according to claim 3 or 6, characterized in that The image preprocessing includes: image denoising processing, threshold segmentation processing, and image enhancement processing; the image preprocessing includes: Performing the image denoising processing and threshold segmentation processing on the initial image to be processed to obtain a denoised initial image; Performing the image enhancement processing on the denoised initial image to obtain a preprocessed target image.

8. An oral three-dimensional scanning system, characterized in that, Including: A host computer, a calibrator and a scanning device respectively electrically connected to the host computer; the calibrator includes a slide table and a target capable of moving along the slide table; a preset checkerboard pattern is provided on the target; the scanning device includes a projection module and an imaging module; the host computer is configured to execute the control method of the oral three-dimensional scanning system according to any one of claims 1 to 7.

9. An electronic device, characterized in that, Including at least one processor and a memory for communicating with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the control method of the oral three-dimensional scanning system according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to execute the control method of the oral three-dimensional scanning system according to any one of claims 1 to 7.

Images