Intraoral three-dimensional imaging device

Abstract

An intraoral three-dimensional imaging device comprises a monocular blue light speckle structured light projector, a white light LED illumination module, a macro RGB camera, a depth decoding module, an inertial measurement unit IMU module and an oral cavity scanning head, the monocular blue light speckle structured light projector emits blue light speckle coding patterns, the white light LED illumination module emits white light, the monocular blue light speckle structured light projector and the white light LED illumination module project alternately, and the blue light speckle coding patterns and the white light are coded. After being reflected by the 45-degree mirror surface of the oral cavity scanning head, the light is irradiated to the oral cavity teeth, then the macro RGB camera collects the irradiated oral cavity tooth image, and RGBD data is output after the monocular structure light depth decoding of the depth decoding module. The device has the advantages of being small in size, good in imaging effect of resisting high reflection or semitransparent of a dental face, high in 3D modeling precision, high in generation speed and capable of removing the influence of water mist in an oral environment.

Description

Technical Field

[0001] The present invention belongs to the field of oral digital diagnosis and treatment, three-dimensional imaging and artificial intelligence technology, and particularly relates to an intraoral three-dimensional imaging device. Background Art

[0002] In the field of dental diagnosis and treatment, digital diagnosis and treatment models are rapidly replacing traditional diagnosis and treatment models and becoming the development trend and mainstream technology of dental diagnosis and treatment. They have become the mainstream of dental restoration, implantation and orthodontics. Currently, more than 90% of patients in my country who need dental restoration must first have a dentist make a dental mold using silicone materials in a dental hospital or clinic. The dental mold can only be successfully made through key steps such as preparing the impression material, impression inspection, impression disinfection and perfusion. The need for repeated impressions may increase manpower, material and environmental costs, and the entire process is long and the model accuracy is low.

[0003] In the field of digital dental diagnosis and treatment, 3D visual perception allows digital dental diagnosis and treatment equipment to obtain three-dimensional depth information of oral teeth, thereby providing more accurate data support for the diagnosis and treatment of oral and dental diseases and the design of dental braces. At the same time, digital dental diagnosis and treatment technology can also provide dentists with more convenient diagnostic methods and treatment plans.

[0004] The intraoral 3D imaging system can perform dynamic real-time 3D scanning of the oral environment, including teeth, soft and hard tissues, etc., directly generate a digital true-color 3D model, and then quickly make a crown mold through CAD and CAM technology, and finally make dentures through a 3D printer, which not only greatly shortens the production time and cost, but also facilitates repair. For doctors, it is simple to operate and easy to use, and can shorten the treatment cycle and reduce costs; for patients, it can not only improve the treatment experience and reduce the pain of treatment, but also reduce the number of medical visits and medical expenses. The existing intraoral 3D imaging system has problems such as large size, slow imaging speed, and low modeling accuracy. Summary of the invention

[0005] In view of this, the present disclosure provides an intraoral three-dimensional imaging device, which includes a monocular blue light speckle structured light projector, a white light LED lighting module, a macro RGB camera, a depth decoding module, an inertial measurement unit IMU module, and an oral scanning head. The monocular blue light speckle structured light projector emits a blue light speckle coding pattern, and the white light LED lighting module emits white light. The two are projected alternately, and then irradiated onto the oral teeth after 45° mirror reflection of the oral scanning head. The irradiated oral teeth image is then collected by the macro RGB camera, and RGBD data is output after monocular structured light depth decoding by the depth decoding module.

[0006] Through the above technical solution, the device has the characteristics of small size, good imaging effect against high reflection or semi-transmission of tooth surface, high 3D modeling accuracy and fast generation speed.

[0007] Compared with the prior art, the intraoral three-dimensional imaging device disclosed in the present invention has the following beneficial effects:

[0008] 1. The intraoral three-dimensional imaging device disclosed in the present invention adopts a monocular blue light speckle structured light projector as an active projection coding light source. Under the condition of the same light source power, the emitted blue light speckle coding pattern is more suitable for three-dimensional measurement of highly reflective, semi-transparent tooth surfaces or braces than traditional color stripe structured light coding projection.

[0009] 2. The present invention is based on blue laser operation. Compared with red light or other visible light sources, the features of the speckle pattern projected on the oral teeth after encoding are more obvious, which solves the problem of three-dimensional measurement of highly reflective and semi-transparent materials.

[0010] 3. The present invention adopts the working mode of surface laser coding macro projection, which has the characteristics of resistance to high reflection, resistance to the influence of semi-transparent materials, fast 3D imaging speed, high point cloud resolution and low device power consumption.

[0011] 4. The oral scanning head of the intraoral three-dimensional imaging device disclosed in the present invention has a self-heating function, which can automatically remove the water mist formed on the reflector in the intraoral environment to achieve a clear imaging effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the structure of an intraoral three-dimensional imaging device provided by one embodiment of the present disclosure;

[0013] Figure 2 is a structural schematic diagram of an oral scanning head provided in one embodiment of the present disclosure;

[0014] Figure 3 It is a schematic diagram of the structure of a blue light speckle coding pattern projected by a monocular blue light speckle structured light projector provided in one embodiment of the present disclosure;

[0015] Figure 4 is a schematic diagram of the structure of a depth decoding module provided by an embodiment of the present disclosure;

[0016] Figure 5 is a schematic diagram of a monocular blue light speckle structured light encoding and decoding method based on speckle image blocks provided by another embodiment of the present disclosure;

[0017] Figure 6 is a front view of an oral scanning head device provided by one embodiment of the present disclosure;

[0018] Figure 7is a schematic diagram of connection between an oral scanning head device and an oral scanner provided by another embodiment of the present disclosure;

[0019] Figure 8 is a disassembled schematic diagram of an oral cavity scanning head device provided by another embodiment of the present disclosure;

[0020] Fig. 9 is an internal structure diagram of an oral scanning head device provided by another embodiment of the present disclosure;

[0021] Fig.10 is a bottom view of an oral scanning head device provided by another embodiment of the present disclosure;

[0022] Fig.11 It is a structural schematic diagram of a macro blue light speckle LD projector provided in one embodiment of the present disclosure;

[0023] Fig.12 is a functional schematic diagram of a macro blue light speckle LD projector provided in one embodiment of the present disclosure;

[0024] Fig.13 This is a schematic diagram of an application of a macro blue light speckle LD projector in a mouth scanning digital impression device provided in one embodiment of the present disclosure;

[0025] Fig.14 is a flow chart of a method for intra-oral depth perception based on monocular blue speckle structured light provided by an embodiment of the present disclosure;

[0026] Fig.15 is a structural diagram of an intra-oral depth perception device of a monocular blue light speckle structured light provided by another embodiment of the present disclosure;

[0027] Fig.16 is a blue light speckle image projected by a monocular blue light speckle projector provided by another embodiment of the present disclosure;

[0028] Fig.17 is a pre-processed blue light speckle image provided by another embodiment of the present disclosure;

[0029] Fig.18 is a physical model of oral teeth provided by another embodiment of the present disclosure;

[0030] Fig.19 Another embodiment of the present disclosure provides a three-dimensional model of oral teeth obtained based on color stripe structured light;

[0031] Fig. 20 Another embodiment of the present disclosure provides a three-dimensional model of oral teeth obtained based on monocular blue light speckle structured light;

[0032] Fig.21is a flow chart of a method for three-dimensional modeling of oral teeth provided by an embodiment of the present disclosure;

[0033] Fig. 22 is a schematic diagram of a point-to-plane ICP registration method provided by another embodiment of the present disclosure;

[0034] Fig.23 is a schematic diagram of a normalization function provided by another embodiment of the present disclosure;

[0035] Fig.24 is a schematic diagram of surface estimation using a light transmission method provided by another embodiment of the present disclosure;

[0036] Fig.25 is a state machine flow chart of different step length rendering provided by another embodiment of the present disclosure;

[0037] The following are the descriptions of the reference numerals:

[0038] 1. Upper structural member; 2. Lower structural member; 2-1. Connecting port; 2-2. Accommodating cavity; 3. Reflector mounting slot; 4. Reflector; 5. Heating wire; 6. Flexible circuit board; 7. Positive and negative contacts; 8. Light outlet hole; 9. Direct light barrier. DETAILED DESCRIPTION

[0039] In order to make those skilled in the art understand the technical solutions disclosed in the present disclosure, the following will be combined with the embodiments and related appended Figures 1 to 25 , the technical solutions of various embodiments are described, and the described embodiments are part of the embodiments of the present disclosure, but not all of the embodiments.

[0040] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present disclosure. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It will be understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0041] See also Figure 1 In one embodiment, the present disclosure discloses an intraoral three-dimensional imaging device, which includes a monocular blue light speckle structured light projector, a white light LED lighting module, a macro RGB camera, a depth decoding module, an inertial measurement unit IMU module, and an oral scanning head. The monocular blue light speckle structured light projector emits a blue light speckle coding pattern, and the white light LED lighting module emits white light, which are projected alternately and irradiated onto the oral teeth after 45° mirror reflection of the oral scanning head. The irradiated oral tooth image is then collected by the macro RGB camera, and RGBD data is output after the monocular structured light depth decoding of the depth decoding module.

[0042] In this embodiment, the intraoral three-dimensional imaging device emits a blue light speckle coding pattern from a monocular blue light speckle structured light projector or white light from a white light LED lighting module, which is then reflected by a 45° mirror of the oral scanning head and irradiated onto the oral teeth. The image is then captured by a macro RGB camera, and the depth information is obtained through the monocular structured light depth decoding of the depth decoding module, and then RGBD data (data including RGB information and depth information) is output through a USB or network interface. During the scanning process, the IMU module has data, which is jointly packaged and output through the depth decoding module.

[0043] The monocular blue light speckle structured light projector is used to project a blue light speckle coding pattern onto the oral teeth, and adopts a blue light LD or VCSEL as a light source. Compared with red light or other visible light, the speckle imaging in the oral environment is clearer and more conducive to the resolution of the RGB camera, because the present disclosure fully takes into account that the blood and blood vessels of the human body are red.

[0044] The white light LED lighting module is used to irradiate white light to the oral teeth as the illumination and fill light of the RGB image; illustratively, it generally uses 3 to 20 LED particles, which are arranged in a circle around the monocular blue light speckle structured light projector and the macro RGB camera; using white light LED as the light source can make the RGB image color positive under white light irradiation, and better restore the original color of the oral teeth.

[0045] The depth decoding module performs monocular structured light depth decoding on the blue light speckle coded image collected by the macro RGB camera. The monocular blue light speckle structured light projector and the white light LED lighting module are controlled in PWM mode by synchronously triggering the Strobe signal, including the selection of the light-emitting module, the switching mode, the light source lighting time and intensity; at the same time, the RGBD image data is packaged and output through the USB or network interface. The module includes a synchronous triggering submodule, a monocular blue light speckle structured light depth calculation submodule, an RGB image preprocessing submodule, and a data embedding and packaging output submodule;

[0046] The IMU module, exemplarily, can be a six-axis IMU or a nine-axis IMU, which is used to detect the posture of the intraoral three-dimensional imaging device, and output the acceleration, angular velocity, and rotation angle values ​​in the XYZ direction, which is helpful for the 3D real-time modeling of oral teeth. Exemplarily, the posture information is output to the depth decoding module through the serial port. That is, the IMU module is used to obtain the posture information of the device (rotation matrix R / translation matrix T), which can be used for 3D reconstruction and repositioning during rescanning. Exemplarily, R / T can also be obtained from the depth point cloud and written. The purpose is to have one more way to obtain R / T, so that the device is more reliable.

[0047] The oral scanning head, the structural diagram is shown in Figure 2 The blue light speckle coding pattern projected by the monocular blue light speckle structured light projector or the white light emitted by the white light LED lighting module is reflected by a 45° mirror and then irradiated onto the oral teeth, and then the irradiated oral teeth image is collected by a macro RGB camera; at the same time, a temperature heating module is attached to the back of the mirror to remove water mist on the mirror, and the wires of the heating module are connected to the depth decoding module through pluggable contacts and heated by electric current.

[0048] As far as this embodiment is concerned, the 45° mirror reflection is adopted because the optical axes of the monocular blue light speckle structured light projector and the macro RGB camera form a 45° angle with the reflecting mirror, so that the projected blue light speckle coding pattern and the received blue light speckle coding image are not deformed (deformation refers to non-linear elongation, shortening or distortion of the image). Deformation will cause inaccurate depth information of the depth decoding module; at the same time, the 45° mirror reflection imaging method can be suitable for the intraoral environment, especially the scanning of the posterior teeth area.

[0049] The present disclosure also includes a button and a main structural member, wherein the button is used for one-touch control of the image acquisition mode and the light source intensity, for example: a short press can pause or start scanning; a long press is used to control the light source illumination intensity or switch the image acquisition mode, wherein the light source illumination intensity, and the intensity of the blue light LD and the white light LED are generally divided into three levels, high brightness, normal, and darker, and the intensity is adjusted to meet the imaging requirements of different scanning applications and materials; wherein the image acquisition mode includes pure blue light speckle, pure white light illumination, and blue light and white light time-division multiplexing switching mode.

[0050] The main structural member is used to fix the monocular blue light speckle structured light projector, the white light LED lighting module, the macro RGB camera, the depth decoding module, the IMU module, and the button. It should be noted that in this embodiment, it is necessary to ensure that the monocular blue light speckle structured light projector and the macro RGB camera are on the same baseline and the optical axes are parallel. In addition, the oral scanning head is covered on the main structural member. In addition, in this embodiment, it is necessary to ensure that the angle between the 45° mirror and the optical axis of the macro RGB camera is 45°.

[0051] In another embodiment, the blue light speckle coding pattern is a single speckle coding pattern composed of a certain number of random scattered spots, and the scattered spots are symmetrical around the center of the pattern.

[0052] In this embodiment, the speckle coding pattern is composed of a certain number of random speckle spots forming a single speckle coding pattern, such as Figure 3 As shown in FIG. 1 , there is symmetry between the speckles around the center of the pattern, that is, the speckle pattern in the upper left corner is symmetrical with the speckle pattern in the lower right corner, and the speckle pattern in the lower left corner is symmetrical with the speckle pattern in the upper right corner. Figure 3The scattered spots A and A' are shown to be symmetrical about the center point.

[0053] In another embodiment, the speckle block composed of a plurality of random speckles in the speckle coding pattern is unique within a certain range in the horizontal or vertical baseline direction. This means that each speckle block can have a unique identifier, which can be used for precise positioning and matching: because each speckle block is unique within a certain range, this unique characteristic, i.e., a unique and unique identifier, can be used to track and match the changes in the speckle block, thereby inferring the depth information.

[0054] In another embodiment, ensure that the monocular blue light speckle structured light projector and the macro RGB camera are on the same baseline and their optical axes are parallel.

[0055] In another embodiment, the monocular blue speckle structured light projector is used to project a blue speckle coding pattern onto oral teeth, and it is equipped with a temperature sensing module, which has multiple functions: on the one hand, a one-to-one correspondence between temperature and the blue speckle coding pattern can be established, and the depth decoding module can find the corresponding reference blue speckle coding image through the temperature value for accurate depth decoding; on the other hand, the temperature can be detected to prevent the temperature from being too high.

[0056] Exemplarily, with respect to this embodiment, the monocular blue light speckle structured light projector is generally a blue light LD speckle projector or a VCSEL speckle projector.

[0057] Among them, the blue light LD speckle projector is a blue light LD (Laser Diode) light source that emits blue light. After being focused by a light hole and a collimator, a speckle coding pattern is projected by a diffractive optical device DOE. The blue light LD light source is a single-point blue light LD laser that emits a wavelength of 400nm~480nm. The smaller the light-emitting aperture, the smaller the speckle spots projected by the diffractive optical device. Its wavelength will drift with temperature changes, causing the projected macro speckle coding pattern to deform. Therefore, at different temperatures, even the reference speckle coding image at a fixed distance will have certain differences. The light hole is a retractable spring, and its diameter can be used to control the size of the light-emitting hole of the blue light LD light source. The collimator focuses and collimates the single-point laser beam emitted by the blue light LD light source, and is generally composed of a convex lens.

[0058] The VCSEL speckle projector is composed of a VCSEL coded light source, a collimator and a diffractive optical element DOE, and projects a speckle coded pattern in a spatial coding manner. The VCSEL coded light source includes a certain number (10 to 5000 points) of VCSEL (vertical cavity surface emitting laser) light points, and the arrangement of these light points is a random coding pattern, that is, the combination of light points is not repeated within a certain range in the horizontal or vertical direction, and all light points constitute a basic coding pattern. The collimator is used to collimate the laser beam emitted by the VCSEL coded light source into parallel light. The collimator and the VCSEL light source and the DOE diffractive optical element can be fixed, or they can be moved up and down within a certain range facing the optical axis to achieve the adjustment of the size of the scattered spots and the degree of focus in the projected speckle spatiotemporal coding pattern. The diffractive optical device DOE replicates the basic coding pattern in an m*n manner, such as 3*3, 5*3, 7*5, etc., to form a speckle coding pattern with more spots, and ensures that the brightness and contrast of the spots in different replicated blocks remain balanced and consistent.

[0059] In another embodiment, the macro RGB camera is used to capture blue light speckle coded images or white light RGB images irradiated on oral teeth.

[0060] Exemplarily, with respect to this embodiment, the depth of field of the macro RGB camera is a macro range of 2 to 10 cm; generally, blue light speckle coded images or white light RGB images are collected frame by frame through time division multiplexing, and the images are output to the depth decoding module in RAW data format, for example; the macro RGB camera can choose to have an embedded blue light filter to enhance the blue light speckle image effect.

[0061] In another embodiment, the depth decoding module includes a synchronization triggering submodule, a monocular blue light speckle structured light depth calculation submodule, an RGB image preprocessing submodule, and a data embedding and packaging output submodule.

[0062] For example, in this embodiment, Figure 4 As shown, the monocular structured light depth decoding is performed on the blue light speckle coded image collected by the macro RGB camera, and the monocular blue light speckle structured light projector and the white light LED lighting module are controlled in PWM mode by synchronously triggering the Strobe signal, including the selection of the light-emitting module, the switching mode, the light source lighting time and intensity; at the same time, the RGBD image data is packaged and output through the USB or network interface; the module includes a synchronous triggering submodule, a monocular blue light speckle structured light depth calculation submodule, an RGB image preprocessing submodule, and a data embedding and packaging output submodule.

[0063] In another embodiment, the synchronization trigger submodule generates a synchronization trigger Strobe signal according to the field synchronization signal input by the macro RGB camera, and controls the monocular blue light speckle structured light projector and the white light LED lighting module in a PWM manner.

[0064] In this embodiment, the control includes the selection of the light module, the switching mode, the light source lighting duration and intensity. The macro RGB camera input has a line / field synchronization signal and data, wherein the field synchronization signal is used to generate a synchronization trigger Strobe signal.

[0065] In another embodiment, the monocular blue light speckle structured light depth calculation submodule obtains the disparity by block matching of the speckle blocks, and then calculates the depth value of each pixel in the blue light speckle image by using the monocular structured light depth calculation formula.

[0066] In this embodiment, the monocular blue light speckle structured light depth calculation submodule is used to calculate the depth value of each pixel in the blue light speckle coded image; the specific calculation process is:

[0067] The depth value d corresponding to each pixel is obtained by comparing the blue light speckle coded image with a reference blue light speckle coded image of a known distance (the reference blue light speckle coded image is projected onto a plane perpendicular to the optical axis of the monocular blue light speckle structured light projector as a reference comparison benchmark for monocular structured light depth calculation. Its vertical reference distance from the projector is is known) to perform block matching disparity calculation, such as Figure 5 As shown, combined with the laser triangulation method, the depth value corresponding to the center of the speckle block (size is m*n, m and n are positive integers) in the blue light speckle coded image is obtained. for:

[0068] in, is the known distance of the reference blue light speckle image, The disparity value corresponding to the optimal matching block is obtained by performing block matching disparity calculation between the preprocessed blue light speckle coded image and the preprocessed reference blue light speckle coded image of known distance. is the baseline distance between the macro RGB camera and the monocular blue speckle structured light projector, is the focal length of the macro RGB camera, is the dot pitch of the macro RGB camera image sensor; It should be noted that before calculating the depth value d, the blue light speckle coded image and the reference blue light speckle coded image may be preprocessed, specifically by using existing methods such as binarization and histogram enhancement, so that the speckle blocks of the blue light speckle coded image and the reference blue light speckle coded image are highly similar.

[0069] In another embodiment, the RGB image preprocessing submodule performs denoising and enhancement on the input blue light speckle coded image to extract blue light speckle spots; and simultaneously performs enhancement processing on the input white light RGB image.

[0070] Exemplarily, in this embodiment, the RGB image preprocessing submodule, on the one hand, performs denoising and enhancement (including binarization or histogram enhancement operations) on the input blue light speckle coded image to extract blue light speckle; on the other hand, it is used for enhancement processing of the input white light RGB image, including Gamma correction, histogram enhancement, HSI color correction, and Y edge enhancement of the YUV space on the RGB image.

[0071] In another embodiment, the data embedding and packaging output submodule jointly packages and outputs the RGB and Depth (ie, depth) data of the embedded auxiliary information.

[0072] Exemplarily, with regard to this embodiment, the data embedding and packaging output submodule reads the temperature T, IMU posture data, and mark bit (for example, 1-bit mark bit, 1 indicates that the current frame is a Depth image, and 0 indicates that the current frame is a white light RGB image) auxiliary information of the monocular blue light speckle structured light projector through the processor or MCU of the depth decoding module, embeds it into the specified position (kth row or lth column) in the RGB image data, and replaces the pixel value of the original position of the RGB image, so as to package the RGB and Depth data of the embedded auxiliary information separately or jointly, and output it through the USB or network interface.

[0073] Auxiliary information is embedded into the data of each frame (not only white light RGB images or blue light speckle coded images or Depth depth maps), and the specified column on the specified row replaces the original RGB or Depth data. For example, in the nth row and the kth column, the hexadecimal value 0F represents the temperature value T of the current frame, replacing the RAW value A2 (hexadecimal value) of the original Depth data.

[0074] In another embodiment, the RAW data input by the macro RGB camera in the depth decoding module and the auxiliary information such as the projector temperature, IMU posture, and marker position are directly packaged and output to the host computer or remote data center through the USB or network interface, bypassing the RGB image preprocessing submodule and the monocular blue light speckle structured light depth calculation submodule. After the image data is packaged to the host computer or remote data center, the RGB image preprocessing submodule and the monocular blue light speckle structured light depth calculation submodule perform image preprocessing and depth calculation in the SDK software mode.

[0075] In another embodiment, the IMU module may not be integrated (i.e., the inertial measurement unit IMU module may not be included in the device of the present disclosure, and the IMU module is not required). If the IMU module is integrated, the posture information of the intraoral three-dimensional imaging device can be directly obtained, and the posture information can be used for three-dimensional modeling of oral teeth or for solving the repositioning problem during supplementary scanning. In addition, in the device of the present disclosure, some modules can be implemented in different physical forms, such as implementing some sub-modules in the depth decoding module in the form of dedicated chips, FPGA or software SDK.

[0076] In addition, the oral scanning head disclosed in the present disclosure relates to the following embodiments: In another embodiment, Figures 6 to 8 As shown, the present disclosure provides a device of an oral scanning head, comprising: The device body includes an upper structure 1 and a lower structure 2 which are detachably connected, wherein: The lower structure 2 includes an integrally formed trumpet-shaped connection port 2-1 and a semi-enclosed groove-type accommodation cavity 2-2, and the connection port 2-1 is connected to the oral scanner; The upper structure 1 covers the outside of the accommodating cavity 2-2, and a reflector mounting groove 3 is provided at one end of the upper structure 1 away from the connecting port 2-1, and a reflector 4 for reflecting a monocular blue light speckle coding pattern or white light projected by the oral scanner is installed on the reflector mounting groove 3.

[0077] The above embodiments constitute a complete technical solution of the oral scanning head device disclosed in the present invention. The cavity scanning head device described in this embodiment is mounted on a monocular blue light speckle projection oral scanner, and can reflect the blue light speckle coding pattern or white light projected by the oral scanner into the oral cavity through a reflector. The diffusely reflected image is refracted by the reflector and then collected by a macro RGB camera in the oral scanner to obtain a blue light speckle image or a white light RGB image in the oral cavity.

[0078] In another embodiment, the angle between the reflector mounting groove 3 and the optical axes of the monocular blue light speckle structured light projector and the macro RGB camera in the oral scanner is 45°.

[0079] In this embodiment, since the angle between the reflector mounting groove and the optical axes of the monocular blue light speckle structured light projector and the macro RGB camera in the oral scanner is 45°, the angle between the reflector mounted on the groove and the optical axes of the monocular blue light speckle structured light projector and the macro RGB camera is naturally also 45°.

[0080] It should be noted that, by setting the angle between the reflector and the optical axes of the monocular blue speckle structured light projector and the macro RGB camera to 45°, this can ensure that the blue light speckle coding pattern projected by the monocular blue speckle structured light projector and the received blue light speckle coding image do not deform (deformation refers to non-linear elongation, shortening or distortion of the image, which will cause inaccurate depth information of the depth decoding module). At the same time, according to clinical tests, the 45° mirror reflection imaging method is most suitable for scanning the intraoral environment, especially the posterior teeth area.

[0081] In another embodiment, a heating wire 5 is disposed in the accommodating cavity 2 - 2 , and the heating wire 5 is attached to a side of the reflector 4 facing the reflector mounting groove 3 .

[0082] In this embodiment, by providing a heating wire, the water mist formed on the reflector can be removed, thereby improving the imaging clarity.

[0083] In another embodiment, if Figure 8 and Fig. 9 As shown, a flexible circuit board 6 is also provided in the accommodating cavity 2 - 2 , and one end of the flexible circuit board 6 is electrically connected to the heating wire 5 .

[0084] Preferably, positive and negative contacts are also provided in the accommodating cavity, one end of the positive and negative contacts is electrically connected to the other end of the flexible circuit board, and the other end of the positive and negative contacts is electrically connected to the positive and negative contacts in the oral scanner.

[0085] Exemplarily, in one embodiment, pluggable positive and negative contacts are also provided in the accommodating cavity of the lower structure, one end of the positive and negative contacts is electrically connected to the other end of the flexible circuit board, and the other end of the positive and negative contacts is electrically connected to the positive and negative contacts in the oral scanner. A 3.3v, 5v or 2.5v DC power supply is provided in the oral scanner, and the DC power supply supplies power to the heating wire through a circuit composed of the positive and negative contacts in the oral scanner, the flexible circuit board, and the positive and negative contacts in the accommodating cavity of the lower structure. When the oral scanner starts to scan the oral cavity, the DC power supply supplies power and heats the heating wire in real time, thereby being able to remove water mist on the reflector in real time.

[0086] In another embodiment, if Fig.10 As shown, a light exit hole 8 is provided at the bottom of the lower structure 2 .

[0087] In this embodiment, the light exit hole is set to be rectangular, and the length and width are set between 5-30 mm. The blue light speckle coding pattern or white light reflected by the reflector can be projected to the posterior tooth area through the light exit hole, thereby ensuring the completeness of the scan.

[0088] In another embodiment, if Fig.10 As shown, a direct light barrier 9 is also provided at the bottom of the lower structure 2 .

[0089] In this embodiment, the direct light blocker is arranged on one side of the light exit hole, and the height is set between 0.5-5mm. The direct light blocker can block the blue light speckle coding pattern projected directly through the light exit hole by the monocular blue light speckle structured light projector in the oral scanner without being reflected by the reflector, thereby avoiding aliasing with the blue light speckle coding pattern reflected on the oral teeth by the reflector (if aliasing occurs, it will cause problems such as depth decoding errors of the oral scanner, increased noise, and inaccurate depth information).

[0090] Preferably, the connecting port is trumpet-shaped.

[0091] Preferably, the accommodating cavity is a semi-enclosed groove-type accommodating cavity.

[0092] In another embodiment, the inner walls of the upper structure and the lower structure are sprayed with a matte black coating.

[0093] In this embodiment, by spraying a matte black coating on the inner wall of the upper structure and the lower structure, the scattering of the blue light speckle coding pattern or white light on the inner wall projected by the monocular blue light speckle structured light projector in the oral scanner can be greatly reduced, thereby reducing the noise of the blue light speckle image and the white light RGB image collected by the macro RGB camera. It should be noted that the matte black coating described in this embodiment is purchased from the market, and the present disclosure is not intended to make any improvements to the coating material itself.

[0094] In summary, the present disclosure provides an oral scanning head device, which has the characteristics of small size, good imaging effect against high reflection or semi-transparency of tooth surface, convenient for scanning the posterior tooth area, and has the following beneficial effects: 1. Use a 45° reflector to reflect the monocular blue light speckle coding pattern or white light projected by the oral scanner onto the oral teeth for imaging and three-dimensional measurement, solving the problem of small intraoral space and difficult measurement of the posterior teeth area.

[0095] 2. A heating wire is provided to automatically remove the water mist formed on the reflector in the intraoral environment, so as to achieve a clear imaging effect.

[0096] 3. The method of spraying matte black coating is adopted to reduce the influence of the inner wall on the blue light speckle coding pattern and the diffuse reflection of LED white light.

[0097] 4. The size of the reflective surface and the opening area can be reduced without changing the main structure, so that the device is suitable for making a smaller oral scanning head device for children's oral scanning applications.

[0098] In addition, the monocular blue light speckle structured light projector disclosed in the present disclosure relates to the following multiple embodiments: See also Fig.11In one embodiment, the present disclosure discloses a macro blue light speckle LD projector, including a blue light LD light source, a light hole, a collimator, a diffractive optical device DOE and an FPC driving board with temperature sensing. The blue light LD light source emits blue light, which is focused by the light hole and the collimator, and then the diffractive optical device DOE projects a macro speckle coding pattern. The blue light LD light source is driven by an FPC driving board with a temperature sensing module, and a corresponding relationship between the temperature of the blue light LD light source and the macro speckle coding pattern is established.

[0099] In this embodiment, a blue light is emitted by a blue light LD (Laser Diode) light source, which is focused by a light aperture and a collimator, and then projected into a macro speckle coding pattern by a diffractive optical device DOE, such as Fig.12 As shown, a driving board with a temperature sensing module is used to drive a blue light LD light source, and a corresponding relationship between temperature and a macro speckle coding pattern is established. One temperature value corresponds to one pattern. It can be found that the projector disclosed in the present invention is not limited to the projected object, for example, not limited to the oral cavity or teeth.

[0100] The blue light LD light source is a single-point blue light LD laser emitting a wavelength of 400nm~480nm. The smaller the light-emitting aperture, the smaller the scattered spots projected by the diffraction optical device. The wavelength will drift with temperature changes, causing the projected macro speckle coding pattern to deform. The light-through hole is a retractable spring, and its tube diameter can be used to control the size of the light-emitting hole of the blue light LD light source. The collimator focuses and collimates the single-point laser beam emitted by the blue light LD light source, and is generally composed of a convex lens.

[0101] The collimating lens is selected from a single lens group or a plurality of lens groups, or a micro lens array corresponds to VCSEL dot matrix in different regions one by one.

[0102] The collimating mirror is made of a material selected from a collimating device made of all plastic, glass-plastic mixture or WLO (Wafer Level Optical) wafer-level process.

[0103] The collimation process refers to making the laser beam with a large divergence angle parallel through a collimator and focusing it to form a parallel light beam.

[0104] The diffractive optical element (DOE) is used to diffract a single-point blue light source into a speckle coding pattern symmetrical at the center point. The depth of field of the projected pattern is in the macro range of 2 to 10 cm, and the depth of field can be adjusted by adjusting the length of the light hole, that is, the distance between the blue light LD light source and the collimator.

[0105] In another embodiment, a mirror holder is also included for fixing the blue light LD light source, the light hole, the collimating mirror and the diffractive optical device DOE.

[0106] In this embodiment, the lens holder is composed of upper and lower cylindrical structural parts with threads, which are used to fix the blue light LD light source, the light hole, the collimator and the diffraction optical device DOE, ensuring that the centers of these devices are on the same optical axis, and the length of the light hole can be adjusted by adjusting the cylindrical structural part up and down with threads, thereby achieving depth of field adjustment of the macro speckle coding pattern.

[0107] In another embodiment, the flexible circuit board FPC driving board with the temperature sensing module is placed under the blue light LD light source and the mirror holder, and the temperature sensing module includes a temperature sensing chip and a thermistor NTC.

[0108] In this embodiment, the FPC (Flexible Printed Circuit) driver board with temperature sensing is composed of a temperature sensing chip or a thermistor NTC and an FPC, and is placed under the blue light LD light source. On the one hand, it is used to detect the temperature of the blue light LD light source in real time. Once the temperature is too high, the driving current can be controlled to protect the LD light source; on the other hand, it is used to detect the change of the projected macro speckle coding pattern and establish a one-to-one correspondence between the temperature and the macro speckle coding pattern. During the depth decoding process of the monocular structured light, the corresponding reference macro speckle coding image can be found through the temperature value for accurate depth decoding. Typically, a table is established in advance and then searched by the table lookup method.

[0109] In another embodiment, the FPC driving board with the temperature sensing module can also be used to detect the temperature of the blue light LD light source in real time, and once the temperature is too high, the driving current can be controlled to protect the LD light source.

[0110] In another embodiment, the macro speckle coding pattern is a single speckle coding pattern composed of a certain number of random speckles.

[0111] In another embodiment, there is symmetry between the scattered spots around the center of the pattern.

[0112] In this embodiment, the macro-speckle coding pattern is composed of a certain number of random speckles forming a single speckle coding pattern. Figure 3 As shown in FIG. 1 , there is symmetry between the speckles around the center of the pattern, that is, the speckle pattern in the upper left corner is symmetrical with the speckle pattern in the lower right corner, and the speckle pattern in the lower left corner is symmetrical with the speckle pattern in the upper right corner. Figure 3 The scattered spots A and A' are shown to be symmetrical about the center point.

[0113] In another embodiment, a speckle block composed of a plurality of random speckles in the macro-speckle coding pattern is unique within a certain range in a horizontal or vertical baseline direction.

[0114] In this embodiment, the speckle block composed of multiple random speckles in the macro speckle coding pattern is unique within a certain range in the horizontal or vertical baseline direction. The certain range refers to a range of N pixels moving left and right in the horizontal direction; a range of M pixels moving up and down in the vertical direction; the values ​​of M and N can be 1 to 500 pixels.

[0115] In another embodiment, the macro speckle coding pattern is fixed, and the number and style of the pattern remain unchanged within the depth of field.

[0116] In this embodiment, the macro speckle coding pattern is fixed, the number of points and style of the pattern remain unchanged within the depth of field, and the speckle spots have certain changes in size and clarity due to different focusing imaging distances. At the same time, the projected macro speckle coding pattern will cause certain deformation with the temperature drift of the blue light LD light source. The reason is that the wavelength of the blue light LD light source will drift with the temperature change, which will cause the projected macro speckle coding pattern to deform. That is, at different temperatures, even the macro speckle coding pattern projected at a fixed distance will have certain differences. Therefore, it is necessary to establish a one-to-one correspondence between the macro speckle coding pattern and the LD light source temperature. This is conducive to the monocular structured light in the depth decoding process can be used to find the corresponding reference macro speckle coding image through the temperature value for accurate depth decoding.

[0117] In another embodiment, the projector can also be applied to infrared light or other bands of visible light.

[0118] In this embodiment, although the macro blue light speckle LD projector takes the blue light wavelength as an example, it can also be applied to other wavelength bands of infrared and visible light.

[0119] A schematic diagram of an application of a macro blue light speckle LD projector in a digital mouth scan impression device provided in an embodiment of the present disclosure is shown in FIG. Fig.13 The macro blue light speckle LD projector is on the same baseline as the macro RGB camera, and the macro RGB camera collects the projected macro speckle coding pattern for monocular structured light depth decoding.

[0120] In summary, the projector disclosed in the present disclosure is different from the existing LD projector. It has a temperature sensing function, realizes the protection of the LD light source, establishes the corresponding relationship between the temperature of the blue light LD light source and the macro speckle coding pattern, and the monocular structured light can find the corresponding reference macro speckle coding image through the temperature value during the depth decoding process for accurate depth decoding. The projector can not only be used for digital diagnosis and treatment of oral teeth (restoration, orthodontics or implantation), but also can be used in the field of industrial high-precision 3D measurement and scanning of the internal structure of cultural relics.

[0121] In addition, the present disclosure also discloses a method for intraoral depth perception based on monocular blue light speckle structured light, which involves the following embodiments: In one embodiment, see Fig.14 The present disclosure also discloses a method for intraoral depth perception based on monocular blue light speckle structured light, comprising the following steps: S100: Through time division multiplexing, blue light speckle coding patterns and LED white light irradiation are projected into the oral cavity, and blue light speckle images are collected and generated respectively (see Fig.16 ) and white light RGB images; S200: preprocessing the blue light speckle image and the white light RGB image respectively; S300: performing depth decoding on the preprocessed blue light speckle image to obtain an intraoral depth image; S400: In each frame of RGB and Depth image, the temperature of the monocular blue light speckle projector, IMU posture parameters (rotation matrix R / translation matrix T), frame marker bits and other auxiliary information synchronized with the frame data are embedded in real time at a fixed position (nth row and mth column) (the auxiliary information is generally read from the monocular blue light speckle projector or IMU inertial navigation unit through a processor or MCU to replace the data value of the original image position), and then the intraoral depth image with embedded auxiliary information and the preprocessed white light RGB image are encoded and compressed to obtain the intraoral RGBD image.

[0122] In another embodiment, in step S200, the blue light speckle image is preprocessed by a histogram enhancement or a binarization operation.

[0123] In this embodiment, the blue light speckle image is preprocessed by histogram enhancement or binarization operation to extract the blue light speckle spots in the image (such as Fig.17 As shown in FIG. 1 ), the acquired blue light speckle image is highly similar to the speckle image blocks of the monocular blue light speckle reference image.

[0124] In another embodiment, in step S200, the white light RGB image is preprocessed by the following method: Gamma correction, histogram enhancement, HSI color correction, Y edge enhancement in YUV space, and UV chromaticity edge enhancement are sequentially performed on the white light RGB image.

[0125] In another embodiment, in step S300, the depth decoding of the pre-processed blue light speckle image comprises the following steps: S201: performing block matching disparity calculation on the preprocessed blue light speckle image and the blue light speckle reference image of known reference distance to obtain a disparity map corresponding to the preprocessed blue light speckle image; In this step, the preprocessed blue light speckle image and the monocular blue light speckle reference image at a known reference distance are block matched to calculate the disparity according to a certain size of speckle image blocks (the size of the speckle image block is m*n, where m and n are positive integers) (for specific calculation, see Figure 5 As shown in Figure 1, the specific strategy is to search for the best matching block with the highest similarity in the M*N (M and N are integers greater than m and n) search window of the monocular blue light speckle reference image with an input speckle image block of size m*n (the search process can refer to the traditional image block matching method) to obtain the disparity value corresponding to the best matching block, and then calculate the disparity map corresponding to the input blue light speckle image block by block.

[0126] S202: Calculate the depth value of the disparity map to obtain an intra-oral depth image.

[0127] In this step, the depth value of the disparity map is calculated according to the following formula:

[0128] in, is the known distance of the monocular blue light speckle reference image (i.e., the reference blue light speckle image mentioned above), The disparity value corresponding to the optimal matching block is obtained by performing block matching disparity calculation on the preprocessed blue light speckle image and the preprocessed monocular blue light speckle reference image of known distance (i.e., the reference blue light speckle image mentioned above). is the baseline distance between the macro RGB camera and the monocular blue light speckle projector, is the focal length of the macro RGB camera, is the pixel pitch of the macro RGB camera image sensor.

[0129] In another embodiment, in step S300, the depth decoding of the pre-processed blue light speckle image further includes the following steps: S303: Post-processing the intra-oral depth image to obtain a post-processed intra-oral depth image.

[0130] In this step, after obtaining the intraoral depth image, it is necessary to perform post-processing such as denoising, removing flying points, and filling holes to obtain a clean depth map.

[0131] In a specific embodiment, the present disclosure combines Figure 18-Figure 20 A comparison is made between the three-dimensional model of oral teeth based on monocular blue light speckle structured light and that obtained by traditional color stripe structured light proposed in the present invention.

[0132] Fig.18 It is a physical model of oral teeth provided by the present disclosure; Fig.19 The present invention provides a color stripe structured light for Fig.18 The three-dimensional model of oral teeth obtained by the model shown; Fig. 20 The present invention provides a monocular blue light speckle structured light for Fig.18 The three-dimensional model of oral teeth obtained by the model shown. By comparison, it can be found that Fig. 20 compared to Fig.19 The three-dimensional modeling details of oral teeth are richer and more accurate, such as better modeling of sharp protrusions, grooves, and nail holes.

[0133] In another embodiment, the present disclosure further provides an intraoral depth perception device, comprising: A projection module, used for projecting a blue light speckle coding pattern into the oral cavity to generate a blue light speckle image; A white light LED lighting module, used for irradiating LED white light into the oral cavity to obtain a white light RGB image; An acquisition module, used for acquiring the generated blue light speckle image and white light RGB image; A preprocessing module, used for preprocessing blue light speckle images and white light RGB images; A depth decoding module, used for performing depth decoding on the preprocessed blue light speckle image to obtain an intraoral depth image; The encoding and compression module is used to encode and compress the intraoral depth image and the preprocessed white light RGB image to obtain the intraoral RGBD image.

[0134] like Fig.15As shown, the projection module adopts a monocular blue light speckle projector, specifically including a blue light LD speckle projector and a blue light VCSEL speckle projector, wherein the blue light LD speckle projector emits blue light from a blue light LD (Laser Diode) light source, which is focused by a light aperture and a collimator, and then a speckle coding pattern is projected by a diffraction optical device DOE. The blue light LD light source is a single-point blue light LD laser emitting a wavelength of 400nm~480nm. The smaller the light-emitting aperture, the smaller the speckle spots projected by the diffraction optical device. The wavelength will drift with temperature changes, causing the projected macro speckle coding pattern to deform. The light aperture is a retractable spring, and its diameter can be used to control the size of the light-emitting aperture of the blue light LD light source. The collimator (Collimator), which focuses and collimates the single-point laser beam emitted by the blue light LD light source, is generally composed of a convex lens. At the same time, the monocular blue light speckle projector is also equipped with a temperature sensing module (thermistor NTC or temperature measurement chip). On the one hand, it can establish a one-to-one correspondence between temperature and blue light speckle coding pattern. The monocular blue light speckle depth decoding can find the corresponding monocular blue light speckle reference image through the temperature value for accurate depth decoding; on the other hand, it can detect the temperature to prevent the temperature from being too high.

[0135] The VCSEL speckle projector consists of a VCSEL coded light source, a collimator and a diffractive optical element DOE, and projects a speckle coded pattern in a spatially coded manner. Among them, the VCSEL coded light source includes a certain number (10 to 5000 points) of VCSEL (vertical cavity surface emitting laser) light points, and the arrangement of these light points is a random coded pattern, that is, the combination of light points within a certain range in the horizontal or vertical direction is not repeated, and all light points constitute the basic coded pattern. The collimator is used to collimate the laser beam emitted by the VCSEL coded light source into parallel light. The collimator and the VCSEL light source and the DOE diffractive optical device can be fixed, or they can be moved up and down within a certain range facing the optical axis to achieve the adjustment of the size of the scattered spots and the degree of focus in the projected spatiotemporal coding pattern. The diffractive optical device DOE replicates the basic coding pattern in an m*n manner, such as 3*3, 5*3, 7*5, etc., to form a speckle coding pattern with more spots, and ensures that the brightness and contrast of the spots in different replicated blocks remain balanced and consistent.

[0136] It should be noted that the speckle coding group is composed of a certain number of random speckle spots to form a single speckle coding pattern, such as Figure 3 As shown in FIG. 1 , there is symmetry between the speckles around the center of the pattern, that is, the speckle pattern in the upper left corner is symmetrical with the speckle pattern in the lower right corner, and the speckle pattern in the lower left corner is symmetrical with the speckle pattern in the upper right corner. Figure 3The scattered spots A and A' are shown to be symmetrical about the center point.

[0137] The white light LED lighting module is used as the illumination and fill light for RGB images. It generally uses 3 to 20 LED particles, which are arranged in a circle around the monocular blue light speckle projector and the macro RGB camera. Using white light LED as the light source can make the RGB image color positive under white light irradiation, and can better restore the original color of oral teeth.

[0138] Furthermore, the acquisition module in this embodiment includes a macro RGB camera with a depth of field of 2 to 10 cm in the macro range; the macro RGB camera generally acquires blue light speckle images or white light RGB images frame by frame in a time division multiplexing manner, and the images are output to the decoding module and the white light RGB image preprocessing module in RAW data format respectively; the macro RGB camera can choose to embed a blue light filter to enhance the blue light speckle image effect and make the blue light speckle spots clearer.

[0139] In another embodiment, the device also includes a synchronization trigger module, which generates a synchronization trigger Strobe signal based on the field synchronization signal input by the macro RGB camera, and controls the monocular blue light speckle projector and the white light LED lighting module in a PWM manner, including the selection of the light-emitting module (speckle projector or LED lighting module), frame switching mode, illumination duration and intensity.

[0140] In another embodiment, the preprocessing module includes: A first preprocessing submodule, used for preprocessing the collected blue light speckle image to obtain a preprocessed blue light speckle image; The second preprocessing submodule is used to preprocess the collected white-light RGB image to obtain a preprocessed white-light RGB image.

[0141] In another embodiment, the depth decoding module includes: A matching disparity calculation submodule is used to perform block matching disparity calculation on the preprocessed blue light speckle image and a blue light speckle reference image of a known reference distance, so as to obtain a disparity map corresponding to the preprocessed blue light speckle image; The depth calculation submodule is used to calculate the depth value of the disparity map to obtain the intraoral depth image.

[0142] In another embodiment, the depth decoding module further includes: The post-processing submodule is used to post-process the intra-oral depth image to obtain a post-processed intra-oral depth image.

[0143] In summary, the present disclosure provides a method for intraoral depth perception based on monocular blue speckle structured light. The method performs depth perception of the oral cavity based on monocular blue speckle structured light. Compared with the traditional color stripe structured light coding projection, the method has the characteristics of fast 3D imaging speed and high point cloud resolution. The beneficial effects are as follows: 1. Using projected blue light speckle coding pattern as the projection coding light source, under the condition of the same light source power, the emitted blue light speckle coding pattern is more suitable for the three-dimensional measurement of highly reflective, semi-transparent tooth surfaces or braces than the traditional color stripe structured light coding projection.

[0144] 2. Through time division multiplexing, the RGBD data inside the mouth can be obtained based on the same macro RGB camera.

[0145] 3. It adopts the working mode of surface structured light speckle coding projection, which has the characteristics of resistance to high reflection, resistance to the influence of semi-transparent materials, fast 3D imaging speed, high point cloud resolution and low power consumption.

[0146] In addition, the present disclosure also discloses a method for three-dimensional modeling of oral teeth based on an intraoral depth camera, which involves the following embodiments: In one embodiment, if Fig.21 As shown, the present disclosure provides a method for three-dimensional modeling of oral teeth based on an intraoral depth camera, comprising the following steps: S100: Obtaining a Depth image of oral teeth based on monocular blue light speckle structured light, and obtaining an RGB image of oral teeth based on LED white light, and preprocessing the RGB image and the Depth image respectively to obtain a preprocessed RGB image and a Depth image; S200: Generate a three-dimensional point cloud based on the pre-processed Depth image calculation; S300: registering and tracking the three-dimensional point cloud to estimate the camera pose; S400: Fusion of the registered 3D point cloud; S500: Rendering and reconstructing the fused three-dimensional point cloud based on the preprocessed RGB image to obtain a three-dimensional model of oral teeth.

[0147] The above embodiments constitute a complete technical solution for the three-dimensional modeling method of oral teeth disclosed in the present invention. In this embodiment, the intraoral depth camera acquires the RGBD data of oral teeth frame by frame based on the depth perception method of monocular blue light speckle structured light, that is, the monocular blue light speckle projector projects blue light speckle and the macro RGB camera collects the blue light speckle image to perform monocular structured light depth decoding, and obtains the Depth map information of the oral teeth; the RGB image information of the oral teeth is obtained by white light LED illumination. The RGBD data is calculated frame by frame based on the time division multiplexing method of the same macro RGB camera. One frame of RGB image corresponds to one frame of Depth map, and there is a one-to-one registration relationship between the RGB image and the Depth map.

[0148] This embodiment uses RGBD data output by a monocular blue light speckle structured light depth camera for modeling. Compared with intraoral depth cameras based on color stripe structured light and confocal microscopy, it is more stable and reliable in 3D imaging of highly reflective and semi-transparent tooth surfaces, has a smaller size, and consumes less power.

[0149] In another embodiment, in step S100, preprocessing the RGB image includes performing denoising filtering, enhancement (gamma correction, YUV space enhancement, histogram enhancement, HIS space tone enhancement) operations on the RGB image, tone correction and clarity enhancement. Preprocessing the Depth image includes performing depth filtering on the Depth depth map, removing flying points and discrete points in a connected domain manner, and performing hole compensation.

[0150] In another embodiment, in step S200, the three-dimensional point cloud is calculated and generated by the following method: Z = depth X = (x-cx)*Z / fx Y = (y-cy)*Z / fy Among them, depth is the depth value, (x, y) is the pixel coordinate of the depth map, cx, cy, fx, fy are the camera intrinsic parameters, and (X, Y, Z) are the coordinates of the calculated three-dimensional point cloud.

[0151] In another embodiment, in step S300, the three-dimensional point cloud is registered and tracked by using an ICP algorithm.

[0152] In this embodiment, in the registration and tracking stage, on the one hand, the camera pose is estimated based on the point cloud (input frame point cloud) generated in step S200, and the registration algorithm used for pose estimation can be an ICP registration algorithm based on depth data or an ICP registration algorithm based on RGBD data.

[0153] The key to the point cloud registration and tracking stage is to estimate the camera pose of the input frame. Generally, there are two methods for camera pose estimation: Depth-based camera pose estimation (DepthTracker) and RGBD-based camera pose estimation (ExtendedTracker).

[0154] Among them, the Depth-based camera pose estimation method uses Fig. 22 The point-to-plane ICP algorithm shown in the figure is used for pose estimation. This method has fast convergence speed and good robustness. When using the point-to-plane ICP algorithm for camera pose estimation, it is necessary to calculate the normal vector of each point in the light transmission result. The method used is to use the cross product between 3D points in adjacent pixels to calculate the normal vector. Finally, after performing light transmission calculation on each pixel, the surface estimation can be completed. The results are stored in the point cloud map and normal vector map of the global 3D model and used to estimate the camera pose of the next frame.

[0155] This method first calculates the point cloud image of the previous frame based on the light transmission in the rendering stage. and normal vector graph And according to the current depth map Calculated point cloud and normal vector graph , as shown in formulas (1-1) and (1-2) respectively. Then the L2 norm is used to define the error function as shown in formula (1-3), and finally the final camera pose is obtained by iterating the Gauss-Newton method.

[0156] (1-1) (1-2) (1-3) In formula (1-1), is the camera internal parameter; is the pixel coordinate; yes The coordinates of are homogenized. In formula (1-2): .

[0157] In formula (1-3), is the transfer matrix from camera to world coordinates, including rotation matrix and translation vector; It is a screening function that will exclude points with too large errors.

[0158] The camera pose estimation method based on RGBD is more accurate than the camera pose estimation method that simply uses the ICP algorithm. First, this method uses a more robust Huber loss when constructing the error function between the target point cloud and the source point cloud; second, when minimizing the Huber loss of the target point cloud and the source point cloud, each point is weighted according to its depth data to minimize the noise interference caused by the increase in distance (the farther the distance, the smaller the weight). This method adds the pixel grayscale difference between the current frame RGB image and the previous frame RGB image to the error optimization. Specifically, the grayscale value of the RGB image is first obtained through the current frame, and then the pixel coordinates of the current frame depth image are back-projected into the world coordinate system. According to the camera pose, the world coordinates of the current frame are projected into the camera coordinate system of the previous frame. Finally, the camera coordinates are projected into the pixel coordinates of RGB and the grayscale value of the previous frame is obtained through bilinear interpolation. Based on this, the pixel difference between the two RGB frames is calculated, and based on the weight of the depth value of the current frame, the Tukey loss function shown in formula (1-4) is used to calculate the gradient vector and Hessian matrix of the grayscale image. Based on the gradient vector and Hessian matrix, the iterative step size of the Levenberg-Marquardt algorithm is updated to minimize the error and estimate the pose.

[0159] (1-4) The results of the ICP optimization phase of the above two camera pose estimations (including the percentage of outlier pixels, the determinant of the Hessian matrix and the residual sum) will be linearly classified by the Support Vector Machine (SVM) in the subsequent steps to determine the tracking effect, which is more robust and stable than simple threshold comparison.

[0160] It should be noted that the present disclosure is not limited to the two disclosed camera pose estimation methods in the point cloud registration and tracking stage. Based on the contents disclosed in the present disclosure, technicians in this field may obtain R and T through other similar camera pose estimation methods, which also falls within the protection scope of the present invention.

[0161] In another embodiment, in step S300, after completing the registration and tracking of the three-dimensional point cloud, it is necessary to classify the registration and tracking effects and determine whether the tracking is successful based on the classification results.

[0162] In this embodiment, before the three-dimensional point cloud registration, the number of input source point clouds and target point clouds, the size of the overlapping area between the source point cloud and the target point cloud obtained after registration, the point cloud matching error, and the point cloud registration effect are input as training data into the SVM linear classifier for the reconstruction scene of the oral teeth. The trained SVM linear classifier is used as the basis for judging the point cloud registration and tracking effect, thereby achieving the classification of the point cloud ICP registration effect of the input frame. On the other hand, while classifying the point cloud registration and tracking effect, this embodiment also maintains a key frame point cloud encoding list for helping to perform repositioning operations. For each frame of input point cloud, the input frame point cloud will be fern-encoded using the Randomized Ferns algorithm. The encoding will represent the input frame point cloud for constructing a key frame point cloud list or repositioning operation.

[0163] After obtaining the trained SVM linear classifier, the ICP registration and tracking effect of the input frame point cloud can be classified. The classification results are divided into three types: good, medium, and bad.

[0164] In another embodiment, after completing the classification of the registration and tracking effects, it is necessary to determine whether the three-dimensional point cloud registration and tracking is successful based on the classification results and execute different strategies.

[0165] For example, in this embodiment, for input frame point clouds with classification results of good and medium, tracking is considered successful, the tracking result and the input frame point cloud are fused, and step S301 is executed simultaneously to determine whether the input frame point cloud can be updated into the key frame point cloud list as a key frame point cloud, that is, in step S302, the fern Ferns code corresponding to the input frame point cloud is updated in the key frame point cloud list; for input frame point clouds with classification results of bad, tracking is considered failed, and step S303 is executed to perform a repositioning operation using the information stored in the current key frame point cloud list.

[0166] Specifically, step S301 to step S303 include: S301: Input frame point cloud determination. Use the Randomized Ferns algorithm to perform Ferns encoding on the input frame point cloud, calculate the Hamming distance between the encoding of the input frame point cloud and the key frame point cloud encoding stored in the key frame point cloud list in turn, and when the distance is greater than or equal to a preset threshold value T, the input frame point cloud is used as a new key frame point cloud, and the input frame ID, Ferns encoding and posture information (rotation matrix R, translation matrix T) are stored in the key frame point cloud encoding list maintained by the present disclosure to update the key frame point cloud list; if the distance is less than the preset threshold value T, the input frame point cloud is not used as a key frame point cloud, so the key frame point cloud does not need to be updated; S302: Update the key frame point cloud list. According to step S301, the input frame point cloud information (ID, fern code and pose information R / T) that meets the conditions is updated as the key frame point cloud to the key frame point cloud list; S303: Repositioning the key frame point cloud list. Use the Randomized Ferns algorithm to perform Ferns encoding on the input frame point cloud that failed to be tracked, calculate the Hamming distance between the input frame point cloud encoding and the key frame point cloud encoding stored in the key frame point cloud list, obtain the key frame point cloud information (including ID, Ferns encoding and posture information R' / T') encoding ID with the minimum Hamming distance, and extract the posture information (R' / T') corresponding to the ID.

[0167] In this step, the Randomized Ferns algorithm performs efficient binary encoding (Ferns encoding) on ​​the RGBD image and uses the Hamming distance as the image similarity metric to determine whether the current input frame point cloud is a key frame point cloud.

[0168] The input frame RGBD image is encoded in the following way. An image includes block encoding, each block has (n=4, RGB and depth) Ferns, each Fern is compared Channel Pixels The pixel value at and the given threshold The Fern binary code stream is determined by the size relationship, as shown in formulas (1-5) to (1-7). and The Hamming distance is used to define the similarity between them, as shown in formula (1-8).

[0169] (1-5) (1-6) (1-7) (1-8) In formula (1-8): It means that formula (1-6) is expressed Apply to images If the binary code streams corresponding to the two blocks are equal, then , otherwise it is equal to 0; Indicates that all The result of adding two binary code streams.

[0170] The key frame point cloud list described above uses a list format to store the frame ID and pose information (R / T) of the key frame point cloud according to the block encoding method. , first calculate each block code , according to its block code, index the frame with the same block code in the above list , the similarity between the two Add 1. For each input frame point cloud, the value of formula (1-9) and the given threshold T can be used to determine whether the frame is a key frame. If it is a key frame, its frame ID and pose information are stored in the above list.

[0171] (1-9) K in formula (1-9) I' The value represents the similarity information between the input frame point cloud and the stored key frame point cloud, K I' The smaller the value, the greater the overlap ratio of the same information between the input frame and the stored key frame, indicating that the camera posture information of the same or similar position view as the input frame has been stored in the key frame list. On the contrary, K I' The larger the value, the more likely the input frame's pose is new camera position information that is not stored in the keyframe list.

[0172] In a specific embodiment, a specific implementation method of relocating a key frame point cloud list includes: sequentially calculating the Hamming distance between the Ferns code of the current input frame and the Ferns code of each key frame in the stored key frame list, the distance value represents the similarity between the current input frame and each key frame in the key frame list maintained by the system, the smaller the Hamming distance value, the higher the similarity between the current input frame and the key frame, which means that the posture of the current input frame is more similar to the posture of the key frame, and taking out the first N key frames with the highest similarity to the current input frame from the key frame list maintained by the system according to the size of the Hamming distance value. D and posture information, sort the first N key frame information taken out according to the similarity with the current input frame, and perform point cloud registration with the N key frames in turn, and classify the registration results through the pre-trained SVM linear classifier. If the classification result is good or medium, it means that the current input frame is successfully registered with the key frame, and the posture information of the current input frame can be determined by registration with the key frame, the key frame sequence is successfully relocated, and the point cloud fusion process continues; if the registration of the current input frame with the N key frames fails, the key frame sequence relocation fails, the current input frame is discarded, and the new input frame is read and processed.

[0173] In another embodiment, in step S400, the registered three-dimensional point clouds are fused using a truncated signed distance field algorithm.

[0174] In this embodiment, the Truncated Signed Distance Function (TSDF) algorithm is used to integrate the input frame point cloud information (RGBD data, D represents point cloud ("Point Cloud" contains depth information. Point cloud is a set of points defined in three-dimensional space. These points usually contain XYZ coordinate information, where X and Y define the position of the point in the horizontal and vertical directions, and the Z axis defines the distance of the point from the observer or sensor, that is, depth information), RGB represents the RGB pixel value corresponding to the point cloud) that is successfully tracked into the oral tooth local point cloud corresponding to the oral tooth local three-dimensional model. The fused new oral tooth local point cloud will be fused with the new input frame point cloud that is determined to be successfully tracked for the next time.

[0175] In a specific embodiment, the point cloud fusion stage integrates the depth information of the current frame into the reconstructed local 3D model based on the pose information obtained in the point cloud registration and tracking stage, and uses the TSDF-based crystal grid model to represent the fused 3D model. TSDF only considers the SDF value within the neighborhood of the surface, that is, the point-to-surface value described by the SDF is normalized to -1~1 using the maximum value (max truncation) in the neighborhood, and the place where 0 is is the surface. The reconstructed space is divided into voxels, each voxel corresponds to a 3D point in the space, and each voxel records two values, using Indicates that ,in Represents the distance from the voxel to the nearest surface (where the TSDF value is 0), represents the weight, which will be used when fusing the next frame of image. Next, the original depth map needs to be TSDF value converted to the world coordinate system, for a known pose Depth map , which is in the global frame Voxel of The calculations are shown in formulas (1-10) to (1-13).

[0176] (1-10) (1-11) (1-12) (1-13) Where: is the nearest neighbor search function, so that the pixel coordinates of the voxel coordinate projection is an integer, then according to Get the corresponding depth, divided by the coefficient Convert the camera to voxel distance to depth and finally pass Normalize. Fig.23 As shown, it is particularly important that the normalization function converts the voxels corresponding to the surface to the sensor direction that exceed μ The value is truncated to 1, and it is considered that there is no object in the space beyond this range; the voxels corresponding to the surface exceeding the distance μ along the ray direction are Value truncated to invalid.

[0177] In getting the Frame depth map After that, it needs to be fused with the previously reconstructed model to update the TSDF value. The TSDF value of the frame depth map fusion is , then it can be updated according to formulas (1-14) and (1-15).

[0178] (1-14) (1-15) As can be seen from formula (1-11), the weight is set to a constant of 1, and a simple average is performed on the update of TSDF, but the actual effect is still good. At the same time, the fusion stage is not called for all voxels in a fixed volume, but only for voxel blocks marked as visible, which can greatly speed up the fusion speed.

[0179] In another embodiment, in step S500, the fused three-dimensional point cloud is rendered and reconstructed by using a light transmission method.

[0180] In this embodiment, the rendering stage is to determine the surface information of the fused oral tooth local point cloud from the TSDF value, and the method adopted is ray casting to reconstruct the oral tooth local three-dimensional model after the point cloud is fused.

[0181] In a specific embodiment, in the rendering and reconstruction stage, it is necessary to determine the surface information of the three-dimensional object from the TSDF value. The method used is the ray transmission method, that is, for each pixel in the image, a ray is projected according to the camera intrinsic parameter. When the ray intersects with the surface of the object, it must pass through two adjacent voxels with a positive and a negative TSDF value (because the TSDF value at the intersection of the ray and the surface should be 0), such as Fig.24As shown. It can be seen that the TSDF value needs to be checked at each voxel along the ray until a zero crossing point is found (TSDF goes from positive to negative), but this will be very computationally intensive, so in practice different step sizes are used for rendering according to different situations.

[0182] Fig.25 is a state machine flow chart of different step length rendering provided by the present disclosure, such as Fig.25 As shown, first check whether the voxel block is allocated. If not, read the next one with the voxel block size (8 voxels) as the step size. If it has been allocated, check whether its TSDF value is within the truncation band. If it is outside the truncation band, use the truncation band size as the step size. If it is within the truncation band, determine whether it is still in front of the surface (whether the TSDF value is positive). If the TSDF is negative, it indicates that it is already behind the surface, and the iteration stops; if it is in front, see if it is approaching the surface (the TSDF value becomes smaller) until the TSDF value is negative, which indicates that a zero crossing point has been found. The exact position of the surface can be found through trilinear interpolation calculation. Another way to improve the efficiency of light transmission is to choose a suitable search range, that is, you can try to render the voxel block through forward projection to obtain the maximum and minimum range of each pixel ray.

[0183] In another embodiment, after the rendering and reconstruction is completed, it is necessary to determine whether the scanning is completed. If it is completed, the current oral teeth local three-dimensional model generated by step S500 is output as the final oral teeth true color three-dimensional model completed by reconstruction; if it is not completed, return to step S100 and re-execute the processing flow of oral teeth three-dimensional modeling.

[0184] As mentioned earlier, Fig.18 It is a physical model of oral teeth provided by the present disclosure; Fig.19 The three-dimensional model of oral teeth obtained based on color stripe structured light provided by the present disclosure; Fig. 20 The present invention provides a three-dimensional model of oral teeth obtained based on monocular blue light speckle structured light.

[0185] By comparing, we can find that Fig. 20 compared to Fig.19 The three-dimensional modeling details of oral teeth are richer and more accurate, such as better modeling of sharp protrusions, grooves, and nail holes.

[0186] In another embodiment, the present disclosure further provides an oral teeth three-dimensional modeling device based on an intraoral depth camera, comprising: An acquisition module is used to acquire an RGB image and a Depth image of oral teeth, and preprocess the RGB image and the Depth image respectively to obtain a preprocessed RGB image and a Depth image; A calculation module is used to calculate and generate a three-dimensional point cloud based on the pre-processed Depth image; The registration module is used to register and track the 3D point cloud to estimate the camera pose; Fusion module, used to fuse the registered 3D point clouds; The 3D reconstruction module is used to render and reconstruct the fused 3D point cloud based on the preprocessed RGB image to obtain a 3D model of oral teeth.

[0187] In summary, the present disclosure provides a method for three-dimensional modeling of oral teeth based on an intraoral depth camera. The method collects RGBD data based on an intraoral monocular blue light speckle structured light depth camera for three-dimensional modeling of oral teeth, and can generate a high-precision true-color three-dimensional model of oral teeth in real time. The beneficial effects are: 1. The present invention adopts RGBD data modeling output by an intraoral monocular blue light speckle structured light depth camera. Compared with intraoral depth cameras based on color stripe structured light models and confocal microscopy, the present invention is more stable and reliable in 3D imaging of highly reflective and semi-transparent tooth surfaces, has a smaller size and lower power consumption.

[0188] 2. The present invention only needs one macro RGB camera to obtain the depth information and RGB information of oral teeth at the same time through time division multiplexing, and realizes a one-to-one relationship between the pixels of the two, thus solving the problem of RGBD registration and directly generating a true color 3D model of oral teeth after modeling.

[0189] 3. The present invention has the characteristics of being resistant to ambient light interference, the influence of high-reflective or semi-transparent tooth surface reflection, fast scanning frame rate, short time for generating oral tooth 3D model, and supporting re-positioning and supplementary scanning.

[0190] In another embodiment, the present disclosure further provides an electronic device, characterized in that it includes: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, any of the above methods is implemented.

[0191] In another embodiment, the present disclosure further provides a computer medium, including: Memory, for storing computer instructions; A processor is used to execute computer instructions to implement any of the above methods.

[0192] Finally, it should be noted that, under the guidance of this specification and without departing from the scope of protection of the claims of this disclosure, ordinary technicians in this field can also make many kinds of changes in form, which are all protected by this disclosure.

Claims

1. An intraoral three-dimensional imaging device, the device include: Monocular blue light speckle structured light projector, white light LED lighting module, macro RGB camera, depth decoding module and oral scanning head, among which, The monocular blue light speckle structured light projector is used to emit blue light speckle coded patterns. White light LED lighting module is used to emit white light, The monocular blue light speckle structured light projector and the white light LED lighting module are projected alternately, and then irradiated onto the oral teeth after being reflected by the 45° mirror of the oral scanning head. The irradiated oral tooth image is then collected by the macro RGB camera, and the RGBD data is output after the monocular structured light depth decoding of the depth decoding module.

2. The device according to claim 1, in, Preferably, the monocular blue speckle structured light projector and the macro RGB camera are on the same baseline and their optical axes are parallel.

3. The device according to claim 1, in, The monocular blue light speckle structured light projector is provided with a temperature sensing module, which can establish a one-to-one correspondence between temperature and the blue light speckle coding pattern, and detect the temperature to prevent the temperature from being too high.

4. The device according to claim 1, in, The macro RGB camera is used to collect blue light speckle coded images and white light RGB images irradiated on oral teeth.

5. The device according to claim 1, in, The depth decoding module includes a synchronization trigger submodule, a monocular blue light speckle structured light depth calculation submodule, an RGB image preprocessing submodule, and a data embedding and packaging output submodule.

6. The device according to claim 5, in, The synchronous trigger submodule generates a synchronous trigger Strobe signal according to the field synchronization signal input by the macro RGB camera, and controls the monocular blue light speckle structured light projector and the white light LED lighting module in a PWM manner.

7. The device according to claim 5, in, The monocular blue light speckle structured light depth calculation submodule obtains the parallax by block matching of the speckle blocks, and then calculates the depth value of each pixel in the blue light speckle image by using the monocular structured light depth calculation formula.

8. The device according to claim 5, in, The RGB image preprocessing submodule performs denoising and enhancement on the input blue light speckle coded image to extract blue light speckle spots; and performs enhancement processing on the input white light RGB image at the same time.

9. The device according to claim 5, in, The data embedding and packaging output submodule jointly packages and outputs the RGB and Depth data of the embedded auxiliary information.

10. The device according to claim 1, in, The blue light speckle coding pattern is a single speckle coding pattern composed of a certain number of random scattered spots, and the scattered spots are symmetrical around the center of the pattern.

Images