Method and apparatus for multi-modal soft tissue diagnosis

Abstract

The invention relates to a method and device for multi-modal imaging of skin and mucosa lesions, characterized in that at least two imaging modalities are used, one of them being a 3D scan of the lesion, the other providing information about the distance and angle between the scanning device and the dermis or mucosa, and at least the second modality is mapped on the 3D data.

Description

[0001] The applicant hereby provides, by reference under PCT Rule 4.18, that priority application EP20176399 is incorporated herein by reference in its entirety, including the description, claims and drawings. Technical Field

[0002] This invention relates to methods and apparatus for imaging skin and mucous membrane lesions. Background Technology

[0003] Traditional soft tissue diagnosis

[0004] Diagnosis begins with the patient's medical history. Data about the patient prior to treatment is collected, such as smoking, alcohol abuse, diabetes, or history of injury. Furthermore, it is important to assess the lesion's occurrence and progression over time (often more so the time of the patient's examination). In cases of acute injuries with typical healing times, this timeframe can be several days, or in cases of chronic soft tissue changes such as lichenification, observation may be required for years. Traditional oral soft tissue diagnosis relies on visual assessment of the lesion, combined with other information such as tactile information or the removability of the whitish discoloration. Several factors are important for visual assessment: the location and size of the lesion, the color of the lesion (reddish or whitish discoloration), the structure and uniformity of the discoloration (spots, networks, etc.). Physicians typically compare the actual lesion to photographs in oral medicine textbooks or other cases observed during known diagnostic practice. Additionally, the congruence of the mucosa is assessed by palpation and its relationship to the underlying bone (whether the lesion can be dislocated relative to the bone under slight pressure or whether the lesion is fixed to the bone or underlying structures such as muscle). Further testing with mechanical abrasion to remove the whitish discoloration helps differentiate between candidiasis and leukoplakia. Bone involvement may require additional X-ray diagnosis, such as in cases of tumors / swelling. The gold standard remains histological examination of tissue samples taken from biopsy materials.

[0005] Extended Diagnostics

[0006] In addition, for “routine diagnosis,” some dentists use blue / UV light to excite tissue autofluorescence and diagnose fluorescent images (e.g., Vizilite, VEL Scope, or similar) or to use Toluidin blue staining. For oral autofluorescence diagnosis, the light source is used to excite endogenous fluorophores such as nicotinamide adenine dinucleotide (NADH) and flavin adenine dinucleotide (FAD). For excitation, wavelengths in the UV / blue range (375 to 440 nm) are used. Emission is observed in the green wavelength range. Healthy mucosa emits pale green autofluorescence when observed with a narrow-band filter that suppresses the excitation wavelength. Abnormal tissues show less fluorescence and appear darker compared to surrounding healthy tissue (see [link to relevant documentation]). Figure 1(1) in the literature questions the ability to distinguish between developmental abnormalities and benign inflammatory lesions. Benign tissue inflammation often shows increased blood supply to the lesion. Increased hemoglobin content may absorb both excitation and emission light and cause a loss of intensity in fluorescence images, which may be mistakenly attributed to tumor formation. Vizilite uses chemical reagents to generate excitation light. To activate it, the capsule is bent to break the glass vial, causing the chemical products to react with each other and produce bluish-white light with wavelengths of 430–580 nm, which lasts for about 10 minutes. VELscope uses blue light with wavelengths between 400 and 460 nm for excitation. Fields of interest are observed via an optical imaging system (see [link]). Figure 1 A meta-analysis of 20 studies on autofluorescence methods for detecting oral developmental abnormalities (see Reference 1) revealed significant variations in sensitivity and specificity among different publications.

[0007] Vizilight: Comparing different studies, the sensitivity and specificity of Vizilight varied between 77%–100% (Sens) and 0%–28% (Spec). For the VELscope system, sensitivity was reported to be 22%–100% and specificity to be 16%–100%. As an example, the sensitivity and specificity of toluidine blue were determined to be 86.36% and 76.9%, respectively. Another non-optical method is the Oral CDx system. Surface cells are collected using a small brush and analyzed in the laboratory. Sensitivity varied between 71.4%–100% and specificity between 32%–100% in different studies.

[0008] In conclusion, autofluorescence alone appears insufficient as a diagnostic aid, especially considering the low prevalence of malignancies and variations in user experience, but it can be an additional diagnostic tool in combination with other methods. Other methods, such as staining or brush biopsy, show moderate to high variability in sensitivity and specificity (staining). Therefore, improvements in diagnostic capability are expected.

[0009] Neural networks for oral disease detection

[0010] Artificial intelligence is becoming increasingly successful as a diagnostic support tool for classifying dermal lesions and X-ray images. For example, AI networks have achieved board-certified dermatologist-level accuracy when trained with >100k images across 700 disease categories (see Reference 2). This resulted in approximately 150 images per category. A review article on the use of deep learning algorithms in dentistry reviewed 25 articles. 2D X-rays, CBCT, QLF, and OCT were used as diagnostic modalities. The review concluded that typical dataset sizes tend to increase from approximately 100 datasets per category up to 1000 datasets. 1000 datasets were reported to achieve approximately 98% accuracy, and more than 4000 datasets were needed to obtain greater than 99% accuracy. Only one of the reviewed articles addressed gingivitis detection using QLF and classification via CNN. No articles addressed intraoral disease classification (Reference 3). Beyond purely image-related classification, other authors reported on the use of CNNs, including contextual factors in oral cancer detection. When 12 out of 35 relevant factors were selected, adding risk factors and socioeconomic factors to clinical symptoms and medical records achieved 99% classification accuracy (Reference 4). In conclusion, neural networks can contribute to improved medical and dental diagnosis if databases with sufficient sample size, features, and quality are available.

[0011] Reference 1: Efficacy of light-based detection systems for early detection of oral cancer and oral potentially malignant disorders: Systematic review, Ravleen Nagi, Yashoda-Bhoomi Reddy-Kantharaj, Nagaraju Rakesh, Sujatha Janardhan-Reddy, Shashikant Sahu Med Oral Patol Oral Cir Bucal. 2016 Jul 1;21(4):e447-55.

[0012] Reference 2: Dermatologist-level classification of skin cancer with deep neural networks, Esteva A. et al. Nature 542 115-118 (2017).

[0013] Reference 3: An overview of deep learning in the field of dentistry, Jae-Joon Hwang 1, Yun-Hoa Jung 1, Bong-Hae Cho 1, Min-SukHeo 2, Imaging Science in Dentistry 2019; 49: 1-7.

[0014] Reference 4: Usage of Probabilistic and General Regression Neural Network for Early Detection and Prevention of Oral Cancer, Neha Sharma, Hari Om, The Scientific World Journal Vol. 2015, Article ID 234191, http: / / dx.doi.org / 10.1155 / 2015 / 234191.

[0015] Reference 5: UV-angeregte Autofluoreszenz: Spektroskopische undfluoreszenzmikroskopische Untersuchungenzur Tumorselektivität endogenerGewebefarbstoffe (UV-excited autofluorescence: spectral and fluorescence microscopy studies of the selectivity of endogenous tissue dyes to tumors), Alexander Hohla Dissertation, LMU, München, 2003. Summary of the Invention

[0016] In research using neural networks, the lack of consistency in images of regions of interest is a concern and often requires manual image preprocessing. Images are typically taken from different distances (and therefore different magnifications), at different viewing angles, and under different lighting conditions. For multimodal diagnostics, achieving a good match (ideally pixel-to-pixel) between different diagnostic modalities is even more important.

[0017] This invention solves all these problems, avoids manual preprocessing, and reduces computation time for AI classification of skin and mucous membrane lesions.

[0018] The object of the present invention is to overcome at least some of the aforementioned problems. This object has been achieved by an apparatus defined according to one example and a method defined according to another example. Other examples relate to further embodiments and improvements. An apparatus that generates a precise 3D surface representation using, for example, confocal, preferably color confocal, time-of-flight, stereometry, or OCT techniques has the advantage of accurately knowing the size of the lesion. The distance between the scanned surface and the 3D imaging device is always precisely known, thus the precise size of the lesion can be calculated. It is also known that the angle of the 3D imaging device relative to the scanned surface is constant, and the illumination conditions are always identical because the illumination is integrated into the device. This is typically a well-defined illumination pattern for 3D measurements. Combining 3D measurements with spectrally resolved 2D images (e.g., 3-channel (RGB) or more) allows for matching of 2D and 3D data. What is unknown to date in the prior art is the use of 3D textures for diagnostic purposes of soft tissue / mucosal lesions, and the mapping of spectrally resolved 2D image data onto such 3D textures of lesions.

[0019] Most practical 3D scanning devices use video-like scanning techniques that combine many sequentially acquired individual 3D images and use 3D landmarks to overlay them in order to correctly overlay the individual images. This is easy for, for example, teeth in the mouth, but becomes more difficult if the surface of the capture area with individual images does not show sufficient 3D landmarks. For example, in extreme cases, flat surfaces or spheres cannot be scanned. In the absence of or with very few landmarks, spectrally resolved 2D data superimposed on the 3D data can support the correct registration of individual 3D images with each other.

[0020] This is helpful when scanning the dermis or mucous membrane in flat areas, because lesions show a different distribution of scattering and absorption coefficients than healthy tissue, which can result in, for example, a whitish staining pattern when the scattering coefficient increases, or a brownish staining pattern if the absorption coefficient increases (see...). Figure 2a Whitening and discoloration and see Figure 2b (It turns brownish).

[0021] 3D image resolution

[0022] For practical 3D scanning systems used, such as in dental applications, a resolution of 10-30 μm may actually have the same order of magnitude of error. This is lower than the resolution of microscopes used for diagnostic histology, but much better than in vivo visual examination. This allows for the calculation of the surface texture of lesions.

[0023] Wavelength selection for 3D image generation:

[0024] Since biological tissues typically exhibit lower illumination penetration depth in the blue or near-UV region (350nm-400nm), primarily due to the increased light scattering coefficient, these wavelengths can be used in combination with scanning methods that suppress volumetric scattering (e.g., confocal and OCT-based methods, which may be combined with depth-of-focus techniques) to produce sharp 3D images of surface texture.

[0025] At wavelengths longer than 840 nm, preferably longer than 980 nm, and most preferably in the range of 1300 nm to 1600 nm, the scattering coefficient is much lower, allowing 3D imaging to depths of tens of millimeters to hundreds of micrometers. This makes the acquisition and imaging of subsurface structures possible.

[0026] Providing at least two wavelengths, one in the 350 to 400 nm range and the other longer than 840 nm, allows for clear 3D surface scanning that can be combined with at least several hundred micrometers of depth structural information from dermal / mucosal lesions. In its simplest case, illumination is sequentially switched between different wavelengths, and the illumination source is coupled into the same optical path using a dichroic mirror. This variation will be applicable to wavelengths detectable by the same sensor (e.g., CMOS 350 nm to 1000 nm).

[0027] For some 3D measurement techniques, if the light source is small enough (e.g., an LED), it is possible to even use slightly different wavelengths of light. Tiny angular deviations will then cause a displacement of the illumination pattern on the sensor, but this can be corrected through calculations (displacement and distortion correction).

[0028] In cases where a second sensor is required, this will be the case when using illumination wavelengths exceeding 1000 nm; at least one beam splitter can be used to separate the optical paths of the different sensors (see [link to relevant documentation]). Figure 3 ).

[0029] Fluorescence imaging

[0030] A further extension of 3D imaging is its combination with fluorescence imaging. As described in the introduction, human tissues exhibit autofluorescence when excited with appropriate wavelengths. Skin / mucosal lesions display autofluorescence of varying intensities. This can be excitation within the UV / blue light range of FAD, NADH, and collagen, but can also be red excitation to excite porphyrins. Combined with 3D imaging, this allows fluorescence image data to be overlaid on 3D texture data.

[0031] The optical beam path can be conventional, with a blocking filter for the excitation light that uses the same light pattern as the UV / blue light wavelength used for 3D imaging, and a blocking filter for fluorescence detection introduced into the imaging path after separation from the illumination path. However, this would require a moving part (the filter) in the device. Depending on the design of the 3D scanner to be expanded, even stronger excitation light power may be required.

[0032] By discarding 3D information in fluorescence image data, the 3D optical path can remain unchanged. A UV-blocking filter can be introduced into the 2D optical path, typically used for 2D imaging in the visible spectrum, while still using the excitation optical path for 3D imaging illumination (see...). Figure 4 ).

[0033] However, the preferred solution is to use a separate excitation source placed on the side of the replaceable cover. A blocking filter can be integrated into the window of the cover. Then, the filter should not suppress the structured light used for 3D measurements. Figure 5 This is possible because many fluorophores in the human body, such as collagen, NADH, FAD, elastin, and keratin, can be excited below 350 nm, and 3D light patterns can be used in the 365 nm–405 nm range (Reference 5). Figure 6 ).

[0034] Another option is to place an excitation light blocking filter in front of the 2D sensor, leaving the 3D optical path unchanged. This filter does not affect imaging within the visible range typically used to produce “color 2D images” because fluorescence emission is also within the visible range (see [link to relevant documentation]). Figure 4 ).

[0035] This improved cover replaces the traditional cover; in any case, the cover is removable for disinfection.

[0036] As an example, but not limited to, the capabilities of intraoral 3D scanning devices such as Primescan or Omnicam (and other anti-volume scattering scanning devices) can be extended using the aforementioned techniques.

[0037] Alternatively, the excitation LEDs can be placed inside a cover, which is more or less an empty shell, and can be sterilized without reducing the LED lifespan; however, this would require further modifications to existing 3D scanning equipment.

[0038] Another advantage of using intraoral scanning equipment as the basis for detecting and classifying intraoral lesions is the form of the equipment, which allows access to all areas of the oral cavity, unlike equipment used to capture images of skin lesions.

[0039] Using the above techniques, the following preferred multimodal imaging options become possible:

[0040] - Use visible wavelength 2D images along with 3D information to calculate the magnification, distance, and angle of the lesion;

[0041] - A 2D image of visible wavelengths overlaid on a 3D texture of the lesion;

[0042] - Use visible wavelength 2D images + fluorescence images with 3D information for recalculating magnification, distance, and angle;

[0043] - Visible wavelength 2D image + fluorescence image overlaid on 3D texture of lesion;

[0044] - Visible wavelength 2D image + subsurface structure information overlaid on the 3D texture of the lesion;

[0045] - Visible wavelength 2D image + fluorescence image superimposed on 3D texture of lesion + subsurface structure information;

[0046] However, any other combination of different imaging modalities, 2D color images, 3D texture images, fluorescence images, and subsurface images with long wavelengths cannot be excluded (see [link to image processing]). Figure 7 ).

[0047] The absolute size and known imaging conditions provided by the combination of at least 3D measurements and 2D color images further enhance the ability to overlay (register) images of the same lesion taken at different times using best-fit algorithms to view even small deviations, which allows for monitoring of lesion progression over time.

[0048] Images captured with such a device can be processed on a processing unit such as a computer that is part of the device and presented to a physician for visual examination on a computer screen. Alternatively, they can be used to build a multimodal image database for training neural networks (via an external network through a cloud-based network training service, or if the internal network has sufficient computing power available), either alone or in combination with further "non-imaging" information as a result of palpation, the removability of the whitish layer, lesion history, and risk factors (smoking, alcohol, etc.). The screen can be a desktop or mobile device display with or without a touchscreen, or a wearable device such as a head-mounted display.

[0049] The device can be equipped with a trained network to provide diagnostic suggestions or recommendations, and if no conclusive diagnostic recommendation can be given, the patient will be referred to an oral specialist for further examination / biopsy (see [link]). Figure 8 ). Attached Figure Description

[0050] In the following description, other aspects and advantageous effects of the invention will be described in more detail by using exemplary embodiments and with reference to the accompanying drawings, wherein

[0051] Figure 1 : Showing a comparison between the photograph and the corresponding autofluorescence image;

[0052] Figure 2a : Shows a whitish pigmented lesion;

[0053] Figure 2b : This indicates a lesion with increased pigmentation;

[0054] Figure 3 This shows the core functional blocks of the 3D scanning device.

[0055] Figure 4 : Shows the function blocks of the 3D scanning device;

[0056] Figure 5 : Shows the front cover of the 3D scanning device;

[0057] Figure 6 : Showing the emission bands of different fluorophores;

[0058] Figure 7 : Indicates a combination of imaging modalities;

[0059] Figure 8 This shows a setup with an artificial neural network for diagnostic support.

[0060] The reference numerals shown in the accompanying drawings denote the elements listed below and will be referenced in the subsequent description of exemplary embodiments.

[0061] 1-1: Lesion

[0062] 3-1: 3D Scanning Optical Devices

[0063] 3-2: Dichroic mirror / beam splitter

[0064] 3-3: Sensors (e.g., CMOS)

[0065] 3-4: Sensors (e.g., InGaAs detectors)

[0066] 4-1: 3D Scanning Optical Devices

[0067] 4-2: Beam splitter

[0068] 4-3: Sensors (e.g., CMOS)

[0069] 4-4: Blocking Filter

[0070] 4-5: Sensors (e.g., CMOS)

[0071] 4-6: Front-end

[0072] 5-1: Front cover

[0073] 5-2: UV LED

[0074] 5-3: Imaging Window

[0075] 7-1: 2D Color Image

[0076] 7-2: Autofluorescence image

[0077] 7-3: 3D Texture Images

[0078] 7-4: Subsurface structure image

[0079] 8-1: Equipment

[0080] 8-2: 2D Images

[0081] 8-3: Database

[0082] 8-4: Brush live tissue examination

[0083] 8-5: X-ray image

[0084] 8-6: Palpation results.

[0085] Figure 1 A comparison of the photograph with the corresponding autofluorescence image is shown, in this case, using a "VELscope" device. The corresponding image clearly shows the good contrast between the lesion (1-1) and healthy tissue in the autofluorescence image.

[0086] Figure 2a The study showed white lesions primarily caused by thickening of the epidermis and thus a significantly increased scattering coefficient. Figure 2b The lesions were shown to have increased pigmentation, which resulted in an increased absorption coefficient.

[0087] Figure 3 The functional blocks of the 3D scanning device are shown. (3-1) is the 3D scanning optics, (3-2) is a dichroic mirror / beam splitter that separates the wavelength bands in the range of 300nm-800nm ​​that reach the CMOS sensor (3-3), while wavelengths longer than 1000nm are mirrored to the sensor (3-4), which can be an InGaAs detector that covers at least the wavelength range of 1000nm-1600nm.

[0088] Figure 4 Functional blocks of a 3D scanning device are shown. (4-6) is the front end, which deflects the image toward the beam splitter (4-2), thereby separating the 2D imaging path from the 3D imaging path. (4-1) is a 3D scanning optics device with a CMOS sensor (4-3) (as a 3D sensor), and (4-4) is a blocking filter that suppresses excitation light so that it does not reach the CMOS sensor (4-5). Alternatively, the CMOS sensor (4-3) can optionally be replaced by a CQD sensor (4-3) with extended sensitivity in the NIR range. The cutoff wavelength of the blocking filter is around 370nm-400nm. This allows spontaneously fluorescing emission light to pass through, and also allows visible wavelengths to pass through for use in color images. The CMOS sensor (4-5) is not limited to a traditional RGB 3-channel sensor, but can contain more channels with better spectral resolution, such as a mosaic-type CMOS sensor with multiple different filters combined with a lens array not shown in the image. This allows the differentiation of different fluorophores, such as Figure 6 As shown, this is because they have emission bands that have maximum values ​​at different wavelengths. Figure 3 The optical components (3-2), (3-3), and (3-4) in the text can be replaced Figure 4 The sensors (4-5) are arranged so that there is a 2D sensor (3-3) for the visible range and another 2D sensor (3-4) for NIR light.

[0089] An apparatus for multimodal imaging of skin and mucous membrane lesions includes: a scanning device (8-1) having an illumination source and sensors (3-3, 3-4, 4-3, 4-5); and at least one processing unit for calculating images based on raw data provided by the scanning device (8-1), adapted to use at least two imaging modalities, wherein a first imaging modality from the two imaging modalities generates 3D data for a 3D image (7-3; 7-4) in a 3D scan of the lesion, wherein the processing unit is adapted to additionally provide 3D information about the distance and angle between the scanning device (8-1) and the dermis or mucous membrane by using an illumination pattern, or stereometry, or time-of-flight, and to map at least one image (7-1; 7-2) generated by a second imaging modality onto the 3D image (7-3; 7-4) of the 3D scan based on the 3D information. The use of an illumination pattern, or stereometry, or time-of-flight is one of various techniques available to those skilled in the art.

[0090] Figure 5 The front cover (5-1) of a 3D scanning device is shown. This cover is typically removable from the rest of the scanning device for sterilization. A UV LED (5-2) is positioned parallel to the imaging window (5-3) to illuminate the region of interest and excite autofluorescence. Backscattered light passes through the imaging window (5-3), which may be covered by an interference filter if the excitation light blocking filter is not located elsewhere in the detection light path. The cover may contain optics including the UV LED, or it may be a shell with more or fewer empty housings covering the optics inside the cover. This avoids subjecting the UV LED to sterilization cycles.

[0091] Figure 6 The different emission bands and maximum values ​​of different fluorophores excited under these conditions at 308 nm are shown. The distinct peaks allow for the separation of different fluorophores. However, this is a normalized image. In reality, the emission intensity of collagen forms a high background signal that can dominate other fluorophores.

[0092] Figure 7 Different combinations of possible imaging modalities using the proposed device are shown, wherein 2D color images (7-1) (e.g., 2D spectrally resolved images), autofluorescence images (7-2), 3D texture images (7-3), and subsurface structure images (7-4) are obtained using longer wavelengths.

[0093] Figure 8 This demonstrates how an artificial neural network can be used to support the setup for diagnosing dermal / mucosal lesions using images captured by the proposed device (8-1). Figure 8 Only 2D images (8-2) are shown, which contain additional 3D information such as distance and angle, but are not limited to these images.Figure 7 All combinations shown or described herein are applicable. A database (8-3) is built from 2D images (8-2) for training the artificial neural network. To improve classification performance, in addition to image data, additional information can be added to the network as brush biopsy results (8-4), X-ray images (8-5), and palpation results (8-6). The images of the palpation results (8-6) are selected only as an example to show the bulge of the gingiva, which may be hard or soft. Of course, multiple cases with these data must be included in the database associated with the corresponding case images used for training.

[0094] Using this invention, due to the known accurate absolute dimensions and known imaging conditions, such as the angle and distance of the lesion surface relative to the imaging plane provided by a combination of at least 3D measurements and 2D color images, it is possible to overlay (register) images of the same lesion taken at different times to see even small deviations, which allows for monitoring the development of the lesion over time.

Claims

1. A device for multimodal imaging of skin and mucous membrane lesions, the device being characterized by comprising: A scanning device (8-1) comprising an illumination source, a beam splitter, and sensors (3-3, 3-4, 4-3, 4-5), wherein the sensors include a visible light 2D sensor and an infrared light 3D sensor, and the beam splitter is configured to specifically split the detected light into wavelengths suitable for the visible light 2D sensor and the infrared light 3D sensor; and at least one processing unit for calculating an image based on raw data provided by the scanning device (8-1), adapted to use at least two imaging modalities, wherein a first imaging modality from the at least two imaging modalities generates 3D data of a 3D image (7-3; 7-4) in a 3D scan of the lesion, wherein the processing unit is adapted to additionally provide 3D information about the distance and angle between the scanning device (8-1) and the dermis or mucosa, and, based on the 3D information, map at least one image (7-1; 7-2) generated by a second imaging modality onto the 3D image (7-3; 7-4) of the 3D scan. The first imaging mode or the second imaging mode is further used for fluorescence imaging using excitation light in the UV / blue range for fluorophore. The scanning device (8-1) includes a separate excitation light source (5-2) positioned on the side of a replaceable cover (5-1), and a blocking filter (4-4) is integrated into a window (5-3) of the replaceable cover (5-1), and the blocking filter (4-4) is designed not to suppress structured light used for 3D measurement. The cover (5-1) covers the fluorescent excitation LED (5-2) to illuminate the region of interest through the window (5-3), wherein the cover (5-1) is removable for sterilization, and the LED (5-2) remains on the rest of the scanning device (8-1) to avoid the LED (5-2) undergoing sterilization cycles.

2. The apparatus according to claim 1, characterized in that, The fluorophores include FAD, NADH, and collagen.

3. The apparatus according to claim 1, characterized in that, The processing unit is also adapted to calculate the precise size of the lesion by using 3D information about the distance and angle between the scanning device (8-1) and the lesion.

4. The apparatus according to claim 1, characterized in that, The processing unit is also adapted to calculate the 3D surface texture of the lesion using the 3D information from the 3D scan.

5. The apparatus according to claim 1, characterized in that, The second imaging modality generates at least one of a 2D image (7-1) or an autofluorescence image (7-2) using a sensor (4-5) as the image (7-1; 7-2), wherein the 2D image (7-1) is resolved using spectral analysis of 3 or more channels.

6. The apparatus according to claim 5, characterized in that, The spectral analysis of the 2D data of the 2D image (7-1) superimposed on the 3D data by the processing unit supports the correct registration of a single 3D image of the 3D data to form a complete 3D image (7-3; 7-4) of the region of interest, wherein the 3D image includes at least one of a 3D texture image (7-3) or a subsurface structure image (7-4).

7. The apparatus according to claim 1, characterized in that, The 3D data from the 3D scan is captured using techniques that suppress volumetric scattering, such as confocal imaging, OCT, or a combination of confocal scanning and depth-of-focus techniques.

8. The apparatus according to claim 1, characterized in that, The scanning device (8-1) is adapted to use a wavelength between 350 nm and 400 nm for the 3D scanning of the lesion by employing a corresponding sensor (4-3) for the 3D image (7-3; 7-4).

9. The apparatus according to claim 1, characterized in that, The scanning device (8-1) is adapted to perform 3D scanning of the subsurface using a wavelength longer than 840 nm by employing a corresponding sensor (4-3) for the subsurface structure image (7-4).

10. The apparatus according to claim 9, characterized in that, The wavelength is longer than 980nm.

11. The apparatus according to claim 10, characterized in that, The wavelength is in the range of 1300nm-1600nm.

12. The apparatus according to any one of claims 8-11, characterized in that, The scanning device (8-1) is adapted to use two wavelength ranges together for 3D scanning of the surface and 3D scanning of the subsurface.

13. The apparatus according to claim 12, characterized in that, The scanning device (8-1) is adapted to sequentially switch illumination between different wavelengths, wherein the illumination source is coupled to the same optical path using a dichroic mirror.

14. The apparatus according to claim 12, characterized in that, The optical paths used for illumination sources of different wavelengths are not entirely consistent.

15. The apparatus according to claim 12, characterized in that, In the scanning device (8-1), when illumination with a wavelength exceeding 1000 nm is used, the scanning device (8-1) includes at least one beam splitter (3-2) to separate the optical paths for different sensors (3-3; 3-4).

16. The apparatus according to claim 1, characterized in that, The scanning device (8-1) is adapted to excite porphyrins using light in the red wavelength range in a second imaging mode, and the processing unit is adapted to overlay a fluorescence image (7-2) onto the 3D data of the 3D scan.

17. The apparatus according to claim 1, characterized in that, The scanning device (8-1) includes a blocking filter (4-4) for using excitation light with the same illumination pattern having UV / blue wavelengths as that used for the 3D imaging, and is adapted to introduce the blocking filter (4-4) for fluorescence detection into the imaging path after separation from the illumination path.

18. The apparatus according to claim 17, characterized in that, The blocking filter (4-4) is located in the imaging path of the 2D image (7-1) of the second imaging mode.

19. The apparatus according to claim 1, characterized in that, The excitation wavelength is below 350nm, and the wavelength of the illumination pattern is in the range of 365nm-405nm.

20. The apparatus according to claim 18, characterized in that, The blocking filter (4-4) is placed in front of a 2D sensor (4-5) for the 2D image (7-1).

21. The apparatus according to claim 1, characterized in that, The scanning device (8-1) has one or more combinations of the following imaging modes: -Use visible wavelength 2D images (7-1; 8-2) along with 3D information for recalculating the magnification, distance, and angle of the lesion; - Visible wavelength 2D images (7-1; 8-2) overlaid on the 3D texture image (7-3) of the lesion; - Use with visible wavelength 2D images (7-1; 8-2) and fluorescence images (7-2) in conjunction with 3D information used to recalculate magnification, distance, and angle; - Visible wavelength 2D images (7-1; 8-2) and fluorescence images (7-2) superimposed on the 3D texture image (7-3) of the lesion; - Visible wavelength 2D images (7-1; 8-2) overlaid on the 3D texture image (7-3) and subsurface structure image (7-4) of the lesion; - Visible wavelength 2D images (7-1; 8-2) and fluorescence images (7-2) are overlaid on the 3D texture image (7-3) and subsurface structure image (7-4) of the lesion.

22. The apparatus according to claim 1, characterized in that, It also includes a display for visualizing the lesion to the doctor / user for visual examination.

23. The apparatus according to claim 1, characterized in that, An artificial neural network is integrated into the device and is adapted to be used for training with images from a multimodal image database (8-3) and to classify the multimodal images (7-1; 7-2; 7-3; 7-4) fed into the artificial neural network.

24. The apparatus according to claim 1, characterized in that, The computed multimodal images can be sent to a cloud-based artificial neural network by means of a computer for training the network and collected in a multimodal image database (8-3), wherein the network can be connected by the device for classifying intraoral lesions captured by the device and provided to the network.

25. The apparatus according to claim 23 or 24, characterized in that, In addition to multimodal imaging data, information such as palpation results, the removability of the white layer, lesion history, and risk factors were used for the training and retrieval of the artificial neural network.

26. The apparatus according to claim 25, characterized in that, The risk factors include smoking and alcohol.

27. The apparatus according to any one of claims 9-11, characterized in that, The scanning device (8-1) includes an InGaAs image sensor (3-4) for covering a wavelength range of at least 1000nm-1600nm.

28. The apparatus according to claim 5, characterized in that, The scanning device (8-1) includes a mosaic CMOS sensor (4-5) with multiple different filters and a lens array for 2D spectral imaging.

29. The apparatus according to any one of claims 9-11, characterized in that, The scanning device (8-1) includes a CQD image sensor (4-3) to extend sensitivity into the NIR range, which in turn covers at least an additional wavelength range from 1000 nm to 1400 nm.

30. The apparatus according to any one of claims 1 to 11, characterized in that, The processing unit is adapted to additionally provide 3D information about the distance and angle between the scanning device (8-1) and the dermis or mucosa by using illumination patterns, stereometry, or time of flight.

Images