Method and apparatus for multimodal soft tissue diagnostics

Abstract

The present invention relates to a method and device for multimodal imaging of dermal and mucosal lesions, characterized by using at least two imaging modalities, one of which is a 3D scan of the lesion, and additionally providing information about the distance and angle between the scanning device and the dermis or mucosa, and mapping at least the second modality onto the 3D data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The applicants hereby provide a statement of incorporation by reference under Rule 4.18 PCT that priority application EP 20 176 399 is incorporated in its entirety into the present international application, including the description, claims and drawings. [Technical Field]

[0002] The present invention relates to a method and apparatus for imaging of dermal and mucosal lesions. [Background technology]

[0003] Traditional soft tissue diagnostic methods The diagnostic method begins with the patient's medical history. Data is collected about the patient's predisposing factors, such as smoking, alcohol abuse, diabetes, or injury history. Furthermore, it is important to assess the time of lesion onset (typically longer than the time of patient detection) and progression over time. This time may be a few days for acute lesions with typical healing times, or may require years of observation for chronic soft tissue changes, such as lichen-type lesions. Traditional oral soft tissue diagnostic methods rely on visual assessment of the lesion combined with other information, such as tactile information or the removability of whitish discoloration. Several factors are important for visual assessment: the location and size of the lesion, the color of the lesion (redness, whitish discoloration), and the structure and homogeneity of the discoloration (plaque, reticular, etc.). Clinicians typically compare the actual lesion with photographs in oral disease textbooks or other cases observed during medical practice using known diagnostic methods. Additionally, the consistency of the mucosa is assessed by palpation and by its relationship to the underlying bone (can the lesion be displaced by slight pressure against the bone, or is it fixed to the bone or underlying structures, e.g., muscle?). Furthermore, removal of whitish discoloration by mechanical friction is tested to distinguish between candidiasis or leukoplakia-type lesions. Bone involvement, e.g., in the case of tumors / swellings, may require additional radiological diagnostics. The gold standard remains tissue obtained from biopsies. Extended Diagnostics

[0004] In addition to traditional diagnostic methods, some dentists use blue / UV light to excite tissue autofluorescence and diagnose fluorescence images (e.g., Vizilite, VELScope, or similar) or toluidine blue staining. For oral autofluorescence diagnosis, a light source is used to excite endogenous fluorophores such as nicotinamide adenine dinucleotide (NADH) and flavin adenine dinucleotide (FAD). Wavelengths within the UV / blue range are used for excitation (375 and 440 nm). Emission is observed within the green wavelength range. Healthy mucosa exhibits pale green autofluorescence when viewed with a narrow-band filter that suppresses the excitation wavelength. Dysplastic tissue exhibits less fluorescence and appears darker compared to surrounding healthy tissue (see (1) in Figure 1). The ability to distinguish between dysplasia and benign inflammatory lesions has been questioned in the literature. Benign tissue inflammation often indicates an increased blood supply to the lesion. Increased hemoglobin content can absorb both excitation and emission light, causing intensity losses in fluorescence images that can be erroneously attributed to neoplasia. Vizilite uses chemicals to generate excitation light. To activate it, the capsule is bent to break the glass vial, causing the chemical products to react with each other and produce a bluish-white light with a wavelength of 430–580 nm that lasts for approximately 10 minutes. VELscope utilizes blue light excitation with a wavelength of 400–460 nm. The field of interest is observed through an optical imaging system (see Figure 1). A meta-analysis of 20 studies on autofluorescence methods for detecting oral dysplasia (see Reference 1) showed a large variation in sensitivity and specificity between different publications.

[0005] Vizilight: Comparing different studies, the sensitivity and specificity of Vizilight vary between 77% and 100% (sensitivity) and 0% and 28% (specificity). For the VELscope system, sensitivities of 22% and 100% and specificities of 16% and 100% have been reported. As an example, the sensitivity and specificity of toluidine blue were determined to be 86.36% and 76.9%, respectively. Another non-optical method is the Oral CDx system. Using a small brush, surface cells are collected and analyzed in the laboratory. Sensitivity varies between 71.4% and 100% in different studies, and specificity varies between 32% and 100%.

[0006] In summary, the use of autofluorescence alone does not appear to be sufficient as a diagnostic aid, especially considering the low prevalence of malignant tumors and the variability in user experience, but it can be an additional diagnostic tool in combination with others. Other methods, such as staining or brush biopsy, show moderate (staining) or high variability in sensitivity and specificity. Therefore, improvements in diagnostic capabilities are desirable. Neural networks for oral disease detection

[0007] Artificial intelligence is becoming increasingly successful as a diagnostic aid for the classification of dermal lesions and X-ray images. AI networks can already achieve the accuracy level of a qualified dermatologist when using, for example, >100k images for 700 disease classes for training (see Reference 2). This results in approximately 150 images per class. A review paper on the use of deep learning algorithms in dentistry reviewed 25 papers. 2D X-ray, CBCT, QLF, and OCT were used as diagnostic modalities. The review concluded that typical dataset sizes tend to increase from approximately 100 datasets per class to 1000 datasets. A dataset of 1000 datasets has been reported to achieve approximately 98% accuracy, while more than 4000 datasets are required for accuracy above 99%. Only one of the reviewed papers mentioned gingivitis detection and classification with CNN using QLF. No paper mentioned classification of oral diseases (Reference 3). Aside from purely image-related classification, other authors have reported the use of CNNs including contextual factors in the detection of oral cancer. A classification accuracy of 99% was achieved by selecting 12 of 35 relevant factors, including clinical symptoms and medical history, as well as risk factors and socioeconomic factors (Reference 4). In conclusion, neural networks can contribute to improving medical and dental diagnosis, provided that databases with sufficient sample size, features, and quality are available.

[0008] Reference 1: Efficacy of light based detection systems for early detection of oral cancer and oral potentially malignant disorders: Systematic review Ravleen Nagi, Yashoda-Bhoomi Reddy-Kantharaj, Nagaraju Rakesh, Sujatha Janardhan-Reddy, Shashikant Sahu Med Oral Patol Oral Cir Bucal. 2016 Jul 1;21 (4):e447-55. Reference 2: Dermatologist-level classification of skin cancer with deep neural networks Esteva A. et al. Nature 542 115-118(2017). Reference 3: An overview of deep learning in the field of dentistry Jae-Joon Hwang 1,Yun-Hoa Jung 1,Bong-Hae Cho 1, Min-Suk Heo 2, Imaging Science in Dentistry 2019; 49: 1-7. Reference 4: Usage of Probabilistic and General Regression Neural Network for Early Detection and Prevention of Oral Cancer Neha Sharma, Hari Om The Scientific World Journal Vol. 2015, Article ID 234191, http: / / dx.doi.org / 10.1155 / 2015 / 234191. Reference 5: UV-angeregte Autofluoreszenz: Spektroskopische und fluoreszenzmikroskopische Untersuchungen zur Tumorselektivitaet endogener Gewebefarbstoffe Alexander Hohla Dissertation, LMU, Muenchen, 2003. Summary of the Invention

[0009] In studies using neural networks, the lack of image consistency in the region of interest is a concern, often requiring manual image preprocessing. Photographs are often taken from different distances (hence different magnifications), different observation angles, and different lighting conditions. In multimodal diagnostic methods, it is even more important to achieve good agreement (ideally pixel-by-pixel) between different diagnostic modalities.

[0010] The present invention addresses all these concerns, avoids manual preprocessing, and can reduce computation time for AI classification of dermal and mucosal lesions.

[0011] The object of the present invention is to overcome at least some of the above-mentioned problems. This object is achieved by the apparatus described in claim 1 and the method described in claim 28. Other claims relate to further embodiments and developments. Using a device that generates accurate 3D surface representations, for example based on confocal, preferably chromatic confocal, time-of-flight, stereogrammetry, or OCT techniques, has the advantage of accurately knowing the dimensions of the lesion. The distance between the scanned surface and the 3D imaging device is always accurately known, from which the exact dimensions of the lesion can be calculated. The angle of the 3D imaging device relative to the scanned surface is also known, and since the illumination is integrated into the device, the lighting conditions are always the same. For 3D measurements, this is typically a well-defined light pattern. Combining 3D measurements with spectrally resolved 2D images (e.g., with three or more channels (RGB)) allows for matching of 2D and 3D data. So far, the prior art is unaware of the use of 3D textures for diagnostic purposes for soft tissue / mucosal lesions and the mapping of spectrally resolved 2D image data onto the 3D texture of such lesions.

[0012] Most practical 3D scanning devices use video-like scanning techniques to combine many sequentially acquired single 3D images and overlay them using 3D landmarks to accurately overlay the single image. This is easy for intraoral teeth, for example, but becomes more difficult when the surface with the capture area of ​​the single image does not exhibit sufficient 3D landmarks. For example, in extreme cases, it is impossible to scan a flat surface or a sphere. In such cases, spectrally resolved 2D data overlaid on the 3D data can support accurate alignment of the single 3D images to each other.

[0013] This is useful when scanning the dermis or mucosa in flat areas, as lesions exhibit different scattering and absorption coefficient distributions than healthy tissue, which for example result in a whitish coloration pattern when the scattering coefficient is increased, or a brownish coloration when the absorption coefficient is increased (see Figure 2a: whitish discoloration, see Figure 2b: brownish discoloration). 3D image resolution

[0014] In practical 3D scanning systems, such as those used in dental applications, a resolution of 10 μm to 30 μm is actually possible with a similar degree of error. This is less than the resolution of microscopes used to diagnose tissue, but is much better than in vivo visual inspection. This allows the surface texture of lesions to be calculated. Wavelength selection for 3D imaging:

[0015] Since biological tissues typically exhibit a lower penetration depth for illumination light in the blue or near-UV region (350 nm-400 nm), mainly due to an increased light scattering coefficient, these wavelengths can be used in combination with scanning methods that suppress volumetric scattered light (e.g., confocal and OCT-based methods that can be combined with depth-of-focus-based techniques) to generate clear surface texture 3D images.

[0016] At wavelengths longer than 840 nm, preferably longer than 980 nm, and most preferably in the range of 1300 nm to 1600 nm, the scattering coefficient is much lower, allowing 3D imaging to depths of tens of mm to hundreds of micrometers, where subsurface structures can be acquired and imaged.

[0017] By providing at least two wavelengths, one in the 350-400 nm range and the other longer than 840 nm, it is possible to combine clear 3D surface scanning with structural information at least hundreds of micrometers deep into dermal / mucosal lesions. In the simplest case, illumination is switched sequentially between different wavelengths, and illumination sources are combined into the same optical path using a dichroic mirror. This variant will work for wavelengths that can be detected using the same sensor (e.g., CMOS 350 nm-1000 nm).

[0018] In some 3D measurement techniques, if the dimensions of the light source are small enough (e.g. LEDs), it may even be possible to use light paths with different wavelengths that do not perfectly coincide. In that case, slight angular deviations will cause a displacement of the illumination pattern on the sensor, which can be corrected by calculation (displacement and distortion correction).

[0019] If a second sensor is required, which may be the case when using illumination wavelengths above 1000 nm, at least one beam splitter can be used to separate the optical paths for the different sensors (see Figure 3). Fluorescence imaging

[0020] A further extension of 3D imaging is its combination with fluorescence imaging. As explained in the previous paragraph, human tissues exhibit autofluorescence when excited with appropriate wavelengths. Dermal / mucosal lesions exhibit autofluorescence of different intensities. This can be excitation in the UV / blue range for FAD, NADH, and collagen, but also red excitation to excite porphyrins. In combination with 3D imaging, this allows the overlay of fluorescence image data onto 3D texture data.

[0021] The optical beam path can be conventional, with a blocking filter for excitation light using the same UV / blue wavelength light pattern as used for 3D imaging, and a blocking filter for fluorescence detection introduced into the imaging path after separation from the illumination path. However, this would require moving parts (filters) in the device. Depending on the design of the 3D scanner being expanded, even stronger excitation light power may be required.

[0022] Although the 3D information in the fluorescence image data is discarded, the 3D optical path can remain unchanged: a UV-blocking filter can be introduced into the 2D optical path typically used for 2D images in the visible spectral range, while still using the excitation optical path for 3D imaging illumination (see Figure 4).

[0023] The most preferable solution, however, is to use a separate excitation light source placed on the side of an interchangeable hood. A blocking filter can be integrated into the hood window. In that case, the filter should not suppress the structured light for 3D measurements (Figure 5). This is possible because many fluorophores in the human body, such as collagen, NADH, FAD, elastin, and keratin, can be excited below 350 nm, allowing the 3D light pattern to use the 365 nm to 405 nm range (Reference 5) (see Figure 6).

[0024] Another option is to place an excitation light-blocking filter in front of the 2D sensor and leave the 3D light path unaltered. The filter does not affect imaging within the visual range, which is typically used to generate "color 2D images," because the fluorescence emission is also within the visual range (see Figure 4).

[0025] This modified hood replaces the conventional hood, which is in any case removable for sterilization.

[0026] By way of example, and not limitation, the capabilities of intraoral 3D scanning devices such as Primescan or Omnicam (and other volume scatter resistant scanning devices) can be extended using the techniques described above.

[0027] Alternatively, the excitation LEDs can be placed inside a hood, which is more or less an empty shell, that can be sterilized without reducing the LED's lifetime, however, it would require more modifications to existing 3D scanning devices.

[0028] A further advantage of using an intraoral scanning device as the basis for detecting and classifying intraoral lesions is the form of the device, which is very different from devices for capturing images of lesions in dermatology, allowing access to all areas of the oral cavity.

[0029] The following preferred multimodal imaging options are enabled by the techniques described above: - visual wavelength 2D images together with 3D information to calculate the magnification, distance and angle of the lesion; - Use of visual wavelength 2D images overlaid on the 3D texture of the lesion, - Visual wavelength 2D image + fluorescent image, with 3D information used for recalculation of magnification, distance, and angle - Visual wavelength 2D image + fluorescence image overlaid on the 3D texture of the lesion, - Visual wavelength 2D images overlaid on the 3D texture + subsurface structural information of the lesion, - Use of visual wavelength 2D images + fluorescence images overlaid on 3D texture + subsurface structural information of the lesion, However, any other combination of different imaging modalities, 2D color images, 3D texture images, fluorescence images, subsurface images with long wavelengths are not excluded (see Figure 7).

[0030] The additional advantage of absolute dimensions and known imaging conditions provided by the combination of at least 3D measurements and 2D color images allows images of the same lesion taken at different times to be overlaid (registered) with a best fit algorithm to see even smaller deviations, which allows the evolution of the lesion over time to be monitored.

[0031] Images captured with such devices can be processed on a processing means, such as a computer that is part of the device, and presented on a computer screen to a physician for visual inspection, or can be used to build a multimodal image database for training a neural network (either an external network via a cloud-based network training service, or an internal network if sufficient computing power is available), either alone or in combination with additional "non-imaging" information such as palpation results, removability of the whitish layer, lesion history, and risk factors (smoking, alcohol, etc.). The screen can be the display of a desktop or mobile device, with or without a touchscreen, or it can be a wearable device such as a head-mounted display.

[0032] The trained network can be implemented in the device to provide a diagnostic suggestion or, if unable to give a final diagnostic suggestion, provide a recommendation to send the patient to an oral disease specialist for further testing / biopsy (see Figure 8).

[0033] In the following description, further aspects and advantages of the invention are explained in more detail by means of illustrative embodiments and with reference to the drawings. [Brief explanation of the drawings]

[0034] [Figure 1] A comparison of the photographs and corresponding autofluorescence images is shown. [Figure 2a] It shows whitish lesions. [Figure 2b]Lesions with increased pigmentation are shown. [Figure 3] The core functional blocks of a 3D scanning device are shown. [Figure 4] 1 shows the functional blocks of a 3D scanning device. [Figure 5] 1 shows the front end hood of the 3D scanning device. [Figure 6] The emission bands of different fluorophores are shown. [Figure 7] 1 shows a combination of imaging modalities. [Figure 8] 1 shows the setup of an artificial neural network for diagnostic support.

[0035] The reference numbers shown in the drawings indicate elements listed below and will be referenced in the subsequent description of the illustrative embodiments. 1-1: Lesion 3-1: 3D scanning optics 3-2: Dichroic mirror / beam splitter 3-3: Sensor (e.g. CMOS) 3-4: Sensor (e.g., InGaAs detector) 4-1: 3D scanning optics 4-2: Beam splitter 4-3: Sensor (e.g. CMOS) 4-4: Blocking filter 4-5: Sensor (e.g. CMOS) 4-6: Front end 5-1: Front hood 5-2:UV LED 5-3: Imaging window 7-1: 2D color image 7-2: Autofluorescence image 7-3: 3D texture image 7-4: Subsurface structure image 8-1: Device 8-2: 2D images 8-3: Database 8-4: Brush biopsy 8-5:X-ray image 8-6: Palpation results DETAILED DESCRIPTION OF THE INVENTION

[0036] Figure 1 shows a comparison of the photograph with the corresponding autofluorescence image, in this case made with the "VELscope" device. The corresponding image shows clearly better contrast between the lesion (1-1) and healthy tissue in the autofluorescence image.

[0037] Figure 2a shows a white lesion caused primarily by a thickening of the epidermal layer, which results in a significantly increased scattering coefficient, while Figure 2b shows a lesion with increased pigmentation, which causes an increased absorption coefficient.

[0038] Figure 3 shows the functional blocks of the 3D scanning device. (3-1) is the 3D scanning optics, (3-2) is a dichroic mirror / beam splitter that separates the wavelength band in the range of 300 nm to 800 nm that reaches the CMOS sensor (3-3), while wavelengths longer than 1000 nm are reflected to the sensor (3-4), which can be an InGaAs detector covering at least the wavelength band from 1000 nm to 1600 nm.

[0039] Figure 4 shows the functional blocks of a 3D scanning device. (4-6) is the front end that deflects the image toward the beam splitter (4-2), separating the 2D imaging path from the 3D imaging path. (4-1) is the 3D scanning optics, which includes a CMOS sensor (4-3) (e.g., a 3D sensor), and (4-4) is a blocking filter that suppresses excitation light that does not reach the CMOS sensor (4-5). Alternatively, the CMOS sensor (4-3) can be optionally replaced with a CQD sensor (4-3) with extended sensitivity in the NIR range. The blocking wavelength of the blocking filter is approximately 370 nm to 400 nm. This allows autofluorescence emission light to pass through, while also allowing visual wavelengths to pass for color imaging. The CMOS sensor (4-5) is not limited to a conventional RGB three-channel sensor but can include more channels with better spectral resolution, such as a mosaic-type CMOS sensor with multiple different filters combined with a lens array (not shown in the image). This makes it possible to distinguish between different fluorophores, as they have emission bands with different wavelength maxima, as shown in Figure 6. Optical components (3-2), (3-3), and (3-4) in Figure 3 can be replaced by component (4-5) in Figure 4 to have a 2D sensor (3-3) for the visible range and another 2D sensor (3-4) for NIR light.

[0040] The apparatus for multimodal imaging of dermal and mucosal lesions comprises a scanning device (8-1) having illumination sources and sensors (3-3, 3-4, 4-3, 4-5) and at least one processing means for calculating an image from raw data provided by the scanning device (8-1), the scanning device (8-1) being adapted to use at least two imaging modalities, a first imaging modality generating 3D data for a 3D image (7-3; 7-4) in a 3D scan of the lesion, the processing means additionally providing 3D information regarding the distance and angle between the scanning device (8-1) and the dermis or mucosa through the use of an illumination pattern, stereogrammetry, or time-of-flight, and adapted to map at least the images (7-1; 7-2) generated by the second imaging modality onto the 3D image (7-3; 7-4) of the 3D scan based on the 3D information. The use of illumination patterns, stereogrammetry, or time-of-flight is one of various techniques that can be used by a person skilled in the art.

[0041] Figure 5 shows the front-end hood (5-1) of a 3D scanning device. The hood is typically removable from the rest of the scanning device for disinfection. UV LEDs (5-2) are positioned parallel to the imaging window (5-3) to illuminate the field of interest and excite autofluorescence. Backscattered light is transmitted through the imaging window (5-3), which may already be covered by an excitation light-blocking filter (interference filter) if this filter is not positioned elsewhere in the detection beam path. The hood may contain optical elements, including the UV LEDs, or it may be a more or less empty shell that covers the optics inside the hood. This avoids exposing the UV LEDs to the sterilization cycle.

[0042] Figure 6 shows the different emission bands and maxima of different fluorophores, excited in this case at 308 nm. The different peaks may allow for the separation of different fluorophores. However, this is a normalized image. In reality, the emission intensity of collagen creates a high background signal that may overwhelm other fluorophores.

[0043] Figure 7 shows the proposed device and different combinations of possible imaging modalities, with 2D color images (7-1) (e.g., 2D spectrally resolved images), autofluorescence images (7-2), 3D texture images (7-3), and subsurface structure images taken at longer wavelengths (7-4).

[0044] Figure 8 shows the setup of an artificial neural network to support the diagnosis of dermal / mucosal lesions using images taken with the proposed device (8-1). Figure 8 shows only 2D images (8-2) that incorporate additional 3D information, such as distance and angle, but is not limited to these images. All combinations shown in Figure 7 or described in the text apply. From the 2D images (8-2), a database (8-3) is constructed and used to train the artificial neural network. To improve classification, additional information other than image data can be added to the network, such as brush biopsy results (8-4), X-ray images (8-5), and palpation results (8-6). The palpation results image (8-6) is merely illustrative, chosen to show gingival ridges, which can be hard or soft. Naturally, a large number of cases with these data must be included in the database connected to the corresponding case images used for training.

[0045] In the present invention, due to the known, precise absolute dimensions, such as the angle and distance of the lesion surface to the imaging plane, and known imaging conditions provided by the combination of at least 3D measurements and 2D color images, it is possible to overlay (align) images of the same lesion taken at different times to see even smaller deviations, which makes it possible to monitor the development of the lesion over time. The following is a summary of the claims as originally filed: [1] An apparatus for multimodal imaging of dermal and mucosal lesions, the apparatus comprising a scanning device (8-1) having an illumination source and sensors (3-3, 3-4, 4-3, 4-5) and at least one processing means for calculation of an image from raw data provided by the scanning device (8-1), characterized in that the scanning device (8-1) is adapted to use at least two imaging modalities, a first imaging modality generating 3D data for a 3D image (7-3; 7-4) in a 3D scanning of the lesion, the processing means additionally providing 3D information on the distance and angle between the scanning device (8-1) and the dermis or mucosa, and adapted to map at least an image (7-1; 7-2) generated by a second imaging modality onto the 3D image (7-3; 7-4) of the 3D scanning based on the 3D information. [2] The apparatus of [1], characterized in that the processing means is further adapted to calculate accurate dimensions of the lesion by using the 3D information regarding the distance and angle between the scanning device (8-1) and the lesion. [3] The apparatus described in [1], characterized in that the processing means is further adapted to calculate a 3D surface texture of the lesion by using the 3D information of the 3D scan. [4] The device described in [1], characterized in that the second imaging modality generates the image (7-1; 7-2) as at least one of a 2D image (7-1) or an autofluorescence image (7-2) by using one sensor (4-5), and the 2D image (7-1) is spectrally resolved in three or more channels. [5] The apparatus described in [4], characterized in that the spectrally resolved 2D data of the 2D image (7-1) superimposed on the 3D data through the processing means supports accurate registration of a single 3D image of the 3D data to form a complete 3D image (7-3; 7-4) of the target area, the 3D image including at least one of a 3D texture image (7-3) or a subsurface structure image (7-4). [6] The apparatus described in [1], characterized in that the 3D data of the 3D scan is captured using a technique that suppresses volume scattering through confocal imaging (OCT) or a combination of confocal scanning or OCT with a depth-of-focus based technique. [7] The apparatus according to [1], characterized in that the scanning device (8-1) is adapted to use wavelengths of 350 to 400 nm for the 3D scanning of the surface by using one corresponding sensor (4-3) for the 3D image (7-3; 7-4). [8] The apparatus according to [1], characterized in that the scanning device (8-1) is adapted to use wavelengths longer than 840 nm, preferably longer than 980 nm, most preferably in the range of 1300 nm to 1600 nm for the 3D scanning of the subsurface by using one corresponding sensor (4-3) for the subsurface structure image (7-4) so ​​as to operate at wavelengths at which dermal and mucosal lesions exhibit smaller scattering coefficients. [9] The apparatus according to [7] or [8], characterized in that the scanning device (8-1) is adapted to use both wavelength ranges together for 3D scanning of the surface and for 3D scanning of the subsurface.

[10] The apparatus according to [9], characterized in that the scanning device (8-1) is adapted to sequentially switch illumination between different wavelengths, and the illumination sources are coupled into the same optical path using a dichroic mirror.

[11] The device according to [9], characterized in that the optical paths of the illumination sources for the different wavelengths do not perfectly coincide if the source dimensions are sufficiently small, and slight angular deviations that cause displacements of the illumination patterns on the associated sensors (3-3; 3-4; 4-3; 4-5) are corrected by calculation.

[12] The apparatus according to [9], characterized in that when the scanning device (8-1) uses illumination with a wavelength above 1000 nm, the scanning device (8-1) comprises at least one beam splitter (3-2) for separating optical paths for the different sensors (3-3; 3-4).

[13] The device described in [1], wherein the first or second imaging modality is further characterized as being for fluorescence imaging using excitation light in the UV / blue range for fluorophores such as FAD, NADH, and collagen.

[14] The apparatus described in [1], characterized in that the scanning device (8-1) is adapted to use light in the red wavelength range for excitation of porphyrins in the second imaging modality, and the processing means is adapted to overlay a fluorescence image (7-2) on the 3D data of the 3D scan.

[15] The apparatus according to

[13] , characterized in that the scanning device (8-1) is provided with a blocking filter (4-4) for excitation light using an illumination pattern having the same UV / blue wavelengths as that used for 3D imaging, and is adapted to introduce the blocking filter (4-4) for fluorescence detection into the imaging path after separation from the illumination path.

[16] The device according to

[15] , characterized in that the blocking filter (4-4) is in a 2D imaging path for the 2D image (7-1) of the second imaging modality.

[17] The apparatus according to

[13] , characterized in that the scanning device (8-1) comprises a separate excitation light source (5-2) arranged on the side of an exchangeable hood (5-1), and a blocking filter (4-4) is integrated into a window (5-3) of the exchangeable hood (5-1), the blocking filter (4-4) being designed not to suppress structured light for 3D measurements.

[18] The device according to

[13] , characterized in that the excitation wavelength is less than 350 nm and the wavelength of the illumination pattern is in the range of 365 nm to 405 nm.

[19] The device according to

[16] , characterized in that the blocking filter (4-4) is arranged in front of one 2D sensor (4-5) for the 2D image (7-1).

[20] The scanning device (8-1) - Visual wavelength 2D images (7-1; 8-2) together with 3D information used to recalculate the magnification, distance, and angle of the lesion; - Use of visual wavelength 2D images (7-1; 8-2) overlaid on a 3D texture image (7-3) of the lesion, - Visual wavelength 2D images (7-1; 8-2) and fluorescence images (7-2) with 3D information used for recalculation of magnification, distance, and angle, - visual wavelength 2D images (7-1; 8-2) and fluorescence images (7-2) overlaid on the 3D texture image (7-3) of the lesion, - visual wavelength 2D images (7-1; 8-2) overlaid on the 3D texture image (7-3) and subsurface structure image (7-4) of the lesion; - Use of visual wavelength 2D images (7-1; 8-2) and fluorescence images (7-2) overlaid on the 3D texture images (7-3) and subsurface structure images (7-4) of the lesion, The device according to [1], characterized in that it has one or more combinations of the above imaging modalities.

[21] The device according to [1], further comprising a display for visualizing the lesion to a doctor / user for visual inspection.

[22] The device according to [1], characterized in that an artificial neural network is integrated into the device and adapted to use images of a multimodal image database (8-3) for training and to classify multimodal images (7-1; 7-2; 7-3; 7-4) fed to the artificial neural network.

[23] The device according to [1], characterized in that the calculated multimodal images can be sent by a computer to a cloud-based artificial neural network for training the artificial neural network and collected in a multimodal image database (8-3), which can be connected by the device for classification of intraoral lesions captured by the device and provided to the artificial neural network.

[24] The device described in

[22] or

[23] , characterized in that in addition to multimodal imaging data, information such as palpation results, removability of the whitish layer, lesion history, and risk factors such as smoking, alcohol, etc. are used for training and searching the artificial neural network.

[25] The apparatus according to [8], characterized in that the scanning device (8-1) comprises one InGaAs image sensor (3-4) for covering at least a wavelength range of 1000 nm to 1600 nm.

[26] The apparatus according to [4], characterized in that the scanning device (8-1) comprises one mosaic-type CMOS sensor (4-5) with a number of different filters combined with a lens array for 2D spectral imaging.

[27] The apparatus according to

[17] , characterized in that the replaceable hood (5-1) covers a fluorescence excitation LED (5-2) for illuminating a target field through the window (5-3), and the replaceable hood (5-1) is removable for sterilization, while the fluorescence excitation LED (5-2) remains in the rest of the scanning device (8-1) to avoid exposing the fluorescence excitation LED (5-2) to a sterilization cycle.

[28] The apparatus described in [8], characterized in that the scanning device (8-1) additionally comprises one CQD image sensor (4-3) for extending sensitivity to the NIR range, covering a wavelength range of at least 1000 nm to 1400 nm.

[29] The apparatus according to one or more of [1] to

[28] , characterized in that the processing means is additionally adapted to provide the 3D information regarding the distance and angle between the scanning device (8-1) and the dermis or mucosa through the use of an illumination pattern, or stereogrammetry, or time of flight.

Claims

1. An apparatus for multimodal imaging of dermal and mucosal lesions, said apparatus comprising a scanning device (8-1) having an illumination source and sensors (3-3, 3-4, 4-3, 4-5) and at least one processing means for the calculation of an image from raw data provided by said scanning device (8-1), said scanning device (8-1) being adapted to use at least two imaging modalities, a first imaging modality generating 3D data for a 3D image (7-3; 7-4) in a 3D scan of the lesion, said processing means additionally providing 3D information on the distance and angle between the scanning device (8-1) and the dermis or mucosa, and calculating a 3D image based on said 3D information. and adapted to map at least an image (7-1; 7-2) generated by a second imaging modality onto the 3D image (7-3; 7-4) of the 3D scan, wherein the first or the second imaging modality is further for fluorescence imaging using excitation light in the UV / blue range for fluorophores, the scanning device (8-1) comprising a separate excitation light source (5-2) arranged at the side of an exchangeable hood (5-1), and a blocking filter (4-4) integrated into a window (5-3) of the exchangeable hood (5-1), the blocking filter (4-4) being designed not to suppress structured light for 3D measurements.

2. 2. The apparatus according to claim 1, wherein the processing means is further adapted to calculate precise dimensions of the lesion by using the 3D information on the distance and angle between the scanning device (8-1) and the lesion.

3. 2. The apparatus of claim 1, wherein the processing means is further adapted to calculate a 3D surface texture of the lesion by using the 3D information of the 3D scan.

4. 2. The device of claim 1, wherein the second imaging modality generates the image (7-1; 7-2) as at least one of a 2D image (7-1) or an autofluorescence image (7-2) by using one sensor (4-5), the 2D image (7-1) being spectrally resolved in three or more channels.

5. 5. The apparatus according to claim 4, characterized in that the spectrally resolved 2D data of the 2D image (7-1) superimposed on the 3D data through the processing means supports accurate registration of a single 3D image of the 3D data to form a complete 3D image (7-3; 7-4) of the area of ​​interest, the 3D image comprising at least one of a 3D texture image (7-3) or a subsurface structure image (7-4).

6. 10. The apparatus of claim 1, wherein the 3D data of the 3D scan is captured with a technique that suppresses volume scattering through confocal imaging (OCT) or a combination of confocal scanning or OCT with a depth-of-focus based technique.

7. 2. The apparatus according to claim 1, characterized in that the scanning device (8-1) is adapted to use wavelengths between 350 and 400 nm for the 3D scanning of a surface by using one corresponding sensor (4-3) for the 3D image (7-3; 7-4).

8. 2. The apparatus according to claim 1, characterized in that the scanning device (8-1) is adapted to use wavelengths in the range longer than 840 nm for the 3D scanning of the subsurface by using one corresponding sensor (4-3) for subsurface structure imaging (7-4) so ​​as to operate at wavelengths where dermal and mucosal lesions exhibit smaller scattering coefficients.

9. Apparatus according to claim 7 or 8, characterized in that the scanning device (8-1) is adapted to use both wavelength ranges together for surface 3D scanning and for subsurface 3D scanning.

10. 10. Apparatus according to claim 9, characterized in that the scanning device (8-1) is adapted to switch illumination sequentially between different wavelengths, and the illumination sources are coupled into the same optical path using dichroic mirrors.

11. The device described in claim 9, characterized in that the optical paths of the illumination light sources for different wavelengths do not perfectly coincide if the light source dimensions are small, and slight angular deviations that cause displacements of the illumination pattern on the associated sensors (3-3; 3-4; 4-3; 4-5) are corrected by calculation.

12. 10. Apparatus according to claim 9, characterized in that in the scanning device (8-1) when using illumination with a wavelength above 1000 nm, the scanning device (8-1) comprises at least one beam splitter (3-2) for separating optical paths for the different sensors (3-3; 3-4).

13. 2. The apparatus according to claim 1, characterized in that the scanning device (8-1) is adapted to use light in the red wavelength range for excitation of porphyrins in the second imaging modality, and the processing means is adapted to overlay a fluorescence image (7-2) on the 3D data of the 3D scan.

14. 2. The apparatus according to claim 1, characterized in that the scanning device (8-1) comprises a blocking filter (4-4) for excitation light using an illumination pattern with UV / blue wavelengths that is the same as that used for 3D imaging, and is adapted to introduce the blocking filter (4-4) for fluorescence detection into the imaging path after separation from the illumination path.

15. 15. The device according to claim 14, characterized in that the blocking filter (4-4) is in the 2D imaging path for the 2D image (7-1) of the second imaging modality.

16. 2. The device of claim 1, wherein the excitation wavelength is less than 350 nm and the wavelength of the illumination pattern is in the range of 365 nm to 405 nm.

17. 16. Device according to claim 15, characterized in that the blocking filter (4-4) is placed in front of one 2D sensor (4-5) for the 2D image (7-1).

18. The scanning device (8-1) - visual wavelength 2D images together with 3D information used to recalculate the magnification, distance and angle of the lesion (7-1; 8-2); - Use of visual wavelength 2D images (7-1; 8-2) overlaid on a 3D texture image of the lesion (7-3), - visual wavelength 2D images (7-1; 8-2) and fluorescence images (7-2) with 3D information used for recalculation of magnification, distance and angle; - visual wavelength 2D images (7-1; 8-2) and fluorescence images (7-2) overlaid on the 3D texture image (7-3) of the lesion, - visual wavelength 2D images (7-1; 8-2) overlaid on the 3D texture image (7-3) and subsurface structure image (7-4) of the lesion, - the use of visual wavelength 2D images (7-1; 8-2) and fluorescence images (7-2) overlaid on the 3D texture images (7-3) and subsurface structure images (7-4) of the lesion; 2. The apparatus of claim 1, comprising one or more combinations of the following imaging modalities:

19. 10. The device of claim 1, further comprising a display for visualization of the lesion to a physician / user for visual inspection.

20. 2. The device according to claim 1, characterized in that an artificial neural network is integrated into the device and adapted to use images of a multimodal image database (8-3) for training and to classify multimodal images (7-1; 7-2; 7-3; 7-4) fed to the artificial neural network.

21. 2. The device according to claim 1, characterized in that the calculated multimodal images can be sent by a computer to a cloud-based artificial neural network for training the artificial neural network and collected in a multimodal image database (8-3), which can be connected by the device for classification of intraoral lesions captured by the device and provided to the artificial neural network.

22. 22. The device according to claim 20 or 21, characterized in that in addition to the multimodal imaging data, information about the palpation results, the removability of the whitish layer, the history of the lesion, and risk factors is used for training and searching the artificial neural network.

23. Apparatus according to claim 8, characterized in that the scanning device (8-1) comprises one InGaAs image sensor (3-4) for covering at least the wavelength range from 1000 nm to 1600 nm.

24. 5. Apparatus according to claim 4, characterized in that the scanning device (8-1) comprises one mosaic-type CMOS sensor (4-5) with a number of different filters combined with a lens array for 2D spectral imaging.

25. 9. The apparatus according to claim 8, characterized in that the scanning device (8-1) comprises one CQD image sensor (4-3) for extending sensitivity to at least the NIR range, additionally covering the wavelength range from 1000 nm to 1400 nm.

26. 26. Apparatus according to one or more of the preceding claims, characterized in that the processing means are additionally adapted to provide the 3D information on the distance and angle between the scanning device (8-1) and the dermis or mucosa through the use of illumination patterns, or stereogrammetry, or time of flight.

Images