System and method for digitally reconstructing an object - Patents.com

Abstract

A system for digitally reconstructing the structure of a region of interest of an object is provided. The system comprises an imaging device for obtaining a sequence of images of the region of interest, and a processor. The processor analyses system data conveying at least information about a position of the imaging device relative to the region of interest. Quasi-periodic behavior is extracted from the system data, the identified quasi-periodic behavior is used to remove redundancies in the sequence of images and / or in the system data, and the sequence of images and the system data are used to reconstruct the structure of the region of interest.

Description

[Technical field]

[0001] The present invention relates to imaging and digital reconstruction of structures of interest, for example as part of a personal care system. [Background technology]

[0002] Recently, there has been growing interest in incorporating cameras or other types of imaging devices into personal care devices, such as oral care devices. Such imaging-assisted oral care devices may enable image-based diagnosis, improved location sensing, treatment planning (e.g., orthodontics, aligners) or treatment monitoring (e.g., periodontitis) that is seamlessly integrated into a user's regular oral hygiene regimen, instead of using separate devices.

[0003] Dental endoscopes or intra-oral scanners exist that can detect various structures and conditions, including diseases of the oral cavity. By forming these scanners as standalone devices, separate from the toothbrush head, for example, there is no problem with properly positioning the sensor relative to the toothbrush head to avoid issues such as interference with bristles position (which is important for toothbrush cleaning effectiveness), interference from various residues of toothpaste and cleaning processes, and suboptimal field of view.

[0004] From such standalone imaging devices, a nearly continuous stream of images can be collected, but it usually requires a separate workflow and scanning procedure performed by either the user or the dental professional. For portable personal care devices with embedded image sensors, this stream of images poses many challenges, the most pressing for everyday use during regular personal care routines being how to reduce the data flow to only the essential data, how to create a 3D localization map from many images (usually collected with a single camera), and how to optimize the data flow to enable near real-time processing and image representation with limited on-device processing resources.

[0005] Images from the imaging device can be consulted for various types of instructions (e.g., directions on how to optimally perform oral care actions, personalized suggestions, etc.). Most, if not all, of such instructions are based on or would greatly benefit from some form of accurate localization of the device within the various images and image sequences. Moreover, a global map of the user's biology of interest (e.g., oral region, head / face / neck skin, etc.) is crucial for visualization purposes, both for reference between current and future scheduled actions, and for pertinent healthcare-related tasks, such as identifying anomalies, pathologies, etc. Summary of the Invention [Problem to be solved by the invention]

[0006] However, the construction of such a comprehensive map typically requires a set of external reference points or reference landmarks, typically generated by a dedicated hardware solution for independent spatial reference.

[0007] Therefore, improved methods for reconstructing structures from a stream of images are needed. Moreover, a solution is needed to enable computationally efficient collection of 3D image maps in the home environment when imaging modalities (e.g., cameras) are integrated during the routine workflow of personal care or personal health device use.

[0008] US Patent Application No. 2016 / 136115 discloses a system for stitching together optical scan data for a first portion of a three-dimensional object and ultrasound data for a second portion of the three-dimensional object, for example a portion of a jaw, and the stitched image is used to aid in the design of, for example, a crown, denture or bridge. [Means for solving the problem]

[0009] The invention is defined by the claims.

[0010] According to an example according to an aspect of the present invention, there is provided a system for digitally reconstructing a structure of a region of interest of an object, the system comprising: an imaging device for obtaining a sequence of images of a region of interest; Analyzing system data, including at least data enabling identification of a position of an imaging device relative to a region of interest, and extracting quasi-periodic behavior from the system data; removing redundancy in the sequence of images and / or system data using the identified quasi-periodic behavior; Reconstructing the structure of the region of interest using the sequence of images and the system data; A processor for performing A system is provided, comprising:

[0011] Many actions performed by humans (and animals) are somewhat periodic (i.e., quasi-periodic). These actions include self-care actions in particular. For example, brushing one's teeth or shaving involve quasi-periodic behaviors. These quasi-periodic behaviors have been identified and shown to be useful for removing redundancy in data of particular structures (i.e., removing redundancy in image or system data).

[0012] The elimination of (at least some of) the redundancies in the data allows the structure to be reconstructed much faster than would be the case without the elimination of redundancies, since less data has to be processed during reconstruction.

[0013] The processor removes redundancy by identifying redundant images in the sequence of images based on the quasi-periodic behavior and removing the redundant images from the sequence of images.

[0014] Due to the quasi-periodic behavior, images in a sequence of images may contain information that is the same or very similar to information in other images in the sequence of images, and therefore redundancy can be removed from the sequence of images by removing images that are similar to other images in the sequence.

[0015] The processor further filters the sequence of images to eliminate images with insufficient quality parameters for reconstruction.

[0016] Often times, a sequence of images may contain images that are of poor quality, e.g., they contain excessive blur, obstructed fields of view, etc. Thus, filtering the sequence of images to remove the poor quality images further reduces the data that needs to be processed during reconstruction.

[0017] The images in the sequence of images may be multispectral images, and the structure of the region of interest is reconstructed separately for each spectral band in the multispectral image.

[0018] Different spectral bands contain different levels of detail for different parts of the image. The particular spectral bands depend on the region of interest being imaged. Thus, the structure of the region of interest can be reconstructed separately for different spectral bands, with each reconstruction containing different levels of detail for different elements within the region of interest.

[0019] The processor reconstructs the structure by identifying structural features in the reconstructed structure of the spectral bands and merging the reconstructed structure of the spectral bands by matching the structural features of the spectral bands.

[0020] To obtain a more complete detailed reconstruction, all the reconstructions from the different spectral bands can be merged together to produce a reconstruction with increased detail relative to the separate reconstructions.

[0021] The processor further estimates a spatial position of a next image in the sequence of images based on the position of the previous image and the quasi-periodic behavior, obtains an actual spatial position of the next image, and adapts the quasi-periodic behavior based on a comparison between the estimated spatial position and the actual spatial position.

[0022] This feedback loop between the identified quasi-periodic behavior and real-world position measurements can refine the quasi-periodic behavior over time and adapt based on changes in the behavior, e.g., the frequency of the quasi-periodic behavior changes over time, e.g., as the user gets tired and performs the behavior slower or with less force.

[0023] The system may further comprise a sensor system, such as an inertial measurement unit (IMU), for obtaining system data corresponding to images in the sequence of images.

[0024] The processor removes redundancies in the system data based on comparing the system data to the quasi-periodic behavior.

[0025] The system data may be used in the reconstruction to obtain a relationship between pixel data in an image and the location of the pixel data in the reconstruction, and thus redundant system data may be eliminated to reduce the data that must be processed to determine the corresponding location of the pixel data in the reconstruction.

[0026] The present invention also provides A personal care unit and a handle; The system defined above and and the area of ​​interest is the oral cavity or a portion thereof.

[0027] The imaging device is located on the handle or on the personal care unit, for example the personal care unit can be a cleaning unit, a shaving unit, an exfoliating unit, etc.

[0028] The personal care device may be a toothbrush comprising a toothbrush head section and bristles, and the imaging device obtains oriented images of the stationary bristles.

[0029] The present invention also provides a computer-implemented method for digitally reconstructing a structure of a region of interest in an object, the computer-implemented method comprising: obtaining a sequence of images of a region of interest using an imaging device; - analyzing system data, including at least data enabling identification of a position of an imaging device relative to a region of interest, and extracting quasi-periodic behavior from the system data; using the identified quasi-periodic behavior to remove redundancy in the sequence of images and / or system data; reconstructing the structure of the region of interest using the sequence of images and the system data; The present invention provides a computer-implemented method comprising:

[0030] Removing redundancy may include identifying redundant images in the sequence of images based on the quasi-periodic behavior, and removing the redundant images from the sequence of images.

[0031] The images in the sequence of images may be multispectral images, and the structure of the region of interest is reconstructed separately for each spectral band in the multispectral image.

[0032] The method may further comprise the steps of estimating a spatial position of a next image in the sequence of images based on the position of the previous image and the quasi-periodic behavior, obtaining an actual spatial position of the next image, and adapting the quasi-periodic behavior based on a comparison between the estimated spatial position and the actual spatial position.

[0033] The invention also provides a computer program comprising computer program code which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of the above-mentioned method.

[0034] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0035] For a better understanding of the present invention, and to show more clearly how the same may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief description of the drawings]

[0036] [Figure 1] FIG. 1 illustrates an example of an oral care system, and in particular a toothbrush head. [Diagram 2] FIG. 1 shows an example of a spectral fingerprint. [Diagram 3] FIG. 1 shows the absorption spectra of several toothpastes obtained by experimental measurements. [Figure 4] FIG. 1 shows an image of teeth from a camera built into a toothbrush. [Diagram 5] FIG. 1 shows the results of hyperspectral approximation from an RGB image. [Figure 6] FIG. 1 shows a schematic of toothpaste in three different states. [Figure 7] FIG. 1 shows a method for segmenting adjuvants from an image. [Figure 8] FIG. 1 illustrates a personal care system for reconstructing the anatomical or physiological structure of oral tissue. [Figure 9] FIG. 1 illustrates exemplary localization data of a toothbrush from an IMU unit during a typical brushing session. [Figure 10] FIG. 13 shows two-dimensional force data of a toothbrush. [Figure 11] FIG. 1 illustrates a spectrogram of system data, particularly position data. [Figure 12] FIG. 1 illustrates a method for reconstructing a biological or physiological structure of a region of interest. [Figure 13] FIG. 2 shows a flowchart of an exemplary method for reconstructing an anatomical or physiological structure of a region of interest. [Figure 14] FIG. 2 shows an example of a reconstructed anatomical or physiological structure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0037] The present invention will now be described with reference to the drawings.

[0038] It should be understood that the detailed description and examples set forth herein, while illustrating exemplary embodiments of the apparatus, systems, and methods, are for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems, and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the figures are schematic only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the figures to denote the same or similar parts.

[0039] The present invention provides a system for digitally reconstructing the structure of a region of interest of an object. The system comprises an imaging device for obtaining a sequence of images of the region of interest, and a processor. The processor analyses system data, which at least conveys information about the position of the imaging device relative to the region of interest. Quasi-periodic behavior is extracted from the system data, the identified quasi-periodic behavior is used to remove redundancies in the sequence of images and / or the system data, and the sequence of images and the system data are used to reconstruct the structure of the region of interest.

[0040] The system also finds use in oral care systems, such as, but not exclusively, toothbrush heads.

[0041] The system further comprises an oral care device, e.g., comprising a handle and / or a cleaning unit. The imaging device can be located anywhere useful for observing anatomy and physiology, e.g., in the handle or cleaning unit.

[0042] The oral care device may be a toothbrush comprising a toothbrush head section and bristles, and the imaging device obtains images primarily, but not exclusively, in the direction of the stationary bristles.

[0043] Alternatively, the oral care device can be a different oral care device, such as, for example, a brushing mouthpiece, an irrigator, a brushing flossing device, a whitening mouthpiece, etc.

[0044] FIG. 1 illustrates an example of an oral care system, and in particular a toothbrush head. The toothbrush head 100 is shown having a head section 102 with bristles 104 protruding therefrom. A camera / imaging device 106 is disposed within the head section 102 and is used to collect oral cavity images 110 or video sequences during regular brushing of the teeth. The camera 106 can be, for example, an RGB camera, an imaging device operating in the visible and near infrared (VISNIR) spectrum with wavelengths ranging from about 400-1100 nm, or a miniature multispectral imaging device. A light emitting device 108 is used to illuminate the user's mouth with visible, infrared, broadband, or single frequency type illumination.

[0045] A processor 112 is used to process the sequence of collected images 110 and use an image reconstruction algorithm to generate an image without the toothpaste. The processor 112 segments the toothpaste 111 from the images 110 and the reconstruction algorithm patches / stitches the images 110 to reconstruct an image of the user's mouth.

[0046] Image 110 is decomposed into red (R), blue (B) and green (G) channel images, which separates each image into three spectral components that are used to color cluster pixels (or groups of pixels) in image 110. Color clustering of these visible spectral bands results in the spectral components of image 110.

[0047] If the images 110 are VISNIR images, they are decomposed into further spectral bands that include wavelengths in the near infrared range. Resolving discrete bands of near infrared wavelengths is sometimes called density clustering (as opposed to color clustering for the visible spectrum). Density clustering of the near infrared (NIR) spectrum yields the spectral content of the images 110.

[0048] If the images 110 are hyperspectral images, they are decomposed from a hyperspectral cube, where the hyperspectral cube gives the spectral components of the images 110.

[0049] Reference spectral measurements 114 of the toothpaste being used are taken at an early point in the cleaning session. The spectral measurements 114 are used to obtain a spectral fingerprint of the toothpaste. The spectral fingerprint enables the processor 112 to identify and segment the toothpaste from the image 110.

[0050] The spectral measurements 114 are updated over time during the event (i.e., brushing the teeth). The images 110 are used to update the spectral measurements 114 so that the spectral fingerprint of the toothbrush can also be updated.

[0051] Alternatively, a database 116 containing the spectral fingerprints of various toothpastes may be maintained online or offline and this data may be retrieved by the processor 112 to identify and segment toothpastes from images.

[0052] Thus, by using spectral fingerprinting, areas occluded by toothpaste can be recognized and segmented from unoccluded image regions using indicative spectral information about the toothpaste measured by the on-board camera 106 before brushing and / or provided by a reference database 116 of previously used or tested toothpastes.

[0053] The processor 112 also outputs a reconstructed map of the anatomical or physiological structure 118 (shown in FIG. 1 as a single tooth) and also outputs at least an occluded image 120. The reconstruction step is described in more detail below.

[0054] Figure 2 shows examples of spectral fingerprints 206 and 208. Two main methods for obtaining spectral fingerprints are described. The first method uses a hyperspectral cube 202, while the second method uses a discrete plane 204 in the hyperspectral space. Hyperspectral space defines a space with two spatial dimensions (x, y), while the z-axis is the wavelength of light.

[0055] In theory, a hyperspectral cube is a continuous cube that contains data for all wavelengths within the cube. In practice, however, continuous measurements for all wavelengths within the cube are impractical or impossible. Therefore, in this context, a hyperspectral cube generally refers to multiple planes (i.e., images) stacked in the z-direction of hyperspectral space. These are sometimes referred to as multispectral cubes.

[0056] For the first method, the collected spectral data of the adjuvant is contained in a hyperspectral cube 202. A plane through the hyperspectral cube 202 at a particular wavelength represents the image seen under such light illumination. A fingerprinting algorithm can then use the entire hyperspectral cube 202 for the adjuvant to the target and segment the adjuvant from the image.

[0057] Statistically significant frequencies that characterize the adjuvant of interest are extracted. This is done by averaging the spectral profiles for all slices of the hyperspectral cube 202, resulting in a curve such as that shown in graph 206.

[0058] Such a process can be performed using a variety of mathematical methods ranging from the classic linear Principal Component Analysis (PCA) to more advanced Independent Component Analysis (ICA) or AI-based methods such as autoencoders and their variants. Additionally, a feature selection pass is added to more reliably extract the most prominent features that contribute to the spectral fingerprint.

[0059] Graph 206 shows an example of a spectral fingerprint for toothpaste. The mean is shown as the solid center line, and the variance is shown as two lines above and below the solid center line. The fingerprint shows the spectrum of a quantity of toothpaste placed on a toothbrush head and collected by an on-board camera. The solid center line represents the average PCA results from a collected hyperspectral cube spanning wavelengths (x-axis) from approximately 400 nm to 1200 nm. Since this measurement is taken directly from imaging the toothpaste, the measurement primarily represents reflectance and scattering from the toothpaste surface.

[0060] The second method is similar to the first method for obtaining a spectral fingerprint. However, the second method uses a relatively small number of discrete planes. For example, this is achieved by separating the image into three separate red, green and blue channels. The discrete planes shown in image 204 are shown to model four discrete spectral components: red (about 420 nm), green (about 550 nm), blue (about 620 nm) and near infrared (about 1050 nm).

[0061] Similarly, for the hyperspectral cube 202, each discrete plane in the image 204 can be averaged, thereby generating color clusters 208 indicative of relative (average) wavelength intensity values.

[0062] The y-axis shows wavelength relative intensity values ​​for both the graph 206 and the color cluster 208 .

[0063] These methods rely on averaging planes corresponding to different wavelengths to obtain a spectral fingerprint. It will therefore be appreciated that the images used to generate the discrete planes in the hyperspectral cube 202 or 204 preferably contain only (or mostly contain) such ancillary materials to reduce unnecessary data.

[0064] For example, when images from a camera built into the head of a toothbrush are used to determine the spectral fingerprint of a toothpaste, the bristles need to be segmented and removed (or filtered out of the spectral data). Similarly, when using lighting devices such as LEDs (light emitting diodes), the glare on the surface of the toothpaste from the lighting also needs to be removed.

[0065] The spectral fingerprints 206 or 208 may then be stored in temporary memory (eg, a look-up table) for use in segmenting the adjuvant from the image, or may be stored permanently (eg, in a database).

[0066] To allow for a database-based definition modality, some examples of toothpaste fingerprints can be used.

[0067] Figure 3 shows the absorption spectra of several toothpastes obtained by experimental measurements, where the measurements were taken when the toothpastes were in the form of a diluted slurry according to ISO (International Organization for Standardization) 11609:2010.

[0068] The toothpaste was diluted in a standard diluent in a 1:5 ratio with vigorous mechanical mixing and stirring until a homogenous slurry medium was formed (usually about 30 seconds). The amount of toothpaste was 1 g as determined by a standard "whole head" amount. In the experiments, the toothpastes had Relative Dentin Abrasivity (RDA) values ​​ranging from about 40 to 145.

[0069] The toothpastes used were layered on the toothbrush head as in normal user operation. A pre-calibrated spectroscopic device was used to obtain the reference absorption spectra. Graph 302 shows the first derivative of the pre-processed absorption spectra from multiple toothpastes. Pre-processing involves Savitzky-Golay smoothing of the absorption spectra.

[0070] Graph 304 shows the raw absorbance spectrum of several toothpaste slurries across the entire spectrum (400-1100 nm). Because the measurements are taken from toothpaste slurries, the measurements primarily represent the absorbance of the bulk of the toothpaste slurry. For this reason, the peak in reflectance shown in graph 206 of FIG. 2 at about 600 nm corresponds to a drop in reflectance at 600 nm in graph 304.

[0071] As can be seen, the spectral fingerprint in graph 304 exhibits both optical absorption characteristics (which produce color) and optical scattering characteristics (which tend to blur the image beneath diluted toothpaste). In particular, it should be noted that toothpaste exhibits high absorption (and high scattering) in the NIR spectral band (approximately 700-1200 nm).

[0072] The absorption properties of toothpaste are generally more useful for color clustering (or other segmentation techniques) because the absorbance of the toothpaste remains fairly constant as the structure of the toothpaste changes, while the scattering properties can change more dramatically. Of course, whether the absorption or scattering properties of the adjuvant are used as the spectral fingerprint depends on the adjuvant itself and whether the scattering and / or absorption properties change over time.

[0073] Both images 306 and 308 show a standard model of a mandible (typodont) with teeth covered in toothpaste slurry obtained by applying a standard brushing procedure with toothpaste and water using an electric toothbrush. Image 306 was obtained using natural lighting, while image 308 was obtained using IR lighting at about 850 nm. Area 310 shown in both images 306 and 308 shows that there is a clear blur in IR image 308 that is not present in image 306, which is naturally lit. The blur is caused by the presence of diluted toothpaste in area 310 absorbing and scattering IR light, which results in a blur in the IR image.

[0074] To better understand the color clusters (or other spectral components) of the image, it is advantageous to have knowledge of the absorption spectra of the different anatomical or physiological structures and components of the image, as well as the spectral fingerprints of the adjuvant. Therefore, the collected spectra were compared to spectra found in the scientific literature for oral tissues.

[0075] According to the literature and scientific publications, for example, human tooth enamel is highly transparent in the NIR band. In fact, the scattering of tooth enamel is very low at NIR frequencies (meaning that infrared light has good penetration in the outermost layer of the tooth). Tooth enamel has been shown to be nearly transparent to NIR light (1053 nm).

[0076] Related literature including spectra for other oral tissues includes Hirmer, M. et al., "Spectroscopic Study of Human Teeth and Blood from Visible to Terahertz Frequencies for Clinical Diagnosis of Dental Pulp Vitality," J Infrared Milli Terahz Waves 33, 366-375 (2012); Daniel Fried, RE (1995) "Nature of light scattering in dental enamel and dentin at visible and near-infrared wavelengths," Appl. Opt., 34, 1278-1285; and Zandona, AF (2019) "Detection and Assessment of Dental Caries," Springer.

[0077] This penetration characteristic of NIR for enamel also exists at about 655 nm, and it is an approximately continuous function of light wavelength up to about 1300 nm. Furthermore, the characteristic spectrum of the gingiva is also used to better segment it using clustering.

[0078] The near infrared spectrum (approximately 700-1100 nm) is particularly advantageous for identifying and segmenting toothpaste within an image because tooth enamel is nearly transparent in the NIR spectrum, while gums exhibit relatively large absorbance. Specifically, toothpaste has an almost unique signature as a feature of low reflectance and high absorbance in the IR spectrum. As a result of the low reflectance and high absorbance (and scattering), camera images of toothpaste at IR frequencies are generally very dark (i.e., relatively low pixel values).

[0079] This can be seen in area 310 of IR image 308, as the gum line appears darker in the right portion of IR image 308 compared to the toothpaste-filled area visible in the left portion of natural light image 306. A similar effect occurs in the interdental spaces where diluted toothpaste is also present. Thus, IR image 308 can be further used to detect the presence of toothpaste by detecting the relative dimming of the image. Images (or portions of images) where there is no or little toothpaste can thus be identified.

[0080] In other cases or personal care applications, particularly advantageous wavelengths will depend on the user's particular area of ​​concern (eg, mouth, skin, hair, etc.) and the adjunct in question.

[0081] 4 shows an image 402 of teeth from a camera built into a toothbrush. Image 402 contains diluted toothpaste slurry on the teeth, particularly in the interdental spaces. The toothpaste has been segmented and removed from image 402. The result is shown in image 404, where the areas in image 402 that contained toothpaste have been segmented and removed. In this case, the toothpaste was segmented by comparing the color clusters for each pixel to the spectral fingerprint of the toothpaste in question.

[0082] Next, various embodiments for segmenting adjuncts from an image are described. In particular, the embodiments refer to toothpaste as an adjunct. However, similar embodiments are valid for other regions of interest and other adjuncts (e.g., shaving cream, skin cream, skin lotion, etc.).

[0083] In a first embodiment, the VIS and NIR images are clustered separately. An algorithm is used to temporally / longitudinally reconstruct oral structures from both the VIS (approximately 400-700 nm) and NIR (approximately 700-1100 nm) images separately. The algorithm leverages color clustering and simple geometric feature detection to identify and collect clean patches (i.e., patches where toothpaste is absent or minimal) from the VIS and NIR images. The collected patches are then located (e.g., using on-board location hardware and / or from the geometric features of the images) and used to reconstruct the entire oral structure by stitching the toothpaste-free images.

[0084] More specifically, the first embodiment involves the following steps. The first step involves defining / obtaining a spectral fingerprint of the adjunct before the start of a brushing session: a camera on the toothbrush identifies when toothpaste is placed on the bristles of the toothbrush.

[0085] The toothpaste placement process is identified by examining one or more of the following: a. The presence of a new object (toothpaste) in the camera's field of view, b. A signal from the location hardware indicating that the user has picked up the toothbrush; or c. User-initiated initiation of a brushing session (eg, pressing a start button).

[0086] The spectral fingerprint is obtained by one or more of the following: i. Collecting one or more initial adjuvant images to establish a spectral fingerprint thereof. ii. Spectral data corroborated from a previous brushing session; or iii. Pre-loaded / defined databases or lookup tables.

[0087] The second step involves collecting the approximate location of the collected images from an inertial measurement unit (or any other position determining device or method).

[0088] The third step involves segmenting the toothpaste from the image by pixel-wise color clustering and identifying patches in the image that are free of toothpaste by matching the color clusters to the spectral fingerprint of the toothpaste.

[0089] Comparison of the spectral content (i.e., color clusters) of the image with the spectral signature of the toothpaste is performed using standard computational geometry algorithms, such as curve comparison algorithms (e.g., Frechet distance, Dynamic Time Warping, or machine learning based algorithms for matching the two spectroscopic fingerprints).

[0090] The final step involves reconstructing the oral structures (geometry, anatomy, physiology) using vertical stitching. At the end of the collection process (e.g., at the end of the brushing session, or possibly during the brushing session, depending on available computational resources), the complete set of collected toothpaste-free patches is used to reconstruct the entire oral anatomical or physiological structure of interest (e.g., teeth, gums, etc.) using the system data.

[0091] If present, areas without patch images can be filled in with data from reference or historical images, for example, from previous brushing sessions. Within a reasonably short amount of time (i.e., one or a few brushing sessions), the entirety of the teeth and gums is reconstructed using only the collected segmented patches.

[0092] This entire reconstruction process is performed on the VIS and NIR image sets separately. Each of the VIS and NIR images generally exhibits different salient geometric features (depending on how light interacts with the oral structures), and thus having both gives two sources of oral structures with slightly different but complementary characteristics.

[0093] In a second embodiment, VIS and NIR images are fused. The fused images generally give better images (i.e., with more, better defined and contrasted details) than the individual VIS and NIR images. These new fused images are then used for the reconstruction process. This procedure is advantageous because the VIS and NIR images have complementary information for the same region of interest (ROI). Complementary information means that details (e.g., type and location of features in the image) that appear in one type of image (i.e., VIS or NIR) may not appear in the other type, and therefore the fused image contains details from both types.

[0094] A multi-scale transformation technique is used to fuse the two images together. Some silicon image sensors are sensitive in the IR band and in the VIS spectrum. Therefore, one of these sensors can be used without an IR filter. Simultaneous VIS and NIR images are collected. The VIS image contains a full RGB image capture, and the NIR image contains the NIR image (approximately 700-1100 nm).

[0095] Alternatively, the NIR image includes a single, relatively narrow IR band, for example, a narrowband IR diode or an IR laser (having a single wavelength) can be used to illuminate the ROI.

[0096] Two images of the same ROI can then be fused together by a non-subsampled contourlet transform built on top of a non-subsampled pyramid and a non-subsampled directional filter bank that splits the image into a base layer and a detail layer. This process allows for improved visual quality of the base layer of the fused image and improved edge detail features and sharpness in the detail layer of the fused image.

[0097] In a third embodiment, the image is a hyperspectral image, either received from a hyperspectral imaging device or reconstructed in software from VIS or VISNIR images, which provides more detail for improved identification and segmentation of toothpaste ingredients within the image.

[0098] In this case, the spectral content of the image includes multiple hyperspectral components, i.e., the spectral content is a continuous function of wavelength versus relative intensity.

[0099] Currently, some hyperspectral imaging sensors may be physically too large for a particular ROI. Therefore, it may be advantageous to reconstruct a hyperspectral image from a VIS or VISNIR image in software. When using a VIS or VISNIR image to reconstruct a hyperspectral image, an algorithm (e.g., AI or machine learning based) is used to create a hyperspectral approximation of the original VIS or VISNIR image.

[0100] A machine learning based algorithm (e.g., a sparse coding algorithm) can be trained using a set of general hyperspectral images (e.g., of general real-world scenes) or using a set of specific hyperspectral images (e.g., of oral cavity structures). The images used to train the machine learning based algorithm are preferably obtained using a hyperspectral imaging sensor.

[0101] Alternatively, deep learning algorithms such as U-Net or autoencoder architectures are also used.

[0102] FIG. 5 shows the result of a hyperspectral approximation from an RGB image 500. The original RGB image 500 is from an RGB camera built into the head section of a toothbrush. The original RGB image 500 shows two teeth (top-down view) surrounded by highly diluted, foamed toothpaste during a real brushing session on a typodont. Image 502 is a slice at 440 nm of the hyperspectral approximation obtained from the original RGB image 500. Similarly, image 504 is a slice at 550 nm and image 506 is a slice at 620 nm.

[0103] The hyperspectral approximation was obtained using a sparse coding algorithm. Many other spectral bands may be, and have been, obtained from the hyperspectral approximation.

[0104] In a third hyperspectral embodiment, the color clustering-based segmentation using spectral fingerprints can be replaced with a hyperspectral-based segmentation also using spectral fingerprints. Thus, the hyperspectral-based segmentation results in cleaner patches (i.e., patches with less and more accurately represented toothpaste) from the image.

[0105] In a fourth embodiment, only the NIR spectral band is used: instead of using the entire multispectral set (i.e., VIS and NIR spectra or a broader spectrum), only frequencies related to the NIR spectrum are used, although the abrasive ingredients in common commercial toothpastes (e.g., silicic acid) have a unique IR spectral signature.

[0106] In this case, the obtained images are also subjected to hyperspectral approximation (in the IR range of about 650-1100 nm) to obtain a NIR-based hyperspectral image.

[0107] Segmentation is then performed, for example, by density clustering NIR spectral bands that are known to be unique in the spectral fingerprint of silica, and thus the presence of toothpaste in the collected image can be identified from the NIR spectrum alone.

[0108] Segmentation involves discarding images that contain any given amount of toothpaste residue. This method may require many more brushing sessions before a complete structure can be reconstructed. However, this method also reduces processing requirements and is sufficient for many use cases.

[0109] In a fifth embodiment, the spectral fingerprint is continuously updated. In this embodiment, the step of defining the spectral fingerprint is continuously repeated during the user's cleaning session, which leads to a periodic update of the target spectral fingerprint of the auxiliary agent. Thus, as the auxiliary agent changes over time, the spectral fingerprint can be updated to allow identification of the auxiliary agent in the collected images, regardless of the state change.

[0110] This makes it possible to track changes in the spectral fingerprint of the auxiliary agent. More particularly, continuously updating the spectral fingerprint consists of a periodic update of the target spectral fingerprint of the auxiliary agent by continuously monitoring the auxiliary agent.

[0111] Figure 6 shows a schematic diagram of toothpaste in three different states. Figure 6(a) shows the toothpaste in its initial state as a paste 612. The paste 612 is generally placed on the bristles 604 of the toothbrush head section 602. In this state, a camera 608 obtains an initial spectral fingerprint by illuminating the paste 612 with an illumination device 610.

[0112] Figure 6(b) shows the toothpaste as a slurry 616 around two teeth 614. The paste 612 shown in Figure 6(a) becomes a slurry 616 after it is mixed with a reference diluent (in experiments on typodons) or, indeed, with saliva in the user's mouth and possibly some moisture from the bristles 604 (if the bristles are pre-wetted before or after the toothpaste is applied). Figure 6(c) shows the toothpaste as a foamed slurry 618 around two teeth 614. The foamed slurry 618 results when the slurry 616 shown in Figure 6(b) is foamed with vigorous mechanical forces; in some cases, foam forms from chemicals within the toothpaste or whitening substances (e.g. peroxide).

[0113] In order to store continuously updated spectral fingerprints, a memory module is used to store the spectral fingerprints and update them by modifying the data of the stored spectral fingerprints when required.

[0114] A linear mixed model is used to estimate the endmembers of the spectral fingerprint of the adjuvant within the collected image pixels, where only pixels containing the adjuvant are monitored to detect changes in the spectral content of such pixels over time.

[0115] A threshold mechanism can be used to detect changes in the spectral content of the auxiliary pixel. If the spectral content exceeds a given threshold, the spectral fingerprint can be updated based on the new detected spectral content. In this regard, the spectral fingerprint is not continuously updated, but the relevant spectral content is continuously monitored and the spectral fingerprint is updated as needed and when required.

[0116] 7 shows a method for segmenting an adjuvant from an image. The method includes, in step 702, obtaining an image of an area of ​​interest. The image may be an RGB image in the visible spectrum (or other type of image in the visible spectrum), a near-infrared image, a hyperspectral image, an approximated hyperspectral image, a multispectral image, etc. The camera is configured to capture at least one distinct spectral band (e.g., a conventional camera captures three bands, i.e., red, green, and blue). The camera may be configured to capture visible light, infrared (or near-infrared) light. Alternatively, the camera may be a multi- or hyperspectral camera.

[0117] The method further comprises obtaining a spectral signature of the auxiliary agent on the region of interest in step 704. The spectral signature comprises at least one spectral component of the auxiliary agent. The spectral signature is a continuous function of the spectral components or a set of discrete spectral components. The spectral components indicate the relative intensity between different wavelengths.

[0118] The method further comprises identifying spectral components in the image in step 706. The spectral components are identified by performing pixel-by-pixel (or group-by-group) color clustering on the RGB image. Alternatively, a hyperspectral cube (i.e., a hyperspectral image) can be generated from the image by a hyperspectral approximation or from an image obtained from a hyperspectral camera.

[0119] Hyperspectral imaging collects information over a broad spectrum of wavelengths such that any object should have a unique spectral signature. Thus, any object can be identified. Of course, if specific unique spectral bands for an adjuvant are known, images for these specific spectral bands can be obtained instead of hyperspectral imaging (e.g., NIR spectral bands for toothpaste).

[0120] Each pixel in the image corresponds to a line in the hyperspectral cube parallel to the wavelength axis at the corresponding pixel location in the position plane (x,y). A curve can therefore be obtained from this line by plotting the relative intensity at each point on the line against wavelength. This curve contains the spectral content of that pixel.

[0121] Thus, the adjuvant can be segmented from the image by comparing the spectral signature of the adjuvant to the spectral content of the pixels of the image. If the spectral content of a pixel (or group of pixels) in the image is sufficiently similar to the corresponding spectral content in the adjuvant's spectral fingerprint, then the pixel can be segmented as the adjuvant. For example, for the spectral content to be sufficiently similar, all pixel spectral content must be within a threshold percentage difference (e.g., within 5%) to the adjuvant spectral content.

[0122] Segmentation of the ancillary matter from the image is particularly advantageous for reconstruction of the anatomical or physiological structure of the ROI. A method for reconstructing the anatomical or physiological structure is now described.

[0123] Reconstructing an anatomical or physiological structure involves obtaining a sequence of images of the entire (or a substantial portion) of the anatomical or physiological structure, and also involves obtaining system data indicating the position of the images relative to at least the location of the anatomical or physiological structure.

[0124] In personal care, it is common to repeat routine actions at regular intervals, such as brushing one's teeth or shaving regularly, to maintain effective care. Humans are very bad at repeating the exact same action or sequence of actions, so such repetition rarely results in the exact same outcome, or a person rarely performs the exact same features in the same order.

[0125] However, the variability in some body structures (e.g., oral cavity, facial skin, etc.) is limited enough due to their geometric constraints to produce a kind of involuntary near-repetition during routine care actions. For example, brushing actions performed daily on the same teeth generally lead to similar action sequences over time.

[0126] Thus, two components of such routine care actions that can be considered are variability due to human interactions and quasi-regularity due to the limited scope of the action.We therefore propose to use knowledge of quasi-regularity in personal care actions to improve the reconstruction process.

[0127] 8 illustrates a personal care system for reconstructing anatomical or physiological structures of oral tissues. The personal care system includes a toothbrush 800 having a head section 802 and bristles 804 protruding from the head section 802. A camera 806 is used to obtain images of the inside of a user's mouth, and a lighting device 808 is used to illuminate the mouth. Additionally, a group of one or more sensors (i.e., a sensor system) is used to obtain system data 814 that conveys at least positional information (e.g., including location and orientation data) corresponding to the images obtained from the camera 806.

[0128] The camera 806 may be an RGB camera or imaging device operating in the VISNIR band (approximately 400-1100 nm) as described above. The camera 806 is used to collect a sequence of images 812 (e.g., individual images or videos) during routine care (i.e., brushing the teeth). Alternatively, the camera 806 may be a miniature multispectral camera or equivalent device capable of outputting a hyperspectral image (hyperspectral cube).

[0129] The sensor system may be an inertial measurement unit (IMU). A continuous flow of data is obtained from the IMU. The IMU may include one or more of an accelerometer, a gyroscope, a compass / magnetometer, a force sensor, a motor current sensor, and a temperature sensor. Alternatively, the sensor system may be any dynamic monitoring device. The sensor system is for generating location, orientation, or brushing motion and force data 814.

[0130] Processor 816 is used to process sequence of images 812. For example, sequence of images 812 is a sequence of unobstructed collected images from which adjuvants have been segmented and removed in the manner described above. Processor 816 analyzes system data 814 to extract quasi-periodic behavior in the user's handling of toothbrush 800.

[0131] Optionally, processor 816 also performs quality screening of sequence of images 812 to remove incomplete, corrupted, out-of-focus or blurry images from sequence 812 .

[0132] Further, the processor 816 is configured to reconstruct the user's oral / anatomical or physiological structures using the sequence of images 812 and the system data 814. For example, the system data 814 includes detailed camera localization data for each image in the sequence 812 using a hyperspectral rigid structure from motion (RSfM) algorithm.

[0133] Including the quasi-periodicity information (i.e., quasi-periodic behavior) allows a processor to remove redundancy from data used during reconstruction (i.e., from the sequence of images 812 or from the system data 814). For example, in Figure 8, redundancy is removed from sequence 812 and a new, less redundant sequence 818 can be used in the reconstruction. Similarly, the system data 814 also contains redundancy that can be removed using the quasi-periodic behavior.

[0134] FIG. 9 shows exemplary location data of a toothbrush from an IMU unit during a normal brushing session. In particular, location data 900 shows the x-component (y-axis) of an accelerometer over time (x-axis). Location data 902 is an enlarged section of location data 900. Exemplary quasi-periodic behavior is indicated by arrows 904, 906, and 908. Similarly, FIG. 10 shows two-dimensional force data of a toothbrush. Data 1002 shows the x-component (y-axis) of a gyroscope in the toothbrush over time (x-axis), and data 1004 shows the force (y-axis) from a force sensor in the toothbrush over time (x-axis). As can be seen from FIG. 9 and FIG. 10, the location and force data can show quasi-periodic behaviors, which can be extracted from the data itself. Of course, the y- and z-components of the accelerometer and gyroscope are also used in the system data. For example, a 9DOF (degrees of freedom) IMU is used. More degrees of freedom may also be available. The location data, force data and / or gyroscope data form part of the so-called "system data", which includes at least data that allows localization of images relative to each other and / or relative to biological regions of interest (e.g. mouth, skin, head, etc.).

[0135] System data can be analyzed to extract similarities from local histories of system data (ie, data from the same toothbrush).

[0136] Periodic behavior (with period ω) can be written as a periodic function f(x+ω)=f(x). Thus, the first linear quasi-periodic analysis utilizes signal peaks in the system data to identify approximate linear reoccurrences of the quasi-period ω in the equation f(x+ω)=Af(x)+B, where A and B are real constants.

[0137] For example, returning to plot 902 in FIG. 9, the peaks are approximately constant, and therefore a real constant A can be assumed to be approximately 1. In this case, the quasi-periodic function can be used approximately in the equation |f(x+ω)-f(x)|=B, where the real constant B acts as a kind of threshold. In plot 902, for ease of calculation, only the peaks are used and compared. Arrows 904, 906 and 908 in FIG. 9 show the correspondence between peaks of similar height. Of course, the more complex the calculation, the more accurate the analysis of the quasi-periodic behavior.

[0138] Optionally, the quasi-periodic behavior (ie, the quasi-periodic function) is iteratively updated by a discrete Fourier transform (DFT) or a short-time Fourier transform (STFT).

[0139] This can be done by setting a starting threshold for identifying peaks in the coefficients of the DFT (or optionally STFT) of the system data. An initial approximation of the frequency (i.e., the quasi-period ω) can be found based on previously identified peaks in the DFT coefficients that are greater than the threshold. The amplitude of the frequency can be found by solving DFT(Qf)=DFT(f), where f is the frequency and Qf is the approximate frequency of the quasi-period. In particular, the following steps are followed: i. Set a starting threshold for collecting peaks in the coefficients of the DFT (or optionally STFT) of the IMU data. ii. Find an initial approximation of the frequency (quasi-periodic, Qf) starting from a peak in the DFT that is greater than a threshold. iii. Find the amplitudes of the frequencies found in the previous step by solving DFT(Qf) = DFT(f). iv. Refine all frequencies and amplitudes simultaneously of the current quasi-periodic approximation of f(x+ω) by solving DFT(Qf)=DFT(f). v. Perform a DFT of the input signal - the current quasi-periodic approximation f(x+ω) obtained in step iv, reduce the threshold and go back to step ii.

[0140] Therefore, all frequencies and amplitudes of the now quasi-periodic approximation f(x+ω) can be simultaneously refined by solving DFT(Qf)=DFT(f).

[0141] To iteratively improve the quasi-periodic function, a difference is performed between the input signal (i.e., system data) and the current quasi-periodic function, and the DFT of the difference is found. Based on the DFT of the difference, the threshold value previously used to identify the peak is updated for the next iteration.

[0142] Figure 11 shows a spectrogram of the system data. Plot 1102 shows the spectrogram obtained via a DFT of the system data, with time (seconds) on the x-axis, frequency (Hz) on the y-axis, and relative intensity (arbitrary units) on the z-axis. Plot 1104 shows the coefficients of spectrogram 1102, with frequency (Hz) on the x-axis, and relative intensity (arbitrary units) on the y-axis. Plot 1106 shows the phase of spectrogram 1102.

[0143] The reconstruction of a 3D map of an anatomical or physiological structure is typically performed using detailed camera localization data corresponding to a series of images taken of a user's anatomy (e.g., oral cavity, facial skin, etc.). These images are taken from approximately the same area multiple times during a time interval (e.g., day, week, etc.). Thus, there is some redundancy in the acquired data that can be exploited in the geometry reconstruction process.

[0144] The reconstruction process can exploit the quasi-periodic behavior of the user to accelerate the RSfM algorithm, for example, by identifying similar sequences of movements that lead to the collection of similar data (i.e., similar images and / or similar system data). In this case, pre-computed values ​​for the RSfM algorithm (e.g., approximate 3D mesh components, or more detailed volumes of the structure) can be reused.

[0145] Similarly, the quasi-periodic behavior can be used to identify similar images in a collected sequence of images, which may allow the RSfM algorithm to skip processing all images if some images are sufficiently similar to previously processed images that they can be replaced with the previously processed image results.

[0146] Furthermore, the use of hyperspectral (or multispectral) images in the reconstruction improves the localization accuracy of the images, as these images have multiple frequency bands available, resulting in the reconstruction of much more detailed 3D maps from sequences extracted from the hyperspectral hypercube.

[0147] The hyperspectral images also have similarities to previous hyperspectral images that have already been processed, leading to a reduction in the number of images that need to be processed.

[0148] Furthermore, hyperspectral images provide more opportunities for the elimination of redundancies in the hyperspectral images by exploiting the quasi-periodic behavior.

[0149] 12 shows a method for reconstructing biological or physiological structures of a region of interest, which involves obtaining system data (e.g., force, motion and / or direction data that provide at least relative position information) in step 1202 and extracting quasi-periodic behavior from the system data in step 1204.

[0150] Further, a hyperspectral image is collected in step 1206. The hyperspectral image is pre-processed on the fly to reduce the computational cost of the reconstruction. Pre-processing involves filtering out images with lower or insufficient quality parameters in step 1208. For example, images with incomplete frames, images with fully or mostly obstructed fields of view, grainy images, or images that are out of focus are filtered out.

[0151] If a prototypical similar image is available, this similar prototypical image can be used to compare the current image using a multiscale structural similarity index / measure (MS-SSIM).

[0152] In the absence of a sufficiently similar image to use as a reference, for example, blind image quality algorithms (e.g., Blind / Referenceless Image Spatial Quality Evaluator (BRISQUE), High Order Statistics Aggregation (HOSA), KonCept512, etc.) may be applied to determine the quality of an image from a set of geometric features extracted from the image.

[0153] Furthermore, the method further comprises detecting similar hyperspectral images in step 1210. This is accomplished using a no-reference hyperspectral algorithm. For example, a differential hashing algorithm can be applied to all bands of the hyperspectral image. These results are then stored locally or remotely for comparison with a list of previously collected images to find if there are previous images with a high similarity to the current image. The set of images to compare can be local and / or remote (e.g., cloud-based). The comparison of the current hash with the stored hashes is performed using an appropriate metric (e.g., Hamming distance, Jaro distance, Monge-Elkan distance, etc.) for all hyperspectral bands, and then the total calculated similarity (i.e., the sum of all band similarities) is calculated as the final score.

[0154] Detecting similar hyperspectral images in step 1210 is accomplished using a reference hyperspectral algorithm. The reference hyperspectral algorithm follows similar steps as the non-reference algorithm, except that the comparison is between the current image and a comparison set (local or remote, as described above). The comparison is performed by using a multi-scale structural similarity index / measure (MS-SSIM). The comparison is also performed for all hyperspectral bands and then summed to obtain a final score.

[0155] If one or more multispectral images similar to the current image are detected, information from the image processing of the previous image (e.g., parameters, results obtained, etc.) is made available to the processing pipeline in order to accelerate or improve the quality of the final result.

[0156] The images also go through an improved localization process (not shown in FIG. 12) that improves the accuracy of the image localization. An initial anatomical or physiological structure is defined at time t=0. The initial anatomical or physiological structure may be defined by a so-called "bootstrapping" process, where a first batch of images collected at different reference locations (i.e., teeth / gingival quadrants, areas for facial skin, etc.) are considered as an initial reference anatomical or physiological structure that includes anatomical and biological features.

[0157] The initial anatomical or physiological structure may also be defined by a so-called "dictionary" process in which a standard medical image dictionary-based set of images is used along with input from the user. For example, the user may select a feature of the anatomical or physiological structure, or a number of features (e.g., number of teeth, number of missing teeth, presence of implants, beard style, skin characteristics, etc.).

[0158] Alternatively, the initial anatomical or physiological structures may be defined using a so-called "clean acquisition" process, in which an independent set of images of the anatomical or physiological structures is collected before (or after) a personal care session in a controlled environment (e.g., uniform lighting, unobstructed, etc.).

[0159] To identify the image from the initial anatomical or physiological structure that most closely matches the current image according to the number of matching scale-invariant feature transform (SIFT) features, a voting scheme is used that identifies the maximum number of matches. Thus, the location in the reference image that most closely matches the current image becomes the approximate location of the current image.

[0160] Since user behavior is generally unpredictable, the camera motion can be estimated between the current image and the reference image. To estimate the camera motion, the view difference between the current image and some (N) closest matching reference images (N is a small integer) can be expressed as a combined act of rotation and translation. To define such a geometric transformation, the 8-point algorithm can be used.

[0161] To improve localization, we integrate the geometric features obtained from the 8-point algorithm with the motion estimates obtained from the quasi-periodic behavior, which can be considered as an approximation of the user motion and can therefore be used to predict the device location in the near future.

[0162] Typically, the set of possible candidate reference images for matching with the current image contains a fairly large number of outliers due to the high variability of the collection process (e.g., mostly user motion during brushing). To reduce such uncertainty (i.e., reduce the number of outliers), additional geometric constraints obtained from the images and motion constraints obtained from the system data can be used. For example, before searching for the closest reference image, a random sample consensus (RANSAC) algorithm can be used to reduce the number of outliers, so that only the "cleanest" reference images are output.

[0163] The RANSAC algorithm can be modified in the search phase to include the quasi-periodic behavior of the user motion and its derivatives in order to speed up the process of identifying outliers.

[0164] During image localization, we assume that the intrinsic parameters of the camera are known so that the so-called fundamental matrix F of the camera can be estimated.

[0165] The method further comprises inputting the hyperspectral image and the system data into a rigid structure from motion (RSfM) algorithm in step 1212 to perform reconstruction of the anatomical or physiological structure.

[0166] Problems arise when associating different hyperspectral bands with the same 3D model, as details disappear from one band but not the other: for example, red features may be very visible in the red band, but disappear in the deep blue or ultraviolet bands.

[0167] Thus, by applying the RSfM algorithm separately to each spectral band of a hyperspectral image, multiple limited, partial 3D model reconstructions with potentially missing details are obtained. The reconstructions can be point sets.

[0168] In step 1214, structural similarity features for each point set corresponding to different spectral bands are defined using the structural component of MS-SSIM. In step 1216, high-level structural features between different point sets of spectral bands are matched, so that the different point sets can be merged together to obtain a more detailed 3D reconstruction of the anatomical or physiological structure.

[0169] Furthermore, the RSfM algorithm and the quasi-periodic behavior may form a feedback loop 1211 in which the difference between the position of the current image calculated by the RSfM algorithm and the predicted position from the quasi-periodic behavior is fed back into the analysis of the quasi-periodic behavior to optimize the search parameters. Alternatively, the prediction error may be fed back, including the MS-SSIM value for the current image and the difference between the positions. The higher the MS-SSIM value (which defines the difference between the current image and the reference image), the higher the difference between the positions is likely to be.

[0170] 13 shows a flow chart for an exemplary method of reconstructing an anatomical or physiological structure of a region of interest. The method generally begins in step 1300 when a user initiates a personal care activity (e.g., brushing teeth) or event. In this case, in step 1302, VIS and NIR images are collected and separately localized into a localized VIS image 1306 and a localized NIR image 1308. The images are localized using system data 1304 obtained, for example, from an IMU on a personal care device (e.g., toothbrush, shaver, etc.).

[0171] The localized VIS image 1306 is subjected to a color clustering process 1310 and the localized NIR image 1308 is subjected to a density clustering process 1312 to obtain the spectral components from the image. The spectral components and the image are passed through a segmentation process 1314 which removes the adjuncts from the image, resulting in localized segmented image patches 1316.

[0172] The initial anatomical or physiological structures 1318 include a set of reference images of the region of interest (e.g., acquired at time t = 0). The reference images can be localized using the system data 1304 to obtain localized anatomical or physiological structures 1320.

[0173] The localized anatomical or physiological structure 1320 can be used in an improved localization process 1322 to improve the accuracy of the localization of the segmented patch. In an anatomical or physiological structure reconstruction process 1326 (e.g., an RSfM algorithm), the improved localized segmented patch 1324 is used to output a better localized reconstructed anatomical or physiological structure 1328 of the region of interest. The reconstructed anatomical or physiological structure 1328 is fed back to the localized anatomical or physiological structure 1320 to improve its accuracy and further improve the improved localization process 1322.

[0174] The reconstructed anatomical or physiological structure may be a 3D model of the anatomical or physiological structure, in particular the oral anatomy including hard and soft tissues. The region of interest may be a part of the body of a user or subject. The sequence of images includes two or more images taken at different times. The system data may include one or more motion signals indicative of a motion of the imaging device relative to the region of interest.

[0175] In some cases, the reconstructed anatomical or physiological structure is a 2D reconstruction.

[0176] 14 shows an example of a reconstructed anatomical or physiological structure. Image 1402 shows a point cloud reconstruction of a single tooth, and image 1404 shows a point cloud reconstruction of a typodont. In this case, the reconstructed structure shown in image 1404 is a 3D jaw (a 3D mouth map of the typodont) with multiple segmented teeth and gingival tissue covering an artificial oral jawbone structure.

[0177] Quasi-periodic behavior includes quasi-periodic functions. Quasi-periodic functions can be modified periodic functions (e.g., f(x+ω)=Af(x)+B). As explained above, a periodic function is defined as f(x+ω)=f(x), where ω is the period of the function.

[0178] The quasi-periodic behavior predicts the future movements of the user. The quasi-periodic behavior predicts whether images or system data obtained in the future will be redundant.

[0179] The reconstruction of anatomical or physiological structures may be performed using the RSfM algorithm. Other algorithms for reconstructing 3D models from image and system data may be used.

[0180] The system data is sometimes derived from the images themselves. For example, geometric features that appear in different images can be identified and the images containing the geometric features located relative to each other. Other methods for locating the images and thereby deriving the system data can also be used.

[0181] Quality parameters used in filtering a sequence of images include the amount of sharpness, the amount of occlusion, the amount of contrast, comparison between similar images, and the like.

[0182] The images in the sequence of images may be multispectral or hyperspectral images.

[0183] An anatomical or physiological structure includes the geometry of said structure.

[0184] The previous examples have shown the reconstruction of oral cavity structures and other biological structures such as skin during shaving, however, it will be appreciated that this method for digitally reconstructing structures of an object can be applied in all modalities requiring digital reconstruction of structures from images.

[0185] For example, consider a user imaging a structure with a camera. The camera movement is constrained by the physically possible movements of the user's arm and the possible movements around the object while still imaging the structure. If the structure is imaged for the purpose of digital reconstruction, the user may attempt to image the structure from different angles and positions to obtain a large amount of data for digital reconstruction. The limited movements available to the user combined with the large amount of data being captured makes it very likely that quasi-periodic behavior will occur.

[0186] In general, imaging an object for digital reconstruction requires a large amount of data, and the act of imaging the object involves a limited amount of motion. Therefore, the analysis of quasi-periodic behavior for digital structure reconstruction can be used in many other applications. The invention is important, for example, in the aerospace industry, in manufacturing and testing in general, and can be applied whenever redundancy in multiple images that are being used to reconstruct a structure of an object or an area of ​​an object is exploited.

[0187] Those skilled in the art can easily develop a processor to perform the methods described herein, and each step of the flowchart thus represents a different action performed by a processor, which is executed by a respective module of the processor.

[0188] As explained above, the system utilizes a processor to perform data processing. The processor can be implemented in numerous ways using software and / or hardware to perform the various functions required. The processor typically employs one or more microprocessors that are programmed using software (e.g., microcode) to perform the required functions. The processor may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0189] Examples of circuitry employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, neural processors, TPUs (tensor processing units), data flow processors, neuromorphic processors, field programmable gate arrays (FPGAs), and application specific integrated circuits (ASICs).

[0190] In various implementations, the processor is associated with one or more storage media, such as volatile and non-volatile computer memories, such as RAM, PROM, EPROM, and EEPROM. The storage media are encoded with one or more programs that, when executed on the one or more processors and / or controllers, perform the required functions. The various storage media may be fixed within the processor or controller or may be portable, such that the one or more programs stored thereon can be loaded into the processor.

[0191] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprises" does not exclude other elements or steps, and singular elements do not exclude a plurality.

[0192] A single processor or other unit may fulfill the functions of several items recited in the claims.

[0193] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0194] The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0195] It should be noted that when the term "adapted to" is used in the claims or this specification, it is intended to be equivalent to the term "configured to."

[0196] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A system for digitally reconstructing a structure of a region of interest of an object, the system comprising a part of a personal care device, the system comprising: an imaging device for obtaining a sequence of images of the region of interest; - analyzing system data, the system data including at least data enabling identification of a position of the imaging device relative to the area of ​​interest, and extracting from the system data quasi-periodic behaviors caused by personal care actions; using the identified quasi-periodic behavior to remove redundancy in the sequence of images and / or in the system data; and reconstructing the structure of the region of interest using the sequence of images and the system data; A processor for performing A system comprising:

2. The processor, identifying redundant images in the sequence of images based on the quasi-periodic behavior; and removing said redundant images from said sequence of images; The system of claim 1 , wherein the redundancy is eliminated by:

3. The system of claim 1 or 2, wherein the processor is further adapted to filter the sequence of images to eliminate images with insufficient quality parameters for the reconstruction.

4. The system of claim 1 , wherein the images in the sequence of images are multispectral images, and the structure of the region of interest is reconstructed separately for each spectral band of the multispectral images.

5. The processor, identifying structural features of the reconstructed structure of the spectral band; and merging the reconstructed structures of the spectral bands by matching the structural features of the spectral bands; The system of claim 4 , further comprising:

6. The processor further comprises: estimating a spatial position of a next image in the sequence of images based on a position of a previous image and the quasi-periodic behavior; obtaining a real spatial location of said next image; adapting the quasi-periodic behavior based on a comparison of the estimated spatial position and the actual spatial position; The system according to claim 1 , further comprising:

7. The system of claim 1 , further comprising a sensor system, such as an inertial measurement unit, for obtaining the system data corresponding to the images in the sequence of images.

8. The system of claim 7 , wherein the processor removes redundancies in the system data based on comparing the system data to the quasi-periodic behavior.

9. A personal care unit and a handle; A system according to any one of claims 1 to 8. and the area of ​​interest is an oral cavity or a portion of the oral cavity.

10. 10. The personal care device of claim 9, comprising a toothbrush having a toothbrush head section and bristles, the imaging device obtaining images in the direction of the bristles that are stationary.

11. 1. A computer-implemented method for digitally reconstructing a structure of a region of interest of an object, the computer-implemented method comprising: obtaining a sequence of images of the area of ​​interest using an imaging device that is part of a personal care device; - analyzing system data, including at least data enabling identification of a position of the imaging device relative to the area of ​​interest, and extracting from the system data quasi-periodic behaviors caused by personal care actions; using the identified quasi-periodic behavior to remove redundancy in the sequence of images and / or in the system data; reconstructing the structure of the region of interest using the sequence of images and the system data; 13. A computer-implemented method comprising:

12. The step of removing redundancy comprises: identifying redundant images in the sequence of images based on the quasi-periodic behavior; removing said redundant images from said sequence of images; 12. The computer-implemented method of claim 11, comprising:

13. 13. The computer-implemented method of claim 11 or 12, wherein images in the sequence of images are multispectral images, and the structure of the region of interest is reconstructed separately for each spectral band of the multispectral images.

14. estimating a spatial position of a next image in the sequence of images based on a position of a previous image and the quasi-periodic behavior; obtaining a real spatial location of the next image; adapting the quasi-periodic behavior based on a comparison of the estimated spatial position and the actual spatial position; 14. The computer-implemented method of claim 11, further comprising:

15. A computer program comprising computer program code which, when executed on a computing device having a processing system, causes said processing system to perform all the steps of the computer implemented method of any one of claims 11 to 14.

Images