Augmented Reality Spectroscopy

Abstract

To provide augmented reality spectroscopy.SOLUTION: In some embodiments, a system comprises: a head-mounted frame removably couplable to a user's head; one or more light sources coupled to the head-mounted frame and configured to emit light with at least two different wavelengths toward a target object in an irradiation field of view of the light sources; one or more electromagnetic radiation detectors coupled to the head-mounted member and configured to receive light reflected after encountering the target object; and a controller operatively coupled to the one or more light sources and detectors and configured to determine and display an output indicating the identity or property of the target object as determined by properties of the light measured by the detectors in relation to properties of the light emitted by the light sources.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] (Priority Claim) This application claims the benefit of priority of U.S. Provisional Application No. 62 / 398,454, filed on September 22, 2016. The above document is incorporated herein by reference.

[0002] (Incorporation by Reference) The present invention incorporates by reference in its entirety each of the following U.S. patent applications: U.S. Patent Application No. 15 / 072,341, U.S. Patent Application No. 14 / 690,401, U.S. Patent Application No. 14 / 555,858, U.S. Application No. 14 / 555,585, U.S. Patent Application No. 13 / 663,466, U.S. Patent Application No. 13 / 684,489, U.S. Patent Application No. 14 / 205,126, U.S. Patent Application No. 14 / 641,376, U.S. Patent Application No. 14 / 212,961, U.S. Provisional Patent Application No. 62 / 298,993 (corresponding to U.S. Patent Application No. 15 / 425,837), and U.S. Patent Application No. 15 / 425,837.

[0003] (Background) (Field of the Invention) The present disclosure relates to systems and methods for augmented reality using wearable components, and more particularly to the configuration of an augmented reality system for identifying materials by the properties of reflected light.

Background Art

[0004] (Description of Related Art) Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, where digitally reproduced images or portions thereof are presented to a user in a manner that appears or can be perceived as being real. Virtual reality, i.e., the "VR" scenario, typically involves the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. Augmented reality or the "AR" scenario typically involves the presentation of digital or virtual image information as an augmentation to the visualization of the actual world around the user while still allowing the user to substantially perceive and view the real world.

[0005] For example, referring to FIG. 1, an augmented reality scene (4) is depicted, and to a user of AR technology, a real-world park-like setting (6) is visible, characterized by people, trees, buildings in the background, and a concrete platform (1120). In addition to these items, the user of AR technology also "sees" a robotic image (1110) standing on the real-world platform (1120) and an avatar character (2) like a flying cartoon that appears anthropomorphic like a honeybee, even though these virtual elements (2, 1110) do not exist within the real world. Stated in conclusion, the human visual perception system is very complex, and the production of VR or AR technology, which promotes a comfortable, natural-like, and rich presentation of virtual image elements among other virtual or real-world image elements, is difficult. For example, a head-mounted AR display (or a helmet-mounted display or smart glasses) is typically mounted at least loosely on the user's head and thus moves as the user's head moves. When the movement of the user's head is detected by the display system, the displayed data can be updated to account for the change in head pose. Certain aspects of a suitable AR system are incorporated herein by reference in their entirety, along with the following additional disclosures regarding augmented and virtual reality systems such as those developed by Magic Leap, Inc. (Fort Lauderdale, Florida), U.S. Patent Application No. 14 / 205,126, titled "System and method for augmented and virtual reality", U.S. Patent Application No. 14 / 641,376, U.S. Patent Application No. 14 / 555,585, U.S. Patent Application No. 14 / 212,961, U.S. Patent Application No. 14 / 690,401, U.S. Patent Application No. 13 / 663,466, U.S. Patent Application No. 13 / 684,489, and U.S. Application No. 62 / 298,993, each incorporated herein by reference in their entirety.

[0006] The systems and methods disclosed herein address various challenges and developments related to AR and VR technologies.

Summary of the Invention

Problems to be Solved by the Invention

[0007] (Summary) The mixed reality system is configured to perform spectroscopy. Mixed reality (alternatively abbreviated as "MR") typically involves virtual objects integrated into and responsive to the natural world. For example, in an MR scenario, AR content can be occluded by real-world objects and / or perceived to interact with other (virtual or real) objects within the real world. Throughout this disclosure, references to AR, VR, or MR are not limitations on the invention, and the techniques may be applied in any context.

Means for Solving the Problems

[0008] Some embodiments are directed to a wearable system for identifying substances (tissue, cells within tissue, or properties within cells / tissue, etc.) as a function of the light wavelengths emitted from a head-mounted member removably attachable to a user's head and subsequently received / reflected / detected thereby. This disclosure primarily refers to tissue or tissue properties as subjects for analysis according to various embodiments, but the technology, techniques, and components are not so limited. Some embodiments utilize one or more light sources, such as electromagnetic radiation emitters, coupled to the head-mounted member and emitting light within one or more wavelengths in a user-selectable direction. Such embodiments enable even continuous and passive measurements. For example, a user wearing a head-mounted system can perform a given activity, and inward-facing sensors can detect the properties of the eyes without interfering with the activity.

[0009] For example, a user may wear a system configured to look inward at the user's eye and identify or measure the properties of eye tissue, such as blood concentration in the blood vessels of the eye. In other embodiments of the inward system, not only the properties of the tissue, but also fluids such as intraocular fluid may be analyzed. In other embodiments, the system may include sensors that look outward toward the outside world and identify or measure the properties of tissues or materials other than the eye, such as the user's limbs or objects in the surrounding environment away from the user.

[0010] In an outward-looking system, an eye-tracking camera coupled to a head-mounted member can determine the directional gaze of the user, and a processor or controller may correlate that gaze with the observation of a real-world target object through an image captured from a real-world capture system (such as a camera or depth sensor) coupled to the head-mounted member. A light source coupled to the head-mounted system emits light, such as infrared light, away from the user, for example, from an electromagnetic radiation emitter, and in some embodiments, emits light and creates an illumination pattern in a direction substantially the same as the gaze direction determined by the eye-tracking camera, thereby emitting it onto the target object.

[0011] In some embodiments, the real-world capture system captures an object. For example, a depth sensor, such as a vertical cavity surface emitting laser, may determine the outline of an object by collecting time-of-flight signals that impinge on the object. Once an object is identified in its contour by such a real-world capture system, it can be highlighted and made available for labeling. In some embodiments, a camera system of a given field of view defines an area available for highlighting and labeling. For example, a camera that correlates with the user's gaze may include suitable increments up to a 5-degree field of view, a 10-degree field of view, or preferably a maximum 30-degree central visual field within which the light source will substantially emit light.

[0012] In some embodiments, such a system further includes one or more electromagnetic radiation detectors or photodetectors coupled to the head-mounted member and configured to receive reflected light emitted from a light source and reflected from a target object, and one or more electromagnetic radiation emitters and a controller operably coupled to the one or more electromagnetic radiation detectors and the one or more electromagnetic radiation emitters, the controller configured to cause the one or more electromagnetic radiation emitters to emit pulses of light and cause the one or more electromagnetic radiation detectors to detect a level of light absorption associated with the emitted pulses of light as a function of any received reflected light of a particular pulsed emission.

[0013] In some embodiments, the system further includes a processor for matching the wavelength of reflected light received from the target object by the detector to a particular material, tissue type, or property of underlying tissue. In some embodiments, other optical properties such as polarization changes or scattering effects on the emitted and detected light are also determined, but for the purposes of this description, wavelength properties are used as exemplary optical properties. For example, in some embodiments, an inward-facing electromagnetic radiation emitter emits light within the infrared spectrum onto a user's retina, receives the reflected light, matches the wavelength of the reflected light, and determines physical properties such as the type of tissue or oxygen saturation within the tissue. In some embodiments, the system includes an outward-facing light source that emits infrared light onto a target object (such as a user's or third party's limb) and receives the reflected light and matches the wavelength of the reflected light to determine the material being observed. For example, such an outward-facing system may detect the presence of cancerous cells among healthy cells. Cancerous or other abnormal cells reflect and absorb light differently than healthy cells, so the reflection of light at a certain wavelength can indicate the presence and amount of the abnormality.

[0014] In some embodiments, the controller receives the captured target object from the real-world capture system and applies a label indicating the identified property to the target object. In some embodiments, the label is a text label or prompt within a display of the head-mounted member. In some embodiments, the label is an audio prompt to the user. In some embodiments, the label is a virtual image of a similar tissue as referenced in a medical book that is superimposed in the vicinity of the target object for easy comparative analysis by the user.

[0015] In some embodiments, the head-mounted member may comprise an eyeglass frame. The eyeglass frame may be a binocular eyeglass frame. The one or more radiation emitters may comprise a light source such as a light-emitting diode. The one or more radiation emitters may comprise a plurality of light sources configured to emit electromagnetic radiation at two or more different wavelengths. The plurality of light sources may be configured to emit electromagnetic radiation at a first wavelength of about 660 nanometers and a second wavelength of about 940 nanometers. The one or more radiation emitters may be configured to emit electromagnetic radiation at two different wavelengths continuously. The one or more radiation emitters may be configured to emit electromagnetic radiation at two different wavelengths simultaneously. The one or more electromagnetic radiation detectors may comprise a device selected from the group consisting of a photodiode, a light detector, and a digital camera sensor. The one or more electromagnetic radiation detectors may be positioned and oriented to receive light reflected after encountering the target object. The one or more electromagnetic radiation detectors may be positioned and oriented to receive light reflected after encountering the tissue or material being observed. That is, the one or more electromagnetic radiation detectors are oriented in substantially the same direction as the one or more electromagnetic radiation emitters, regardless of whether they face inward toward the user's eyes or outward toward the user's environment.

[0016] The controller may further be configured to cause the plurality of light sources to emit a cycling pattern in which one or more electromagnetic radiation detectors separately detect a first and a second wavelength, turning on the first wavelength, then turning on the second wavelength, and then turning off both wavelengths. The controller may be configured to cause the plurality of light emitting diodes to emit a cycling pattern in which the first wavelength is turned on, then the second wavelength is turned on, and then both wavelengths are turned off, in a cycling pulse pattern of about 30 times per second.

[0017] In some embodiments, the controller may be configured to calculate a ratio of the first wavelength light measurement to the second wavelength light measurement. In some embodiments, this ratio may further be converted to an oxygen saturation reading, at least in part, via a look-up table, based on Lambert-Beer's law. In some embodiments, the ratio is converted to a material identifier in an external look-up table, such as stored within an absorption database module on the head-mounted member or coupled to the head-mounted member on a local or remote processing module. For example, an absorption database module regarding the absorption rate or wavelength reflectance of a particular tissue may be stored within a "cloud" storage system accessible by a healthcare provider and accessed through a remote processing module. In some embodiments, the absorption database module may store absorption properties (such as wavelength rate or wavelength reflectance) regarding a certain food item and be permanently stored on the local processing module of the head-mounted member.

[0018] In this way, the controller may be configured to operate one or more electromagnetic radiation emitters and one or more electromagnetic radiation detectors to function as a head-mounted spectrometer for a wide range of uses. The controller may be coupled to a head-mounted member and operably coupled to an optical element such that the output of the controller, which indicates the wavelength properties indicative of the nature of a particular tissue or other material, can be visually recognized by the user through the optical element and is visible to the user. The one or more electromagnetic radiation detectors may comprise a digital image sensor having a plurality of pixels, and the controller may be configured to automatically detect a subset of pixels that receive, for example, light reflected after encountering tissue or cells within the tissue. In some embodiments, such a subset of pixels is used to produce an output representative of a target object within the field of view of the digital image sensor. For example, the output may be a display label indicating the absorption level of the tissue. In some embodiments, a comparison value is displayed as the output. For example, the output may be the percentage saturation of oxygen in the blood from a first analysis time and the percentage saturation of oxygen at a second analysis time, and the rate of change is described between the two times. In these embodiments, a disease such as diabetic retinopathy may be detected by recognizing the change over time of the measured property.

[0019] In some embodiments, the controller may be configured to automatically detect a subset of pixels based at least in part on the reflected light luminance difference between the signals associated with the pixels. The controller may be configured to automatically detect a subset of pixels based at least in part on the reflected light absorbance difference between the signals associated with the pixels. In such embodiments, such a subset may be isolated pixels, flagged for further analysis such as additional irradiation or mapping, or a virtual image may be overlaid on such pixels to provide a visual contrast with the isolated pixels displaying other properties and serve as a notification to the user of the different properties of the sub-pixels identified by the system.

[0020] In some embodiments, system data collection is not only time multiplexed to pulse and record optical pulses, but also collected passively multiple times a day. In some embodiments, a GPS or other similar mapping system is coupled to the system to correlate the user's location or time with certain physiological data collected. For example, a user may track their physiological responses to a location or activity throughout the day.

[0021] These and many other features and advantages of the present invention will be understood when the following figures and description are further considered. This specification also provides, for example, the following items. (Item 1) A wearable spectroscopy system, A head-mounted display system removably attachable to a user's head, At least one eye-tracking camera configured to detect the user's gaze, One or more light sources coupled to the head-mounted display system and configured to emit light with at least two different wavelengths within a field of view irradiated in substantially the same direction as the detected gaze, One or more electromagnetic radiation detectors coupled to the head-mounted member and configured to receive light reflected from a target object within the irradiated field of view, A controller operably coupled to the one or more light sources and the one or more electromagnetic radiation detectors, the controller being configured to cause the one or more light sources to emit pulses of light and to cause the one or more electromagnetic radiation detectors to detect levels of light absorption associated with the emitted pulses of light and the light reflected from the target object, An absorption database of the absorption properties of at least one material, A graphic processor unit for displaying an output to the user and a system comprising. (Item 2) The system according to item 1, wherein the one or more light sources comprise a plurality of light emitting diodes. (Item 3) The system according to item 1, wherein the one or more light sources are configured to emit electromagnetic radiation at two or more predetermined wavelengths. (Item 4) The system according to item 3, wherein the one or more light sources are configured to emit electromagnetic radiation at a first wavelength of about 660 nanometers and a second wavelength of about 940 nanometers. (Item 5) The system according to item 3, wherein the one or more light sources are configured to continuously emit electromagnetic radiation at the two predetermined wavelengths. (Item 6) The system according to item 3, wherein the one or more light sources are configured to simultaneously emit electromagnetic radiation at the two predetermined wavelengths. (Item 7) The system according to item 1, wherein the controller is further configured to cause the one or more light sources to emit a cycling pattern in which the first wavelength is turned on, then the second wavelength is turned on, and then both the first and second wavelengths are turned off, such that the one or more electromagnetic radiation detectors detect the first and second wavelengths separately. (Item 8) The system according to item 1, wherein the controller is configured to calculate a ratio of a first wavelength light measurement to a second wavelength light measurement, and the system is configured to convert the ratio into a property of the tissue based on the absorption database. (Item 9) The system according to item 8, wherein the controller is operably coupled to an optical element that is coupled to the head-mounted member and visible to the user, and the system is configured to provide an output based on the property of the tissue, the output being visible to the user through the optical element. (Item 10) The system according to item 1, wherein the one or more electromagnetic radiation detectors comprise a device selected from the group consisting of a photodiode and a photodetector. (Item 11) The system according to item 1, wherein the one or more electromagnetic radiation detectors comprise a digital image sensor. (Item 12) The system according to item 11, wherein the digital image sensor comprises a plurality of pixels, and the controller is configured to automatically detect a subset of the pixels that receive the light reflected after encountering the property of a predetermined tissue, and produce an output that displays the location of the subset of the pixels indicating the property of the predetermined tissue. (Item 13) The system according to item 1, wherein the head-mounted member further comprises an inertial measurement unit positioning system. (Item 14) The system according to item 13, wherein the inertial measurement system determines the orientation of the user's head. (Item 15) The system according to item 14, wherein the illuminated field of view is at least as wide as the orientation. (Item 16) The system according to item 1, wherein the head-mounted display system comprises a waveguide stack configured to output light with a selectively variable level of wavefront divergence. (Item 17) The system according to item 16, wherein the waveguide stack comprises a waveguide having a refractive power.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

[0023] (Detailed Description) Some AR and VR systems include a processing capability such as a controller or microcontroller, and also a power source for powering various configured functions. Due to the fact that at least some of the components within wearable computing systems such as AR and VR systems are in close proximity to the body of the user operating them, there is an opportunity to utilize some of these system components to perform certain physiological monitoring tasks related to the user. For example, physiological monitoring may be performed by measuring light absorption.

[0024] In conventional light absorption measurement techniques (e.g., a pulse oximeter attachable to a person's finger as in FIG. 4A or glucose detection), light is emitted in a controlled and fixed direction and received at a controlled and fixed receiver. The light is pulsed at different wavelengths through the surrounding tissue structure and detected on the other side of the tissue structure (thus measuring properties of the light such as absorption and scattering). In such systems, the measurement of the emitted light compared to the measurement of the detected light is proportional to, or read as, an output of an estimated tissue or tissue property (e.g., the estimated blood oxygen saturation level with respect to a pulse oximeter), or simply provides some other material or tissue type. A calibration curve depicting the ratio of the incident light to other light can also predict the properties of the underlying tissue as a function of the light incident thereon, as shown in FIG. 4D.

[0025] Raman spectroscopy is another technique that measures the inelastic scattering of photons released by irradiated molecules. When a specific molecule is irradiated, it will exhibit a specific shift in wavelength, thereby presenting a unique scattering effect that can be used to measure and quantify the molecules in the sample.

[0026] Figure 4B illustrates a chart of the absorption spectra of hemoglobin for oxygenation (806) versus deoxygenation (808). As shown in such plots (806, 808), within the red light wavelength range of the electromagnetic spectrum such as approximately 660 nm, there are significant differences in absorption regarding oxygenated versus deoxygenated hemoglobin, while at approximately 940 nm within the infrared wavelength range, there are inverted differences. Detection using pulsed radiation and a pulse oximeter at such wavelengths is known to utilize such known absorption differences in the determination of oxygen saturation for a particular user.

[0027] The pulse oximeter (802) is typically configured to at least partially enclose a tissue structure such as a finger (804) or an earlobe, etc., but there are desktop systems such as that depicted in Figure 4C (812) that have been proposed to observe absorption differences within the blood vessels of the eye such as the retinal vasculature, and similarly, it may be configured to detect the properties of other tissues.

[0028] Such a configuration (812) may be referred to as a flow oximeter or a spectroscopy system and, as shown, may comprise components including a camera (816), a zoom lens (822), first (818) and second light emitting diodes (LEDs), and one or more beam splitters (814). For a user such as a high altitude hiker, an athlete, or a person with certain cardiovascular or respiratory problems, it would be beneficial to be able to read out information on their blood oxygen saturation as they move throughout the day and engage in their activities, or for a caregiver, to be able to analyze tissue in real time with respect to underlying abnormalities, but most configurations are not portable or wearable, involve somewhat inconvenient encapsulation of the tissue structure, do not account for other absorbent properties indicative of other tissue states or materials, or do not correlate the gaze of the user as part of the orientation of the sensor (in other words, lack selectivity of the target object for spectroscopic identification and analysis).

[0029] Advantageously, in some embodiments, a solution is presented herein that combines the convenience of wearable computing in the form of an AR or VR system with imaging means for determining additional tissue identification and properties within the user's field of view in real time.

[0030] Referring to FIGS. 2A - 2D, some general component options are illustrated. In the section of the detailed description that follows the discussion of FIGS. 2A - 2D, various systems, subsystems, and components are presented to address the purpose of providing a high quality and comfortably perceivable display system for human VR and / or AR that accesses and creates external information sources.

[0031] As shown in FIG. 2A, an AR system user (60) is depicted as wearing a head-mounted component (58) characterized by a frame (64) structure coupled to a display system (62) positioned in front of the user's eyes. A speaker (66) is coupled to the frame (64) in the depicted configuration and positioned adjacent to the user's external ear canal (in one embodiment, another speaker, not shown, is positioned adjacent to the user's other external ear canal to provide stereo / formable sound control). The display (62) is operably coupled (68) to a local processing and data module (70) by means such as wired leads or wireless connectivity, which may be mounted in various configurations, such as fixed to the frame (64), fixed to a helmet or hat (80) as shown in the embodiment of FIG. 2B, built into headphones, removably attached to the user's torso (82) in a backpack-style configuration as shown in the embodiment of FIG. 2C, or removably attached to the user's hip (84) in a belt-coupled configuration as shown in the embodiment of FIG. 2D.

[0032] The local processing and data module (70) may comprise a processor or controller (e.g., a power-efficient processor or controller) and digital memory such as flash memory, both of which may be used to assist in the processing, caching, and storage of data that is (a) captured from sensors such as electromagnetic emitters and detectors, image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes, etc., that may be operably coupled to the frame (64), and / or (b) possibly obtained and / or processed using the remote processing module (72) and / or remote data repository (74) for passage to the display (62) after processing or reading. The local processing and data module (70) may be operably coupled (76, 78) to these remote processing modules (72) and remote data repositories (74) such that the remote modules (72, 74) are operably coupled to each other and available as resources to the local processing and data module (70) via a wired or wireless communication link or the like.

[0033] In one embodiment, the remote processing module (72) may comprise one or more relatively high-performance processors or controllers configured to analyze and process data, the properties of emitted or received light, and / or image information. In one embodiment, the remote data repository (74) may comprise a relatively large-scale digital data storage facility, which may be available through other networking configurations in an Internet or "cloud" resource configuration. In one embodiment, all data is stored and all calculations are performed in the local processing and data module, allowing for completely autonomous use from any remote module.

[0034] Referring now to FIG. 3, the schematic illustrates the cooperation between a cloud computing asset (46) and a local processing asset that may reside, for example, within a head-mounted component (58) coupled to a user's head (120) and a local processing and data module (70) coupled to the user's belt (308). Thus, component (70) may also be referred to as a "belt pack" (70), as shown in FIG. 3. In one embodiment, cloud (46) assets such as one or more cloud server systems (110) are operatively coupled (115) directly to one or both of the local computing assets (40, 42), such as a processor and memory configuration, coupled to the user's head (120) and belt (308) as described above, via a wired or wireless networking, etc. (wireless is generally preferred for mobile types, and wired is generally preferred for certain high bandwidth or high data volume transfers that may be desired). These local computing assets local to the user may also be operatively coupled to each other via wired and / or wireless connectivity configurations (44), such as the wired connection (68) discussed below with reference to FIG. 8.

[0035] In one embodiment, to maintain a low inertia and small subsystem mounted on the user's head (120), the primary transfer between the user and the cloud (46) may be via a link between the subsystem mounted on the belt (308) and the cloud, and the head-mounted subsystem (120) is primarily data tethered to the belt-based subsystem (308) using a wireless connectivity such as ultra-wideband ("UWB") connectivity, as currently employed, for example, in personal computing peripheral connectivity applications.

[0036] By using efficient local and remote processing coordination and an appropriate display device for the user, such as the user interface or user display system (62) shown in FIG. 2A or a variant thereof, one aspect of the world regarding the user's current actual or virtual location can be transferred or "passed" to the user and updated in an efficient manner. In other words, the map of the world can be continuously updated at a storage location that can be partially resident, for example, on the user's AR system and partially resident within cloud resources. The map (also referred to as the "navigable world model") may be a large database comprising raster images, 3-D and 2-D points, parameter information, and other information about the real world. As more and more AR users continuously capture information about their physical environment (e.g., through cameras, sensors, IMUs, etc.), the map becomes increasingly accurate and complete.

[0037] By using the configuration as described above, in which there is one world model that resides on and can be delivered from cloud computing resources, such a world can be "passable" to one or more users in a relatively low bandwidth form that is preferred for attempting to stream real-time video data or the like. In some embodiments, the augmented experience of a person standing near the image (i.e., as shown in FIG. 1) may be provided by a cloud-based world model, a subset of which may be passed to them and their local display device to complete the view. A person sitting facing a remote display device, which can be something as simple as a personal computer on an airplane, can efficiently download the same section of that information from the cloud and render it on that display. In fact, a person actually present in a park near the image can take a remote friend for a walk in that park, and the friend may join through virtual and augmented reality. The system will need to know the location of the street, the trees, the image, but by using that information on the cloud, the joining friend can download from the cloud side of the scenario and then start the walk as local augmented reality for the person actually in the park.

[0038] 3-D points may be captured from the environment, and the pose of the camera (i.e., the vector and / or in-situ information relative to the world) that captures those images or points may be determined so that these points or images can be "tagged" or associated with this pose information. Then, the points captured by a second camera may be used to determine the pose of the second camera. In other words, the second camera can be oriented and / or located based on a comparison with the tagged images from the first camera. This knowledge may then be used to extract texture, create a map, and create a virtual copy of the real world (at which point there are two cameras aligned to the surroundings).

[0039] Accordingly, at a basic level, in some embodiments, a human-worn system can be utilized to capture both 3-D points and the 2-D images that generated those points, and these points and images may be transmitted to cloud storage and processing resources. They may also be cached locally (i.e., cache tagged images) along with built-in pose information, and thus the cloud can have available (i.e., in the available cache) the 3-D points along with the tagged 2-D images (i.e., tagged with 3-D pose). When the user is observing something dynamic (e.g., a scene with moving objects or features), additional information regarding the motion may also be sent to the cloud (e.g., when looking at another person's face, the user can take a texture map of the face and push it at an optimized frequency even if the surrounding world is otherwise basically static). As described above, further information regarding object recognition devices and passable world models is incorporated herein by reference in its entirety along with the following additional disclosures related to augmented and virtual reality systems such as those developed by Magic Leap, Inc. (Fort Lauderdale, Florida): U.S. Patent Application No. 14 / 205,126, entitled "System and method for augmented and virtual reality"; U.S. Patent Application No. 14 / 641,376; U.S. Patent Application No. 14 / 555,585; U.S. Patent Application No. 14 / 212,961; U.S. Patent Application No. 14 / 690,401; U.S. Patent Application No. 13 / 663,466; U.S. Patent Application No. 13 / 684,489; and U.S. Patent Application No. 62 / 298,993, each of which is incorporated herein by reference in its entirety.

[0040] In some embodiments, the use of such passable world information may enable the spectroscopic identification and labeling of objects to then be passed between users. For example, in a clinical setting, a first caregiver operating a device implementing the features of the present disclosure may map and detect cancerous tissue on a patient and assign and apply a virtual label, such as a metatag, to the tissue. Similarly, a second caregiver wearing such a device may then, independently, receive a notification of the virtual label identifying such cells by looking at the same cancerous tissue cell cluster without having to be involved in one or more of emitting light, receiving light, matching absorption features to the tissue, or labeling the tissue.

[0041] GPS and other location-specific information may be utilized as an input to such processing. It should be understood that very accurate location-specification of a user's head, totem, hand gestures, haptic devices, etc. can facilitate the display of appropriate virtual content to the user or passable virtual or augmented content between users within the passable world.

[0042] Referring to FIG. 5, an orthogonal top view of a head - mountable component (58) of a wearable computing configuration is illustrated, featuring various integrated components for an exemplary spectroscopy system. This configuration includes two display elements (62 - binocular - one for each eye), two forward - facing cameras (124) for observing and detecting the world around the user, each camera (124) having an associated field of view (18, 22), at least one spectroscopy array (126, described in more detail in FIG. 6) with an associated field of view (20), and also a forward - facing relatively high - resolution photographic camera (156) with an associated field of view (26), as described in the disclosure incorporated above by reference, one or more inertial measurement units (102), and a depth sensor (154) with an associated field of view (24). Facing towards the user's eyes (12, 13) and coupled to the head - mounted component (58) frame are an eye - tracking camera (828, 830) and inward - facing emitters and receivers (832, 834). Those skilled in the art will understand that the inward - facing emitters and receivers (832, 834) emit and receive light directed towards the eyes in an irradiation pattern (824, 826) in substantially the same manner as the spectroscopy array (126) does for outward - facing objects within its field of view (20). These components or combinations including all or less than all of these components are operably coupled (848) to a controller (844), which is operably coupled to a power source (846), such as a battery, by wire conductors or the like.

[0043] In some embodiments, the display element (62) includes one or more waveguides (e.g., a stack of waveguides), which are optically transmissive and enable the user to "see" the world by receiving light from the world. The waveguide also receives light containing display information and propagates and emits the light to the user's eyes (12, 13), thereby displaying an image to the user. Preferably, the light propagating from the waveguide provides a specific defined level of wavefront divergence corresponding to different depth planes (e.g., the light forming an image of an object at a specific distance from the user has a wavefront divergence corresponding or substantially matching the wavefront divergence that would reach the user from that object if it were real). For example, the waveguide may have a refractive power and may be configured to output light with a selectively variable level of wavefront divergence. It should be understood that this wavefront divergence provides a cue for the eyes (12, 13) to perform accommodation. In addition, the display element (62) utilizes binocular disparity and further provides a depth cue, e.g., a cue for the binocular vergence movement of the eyes (12, 13). Advantageously, the cue for accommodation and the cue for binocular vergence movement can be coordinated, e.g., such that both of them correspond to an object at the same distance from the user. This accommodation-binocular vergence coordination facilitates the long-term wearability of systems that utilize the head-mounted member (58).

[0044] Continuing to refer to FIG. 5, preferably, each emitter (126, 832, 834) is configured to controllably emit electromagnetic radiation at two or more wavelengths, such as about 660 nm and about 940 nm, by an LED or the like, and preferably, the irradiation fields (824, 826) are oriented to irradiate a target object or surface. In some embodiments, the target object is inward, such as the eyes (12, 13), and the irradiation patterns (824, 826) may be fixed or widened / narrowed with respect to the target-specific area of the eye in response to eye-tracking camera data points. In some embodiments, the target object is outward (e.g., away from the user), and the irradiation pattern within the field of view (20) of the spectrometer array (126) conforms to the gaze of the eyes (12, 13) determined from the eye-tracking cameras (828, 830).

[0045] In some embodiments, the gaze can be understood as a vector extending from the user's eye, such as extending through the lens from the fovea of the eye, and the emitters (832, 834) may output infrared light onto the user's eye, and the reflection from the eye (e.g., corneal reflection) may be monitored. The vector between the pupil center of the eye (e.g., the display system may determine the centroid of the pupil, for example, through an infrared image) and the reflection from the eye may be used to determine the gaze of the eye. In some embodiments, when estimating the position of the eye, since the eye has a sclera and an eyeball, the geometry can be represented as two overlapping circles. The vector pointing to the eye may be determined or calculated based on this information. Also, since the cross-section of the eye is circular and the sclera swings through a specific angle, the rotation center of the eye may also be estimated. This can result in a vector distance due to the autocorrelation of the received signal with respect to the known transmitted signal, rather than simply ray tracing. The output can be seen as a Purkinje image 1400, which can in turn be used to track the movement of the eye.

[0046] One of ordinary skill in the art will also understand other methods for determining the irradiation pattern within the field of view (20) based on head pose information and the like determined by one or more of the IMUs (102).

[0047] In some embodiments, the emitter may be configured to emit wavelengths using a controlled pulsed emission cycle, either simultaneously or sequentially. One or more detectors (126, 828, 830) may comprise photodiodes, light detectors, and / or digital camera sensors, and are preferably positioned and oriented to receive radiation that has encountered the target tissue or other material or object. One or more electromagnetic radiation detectors (126, 828, 830) may comprise a digital image sensor having a plurality of pixels, and the controller (844) is configured to automatically detect a subset of the pixels that receive the reflected light after it has encountered the target object, and to use such a subset of pixels to produce an output.

[0048] In some embodiments, the output is a function of the received light that matches from a database of materials and material properties to the light emitted to the target. For example, in some embodiments, the absorption database comprises a plurality of absorption charts such as depicted in FIGS. 7A and 7B. The database comprising the charts may include an electronic representation or conversion of the information within the charts, and it should be understood that the use of the term "chart" herein includes such representation or conversion. FIGS. 7A and 7B are used merely as examples, but demonstrate the properties of various tissues that can be detected from a given system that emits light from a specific light source, receives light of a specific wavelength and / or properties of light, and determines the probability that the observed target is a specific tissue or has a specific property within the tissue. Other charts, such as either a saturation curve or a calibration curve, may also be selectively accessed by the user. For example, the user may select an absorption database for a specific light source or wavelength pattern and then search until identifying a material that matches the properties required by the spectroscopic system. Such embodiments may be referred to as "closed searches," i.e., they examine specific properties, as opposed to "open searches" that examine any target and then search a database for a match to the properties of the detected light.

[0049] The controller (844) may be configured to automatically detect a subset of pixels within the field of view (124, or 126, or 824, 826, FIG. 5) based at least in part on the difference in the properties of the reflected light between the signals associated with the pixels. For example, the controller (844) may be configured to automatically detect a subset of pixels based at least in part on the difference in the absorption of the reflected light between the signals associated with the pixels. Without being limited by theory, light striking an object will be reflected, transmitted (absorbed), or scattered in response to striking the object such that R+T+S=1 (where R = reflection from the object, T = transmission / absorption into the object, and S = scattering from the object). If a particular subset of pixels reflects a higher percentage of light relative to the surrounding sub-pixels, the controller may isolate these sub-pixels or record or register the pixel locations regarding these different properties within a memory system. In some embodiments, the pixel locations are stored within a passable world mapping system as dense or sparse mapping points such that an additional user of the head-mounted display system can access the map and the subset of pixels can be passed to the additional user and accessed and / or displayed on the display of the second user.

[0050] Referring to FIG. 6, the spectroscopic array (126) may comprise a light source (612) that emits light (613) towards the target object (620). In some embodiments, the light source (612) is an electromagnetic emitter such as a light emitting diode. In some embodiments, the direction of the emitted light (613) is substantially the same as the user's (60) gaze orientation or the user's (60) head pose orientation. In some embodiments, the light detector (614) captures the light (615) reflected from the target object. In some embodiments, the processor (610), which may be the controller (844) depicted in FIG. 5, determines the absorption properties between the emitted light (613) and the reflected light (615) and matches the properties to the properties from the absorption database (630). In some embodiments, the absorption database (630) is stored on a local processing module such as the module (70) depicted in FIG. 2A, for example. In some embodiments, the absorption database (630) is stored on a remote processing module (72) such as that depicted in FIG. 2A.

[0051] The object (620) is depicted as an apple in FIG. 6 for convenience, but the food property has its individual light absorption property, and embodiments of the present invention can be used to identify foods by their light properties, but more advanced uses are also envisioned. In some embodiments, the outward-facing spectroscopic array (126) identifies a tissue source (624), for example, as depicted for illustrative purposes, an arm. The emitted light (613) can impinge on the tissue source (624), and the reflected light (615) can indicate the presence of irregular cells (626) among regular cells (625). As the light source (612) irradiates the tissue source (624), the irregular cells (626) will return light properties different from those of the regular cells (625) to the photodetector (614). The irregular cells (626) can be cancerous, part of scar tissue, or simply healthy cells between tissues that show or have a difference from surrounding cells, indicating where, for example, blood vessels or bone within the tissue source (624) can be located. In some embodiments, the regular cells constitute the majority of the cells in the sample being analyzed, the irregular cells constitute a small portion of the cells of the sample, and the irregular cells exhibit detection properties different from those of the regular cells. In some embodiments, a real-world camera that captures an image at the pixel level may mark such irregular cells (626). As described above, one such marking may be a labeling system that applies a text image in proximity to the irregular cells (626), and another such labeling system may be a color overlay on the irregular cells (626) as seen through the display element 62 (FIG. 5).

[0052] Accordingly, referring again to FIG. 5, a head-mounted member (58) removably attachable to a user's head, one or more electromagnetic radiation emitters (126, 832, 834) coupled to the head-mounted member and configured to emit light with at least two different wavelengths in an inward or outward direction, one or more electromagnetic radiation detectors (126, 828, 830) coupled to the head-mounted member and configured to receive the reflected light after encountering a target object, and one or more electromagnetic radiation emitters (126, 832, 834) and one or more electromagnetic radiation detectors (126, 828, 830) operatively coupled, the one or more electromagnetic radiation emitters being caused to emit a pulse of light while the one or more electromagnetic radiation detectors are caused to detect a level of light absorption associated with the emitted pulse of light and produce a displayable output, a controller (844), a system for determining the nature of tissue or other materials is presented through a wearable computing system such as for AR or VR.

[0053] The head-mounted member (58) may comprise a frame configured to fit on a user's head, such as an eyeglass frame. The eyeglass frame may be a binocular eyeglass frame, and alternative embodiments may be monocular. The one or more emitters (126, 832, 834) may comprise a light source that emits light at a plurality of wavelengths, such as at least one light-emitting diode or other electromagnetic radiation emitter. The plurality of light sources may preferably be configured to emit light at two wavelengths, for example, a first wavelength of about 660 nanometers and a second wavelength of about 940 nanometers.

[0054] In some embodiments, one or more emitters (126, 832, 834) may be configured to emit light at successive, individual wavelengths. In some embodiments, one or more emitters (126, 832, 834) may be configured to emit light at individual wavelengths simultaneously. The one or more electromagnetic radiation detectors (126, 828, 830) may comprise a device selected from the group consisting of a photodiode, a light detector, and a digital camera sensor. The controller (844) may further be configured to cause the plurality of light emitting diodes to emit a periodic pattern of first wavelength on, then second wavelength on, then both wavelengths off, such that the one or more electromagnetic radiation detectors detect the first and second wavelengths separately. The controller (844) may be configured to cause the plurality of light emitting diodes to emit a periodic pattern of first wavelength on, then second wavelength on, then both wavelengths off, in a periodic pulsed pattern of about 30 times per second. The controller (844) may be configured to calculate a ratio of the first wavelength light measurement to the second wavelength light measurement, which ratio is converted to an oxygen saturation reading via a look-up table, at least in part based on Lambert-Beer's law.

[0055] The controller (844) may be configured to operate one or more emitters (126, 832, 834) and one or more electromagnetic radiation detectors (126, 828, 830) to function as a head-mounted spectrometer. The controller (844) may be operably coupled to an optical element (62) that is visible to a user through a head-mounted member (58), such that an output of the controller (844), indicative of a particular material property or tissue property, is visible to the user through the optical element (62).

[0056] FIG. 7A is an exemplary light property absorption chart that can be referenced by an absorption database (630, FIG. 6). As depicted, various light source types such as IR, NIR, or light emitting diodes within the visible spectrum may be optimal for detecting certain tissues and properties within the tissue. In some embodiments, the absorption rate or scatter in the calibration curve is calculated from the emitted light versus the reflected light and applied to a given absorption database (630) as depicted in FIG. 7A to determine underlying tissue and / or properties therein, or to detect anomalies.

[0057] FIG. 7B depicts a potential “overlap” of wavelengths. As depicted, “oxygenated blood” overlaps “deoxygenated blood” at a certain wavelength, which can weaken the results that the spectroscopy process can provide. To avoid this potential overlap, in some embodiments, light at a second, different wavelength is emitted to provide a second light source for measurement and comparison.

[0058] FIG. 8 illustrates a method (850) for using a wearable AR / VR system characterized by spectroscopic components for identifying tissues or properties within a tissue. The method (850) begins with the system orienting a light source towards a target object (851). In some embodiments, the orientation may have a light source that is directed inwardly towards the user's eye, is fixed, or scans the eye such as retinal scanning. In some embodiments, the orientation is performed by determining the user's eye fixation or head pose and orienting the light source in substantially the same direction towards a target object within such fixation or pose field of view, or towards a fiducial or target object.

[0059] In some embodiments, at (852), the light source emits light in an illumination pattern towards the target object or surface. In some embodiments, the light is pulsed in intervals timed by a timer. In some embodiments, the light source emits light of at least one wavelength, and at (854), a radiation detector such as a photodetector receives the reflected light. In some embodiments, the detector is also operably coupled to a timer to indicate whether the received light was initially pulsed at a certain time and to determine a change in the nature of the light in response to reflection on the target object. In some embodiments, (852) starts simultaneously with the mapping at (853), although this sequence may not necessarily be so.

[0060] In some embodiments, the real-world capture system may begin mapping the target object at (853). In some embodiments, such mapping may include receiving passable world data of the target object. In some embodiments, the mapping may include depth sensor analysis of the contour of the target object. In some embodiments, the mapping may include constructing a mesh model of the items within the field of view and referring to them for potential labeling. In some embodiments, the target object is not a specific object within the field of view that can be captured by a depth sensor, but rather a depth plane within the field of view itself.

[0061] In some embodiments, at (855), the controller analyzes the emitted light compared to the received light, such as based on Lambert-Beer's law or optical density relationships (described below) or the scattering pattern of a calibration curve. In some embodiments, at (856), the properties of the light being compared are referenced within an absorption database, which is either stored locally on the system or accessed remotely through the system, to identify the tissue or properties of the tissue of the target object. In some embodiments, the absorption database may comprise a saturation light chart such as that depicted in FIG. 4B or may comprise a calibration curve for a specific light wavelength.

[0062] In some embodiments, at (854), the radiation detector does not receive light of a wavelength different from the wavelength of the light emitted at (852), and the controller cannot perform spectroscopic analysis. In such cases, as in FIG. 7B, it would occur with the wavelength overlap within a range for oxygenated and deoxygenated blood. In some embodiments, at (854a), no wavelength difference is detected between the emitted light and the received light, and sub-step (854b) is initiated by emitting light at a different wavelength from that emitted at (852). The information on the newly emitted and received light is then delivered to the controller at (855).

[0063] In some embodiments, the real-world camera may additionally identify sub-pixels within the field of view that exhibit irregularities at (857), potentially simultaneously with each of (852 - 856), after mapping the target object (853). For example, in some embodiments, the color contrast between pixels is detected during real-world capture at (853), and at (857), these pixels are further modified to potentially highlight such contrast as unhealthy cells. In some embodiments, real-world capture (853) detects irregular lines between pixel clusters, and at (857), the pixels bounded by the irregular lines are marked on the user display (e.g., by a virtual color overlay).

[0064] In some embodiments, the method (850) ends at (858) with the system displaying to the user the tissue or the material properties of the tissue. In some embodiments, the display may comprise a text label that is virtually displayed proximate to the target object, an audio label that describes the target object as determined from the absorption database (630), or a virtual image of a similar tissue or object identified by the absorption database (630) that is juxtaposed proximate to the target object.

[0065] In some embodiments, a significant amount of spectral activity is implemented using software operated by a controller (844) such that an initial task of locating a desired target (e.g., a blood vessel, muscle tissue, bone tissue, or other tissue, and at a desired depth) is performed using digital image processing (using various filters, such as by color, grayscale, and / or intensity threshold analysis, etc.). Such targeting may be performed using pattern, shape recognition, or texture recognition. Cancerous or otherwise irregular cells generally have irregular boundaries. The camera system may identify a series of pixels within the camera field of view (camera 124 and fields of view 18, 22, etc. of FIG. 5) with irregular non-linear patterns and salient points, and may identify boundaries with potentially unhealthy cells, etc. Alternatively, the software and controller may be configured to use the intensity of the center of the target object and the intensity of the surrounding object / tissue to determine the contrast / optical density with the target object and to determine anomalies. Such measurements may simply be used to identify the area of interest for spectral scanning consistent with the present disclosure and are not necessarily a means of identifying the tissue itself. Further, as described above with reference to the irregular cells (626) in FIG. 6, the augmented reality system may overlay a label or color pattern within the boundaries of potentially unhealthy cells and flag / highlight them relative to the surrounding healthy cells.

[0066] In some embodiments, the controller (844) may calculate a density ratio (contrast) and utilize it to calculate oxygen saturation from density ratios of various pulse oximetry properties within a blood vessel. The vascular optical density ("O.D.") at each of two or more emitted wavelengths may be calculated using the following equation. OD 脈管 =-log 10 (Iv / It)

[0067] Where OD 脈管 is the optical density of the blood vessel, Iv is the blood vessel intensity, and It is the surrounding tissue intensity.

[0068] Intravascular oxygen saturation (also referred to as "SO2") may be calculated as the linear ratio of vascular optical density (OD ratio or "ODR") at two wavelengths, as follows. SO2 = ODR = OD 第1の波長 / OD 第2の波長

[0069] In one embodiment, wavelengths of approximately 570 nm (sensitive to deoxygenated hemoglobin) and approximately 600 nm (sensitive to oxygenated hemoglobin) may be used in retinal vascular oximetry as S02 = ODR = OD 600nm / 0D570nm, for example, without considering adjusting the ratio by a calibration factor.

[0070] The foregoing equations are merely exemplary references for calculating material properties. Those skilled in the art will understand that a controller can determine many other tissue properties and relationships. It should be understood that using the controller (844) to perform calculations and / or make determinations may involve performing the calculations locally on a processor within the controller (844). In some other embodiments, using the controller (844) to perform calculations and / or make determinations may involve interfacing with external computing resources, such as resources within a cloud (46) of a server (110) or the like, using the controller.

[0071] (Computer Vision) As discussed above, the spectroscopic system may be configured to detect objects or object characteristics (e.g., properties) within an environment surrounding a user. In some embodiments, objects or object properties present within the environment may be detected using computer vision techniques. For example, as disclosed herein, a camera facing forward of the spectroscopic system may be configured to image an object, and the system may be configured to perform image analysis on the image to determine the presence of features on the object. The system may analyze images obtained by an outward-facing imaging system, absorption determinations, and / or measurements of reflected and / or scattered light to perform object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, etc. One or more computer vision algorithms may be selected as needed and used to perform these tasks. Non-limiting examples of computer vision algorithms include Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retina Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces approach, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, Visual Simultaneous Localization and Mapping (vSLAM) techniques, Sequential Bayesian estimators (e.g., Kalman filter, Extended Kalman filter, etc.), bundle adjustment, adaptive thresholding (and other thresholding techniques), Iterative Closest Point (ICP), Semi-Global Matching (SGM), Semi-Global Block Matching (SGBM), feature point histograms, various machine learning algorithms (e.g., support vector machine, k-nearest neighbor algorithm, naive Bayes, neural networks (including convolutional or deep neural networks), or other supervised / unsupervised models, etc.).

[0072] As discussed herein, an object or a feature (including properties) of an object may be detected based on one or more criteria (e.g., absorption, light reflection, and / or light scattering at one or more wavelengths). When a spectroscopy system uses computer vision algorithms or data received from one or more sensor assemblies (which may or may not be part of the spectroscopy system) to detect the presence or absence of a criterion in the ambient environment, the spectroscopy system may then signal the presence of the object or feature.

[0073] One or more of these computer vision techniques may also be used in combination with data obtained from other environmental sensors (e.g., microphones, GPS sensors, etc.) to detect and determine various properties of the objects detected by the sensors.

[0074] (Machine learning) Various machine learning algorithms may be used to learn to identify the presence of an object or features of an object. Once trained, the machine learning algorithm may be stored by the spectroscopic system. Some examples of machine learning algorithms can include supervised or unsupervised machine learning algorithms, regression algorithms (e.g., ordinary least squares regression, etc.), instance-based algorithms (e.g., learning vector quantization, etc.), decision tree algorithms (e.g., classification and regression trees, etc.), Bayesian algorithms (e.g., naive Bayes, etc.), clustering algorithms (e.g., k-means clustering, etc.), association rule learning algorithms (e.g., Apriori algorithm, etc.), artificial neural network algorithms (e.g., Perceptron, etc.), deep learning algorithms (e.g., Deep Boltzmann Machine, i.e., deep neural network, etc.), dimensionality reduction algorithms (e.g., principal component analysis, etc.), ensemble algorithms (e.g., Stacked Generalization, etc.), and / or other machine learning algorithms. In some embodiments, individual models can be customized for individual datasets. For example, a wearable device can generate or store a base model. The base model is used as a starting point and may generate additional models specific to a data type (e.g., a particular user), a dataset (e.g., a set of absorption rates, light reflectance, and / or light scattering values obtained at one or more wavelengths), a conditional situation, or other variations. In some embodiments, the spectroscopic system can be configured to utilize multiple techniques to generate models for the analysis of aggregated data. Other techniques may include using predefined thresholds or data values.

[0075] The criteria for detecting an object or a feature of an object may include one or more threshold conditions. If the analysis of data obtained by a sensor (e.g., a camera or a photodetector) indicates that the threshold conditions are met, the spectroscopic system may provide a signal indicating the detection of the presence of an object in the ambient environment. The threshold conditions may involve quantitative and / or qualitative measurements. For example, the threshold conditions may include a score or percentage associated with the likelihood of the presence of an object and / or a feature. The spectroscopic system may compare a score calculated from the sensor's data with a threshold score. If the score is higher than the threshold level, the spectroscopic system may signal the detection of the presence of an object or an object feature. In some other embodiments, the spectroscopic system may signal the absence of an object or a feature if the score is lower than the threshold.

[0076] It should be understood that the processes, methods, and algorithms described herein and / or depicted in the accompanying figures are each embodied in one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, thereby being fully or partially automated. The code modules can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some implementations, certain operations and methods can be performed by circuitry specific to a given function. In some embodiments, the code modules may be executed by hardware within a controller (844) (FIG. 5) and / or in the cloud (46) (e.g., a server (110)).

[0077] Furthermore, because the functional implementations of the present disclosure are sufficiently mathematically, computationally, or technically complex, special-purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, for example, due to the amount or complexity of the calculations involved or to provide the results substantially in real time. For example, video may include many frames, each of which may have millions of pixels, and specifically programmed computer hardware is required to process the video data to provide the desired image processing tasks or applications in a commercially reasonable amount of time.

[0078] A code module or any type of data may be stored on any type of non-transitory computer-readable medium such as a physical computer storage device including a hard drive, solid state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage device, combinations of the same, and / or equivalents. In some embodiments, the non-transitory computer-readable medium may be part of one or more of a local processing and data module (70, FIG. 2C), a remote processing module (72, FIG. 2D), and a remote data repository (74, FIG. 2D). The methods and modules (or data) may also be transmitted as data signals generated on various computer-readable transmission media including wireless-based and wire / cable-based media (e.g., as part of a carrier wave or other analog or digital propagated signal) and may take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). The results of the disclosed process or process steps may be persistently or otherwise stored within any type of non-transitory tangible computer storage device or communicated via a computer-readable transmission medium.

[0079] Any process, block, state, step, or functionality in a flowchart described herein and / or depicted in the accompanying figures is to be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in a process. The various processes, blocks, states, steps, or functionality can be combined, rearranged, added, deleted, modified, or otherwise changed from the exemplary embodiments provided herein. In some embodiments, additional or different computing systems, or code modules, may implement some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the associated blocks, steps, or states can be performed in other suitable sequences, e.g., continuously, in parallel, or in some other manner. Tasks or events can be added to or removed from the disclosed exemplary embodiments. Further, the separation of the various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems can generally be integrated together in a single computer product or packaged in multiple computer products.

[0080] Various exemplary embodiments of the present invention are described herein. By way of non-limiting illustration, reference is made to these examples. They are provided to illustrate more broadly applicable aspects of the present invention. Various changes may be made to the invention described, and equivalents may be substituted without departing from the true spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation, material, composition, process, act or step of the process to the objective, spirit or scope of the present invention. Further, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein can be readily separated from or combined with features of any of the other several embodiments without departing from the scope or spirit of the present invention. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0081] The present invention includes methods that can be performed using a target device. The method may include the act of providing such a suitable device. Such provision may be performed by an end user. In other words, the act of "providing" simply requires that the end user acquire, access, approach, position, configure, activate, power on, or otherwise act so as to provide the essential device in the target method. The methods described herein may be performed in any order of the described events that is logically possible, as well as in the order of the described events.

[0082] Exemplary aspects of the present invention are described above, along with details regarding material selection and manufacturing. Regarding other details of the present invention, these are understood in relation to the above-referenced patents and publications and can generally be grasped or understood by those skilled in the art; the same can apply to the method-based aspects of the present invention from the perspective of additional acts as generally or theoretically employed.

[0083] In addition, although the present invention has been described with reference to several embodiments incorporating various features optionally, the present invention is not limited to what is described or indicated as being contemplated with respect to each variation of the present invention. Various modifications may be made to the invention described herein, and equivalents may be substituted (whether or not described herein or not included for the sake of brevity) without departing from the true spirit and scope of the present invention. In addition, when ranges of values are provided, it is understood that all intervening values between the upper and lower limits of that range, as well as any other defined or intervening values within that defined range, are included within the present invention.

[0084] Also contemplated is that any optional feature of a variation of the invention described herein may be described and claimed independently or in combination with any one or more of the features described herein. References to items in the singular include the possibility that there are multiple identical items. More specifically, as used herein and in the claims associated therewith, the singular forms "a", "an", "said", and "the" include plural referents unless specifically stated otherwise. In other words, the use of the article enables "at least one" of the item in question in the above description as well as in the claims associated with this disclosure. Further, note that such claims may be drafted to exclude any optional element. Accordingly, this description serves the purpose of acting as a preamble for the use of such exclusive terms as "only", "solely", and equivalents thereof, or the use of "negative" limitations, in connection with the recitation of claim elements.

[0085] Without using such exclusive terms, the term "comprising" in the claims associated with this disclosure is intended to allow the inclusion of any additional elements, whether or not a given number of elements are recited in such claims or an addition of features can be regarded as transforming the nature of the elements recited in such claims. Unless otherwise specifically defined herein, all technical and scientific terms used herein are to be given the broadest possible generally understood meaning while maintaining the validity of the claims.

[0086] The scope of the present invention is not limited to the provided examples and / or this specification, but rather is limited only by the scope of the claims associated with this disclosure.

Claims

**Claim 1**: A spectroscopic method, wherein the spectroscopic method is under the control of one or more processors of a wearable spectroscopic system, determining a target object through at least one of digital image processing or intensity threshold analysis on an image captured by a digital image sensor; causing one or more light sources of the wearable spectroscopic system to irradiate the target object by emitting pulses of light having at least two different wavelengths; receiving one or more signals from at least one electromagnetic radiation detector of the wearable spectroscopic system, wherein the one or more signals indicate a detected level of light absorption, and the detected level of light absorption is related to the emitted light pulses and the light reflected from the target object irradiated by the emitted light pulses; comparing the detected level of light absorption with stored absorption data including absorption properties of a plurality of materials, and determining an output based on matching the detected level of light absorption with one or more of the plurality of materials of the stored absorption data; causing the wearable spectroscopic system to display, via a head-mounted display of the wearable spectroscopic system, an output based on the detected level of light absorption to a user wearing the wearable spectroscopic system A spectroscopic method comprising the above. **Claim 2**: The method according to claim 1, wherein the absorption data is stored in a local memory connected to the wearable spectroscopic system. **Claim 3**: The method according to claim 1, wherein the absorption data is stored in a memory of a remote computing device. **Claim 4**: The method according to claim 1, wherein the one or more light sources include a plurality of light-emitting diodes. **Claim 5**: The method according to claim 1, wherein the at least two different wavelengths include a first wavelength of about 660 nanometers and a second wavelength of about 940 nanometers. **Claim 6**: The method according to claim 1, wherein the at least two different wavelengths are emitted continuously. **Claim 7**: The method according to claim 1, wherein the at least two different wavelengths are emitted simultaneously. **Claim 8**: The method according to claim 1, wherein the at least two different wavelengths are emitted in a cycle pattern in which the first wavelength is turned on, then the second wavelength is turned on, and then both the first wavelength and the second wavelength are turned off, such that the at least one electromagnetic radiation detector detects the first wavelength and the second wavelength at a plurality of different times. **Claim 9**: The method according to claim 1, further comprising calculating a ratio of the first wavelength light measurement to the second wavelength light measurement and converting the ratio into a property of the tissue. **Claim 10**: The method according to claim 9, wherein the output includes the property of the tissue. **Claim 11**: The method according to claim 9, wherein the property of the tissue includes at least one property selected from the group consisting of an estimated blood saturation level, the presence of abnormal cells, and the presence of cancerous cells. **Claim 12**: The method according to claim 1, wherein the at least one electromagnetic radiation detector includes a device selected from the group consisting of a photodiode and a light detector. **Claim 13**: The method according to claim 1, wherein the at least one electromagnetic radiation detector includes a digital image sensor. **Claim 14**: The digital image sensor includes a plurality of pixels, and the one or more processors are configured to automatically detect a subset of the pixels that receive the light reflected after encountering a predetermined tissue and generate an output that displays the location of the subset of the pixels indicating the predetermined tissue. The method according to claim 13. **Claim 15**: The method according to claim 1, further comprising determining a posture of the user's head via an inertial measurement unit positioning system of the wearable spectroscopy system. **Claim 16**: The method according to claim 15, wherein the one or more light sources emit the light in a direction corresponding to the posture of the user's head. **Claim 17**: Causing the wearable spectroscopy system to display an output based on the detected level of light absorption includes rendering an output representing the target object on an optical element of a head-mounted display of the wearable spectroscopy system. The method according to claim 1. The method according to claim 17, wherein rendering the output representing the target object includes rendering a display label indicating an absorption level of the target object. The method according to claim 1, wherein determining the target object includes receiving an image captured by a camera of the wearable spectroscopy system and identifying a series of pixels within a camera field of view having an irregular pattern or a non-linear pattern as representing the target object.

Images