User heart rate detection system and method using fusion of multi-sensor data

Abstract

Systems and methods for fusing multiple types of sensor data to determine a heart rate of a user. An accelerometer obtains accelerometer data associated with the user over a time period, and a gyroscope obtains gyroscope data associated with the user over the time period. Also, a camera obtains a plurality of images of the user's eye over the time period. The plurality images are analyzed to generate image data of the user's eyelid over the time period. The accelerometer data, the gyroscope data, and the image data are fused into fused sensor data, and a heart rate of the user is determined from the fused sensor data.

Description

USER HEART RATE DETECTION SYSTEM AND METHOD USING FUSION OF MULTI-SENSOR DATATechnical Field

[0001] The present disclosure relates generally to health detection mechanisms, and more particularly to detecting a person’s heart rate from a fusion of different sensor data.BACKGROUNDDescription of the Related Art

[0002] Clinical assessment of a person’s health commonly includes measurement of a person’s heart rate, which may also include heart rate variability or respiration rate. Commonly, a person’s heart rate is determined at a clinic or care facility using a heart rate monitor or by a trained medical provider’s quasi-objective assessment of arterial palpation. Likewise, a person’s respiration rate is commonly determined at a clinic or care facility by a trained medical provider using a stethoscope. A person’s heart rate, or heart rate variability or respiration rate or a combination thereof, provide information to the medical provider that can be used to assess a wide range of health conditions or disorders. Obtaining a person’s heart rate, heart rate variability, or respiration rate when the person is not at a clinic or care facility or is not in the presence of a medical provider can be difficult to determine. It is with respect to these and other considerations that the embodiments described herein have been made.BRIEF SUMMARY

[0003] Embodiments are generally directed to systems and methods for fusing different types of sensor data to determine a user’s heart rate, as well as the user’s heart rate variability and the user’s respiration rate. In some embodiments, the sensor data is obtained from sensors embedded or integrated into an augmented-reality or a virtual-reality headset that is worn by a user. Accelerometer data and gyroscope data associated with the user is obtained. A plurality of images of the user’s eye are also obtained. The images may be pre-processed to identify an area of interest, such as the lower eyelid. Image data, such as infrared reflective intensity, may be generated from the area of interest within the plurality of images. The accelerometer data, the gyroscope data, and the image data are then fused into fused sensor data. In some embodiments, the accelerometer data, the gyroscope data, and the image data are weighted before being fusedinto the fused sensor data. A heart rate of the user is determined from the fused sensor data. In various embodiments, an artificial intelligence model may be employed on the fused sensor data to output the heart rate, wherein the artificial intelligence model is trained on fused historical accelerometer data, historical gyroscope data, historical image data, and historical accurate heart rates obtained from a plurality of users. In some embodiments, a heart rate variability of the user may be determined from the determined heart rate or from the fused sensor data. In other embodiments, a respiration rate of the user may be determined from the fused sensor data.

[0004] Embodiments described herein fuse inertial sensor data and camera data, together with machine learning or artificial intelligence techniques, to provide an objective assessment of a user’s heart rate, or heart rate variability or respiration rate.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0005] Non-limiting and non-exhaustive embodiments are described with reference to the following drawings. In the drawings, like reference numerals refer to like parts throughout the various figures unless otherwise specified.

[0006] For a better understanding of the present invention, reference will be made to the following Detailed Description, which is to be read in association with the accompanying drawings:

[0007] Figure 1 illustrates a context diagram of an environment for determining biological information of a person from fused sensor data in accordance with embodiments described herein;

[0008] Figure 2 is a context diagram of non-limiting embodiments of systems for obtaining and fusing different types of sensor data to determine a person’s biological information in accordance with embodiments described herein;

[0009] Figure 3 is a use-case example illustrating fusion of different types of sensor data to determine a person’s biological information in accordance with embodiments described herein;

[0010] Figure 4 is a context diagram of non-limiting embodiments for weighting and fusing different types of sensor data to determine a person’s biological information in accordance with embodiments described herein;

[0011] Figure 5 illustrates a logical flow diagram showing one embodiment of a process for obtaining and fusing different types of sensor data to determine a person’s biological information in accordance with embodiments described herein; and

[0012] Figure 6 shows a system diagram that describes various implementations of computing systems for implementing embodiments described herein.DETAILED DESCRIPTION

[0013] The following description, along with the accompanying drawings, sets forth certain specific details in order to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant art will recognize that the disclosed embodiments may be practiced in various combinations, without one or more of these specific details, or with other methods, components, devices, materials, etc. In other instances, well- known structures or components that are associated with the environment of the present disclosure, including but not limited to the communication systems and networks, have not been shown or described in order to avoid unnecessarily obscuring descriptions of the embodiments. Additionally, the various embodiments may be methods, systems, media, or devices. Accordingly, the various embodiments may be entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects.

[0014] Throughout the specification, claims, and drawings, the following terms take the meaning explicitly associated herein, unless the context clearly dictates otherwise. The term “herein” refers to the specification, claims, and drawings associated with the current application. The phrases “in one embodiment,” “in another embodiment,” “in various embodiments,” “in some embodiments,” “in other embodiments,” and other variations thereof refer to one or more features, structures, functions, limitations, or characteristics of the present disclosure, and are not limited to the same or different embodiments unless the context clearly dictates otherwise. As used herein, the term “or” is an inclusive “or” operator, and is equivalent to the phrases “A or B, or both” or “A or B or C, or any combination thereof,” and lists with additional elements are similarly treated. The term “based on” is not exclusive and allows for being based on additional features, functions, aspects, or limitations not described, unless the context clearly dictates otherwise. In addition, throughout the specification, the meaning of “a,” “an,” and “the” include singular and plural references.

[0015] Figure 1 illustrates a context diagram of an environment 100 for determining biological information of a person from fused sensor data in accordance with embodiments described herein. Biological information may include a person’s heart rate, heart rate variability, or respiration rate, or some combination thereof. Moreover, the terms person, patient, and user may be used interchangeably herein.

[0016] Environment 100 includes a user computing device 102 and a biological- information detection computing device 104. Although illustrated separately, the user computing device 102 and the biological-information detection computing device 104 may be integrated into a single computing device, the same computing device, or have their functionality performed by a plurality of computing devices.

[0017] The user computing device 102 is a computer device or system that is held or worn by a user and configured to obtain sensor data related to the user. Examples of the user computing device 102 may include an augmented-reality headset, a virtual-reality headset, a mixed-reality headset, a smartphone, a tablet computer, a smartwatch, or other user-computing device positioned to be in contact with the user or in communication with sensors positioned to capture sensor data of the user. The user computing device 102 includes sensors 112, which are configured to capture, record, or obtain a plurality of different types of sensor data. Sensors 112 may include one or more inertial measurement units or sensors (e.g., an accelerometer, a gyroscope, an altimeter, etc.) and one or more cameras (e.g., infrared camera, near-infrared camera, visible-light camera, etc.). In some embodiments, one or more of sensors 112 may be embedded into or attached to the user computing device 102. In other embodiments, one or more of sensors 112 may be separate from the user computing device 102 and configured to attach to or be worn by the user independent of the user computing device 102, or otherwise collect sensor data of the user.

[0018] The biological-information detection computing device 104 is a computer device or system that is configured to obtain sensor data and determine biological information of a user. In various embodiments, the biological-information detection computing device 104 includes a biological-information detection system 114 that receives or obtains sensor data from the user computing device 102. The biological-information detection system 114 fuses the sensor data to determine biological information of the user of the user computing device 102. The biological information determined by the biological-information detection system 114 may include the user’s heart rate, the user’s heart rate variability, or the user’s respiration rate, or some combination thereof. In some embodiments, the biological-information detection computing device 104 may output the user’s heart rate, the user’s heart rate variability, or the user’s respiration rate, or some combination thereof, to a display device (not illustrated). In other embodiments, the biological-information detection computing device 104 may transmit the user’s heart rate, the user’s heart rate variability, or the user’s respiration rate, or somecombination thereof, to another computing device (not illustrated), such as a computing device of a medical provider.

[0019] Figure 2 is a context diagram of non-limiting embodiments of example systems 200 for obtaining and fusing different types of sensor data to determine a person’s biological information in accordance with embodiments described herein. Example 200 includes a user computing device 102 and a biological-information detection computing device 104, similar to environment 100 in Figure 1.

[0020] The user computing device 102 includes sensors 112, which includes accelerometer 204, gyroscope 206, and camera 208. The user computing device 102 is configured to obtain accelerometer data from the accelerometer 204, gyroscope data from the gyroscope 206, and image data from the camera 208, which may be individually or collectively referred to as sensor data. The user computing device 102 provides the obtained sensor data to the biological -information detection computing device 104.

[0021] The accelerometer 204 is a configured to capture or determine three-dimensional acceleration of motion associated with the user of the user computing device 102. The gyroscope 206 is configured to capture or determine three-dimensional orientation or angular velocity associated with the user of the user computing device 102. In various embodiments, the accelerometer 204 or the gyroscope 206, or both, may be implemented as an embedded electromechanical sensor within the user computing device 102. In other embodiments, the accelerometer 204 or the gyroscope 206, or both, may be implemented as one or more standalone sensors that are affixed to the user of the user computing device 102.

[0022] The camera 208 is positioned and configured to capture a plurality of sequential images or frames of the user’s eye or an area adjacent to the user’s eye, or both. In some embodiments, the camera 208 may capture infrared images, near-infrared images, visible-light images (e.g., RGB images), or some combination thereof. In some embodiments, the camera 208 may be embedded or attached to the user computing device 102. In other embodiments, the camera 208 may be a standalone device that is separate from the user computing device 102.

[0023] The biological-information detection computing device 104 is configured to receive the accelerometer data, gyroscope data, and image data from the user computing device 102. In various embodiments, the biological-information detection computing device 104 includes a biological-information detection system 114, which includes a sensor pre-processing and filtering module 214, a sensor-data fusion module 216, a heart-rate detection module 218, a heart-rate-variability detection module 220, and a respiration-rate detection module 222. Insome embodiments, the heart-rate-variability detection module 220 or the respiration-rate detection module 222 may be optional and may not be included in biological-information detection computing device 104.

[0024] The sensor pre-processing and filtering module 214 is configured to receive the sensor data - i.e., the accelerometer data, the gyroscope data, the image data, or some combination thereof - from the user computing device 102 and to pre-process and filter the sensor data for fusion by the sensor-data fusion module 216. In some embodiments, the sensor pre-processing and filtering module 214 pre-processes the sensor data to remove noise or corrupted data. The sensor pre-processing and filtering module 214 may also perform other amplitude or temporal filtering on the sensor data.

[0025] In some embodiments, the sensor pre-processing and filtering module 214 may perform other pre-processing or filtering on the sensor data. For example, the sensor preprocessing and filtering module 214 may analyze the image data captured by camera 208 to identify an area of interest around the user’s eye, such as a portion of the user’s lower eyelid. In at least one embodiment, one or more machine learning or artificial intelligence techniques may be utilized on each image of the image data to identify the area of interest. The sensor preprocessing and filtering module 214 may also extract other features from the image data, such as infrared reflectance of the area of interest of the eyelid, pixel intensity of area of interest, pupil diameter, glint intensity, etc. these extracted features may be obtained from a plurality of camera sensor frames or images to generate the image data (e.g., a signal representing infrared reflectance over time). The sensor pre-processing and filtering module 214 outputs clean accelerometer data, clean gyroscope data, and clean image data to the sensor-data fusion module 216.

[0026] The sensor-data fusion module 216 is configured to fuse the received sensor data - i.e., the clean accelerometer data, the clean gyroscope data, and the clean image data - to generate and output R-peaks data. The R-peaks data is a combined or fused data output from which the user’s heart rate, heart rate variability, or respiration rate can be determined. In various embodiments, the sensor-data fusion module 216 may weigh the sensor data before it is fused or combined to generate the R-peaks data, which is described in more detail below in conjunction with Figure 4.

[0027] The heart-rate detection module 218 is configured to receive the R-peaks data from the sensor-data fusion module 216 and to determine the user’s heart rate from the R-peaks data. In various embodiments, heart-rate detection module 218 may employ one or moremachine learning or artificial intelligence techniques on the R-peaks data to determine the user’s heart rate. For example, the heart-rate detection module 218 may input the R-peaks data into a trained artificial intelligence model to output the user’s heart rate. In some embodiments, the artificial intelligence model may be trained with historical R-peaks data (e.g., a fusion of accelerometer data, gyroscope data, and image data, as described herein) and known heat rate data from a plurality of other users. In at least one embodiment, a plurality of artificial intelligence models may be trained for different demographics of users. In this way, the heartrate detection module 218 can utilize the artificial intelligence model that was trained using data from the same or similar demographic as the user of the user computing device 102.

[0028] The heart-rate-variability detection module 220 is configured to receive the determined heart rate from the heart-rate detection module 218 and to calculate variations of the user’s heart rate over time.

[0029] The respiration-rate detection module 222 is configured to receive the R-peaks data from the sensor-data fusion module 216 and to determine the user’s respiration rate from the R-peaks data. In various embodiments, respiration-rate detection module 222 may employ one or more machine learning or artificial intelligence techniques on the R-peaks data to determine the user’s respiration rate. For example, the respiration-rate detection module 222 may input the R-peaks data into a trained artificial intelligence model to output the user’s respiration rate. In some embodiments, the artificial intelligence model may be trained with historical R-peaks data (e.g., a fusion of accelerometer data, gyroscope data, and image data, as described herein) and known respiration rate data from a plurality of other users. In at least one embodiment, a plurality of artificial intelligence models may be trained for different demographics of users. In this way, the respiration-rate detection module 222 can utilize the artificial intelligence model that was trained using data from the same or similar demographic as the user of the user computing device 102.

[0030] Although embodiments are described herein as fusing a combination of accelerometer data, gyroscope data, and image data, embodiments are not so limited. For example, in some embodiments, the sensor-data fusion module 216 may fuse only accelerometer and image data to generate the R-peaks data from which the heart-rate detection module 218 determines the user’s heart rate. As another example, the sensor-data fusion module 216 may fuse only gyroscope data and image data to generate the R-peaks data from which the heart-rate detection module 218 determines the user’s heart rate.

[0031] Moreover, although sensors 112 illustrates a single accelerometer, a single gyroscope, and a single camera, embodiments are not so limited. In some embodiments, one or more accelerometers, one or more gyroscopes, and one or more cameras may be utilized to generate one or more sets of accelerometer data, one or more sets of gyroscope data, and one or more sets of image data. Each set of data can then be pre-processed and filtered by the sensor pre-processing and filtering module 214, such that a clean version of each set of data is then fused by the sensor-data fusion module 216 to generate the R peaks data, as described herein.

[0032] The biological-information detection system 114 can output or use the heart rate determined by the heart-rate detection module 218, the heart rate variability determined by the heart-rate-variability detection module 220, or the respiration rate determined by the respirationrate detection module 222, which may collectively be referred to as the determined biological information. For example, in some embodiments, the biological-information detection system 114 can output or transmit the determined biological information to a medical provider. In other embodiments, biological -information detection system 114 may determine a stress level of the user, an excitement level of the user, or an engagement level of the user based on the determined biological information. This additional information can then be displayed to the user or utilized by other systems to provide output to the user (e.g., modifying game play of a video game being operated by the user as user computing device 102 is collecting sensor data and the biological- information detection system 114 is determining the user’s heart rate, as described herein).

[0033] Figure 3 is a use-case example 300 illustrating fusion of different types of sensor data to determine a person’s biological information in accordance with embodiments described herein. Example 300 shows example signals of clean image data, clean accelerometer data, and clean gyroscope data. The clean image data may be a signal representing infrared reflectance of an area of interest of a user’s eyelid over time. The clean accelerometer data may be a signal representing the user’s movement over time. Because the accelerometer data may include three- dimensional data, the clean accelerometer data may include a separate signal for each separate dimension resulting in three separate motion signals, not illustrated. The clean gyroscope data may be a signal representing the user’s orientation over time. Because the gyroscope data may include three-dimensional data, the clean gyroscope data may include a separate signal for each separate dimension resulting in three separate orientation signals, not illustrated.

[0034] By employing embodiments herein, the clean image data, the clean accelerometer data, and the clean gyroscope data is fused to generate the illustrated R-peaks data. From the R-peaks data, the user’s heart rate, heart rate variability, or respiration rate, or some combination thereof, can be determined, as described herein.

[0035] Figure 4 is a context diagram of non-limiting embodiments for weighting and fusing different types of sensor data by the sensor-data fusion module 216 to determine a person’s heart rate in accordance with embodiments described herein. The sensor-data fusion module 216 receives clean accelerometer data 402, clean gyroscope data 412, and clean image data 422, such as from sensor pre-processing and filtering module 214 in Figure 2.

[0036] From the clean accelerometer data 402, the sensor-data fusion module 216 generates estimated variables 404 and a signal quality measure 406. The estimated variables 404 may be numerical predictions or estimates of the clean accelerometer data. In some embodiments, the estimated variables 404 may be normalized with respect to the gyroscope data and the image data. The signal quality measure 406 may be a numerical value identifying a quality of the clean accelerometer data. The sensor-data fusion module 216 generates a weight 408 from the signal quality measure 406, which then applied to the estimated variables 404 at operation 410 to generate weighted accelerometer data. In some embodiments, operation 410 is a multiplication operation.

[0037] From the clean gyroscope data 412, the sensor-data fusion module 216 generates estimated variables 414 and a signal quality measure 416. The estimated variables 414 may be numerical predictions or estimates of the clean gyroscope data. In some embodiments, the estimated variables 414 may be normalized with respect to the accelerometer data and the image data. The signal quality measure 416 may be a numerical value identifying a quality of the clean gyroscope data. The sensor-data fusion module 216 generates a weight 418 from the signal quality measure 416, which then applied to the estimated variables 414 at operation 420 to generate weighted gyroscope data. In some embodiments, operation 420 is a multiplication operation.

[0038] From the clean image data 422, the sensor-data fusion module 216 generates estimated variables 424 and a signal quality measure 426. The estimated variables 424 may be numerical predictions or estimates of the clean image data (e.g., infrared reflectance intensity). In some embodiments, the estimated variables 424 may be normalized with respect to the accelerometer data and the gyroscope data. The signal quality measure 426 may be a numerical value identifying a quality of the clean image data. The sensor-data fusion module 216 generates a weight 428 from the signal quality measure 426, which then applied to the estimated variables424 at operation 430 to generate weighted image data. In some embodiments, operation 430 is a multiplication operation.

[0039] The weighted accelerometer data from operation 410, the weighted gyroscope data from operation 420, and the weighted image data from operation 430 are then fused or combined together at operation 434 to generate the R-peaks data. In some embodiments, operation 434 is an addition operation. As described herein, the R-peaks data is then used by one or more machine learning or artificial intelligence techniques to determine the user’s heart rate, heart rate variability, or respiration rate, or some combination thereof.

[0040] Figure 5 illustrates a logical flow diagram showing one embodiment of a process 500 for obtaining and fusing different types of sensor data to determine a person’s heart rate in accordance with embodiments described herein.

[0041] Process 500 begins, after a start block, at block 402, where multiple types of sensor data of a user are obtained. As described herein, the obtained sensor data may include accelerometer data from one or more accelerometers in contact with a user, gyroscope data from one or more gyroscopes in contact with a user, image data of the user’s eye, or some combination thereof.

[0042] Process 500 proceeds after block 502 to block 504, where the sensor data is pre- processed to generate clean sensor data. In some embodiments, the sensor data is filtered to remove corrupt or duplicative data. In other embodiments, the sensor data is pre-processed to extract features from the sensor data, as described herein. For example, images of the user’s eye may be analyzed to detect a lower eyelid of the user and an area of interest on the eyelid. A reflective intensity of the area of interest is then determined from each of the images, from which clean image data is generated.

[0043] Process 500 continues after block 504 at block 506, where the clean sensor data is weighted. In various embodiments, separate weights are applied to the different types of sensor data. In some embodiments, the separate weights are determined by a developer or an administrator. In other embodiments, the separate weights may be determined by the quality of the received sensor data.

[0044] Process 500 proceeds next after block 506 to block 508, where the weighted sensor data is fused or otherwise combined to generate R-peaks data.

[0045] Process 500 continues next after block 508 at block 510, where the user’s heart rate is determined from the fused sensor data. As described herein, one or more machinelearning or artificial intelligence techniques may be trained and employed on the R-peaks data to determine the user’s heart rate.

[0046] Process 500 proceeds after block 510 to block 512, where the user’s heart rate variability is determined from the determined heart rate. In various embodiments, the heart rate variability tracks changes to the user’s heart rate over time.

[0047] Process 500 continues after block 512 at block 514, where the user’s respiration rate is determined from the fused sensor data. As described herein, one or more machine learning or artificial intelligence techniques may be trained and employed on the R-peaks data to determine the user’s respiration rate.

[0048] Process 500 proceeds next after block 514 to block 516, where the determined heart rate, the determined heart rate variability, and the determined respiration rate are output. In some embodiments, this determined biological information is transmitted or displayed to a medical provider. In other embodiments, this determined biological information is provided to other systems for providing other information to the user.

[0049] After block 516, process 500 terminates or otherwise returns to a calling process to perform other actions. Although not illustrated, process 500 may loop after block 516 to block 502 to continue to receive sensor data, fuse the sensor data, and determine biological information of the user. Moreover, although process 500 is illustrated as a sequential process, embodiments are not so limited. Rather, some operations may be performed in parallel or in an order other than what is illustrated. For example, blocks 510 and 514 may be performed in parallel, or block 514 may be performed before block 510.Figure 6 shows a system diagram that describes various implementations of computing systems for implementing embodiments described herein. As described herein, the system includes a user computing device 102 and a biological-information detection computing device 104.The biological-information detection computing device 104 receives multiple types of sensor data from the user computing device 102, fuses the sensor data together, and then determines the user’s heart rate (or heart rate variability or respiration rate) from the fused data, as described herein. One or more special-purpose computing systems may be used to implement the biological -information detection computing device 104. Accordingly, various embodiments described herein may be implemented in software, hardware, firmware, or in some combination thereof. The biological-information detection computing device 104 may include memory 602,processor 614, I / O interfaces 618, other computer-readable media 620, and network connections 622.Processor 614 may include one or more central processing units, circuitry, or other computing components or units - collectively referred to as a processor or one or more processors - that are configured to performed embodiments herein or to execute computer instructions to perform embodiments described herein. In some embodiments, a single processor may operate individually to perform embodiments described herein. In other embodiments, a plurality of processors may operate to collectively perform embodiments described herein, such that one or more processors may operate to perform some, but not all, of the embodiments described herein.Memory 602 may include one or more various types of non-volatile and / or volatile storage technologies. Examples of memory 602 may include, but are not limited to, flash memory, hard disk drives, optical drives, solid-state drives, various types of random access memory (RAM), various types of read-only memory (ROM), other computer-readable storage media (also referred to as processor-readable storage media), or the like, or any combination thereof. Memory 602 may be utilized to store information, including computer-readable instructions that are utilized by processor 614 to operate or perform actions, including embodiments described herein.Memory 602 may have stored thereon a biological detection system 114 and other programs and data 610. The biological-information detection system 114 may include a sensorpre-processing and filtering module 214, a sensor-data fusion module 216, a heart-rate detection module 218, a heart-rate-variability detection module 220, and a respiration-rate detection module 222, which are described herein. Although the sensor-pre-processing and filtering module 214, the sensor-data fusion module 216, the heart-rate detection module 218, the heart- rate-variability detection module 220, and the respiration-rate detection module 222 are illustrated as separate components, embodiments are not so limited. Rather, one or more computing components or modules may individually or collectively be employed to perform the functionality of the sensor-pre-processing and filtering module 214, the sensor-data fusion module 216, the heart-rate detection module 218, the heart-rate-variability detection module 220, and the respiration-rate detection module 222.The other programs and data 610 may include operating systems or other information. For example, in some embodiments, the other programs and data 610 may store the receivedsensor data, the determined heart rate, the determined heart rate variability, or the determined respiration rate.In various embodiments, the network connections 622 include transmitters and receivers (not illustrated) to communicate with the user computing device 102 to receive sensor data. I / O interfaces 618 may include one or more data input or output interfaces that can be used to receive information from or display information to a user or a medical provider. For example, the I / O interfaces 618 may include an input interface to receive demographic information about the user and an output interface to display the determined biological information of the user. Other computer-readable media 620 may include other types of stationary or removable computer- readable media, such as removable flash drives, external hard drives, or the like.The user computing device 102 collects or obtains multiple types of sensor data from a user and provides the sensor data to the biological-information detection computing device 104, as described herein. One or more special-purpose computing systems may be used to implement the user computing device 102. Accordingly, various embodiments described herein may be implemented in software, hardware, firmware, or in some combination thereof. The user computing device 102 includes sensors 112 and may also include memory 652, processor 664, I / O interfaces 668, other computer-readable media 670, and network connections 672, which may be similar to memory 602, processor 614, I / O interfaces 618, other computer-readable media 620, and network connections 622 of the biological-information detection computing device 104.

[0050] The following is a summarization of the claims as filed.

[0051] A system may be summarized as comprising: an accelerometer configured to obtain accelerometer data associated with a user over a time period; a gyroscope configured to obtain gyroscope data associated with the user over the time period; a camera configured to obtain a plurality of images of the user’s eye over the time period; and at least one processor. The at least one processor may be configured to: receive the accelerometer data from the accelerometer; receive the gyroscope data from the gyroscope; receive the plurality of images from the camera; analyze the plurality images to generate image data of the user’ s eye over the time period; fuse the accelerometer data, the gyroscope data, and the image data into fused sensor data; and determine a heart rate of the user from the fused sensor data.

[0052] The at least one processor may fuse the accelerometer data, the gyroscope data, and the image data into the fused sensor data by being further configured to: weight theaccelerometer data, the gyroscope data, and the image data; and fuse the weighted accelerometer data, the weighted gyroscope data, and the weighted image data into the fused sensor data.

[0053] In some embodiments, the accelerometer, the gyroscope, and the camera are embedded into an augmented or virtual-reality headset.

[0054] The at least one processor may be further configured to determine a heart rate variability of the user from the fused sensor data.

[0055] The at least one processor may be further configured to determine a respiration rate of the user from the fused sensor data.

[0056] The at least one processor may analyze the plurality of images to generate the image data by being further configured to: detect a lower eyelid in each of the plurality of images; determine a reflective intensity of an area of interest of the lower eyelid in each of the plurality of images; and generate the image data from the reflective intensity determined for each of the plurality of images.

[0057] The at least one processor may determine the heart rate of the user from the fused sensor data by being further configured to employ an artificial intelligence model on the fused sensor data to output the heart rate, wherein the artificial intelligence model is trained on fused historical accelerometer data, historical gyroscope data, historical image data, and historical accurate heart rates obtained from a plurality of users.

[0058] A method may be summarized as comprising: obtaining first sensor data over time from a first sensor of a headset computing device that is being worn by a user; obtaining second sensor data over time from a second sensor in the headset; weighting the first sensor data and the second sensor data; fusing the weighted first sensor data and the weighted second sensor data to generate fused sensor data; and determining a heart rate of the user from the fused sensor data.

[0059] The method may further comprise pre-processing the first sensor data and the second sensor data prior to weighting the first data and the second sensor data.

[0060] The method may further comprise removing noise from the first sensor data prior to weighting the first data and the second sensor data.

[0061] The method may further comprise determining a heart rate variability of the user from the fused sensor data.

[0062] The method may further comprise determining a respiration rate of the user from the fused sensor data.

[0063] In some embodiments, obtaining the first sensor data over time from the first sensor may comprise obtaining accelerometer data over time from an accelerometer embedded in the headset.

[0064] In some embodiments, obtaining the first sensor data over time from the first sensor may comprise obtaining gyroscope data over time from a gyroscope embedded in the headset.

[0065] In some embodiments, obtaining the first sensor data over time from the first sensor may comprise obtaining image data of the user’s eyes over time from a camera embedded in the headset.

[0066] In some embodiments, obtaining the first sensor data over time from the first sensor and obtaining the second sensor data over time from the second sensor may comprise: obtaining accelerometer data over time from an accelerometer embedded in the headset; and obtaining gyroscope data over time from a gyroscope embedded in the headset.

[0067] A computing device may be summarized as comprising: a memory configured to store computer instructions; and at least one processor. The at least one processor may execute the computer instructions to perform actions, the actions including: obtaining accelerometer data associated with a user over a time segment; obtaining gyroscope data associated with the user over the time segment; obtaining a plurality of images of the user’s eye over the time segment; analyzing the plurality images to generate image data over the time period, including: detecting an area of interest associated with the user’s eye in each of the plurality of images; determining a reflective intensity of the area of interest in each of the plurality of images; and generating the image data from the reflective intensity determined for each of the plurality of images; weighting the accelerometer data, the gyroscope data, and the image data; fusing the weighted accelerometer data, the weighted gyroscope data, and the weighted image data into fused sensor data; and determining a heart rate of the user from the fused sensor data.

[0068] The at least one processor may determine the heart rate of the user from the fused sensor data by executing the computer instructions to perform further actions, the further actions including employing an artificial intelligence model on the fused sensor data to output the heart rate, wherein the artificial intelligence model is trained on fused historical accelerometer data, historical gyroscope data, historical image data, and historical accurate heart rates obtained from a plurality of users.

[0069] The at least one processor may execute the computer instructions to perform further actions, the further actions including employing an artificial intelligence model on thefused sensor data to output a heart rate variability of the user, wherein the artificial intelligence model is trained on fused historical accelerometer data, historical gyroscope data, historical image data, and historical accurate heart rate variabilities obtained from a plurality of users.

[0070] The at least one processor may execute the computer instructions to perform further actions, the further actions including employing an artificial intelligence model on the fused sensor data to output a respiration rate, wherein the artificial intelligence model is trained on fused historical accelerometer data, historical gyroscope data, historical image data, and historical accurate respiration rates obtained from a plurality of users.

[0071] Aspects of the various embodiments described above can be combined to provide further embodiments. These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled.

Claims

CLAIMS1. A system, comprising: an accelerometer configured to obtain accelerometer data associated with a user over a time period; a gyroscope configured to obtain gyroscope data associated with the user over the time period; a camera configured to obtain a plurality of images of the user’s eye over the time period; and at least one processor configured to: receive the accelerometer data from the accelerometer; receive the gyroscope data from the gyroscope; receive the plurality of images from the camera; analyze the plurality images to generate image data of the user’ s eye over the time period; fuse the accelerometer data, the gyroscope data, and the image data into fused sensor data; and determine a heart rate of the user from the fused sensor data.

2. The system of claim 1, wherein the at least one processor fuses the accelerometer data, the gyroscope data, and the image data into the fused sensor data by being further configured to: weight the accelerometer data, the gyroscope data, and the image data; and fuse the weighted accelerometer data, the weighted gyroscope data, and the weighted image data into the fused sensor data.

3. The system of claim 1, wherein the accelerometer, the gyroscope, and the camera are embedded into an augmented or virtual-reality headset.

4. The system of claim 1, wherein the at least one processor is further configured to: determine a heart rate variability of the user from the fused sensor data.

5. The system of claim 1, wherein the at least one processor is further configured to: determine a respiration rate of the user from the fused sensor data.

6. The system of claim 1, wherein the at least one processor analyzes the plurality of images to generate the image data by being further configured to: detect a lower eyelid in each of the plurality of images; determine a reflective intensity of an area of interest of the lower eyelid in each of the plurality of images; and generate the image data from the reflective intensity determined for each of the plurality of images.

7. The system of claim 1, wherein the at least one processor determines the heart rate of the user from the fused sensor data by being further configured to: employ an artificial intelligence model on the fused sensor data to output the heart rate, wherein the artificial intelligence model is trained on fused historical accelerometer data, historical gyroscope data, historical image data, and historical accurate heart rates obtained from a plurality of users.

8. A method, comprising: obtaining first sensor data over time from a first sensor of a headset computing device that is being worn by a user; obtaining second sensor data over time from a second sensor in the headset; weighting the first sensor data and the second sensor data; fusing the weighted first sensor data and the weighted second sensor data to generate fused sensor data; and determining a heart rate of the user from the fused sensor data.

9. The method of claim 8, further comprising: pre-processing the first sensor data and the second sensor data prior to weighting the first data and the second sensor data.

10. The method of claim 8, further comprising: removing noise from the first sensor data prior to weighting the first data and the second sensor data.

11. The method of claim 8, further comprising: determining a heart rate variability of the user from the fused sensor data.

12. The method of claim 8, further comprising: determining a respiration rate of the user from the fused sensor data.

13. The method of claim 8, wherein obtaining the first sensor data over time from the first sensor comprises: obtaining accelerometer data over time from an accelerometer embedded in the headset.

14. The method of claim 8, wherein obtaining the first sensor data over time from the first sensor comprises: obtaining gyroscope data over time from a gyroscope embedded in the headset.

15. The method of claim 8, wherein obtaining the first sensor data over time from the first sensor comprises: obtaining image data of the user’s eyes over time from a camera embedded in the headset.

16. The method of claim 8, wherein obtaining the first sensor data over time from the first sensor and obtaining the second sensor data over time from the second sensor comprises: obtaining accelerometer data over time from an accelerometer embedded in the headset; and obtaining gyroscope data over time from a gyroscope embedded in the headset.

17. A computing device, comprising: a memory configured to store computer instructions; and at least one processor that executes the computer instructions to perform actions, the actions including:obtaining accelerometer data associated with a user over a time segment; obtaining gyroscope data associated with the user over the time segment; obtaining a plurality of images of the user’ s eye over the time segment; analyzing the plurality images to generate image data over the time period, including: detecting an area of interest associated with the user’s eye in each of the plurality of images; determining a reflective intensity of the area of interest in each of the plurality of images; and generating the image data from the reflective intensity determined for each of the plurality of images; weighting the accelerometer data, the gyroscope data, and the image data; fusing the weighted accelerometer data, the weighted gyroscope data, and the weighted image data into fused sensor data; and determining a heart rate of the user from the fused sensor data.

18. The computing device of claim 17, wherein at least one processor determines the heart rate of the user from the fused sensor data by executing the computer instructions to perform further actions, the further actions including: employing an artificial intelligence model on the fused sensor data to output the heart rate, wherein the artificial intelligence model is trained on fused historical accelerometer data, historical gyroscope data, historical image data, and historical accurate heart rates obtained from a plurality of users.

19. The computing device of claim 17, wherein at least one processor executes the computer instructions to perform further actions, the further actions including: employing an artificial intelligence model on the fused sensor data to output a heart rate variability of the user, wherein the artificial intelligence model is trained on fused historical accelerometer data, historical gyroscope data, historical image data, and historical accurate heart rate variabilities obtained from a plurality of users.

20. The computing device of claim 17, wherein at least one processor executes the computer instructions to perform further actions, the further actions including: employing an artificial intelligence model on the fused sensor data to output a respiration rate, wherein the artificial intelligence model is trained on fused historical accelerometer data, historical gyroscope data, historical image data, and historical accurate respiration rates obtained from a plurality of users.

Images