Systems and methods for implementing behavioral detection and behavioral intervention - Patents.com
XR devices facilitate personalized behavioral interventions by detecting user behaviors and assessing their effectiveness, addressing the challenge of preventing chronic diseases through complex lifestyle changes.
Patent Information
- Application Number
- JP2024520793
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2021-10-29
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-10-29
AI Technical Summary
Chronic diseases, primarily caused by lifestyle factors, are difficult to prevent through behavioral changes due to their complexity and time-consuming nature, requiring individuals to interrupt addictive habits and adopt unfamiliar healthy behaviors.
Extended reality (XR) devices are used to detect behavioral information, determine the likelihood of certain behaviors, generate interventions, and assess their effectiveness, allowing for personalized and context-aware interventions to influence user behavior towards healthier choices.
The system effectively tailors interventions to individual user characteristics and contexts, increasing the likelihood of adopting healthy behaviors and reducing the occurrence of unhealthy ones, thereby potentially preventing chronic diseases.
Smart Images

Figure 2024541820000001_ABST
Abstract
Description
[Technical field]
[0001]
[0001] The present disclosure relates generally to behavior detection and intervention. In some examples, aspects of the present disclosure relate to systems and techniques for implementing behavior detection and intervention to affect behavior. [Background technology]
[0002]
[0002] Chronic diseases are the leading causes of death and disability in the world and contribute approximately two-thirds of the global burden of disease, with enormous healthcare costs for societies and governments.
[0003]
[0003] The majority of chronic diseases can be attributed to lifestyle factors and are therefore preventable by adopting healthy behaviors. For example, among US adults, over 90% of type 2 diabetes, 80% of cardiovascular disease, 70% of strokes, and 70% of colon cancer are potentially preventable by a combination of quitting smoking, maintaining a healthy weight, engaging in moderate physical activity, maintaining a healthy diet, and adhering to moderate alcohol intake. Making behavioral changes is complex and time-consuming, requiring people to interrupt addictive lifestyle habits while simultaneously encouraging a new, perhaps unfamiliar, set of actions. Summary of the Invention
[0004]
[0004] In some examples, systems and techniques are described for implementing behavior detection and behavior interventions to affect user behavior according to a user's goal (or target behavior). According to at least one illustrative example, a method of generating one or more interventions is provided. The method includes: acquiring, by an extended reality (XR) device, behavior information associated with a user of the XR device; determining, by the XR device, a likelihood that the user will take a certain behavior based on the behavior information; determining, by the XR device, an intervention based on the determined likelihood exceeding a likelihood threshold; generating, by the XR device, the intervention; determining whether the user has taken the behavior after generating the intervention; determining effectiveness of the intervention based on whether the user has taken the behavior; and transmitting an indication of effectiveness of the intervention to a server for use in determining an intervention for one or more additional users.
[0005] In another illustrative example, an apparatus (e.g., an extended reality (XR) device) for generating one or more interventions is provided. The apparatus includes at least one memory (e.g., configured to store data, such as sensor data, one or more images, etc.) and at least one processor (e.g., implemented in a circuit) coupled to the at least one memory. The at least one processor is configured to obtain behavioral information associated with a user of the apparatus, determine a likelihood that the user will take an action based on the behavioral information, determine an intervention associated with the action based on the determined likelihood exceeding a likelihood threshold, generate the intervention, and after outputting the intervention, determine whether the user took the action, determine effectiveness of the intervention based on whether the user took the action, and transmit an indication of the effectiveness of the intervention to a server for use in determining an intervention for one or more additional users.
[0006]
[0006] In another example, a non-transitory computer readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to obtain behavioral information associated with a user of an XR device, determine a likelihood that the user will take a certain behavior based on the behavioral information, determine an intervention associated with the behavior based on the determined likelihood exceeding a likelihood threshold, generate the intervention, output the intervention, then determine whether the user took the behavior, determine effectiveness of the intervention based on whether the user took the behavior, and transmit an indication of the effectiveness of the intervention to a server for use in determining an intervention for one or more additional users.
[0007] In another example, an apparatus (e.g., an extended reality (XR) device) for processing one or more frames is provided, the apparatus including: means for obtaining behavioral information associated with a user of the XR device, means for determining a likelihood that the user will take an action based on the behavioral information, means for determining an intervention associated with the action based on the determined likelihood exceeding a likelihood threshold, means for generating the intervention, means for determining whether the user took the action after outputting the intervention, means for determining effectiveness of the intervention based on whether the user took the action, and means for transmitting an indication of effectiveness of the intervention to a server for use in determining an intervention for one or more additional users.
[0008]
[0008] In some aspects, the methods, devices, and computer-readable media described above may include transmitting contextual information associated with the intervention to a server, where the contextual information associated with the intervention includes at least one of a time of day, one or more actions by a user of the XR device (or apparatus) prior to the intervention, behavioral information associated with the user of the XR device (or apparatus), a location of the user of the XR device, and a proximity of the user of the XR device (or apparatus) to one or more individuals.
[0009]
[0009] In some aspects, the methods, apparatus, and computer-readable media described above may include transmitting one or more characteristics associated with a user of the XR device (or apparatus) to a server, where the one or more characteristics associated with the user of the XR device (or apparatus) include at least one of gender, age, family status, target behavior, country, culture, location, personality type, one or more health conditions, one or more dietary restrictions, and one or more physical abilities.
[0010]
[0010] In some aspects, to obtain behavioral information associated with a user of an XR device, the methods, apparatus, and computer-readable media described above may include determining one or more behavioral triggers that are precursors to a behavior, the one or more behavioral triggers including at least one of the user's stress level, the user's heart rate, objects within a field of view of the XR device (or apparatus), a location where the user is located, a time at which the behavioral information is obtained, one or more people in proximity to the user, and an activity performed by the user.
[0011]
[0011] In some aspects, to obtain behavioral information associated with a user of an XR device (or apparatus), the methods, apparatus, and computer-readable media described above may include determining one or more pre-actions that indicate the likelihood that the user will take that action.
[0012]
[0012] In some aspects, to obtain behavioral information associated with a user of an XR device (or apparatus), the methods, apparatus, and computer-readable media described above may include detecting one or more behavioral artifacts associated with the behavior in one or more images acquired by the XR device (or apparatus).
[0013]
[0013] In some aspects, to generate an intervention, the methods, devices, and computer-readable media described above may include displaying virtual content on a display of the XR device (or apparatus), wherein the real-world environment can be viewed through the display of the XR device (or apparatus) as the virtual content is displayed by the display.
[0014] According to another illustrative example, a method of generating one or more interventions is provided, the method including obtaining, by a server, first intervention information associated with a first user and the first intervention, updating one or more parameters of an intervention library based at least in part on second intervention information associated with a second user and the second intervention based on the first intervention information, and determining a third intervention for a third user based on the updated one or more parameters of the intervention library.
[0015] In another illustrative example, a system (e.g., including at least one server) for generating one or more interventions is provided. The system includes at least one memory (e.g., configured to store data, such as sensor data, one or more images, etc.) and at least one processor (e.g., implemented in a circuit) coupled to the at least one memory. The at least one processor is configured to: obtain first intervention information associated with a first user and the first intervention; update one or more parameters of an intervention library based at least in part on second intervention information associated with a second user and the second intervention based on the first intervention information; and determine a third intervention for a third user based on the updated one or more parameters of the intervention library.
[0016]
[0016] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to obtain first intervention information associated with a first user and the first intervention, update one or more parameters of an intervention library based at least in part on second intervention information associated with a second user and the second intervention based on the first intervention information, and determine a third intervention for a third user based on the updated one or more parameters of the intervention library.
[0017] In another example, an apparatus for processing one or more frames is provided, the apparatus comprising: means for obtaining first intervention information associated with a first user and a first intervention, means for updating one or more parameters of an intervention library based at least in part on second intervention information associated with a second user and the second intervention based on the first intervention information, and means for determining a third intervention for a third user based on the updated one or more parameters of the intervention library.
[0018]
[0018] In some aspects, the first intervention information associated with the first user includes at least one of an intervention type, an indication of effectiveness of the first intervention, an intervention context associated with the first intervention, and one or more characteristics associated with the first user.
[0019]
[0019] In some aspects, to determine a third intervention for a third user based on one or more updated parameters of the intervention library, the methods, apparatus, and computer-readable media described above may include obtaining, by a server, third intervention information associated with the third user; determining a correlation between the third intervention information associated with the third user and fourth intervention information associated with the intervention library; determining the third intervention based on the correlation between the third intervention information and the fourth intervention information exceeding a correlation threshold; and transmitting the third intervention to a device associated with the third user.
[0020] In some aspects, the first intervention information includes contextual information associated with the first intervention.
[0021]
[0021] In some aspects, the contextual information associated with the first intervention includes at least one of a time of day, one or more actions by the first user prior to the first intervention, first intervention information associated with the first user, a location of the first user, and a proximity of the first user to one or more individuals.
[0022] In some aspects, the first intervention information includes one or more characteristics associated with the first user.
[0023] In some aspects, the one or more characteristics associated with the first user include at least one of gender, age, family status, target behavior, country, culture, location, and personality type.
[0024]
[0024] In some aspects, the methods, apparatus, and computer-readable media described above may include obtaining, by a server, fifth behavioral information associated with a fifth user and a fifth behavior; updating one or more parameters of a behavioral library based at least in part on the sixth behavioral information associated with the sixth user based on the fifth behavioral information; and determining one or more behavioral parameters for the seventh user based on the updated one or more parameters of the intervention library.
[0025] In some aspects, the one or more behavioral parameters for the seventh user include at least one of a behavioral trigger, a pre-action, and a behavioral artifact associated with the fifth action.
[0026]
[0026] In some aspects, the one or more behavioral parameters for the seventh user include one or more weightings associated with determining the likelihood that the seventh user will perform or not perform the fifth behavior.
[0027]
[0027] In some aspects, the fifth behavioral information includes one or more characteristics associated with the fifth user.
[0028] In some aspects, the fifth behavioral information includes contextual information associated with the fifth behavior.
[0029] In some aspects, one or more of the devices described above are part of or include a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a vehicle (e.g., a computing device of a vehicle), or other device. In some aspects, the device includes a camera or cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors. In some cases, the one or more sensors may be used to determine the location and / or pose of the device, the state of the device, and / or for other purposes. In some aspects, the device may include one or more microphones (e.g., for capturing auditory input and / or other sound or audio). In some aspects, the device may include one or more speakers (eg, for providing auditory feedback or other audio output).
[0030]
[0030] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.
[0031]
[0031] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings. [Brief description of the drawings]
[0032]
[0032] Exemplary embodiments of the present application are described in detail below with reference to the following figures. [Figure 1A]
[0033] 1 is an image illustrating an exemplary intervention, according to some examples of the present disclosure. [Figure 1B] 1 is an image illustrating an exemplary intervention, according to some examples of the present disclosure. [Figure 1C] 1 is an image illustrating an exemplary intervention, according to some examples of the present disclosure. [Figure 1D] 1 is an image illustrating an exemplary intervention, according to some examples of the present disclosure. [Figure 1E] 1 is an image illustrating an exemplary intervention, according to some examples of the present disclosure. [Figure 1F] 1 is an image illustrating an exemplary intervention, according to some examples of the present disclosure. [Figure 1G] 1 is an image illustrating an exemplary intervention, according to some examples of the present disclosure. [Diagram 2]
[0034] FIG. 1 is a simplified block diagram illustrating an example extended reality (XR) system, in accordance with some examples of the present disclosure. [Diagram 3]
[0035] FIG. 1 illustrates an example of an XR system being worn by a user, according to some examples of the present disclosure. [Figure 4]
[0036] FIG. 1 is a block diagram illustrating an example learning system, according to some examples of the present disclosure. [Diagram 5]
[0037] FIG. 1 is a flow diagram illustrating an example of a process for generating one or more interventions, according to some examples. [Figure 6]
[0038] FIG. 11 is a flow diagram illustrating another example of a process for generating one or more interventions, according to some examples. [Figure 7]
[0039] FIG. 1 is a flow diagram illustrating an example of a process for predicting one or more actions, according to some examples. [Figure 8]
[0040] FIG. 1 is a block diagram illustrating an example of a deep learning network, in accordance with some examples. [Figure 9]
[0041] FIG. 1 is a block diagram illustrating an example of a convolutional neural network, in accordance with some examples. [Figure 10]
[0042] FIG. 1 illustrates an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0033]
[0043] Specific aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not intended to be limiting.
[0034]
[0044] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope of the present application as set forth in the appended claims.
[0035]
[0045] As mentioned above, chronic diseases are the leading cause of death and disability in the world. Many of such chronic diseases can be attributed to lifestyle factors and can be prevented or managed by implementing healthy behaviors. Most chronic diseases (e.g., type 2 diabetes, cardiovascular disease, stroke, etc.) can be potentially prevented or ameliorated by leading a healthy lifestyle, such as quitting smoking, maintaining a healthy weight, engaging in moderate physical activity, maintaining a healthy diet, and adhering to moderate alcohol intake.
[0036]
[0046] However, making behavioral changes is complex and time-consuming, requiring a person to interrupt addictive lifestyle habits while simultaneously initiating a new, possibly unfamiliar, course of action. It is difficult for a person to successfully change behavior by relying on willpower alone. People tend to behave in pursuit of what they want or need at any given moment. A person's intentions and beliefs about what is good or bad influence a person's actions only if they generate a strong enough desire to take the target behavior at the appropriate moment.
[0037]
[0047] There are techniques that can be used to help influence people to adopt healthier behaviors. A wide range of behavior change techniques have been identified in psychological studies, including environmental restructuring (e.g., not having unhealthy snacks in the house, having hand sanitizer in a place where people can easily see and access it), prompts / cues (e.g., putting a sticker on the door to remind yourself to bring your workout clothes and shoes for your after-work fitness class), etc.
[0038]
[0048] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively "systems and techniques") for identifying behaviors that are likely to occur and for identifying and generating behavioral interventions that influence the likely behaviors. For example, by using computing devices and context-aware techniques, behavioral interventions adapted to the particular situation faced by a user can be delivered in real time (Just In Time Adaptive Intervention (JITAI)) and more effectively (e.g., by selecting an intervention that is better tailored to the user and / or by providing an intervention that is harder to ignore) to help the user address moments when they are prone to unhealthy behaviors or seize opportunities to perform healthy behaviors. In an illustrative example, a multi-user intervention system (e.g., a server) can learn the effectiveness of an intervention across different users. In some cases, the multi-user intervention system can learn the effectiveness of an intervention across multiple different users. In some cases, learning the effectiveness of an intervention across multiple different users can involve learning the effectiveness of an intervention across multiple different users, such as for a subset of users who share similar characteristics and / or experience similar contexts. Additionally or alternatively, the multi-user intervention system can learn to predict user behavior based on the behavioral information of the different users and the indication of behavior prediction effectiveness. In some aspects, learning to predict user behavior can include determining the effectiveness of predicting user behavior for a subset of users who share similar characteristics and / or experience a similar context. Technical effects of a multi-user intervention system learning effectiveness of interventions and / or learning to predict user behavior across users include, but are not limited to, providing interventions that are more likely to be effective, constraining interventions that may be ineffective for a particular user, more accurately predicting user behavior, etc.
[0039]
[0049] In some cases, the systems and techniques can be implemented using an extended reality (XR) system or device. The XR system or device can provide virtual content to a user and / or combine a real-world or physical environment with a virtual environment (consisting of virtual content) to provide an XR experience to the user. The real-world environment can include real-world objects (also called physical objects), such as people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. The XR system or device can facilitate interaction with different types of XR environments (e.g., a user can interact with an XR environment using the XR system or device). The XR system can include a VR system that facilitates interaction with a virtual reality (VR) environment, an AR system that facilitates interaction with an augmented reality (AR) environment, an MR system that facilitates interaction with a mixed reality (MR) environment, and / or other XR systems. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses (e.g., AR glasses), among others. In some cases, the XR system can track parts of the user (e.g., the user's hands and / or fingertips) to enable the user to interact with items of virtual content.
[0040]
[0050] In some cases, an XR system may include an optical "see-through" or "pass-through" display (e.g., a see-through or pass-through AR HMD or AR glasses), allowing the XR system to display XR content (e.g., AR content) directly on the real-world view without displaying video content. For example, a user may see a physical object through a display (e.g., a glass or lens), and the AR system may display AR content on the display to provide the user with an enhanced visual perception of one or more real-world objects. In one example, the display of an optical see-through AR system may include a lens or glass in front of each eye (or a single lens or glass spanning both eyes). The see-through display may allow a user to see directly into the real world or physical object, and may display (e.g., project or otherwise display) an enhanced image of the object or additional AR content to augment the user's visual perception of the real world.
[0041]
[0051] See-through or pass-through XR systems are intended to be worn while the user is engaged in the real world (as opposed to VR, where the user is immersed in virtual content and the real world is completely occluded). Unlike smartphones, PCs, and other computing devices, head-mounted XR devices (e.g., smart glasses, HMDs, etc.) are worn on the face and directly mediate the user's visual and auditory sensory channels.
[0042]
[0052] In some examples, the XR system (or a server in communication with the XR system) can identify behavioral information about a user of the XR system. As used herein, behavioral information can include behavioral triggers, prior actions, behavioral artifacts, any combination thereof, and / or other behavioral information. The XR system or another device (e.g., a mobile device) can identify behavioral triggers (e.g., the user's stress level, the user's heart rate, the location where the user is located, the time the behavioral information is obtained, one or more people in proximity to the user, an activity the user is engaged in, or other actions - in some cases these may not be apparent to the user) that increase the likelihood that the user of the XR system will take a particular action. The XR system (or a server or other device) can additionally or alternatively identify a prior action of the user of the XR system (e.g., walking to a cabinet containing alcohol). In some cases, the prior action can indicate that the user has already decided to take an unhealthy action or to avoid taking a healthy action. The XR system (or server) may additionally or alternatively identify pre-behaviors and / or triggers based on attention tracking (e.g., detecting eye movements to measure visual attention) and / or based on measuring user biometrics (e.g., identifying desire). In some cases, the XR system (or server) may correlate attention tracking and / or biometrics with actual behavior to validate a pre-behavior model. The XR system (or server) may additionally or alternatively identify behavior artifacts, which may include physical objects associated with the behavior (e.g., a bottle of alcohol, a pack of cigarettes, running shoes, etc.).In some cases, the user can input an action plan for one or more target behavioral changes (e.g., not smoking after meals, not eating sugary foods after 4 p.m., running in the morning three times a week, etc.), and the XR system can obtain the above-mentioned behavioral information (e.g., pre-actions, behavioral triggers, and / or behavioral artifacts) for each target behavioral change (e.g., behavioral information for quitting smoking, behavioral information for exercising more frequently, etc.).
[0043]
[0053] The term "target behavior" is used herein to describe a behavior that a user selects to change, either by stopping or reducing the behavior (e.g., quitting drinking alcohol) or by starting or increasing the behavior (e.g., exercising three days a week). In some cases, the target behavior may be selected by the user, and in some cases, the target behavior may be selected by a third party, such as a doctor, insurance company, etc.
[0044]
[0054] The XR system (or server) can use any combination of behavioral information to determine (e.g., using a machine-learning (ML)-based model for each person) the likelihood that a person will take a target behavior (or not take a target behavior). For example, a person walking toward an alcohol cabinet has an increasing likelihood of consuming an alcoholic beverage as they get closer to the alcohol cabinet. The XR system (or server) can generate and possibly implement (e.g., display, output as audio, provide haptic feedback, and / or provide other feedback) an appropriate intervention based on the likelihood that the user will take the target behavior (or not take the target behavior), based on the effectiveness of interventions (e.g., interventions that worked or did not work) for that user or other users at different times in the past, based on user input indicating one or more interventions that the user prefers, and / or based on other factors. In some implementations, the XR system (or server) can transmit the generated intervention to be implemented by another device. In some aspects, the generated intervention can range from no intervention to completely obscuring the object and alerting a designated assistant. As used herein, references to generating an intervention can include the XR system outputting and / or performing the intervention to the user, as well as the XR system sending the intervention to be output and / or performed by another device. In some cases, the intervention can be stronger or otherwise emphasized as the user approaches the behavior artifact. The intervention can be positive (e.g., a positive intervention with the message "Yes! Put on your shoes and go for a jog") or negative (e.g., a negative intervention with the message "Put down your cigarette"). In some cases, the user's biometrics (e.g., heart rate, body temperature, etc.) can be analyzed to determine whether the user is likely to take the behavior.
[0045]
[0055] In some examples, the XR system can utilize a learning system (which can be ML-based, e.g., using one or more neural networks) to determine the most effective intervention across multiple users. The learning system can apply the learned intervention to a population of users based on their individual characteristics. In some cases, the learning system can be implemented as one or more servers in communication with the XR system.
[0046]
[0056] In some examples, the learning system can maintain a library of interventions. In some examples, the learning system can score interventions for effectiveness. The learning system (e.g., using ML models such as one or more neural networks) can associate user characteristics, goals, and / or contextual information with the likelihood of success of a particular intervention.
[0047]
[0057] Examples of user characteristics include gender, biological sex, age, family status, target behavior, severity of a particular problem (e.g., smoking, overeating, etc.), country / culture, previously effective interventions, personality type (e.g., fear response, aversion response, etc.), one or more current and / or past health conditions of the user (e.g., high blood pressure, weight problems, etc.), family medical history, one or more dietary restrictions of the user (e.g., lactose intolerance, high fiber diet, low calorie diet, etc.), one or more physical abilities and / or limitations of the user (e.g., weak or injured hip, paralysis, etc.), The learning system may include the following characteristics: type of work (e.g., shift work, fixed daily schedule, number of jobs), education level, content history (e.g., whether the user has viewed articles or videos related to the intervention and / or goal), health knowledge, previously effective interventions of friends and family, recent health behaviors (e.g., exercise, sleep, nutritional intake in past days), personal preferences of the user (e.g., preferred food types, preferred exercise types, preferred intervention types), social support, motivation level, any combination thereof, and / or other characteristics. The learning system may monitor whether a particular intervention is successful. The learning system may update the multi-user intervention library based on whether a particular intervention is successful. In some examples, the learning system may be implemented on an XR system. In some examples, the learning system may send updates to a multi-user intervention library, which may be referred to as a Global Intervention Efficacy Analysis (GIEA) server. In some cases, the learning system may be used to learn precursor behaviors for a particular user type (e.g., demographic-based, problem severity, etc.).
[0048]
[0058] Context information can include, but is not limited to, time of day, one or more actions by the user prior to the intervention, behavioral information associated with the user and / or the intervention, a location and / or environment associated with the location of the user (and / or XR device) before and / or during the intervention, the user's proximity to one or more individuals, and a combination of technologies / sensors being used by the user at the time of the intervention. For example, the environment associated with the user's location can include whether there are stairs available for a person to use, whether the user is in a kitchen, whether there is space available for the user to exercise, the weather at the user's location (e.g., hot, cold, rainy, windy, etc.), the presence of behavioral artifacts (e.g., cigarettes, running shoes), etc.
[0049]
[0059] 1A-1G are images illustrating exemplary interventions. In the illustrated example of FIG. 1A-1D, the target behavior may be a user's desire to quit smoking. FIG. 1A illustrates an exemplary cigarette package with a general label 104 shown on the visible side of the cigarette package. FIG. 1B illustrates an exemplary progress update intervention 106 projected on the visible side of the cigarette package 102, indicating the user's progress toward a goal, reading "14 Days Cigarette-Free." FIG. 1C illustrates an exemplary personal message intervention 108 projected on the visible side of the cigarette package 102, including a message from the user's child reading "I love you, Mommy!". FIG. 1D illustrates an exemplary image intervention 110 projected on the visible side of the cigarette package 102. In the illustrated example of FIG. 1D, the image intervention 110 includes a picture of a parent (e.g., a user of an XR device) holding the parent's young child. In some cases, the exemplary image interventions 106, 108, 110 can be examples of multiple available interventions, including other types of interventions described throughout this disclosure, as well as any other types of interventions that the XR system can provide to influence user behavior. In some aspects, the XR system can learn which of the available intervention options are most likely to be effective and provide the user with the best available intervention depending on one or more of the user's characteristics, the context of the intervention, the user's characteristics, or any other relevant factors determined by the XR system. In one illustrative example, the learning system may learn that for users with children in the 0-14 age range, overlaying a picture of the user's child on a pack of cigarettes (e.g., image intervention 110 shown in FIG. 1D ) is effective, but for users with older children, a message indicating progress (e.g., "14 days without cigarettes") is most effective. Instead of using general assumptions about behavior and / or response to the intervention, the system can learn which type of intervention is most effective, thus increasing overall system effectiveness.
[0050]
[0060] 1E-1F show additional exemplary interventions. In some cases, obscuring interventions can be used to deter a user from performing an undesirable behavior by obscuring a behavioral artifact (e.g., a physical object associated with the behavior). The image 105 shown in FIG. 1E shows a cabinet where an intervention 112 obscures a particular cabinet that the XR system learned contains alcohol. The intervention 112 also includes a progress update that reads "4 days without alcohol." FIG. 1F shows additional obscuring interventions. In the left image 115, the shelf containing alcohol is obscured by the intervention 124, while the sweet foods 120, 122 remain visible. In the right image 125, the shelf containing alcohol 130 is visible, but interventions 126, 128 are provided to obscure the sweet foods depending on the user's target behavior and / or goals.
[0051]
[0061] FIG. 1G illustrates another exemplary intervention. In the image 140 shown in FIG. 1G, an intervention 142 is shown highlighting the healthy food option of berries in the market. The intervention 142 may also include an informational message 144 to explain to the user why choosing berries may be beneficial to the user's health. The intervention 142 shown in FIG. 1G may be provided for multiple different target behaviors. In one illustrative example, the target behavior may be to stop eating processed foods. The XR system may provide the intervention 142 to encourage the user to perform an alternative behavior of eating healthy foods. For example, the target behavior may be to eat low sugar foods (e.g., eat more fruits and vegetables). In such a case, the XR system may provide the same intervention 142 to the user even though the stated goal is different. In some cases, users with different target behaviors may share one or more common characteristics, and the XR system (or server) may determine that the same intervention is likely to be effective for those users based on their common characteristics.
[0052]
[0062] FIG. 2 illustrates an example XR system 200 according to some aspects of the disclosure. The XR system 200 can run (or execute) XR applications and implement XR actions. In some examples, the XR system 200 can perform tracking and locating, mapping of the physical world (e.g., a scene), and placing and rendering of virtual content on a display 209 (e.g., a screen, a visible plane / area, and / or other display) as part of an XR experience. For example, the XR system 200 can generate a map (e.g., a three-dimensional (3D) map) of a scene in the physical world, track a pose (e.g., location and position) of the XR system 200 relative to the scene (e.g., relative to the 3D map of the scene), place and / or anchor virtual content at specific locations on the map of the scene, and render the virtual content on the display 209 such that the virtual content appears to be at a location in the scene that corresponds to the specific location on the map of the scene where the virtual content is placed and / or anchored. The display 209 may include glasses, screens, lenses, projectors, and / or other display mechanisms that allow the user to view the real-world environment and also allow the XR content to be overlaid, overlapped, blended, or otherwise displayed. As described below, the XR system 200 may implement attention tracking.
[0053]
[0063] 2, the XR system 200 includes one or more image sensors 202, an accelerometer 204, a gyroscope 206, storage 208, a display 209, a compute component 210, an XR engine 220, a behavioral intervention management engine 222, an image processing engine 224, and a rendering engine 226. It should be noted that the components 202-226 shown in FIG. 2 are non-limiting examples provided for illustrative and descriptive purposes, and other examples may include more, fewer, or different components than those shown in FIG. 2. For example, in some cases, the XR system 200 may include one or more other sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more eye tracking sensors, one or more speakers (e.g., for providing auditory feedback or other audio output), one or more microphones (e.g., for capturing auditory input and / or other voice or audio), one or more other hardware components, and / or one or more other software and / or hardware components not shown in FIG. 2. An example architecture and example hardware components that may be implemented by the XR system 200 are further described below with respect to the computing system 1000 shown in FIG. 10.
[0054]
[0064] Further, for simplicity and illustrative purposes, one or more image sensors 202 will be referred to herein (e.g., in the singular) as image sensor 202. However, one of ordinary skill in the art will recognize that XR system 200 can include a single image sensor or multiple image sensors. Additionally, reference to any of the components of XR system 200 (e.g., 202-226) in the singular or plural should not be construed as limiting the number of such components implemented by XR system 200 to one or more than one. For example, reference to accelerometer 204 in the singular should not be construed as limiting the number of accelerometers implemented by XR system 200 to one. One of ordinary skill in the art will recognize that for any of the components 202-226 shown in FIG. 2, XR system 200 can include only one of such components or more than two of such components.
[0055]
[0065] The XR system 200 can be part of or implemented in a single computing device or multiple computing devices. In some examples, the XR system 200 can be part of an electronic device or devices, such as a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smartphone, cellular phone, conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart television, a display device, a game console, a video streaming device, an Internet-of-Things (IoT) device, a smart wearable device (e.g., a head-mounted display (HMD), smart glasses, etc.), or any other suitable electronic device.
[0056]
[0066] In some implementations, one or more of the image sensor 202, the accelerometer 204, the gyroscope 206, the storage 208, the display 209, the compute component 210, the XR engine 220, the behavioral intervention management engine 222, the image processing engine 224, the rendering engine 226, and any other components (e.g., microphone, speaker, eye tracking sensor, etc.) shown herein in Figure 2 can be part of the same computing device. For example, in some cases, one or more of the image sensor 202, the accelerometer 204, the gyroscope 206, the storage 208, the compute component 210, the XR engine 220, the behavioral intervention management engine 222, the image processing engine 224, and the rendering engine 226 can be integrated into a smartphone, a laptop, a tablet computer, a smart wearable device, a gaming system, and / or any other computing device. However, in some implementations, one or more of the image sensor 202, accelerometer 204, gyroscope 206, storage 208, compute component 210, XR engine 220, behavioral intervention management engine 222, image processing engine 224, and rendering engine 226 can be part of two or more separate computing devices. For example, in some cases, some of the components 202-226 can be part of or implemented by one computing device, and remaining components can be part of or implemented by one or more other computing devices.
[0057]
[0067] The image sensor 202 may include any image and / or video sensor or capture device, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device on an electronic device such as a television or computer, a camera, etc. In some cases, the image sensor 202 may be part of a camera or computing device, such as an XR device (e.g., HMD, smart glasses, etc.), a digital camera, a smartphone, a smart television, a gaming system, etc. In some examples, the image sensor 202 may be part of a multiple camera assembly, such as a dual camera assembly. The image sensor 202 may capture images and / or video content (e.g., raw images and / or video data), which may then be processed by the compute component 210, the XR engine 220, the behavioral intervention management engine 222, the image processing engine 224, and / or the rendering engine 226 described herein.
[0058]
[0068] In some examples, the image sensor 202 can capture image data, generate frames based on the image data, and / or provide image data or frames to the XR engine 220, the behavioral intervention management engine 222, the image processing engine 224, and / or the rendering engine 226 for processing. A frame can include a video frame or a still image of a video sequence. A frame can include a pixel array representing a scene. For example, a frame can be a red-green-blue (RGB) frame with red, green, and blue color components per pixel, a luminance, red difference, blue difference (YCbCr) frame with one luminance component and two chrominance (color) components (red chrominance and blue chrominance) per pixel, or any other suitable type of color or monochrome image.
[0059]
[0069] For example, the accelerometer 204 can detect acceleration by the XR system 200 and generate acceleration measurements based on the detected acceleration. The gyroscope 206 can detect and measure the orientation and angular velocity of the XR system 200. For example, the gyroscope 206 can be used to measure the pitch, roll, and yaw of the XR system 200. In some examples, the image sensor 202 and / or the XR engine 220 can use the measurements obtained by the accelerometer 204 and the gyroscope 206 to calculate the pose of the XR system 200. As mentioned above, in other examples, the XR system 200 can also include other sensors, such as magnetometers, machine vision sensors, smart scene sensors, voice recognition sensors, impact sensors, shock sensors, position sensors, tilt sensors, etc.
[0060]
[0070] The storage 208 can be any storage device for storing data. Moreover, the storage 208 can store data from any of the components of the XR system 200. For example, the storage 208 can store data from the image sensor 202 (e.g., image or video data), data from the accelerometer 204 (e.g., measurements), data from the gyroscope 206 (e.g., measurements), data from the compute component 210 (e.g., processing parameters, preferences, virtual content, rendering content, scene maps, tracking and localization data, object detection data, privacy data, XR application data, face recognition data, occlusion data, etc.), data from the XR engine 220, data from the behavioral intervention management engine 222, data from the image processing engine 224, and / or data from the rendering engine 226 (e.g., output frames). In some examples, the storage 208 can include a buffer for storing frames for processing by the compute component 210.
[0061]
[0071] The one or more compute components 210 may include a central processing unit (CPU) 212, a graphics processing unit (GPU) 214, a digital signal processor (DSP) 216, and / or an image signal processor (ISP) 218. The compute components 210 may perform various operations, such as image enhancement, computer vision, graphics rendering, XR (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image / video processing, sensor processing, recognition (e.g., text recognition, face recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), machine learning, filtering, and any of the various operations described herein. In this example, the compute components 210 implement an XR engine 220, a behavioral intervention management engine 222, an image processing engine 224, and a rendering engine 226. In other examples, the compute components 210 may also implement one or more other processing engines.
[0062]
[0072] The operations for the XR engine 220, the behavioral intervention management engine 222, the image processing engine 224, and the rendering engine 226 (and any image processing engine) may be implemented by any of the compute components 210. In an illustrative example, the operations of the rendering engine 226 may be implemented by the GPU 214, and the operations of the XR engine 220, the behavioral intervention management engine 222, and the image processing engine 224 may be implemented by the CPU 212, the DSP 216, and / or the ISP 218. In some cases, the compute components 210 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof for performing any of the various operations described herein.
[0063]
[0073] In some examples, the XR engine 220 can perform XR operations based on data from the image sensor 202, the accelerometer 204, the gyroscope 206, and / or one or more sensors on the XR system 200, such as one or more IMUs, radar, etc. In some examples, the XR engine 220 can perform tracking, localization, pose estimation, mapping, content anchoring operations, and / or any other XR operations / functions.
[0064]
[0074] The behavioral intervention management engine 222 can perform behavioral detection and generate behavioral interventions. In some cases, when the behavioral intervention management engine 222 determines (e.g., using a likelihood algorithm) that an undesirable behavior is likely to occur (or that a desirable behavior is unlikely to occur) and / or that a healthy behavior is likely to occur, the behavioral intervention management engine 222 can generate and implement an intervention. In some cases, the behavioral intervention management engine 222 can include one or more learning systems that can improve the effectiveness of the intervention and / or the accuracy of the behavior detection. For example, the behavioral intervention management engine 222 can determine whether an implemented intervention successfully influenced the user's behavior (e.g., performing a healthy behavior, not performing an unhealthy behavior, etc.). In some cases, based on whether the intervention was successful, the behavioral intervention management engine 222 can update the effectiveness of the intervention in an intervention library.
[0065]
[0075] In some cases, the behavioral intervention management engine 222 can also improve the accuracy of the behavioral prediction. For example, if the behavioral intervention management engine 222 determines that an undesirable behavior is unlikely to occur and decides not to generate an intervention, the user may still perform the undesirable behavior. The behavioral management engine 222 can use the inaccurate behavioral prediction to update the behavioral likelihood algorithm. In one illustrative example, the behavioral intervention management system can update weights associated with the behavioral information (e.g., behavioral triggers, prior actions, and / or behavioral artifacts) used to determine the likelihood that a behavior will (or will not) occur. In another illustrative example, the behavioral intervention management engine 222 can determine whether additional behavioral information specific to the user was omitted from the likelihood determination. For example, the behavioral intervention management engine 222 can determine that a particular user has a unique behavioral trigger and include the unique behavioral trigger when determining a subsequent likelihood determination.
[0066]
[0076] In some aspects, the behavioral intervention management engine 222 can communicate with a server to transmit an indication of the effectiveness of the intervention on the user's behavior and / or an indication of the accuracy with which the user determines the likelihood of performing (or not performing) the target behavior. In some cases, the behavioral intervention management engine 222 can also transmit to the server the characteristics of the user (e.g., gender, age, family status), the context of the behavior and / or intervention (e.g., location, technology / sensor combination in use), behavioral information associated with determining the likelihood of the behavior and / or determining which intervention to generate for the user (e.g., behavioral triggers, prior behaviors), any combination thereof, and / or other information. In some cases, the behavioral intervention management engine 222 can receive an intervention and / or behavioral prediction model from the server. In some cases, the received intervention and / or behavioral prediction model can be based on characteristics, context, and / or behavioral information determined from multiple users. In some cases, the behavioral intervention management engine 222 can receive an intervention and / or behavioral prediction model from the server associated with characteristics and / or context shared between the user and a subset of other users.
[0067]
[0077] The image processing engine 224 can perform one or more image processing operations. In some examples, the image processing engine 224 can perform image processing operations based on data from the image sensor 202. The rendering engine 226 can obtain image data generated and / or processed by the compute component 210, the image sensor 202, the XR engine 220, and / or the image processing engine 224, and can render video and / or image frames for presentation on a display device.
[0068]
[0078] FIG. 3 illustrates an example of an XR system 320 worn by a user 300. The XR system 320 may include some or all of the same components as the XR system 200 shown in FIG. 2 and described above, and may perform some or all of the same functions as the XR system 200. Although the XR system 320 is illustrated in FIG. 2 as AR glasses, the XR system 320 may include any suitable type of XR device, such as an HMD or other XR device. The XR system 320 is described as an optical see-through AR device that allows the user 300 to see the real world while wearing the XR system 320. For example, the user 300 can see an object 302 in a real-world environment on a plane 304 at a distance from the user 300. The XR system 320 has an image sensor 318 and a display 310 (e.g., glasses, a screen, a lens, or other display) that allows the user 300 to see the real-world environment and also allows AR content to be displayed. The image sensor 318 can be similar or identical to the image sensor 202 shown in FIG. 2. Although one image sensor 318 and one display 310 are shown in FIG. 3, the XR system 320 can include multiple cameras and / or multiple displays (e.g., a display for the right eye and a display for the left eye) in some implementations. AR content (e.g., images, videos, graphics, virtual or AR objects, or other AR content) can be projected or displayed on the display 310. In one example, the AR content can include an augmented version of the object 302. In another example, the AR content can include additional AR content related to the object 302 or related to one or more other objects in the real-world environment.
[0069]
[0079] As shown in FIG. 3 , the XR system 320 can include or be in wired or wireless communication with a compute component 316 and a memory 312. The compute component 316 and the memory 312 can store and execute instructions used to implement the techniques described herein. In implementations in which the XR system 320 is in communication (wired or wireless) with the memory 312 and the compute component 316, the device housing the memory 312 and the compute component 316 can be a computing device such as a desktop computer, a laptop computer, a mobile phone, a tablet, a game console, or other suitable device. The XR system 320 also includes or is in communication (wired or wireless) with an input device 314. The input device 314 can include any suitable input device, such as a touch screen, a pen or other pointer device, a keyboard, mouse buttons or keys, a microphone for receiving voice commands, a gesture input device for receiving gesture commands, any combination thereof, and / or other input devices. In some cases, the image sensor 318 can capture images that can be processed to interpret gesture commands.
[0070]
[0080] The image sensor 318 may capture color images (e.g., images having red-green-blue (RGB) color components, images having luma (Y) and chroma (C) color components such as YCbCr images, or other color images) and / or grayscale images. As mentioned above, in some cases, the XR system 320 may include multiple cameras, such as dual front cameras and / or one or more front-facing cameras and one or more rear-facing cameras, which may incorporate a variety of sensors. In some cases, the image sensor 318 (and / or other cameras of the XR system 320) may capture still images and / or video, including multiple video frames (or images). In some cases, the image data received by the image sensor 318 (and / or other cameras) may be in a raw, uncompressed format and may be compressed and / or otherwise processed (e.g., by an image signal processor (ISP) or other processor of the XR system 320) before being further processed and / or stored in the memory 312. In some cases, image compression may be performed by compute component 316 using lossless or lossy compression techniques (eg, any suitable video or image compression techniques).
[0071]
[0081] In some cases, the image sensor 318 (and / or other cameras of the XR system 320) can be configured to also capture depth information. For example, in some implementations, the image sensor 318 (and / or other cameras) can include an RGB-depth (RGB-D) camera. In some cases, the XR system 320 can include one or more depth sensors (not shown) that are separate from the image sensor 318 (and / or other cameras) and can capture depth information. For example, such depth sensors can obtain depth information independent of the image sensor 318. In some examples, the depth sensor can be physically located in the same general location as the image sensor 318, but can operate at a different frequency or frame rate than the image sensor 318. In some examples, the depth sensor can take the form of a light source that can project a structured or textured light pattern, which may include one or more narrow bands of light, onto one or more objects in the scene. The depth information can then be obtained by exploiting geometric distortions of the projected pattern caused by the surface shape of the object. In one example, depth information may be obtained from a stereo sensor, such as a combination of an infrared structured light projector and an infrared camera aligned with a camera (e.g., an RGB camera).
[0072]
[0082] In some implementations, the XR system 320 includes one or more sensors. The one or more sensors can include one or more accelerometers (e.g., the accelerometer 204 shown in FIG. 2), one or more gyroscopes (e.g., the gyroscope 206 shown in FIG. 2), and / or other sensors. The one or more sensors can provide velocity, orientation, and / or other position-related information to the compute component 316. In some cases, the one or more sensors can include at least one inertial measurement unit (IMU). An IMU is an electronic device that uses a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers to measure a specific force, angular velocity, and / or orientation of the XR system 320. In some examples, the one or more sensors may output measured information associated with the capture of images captured by the image sensor 318 (and / or other cameras of the XR system 320) and / or depth information obtained using one or more depth sensors of the XR system 320.
[0073]
[0083] The output of one or more sensors (e.g., one or more IMUs) can be used by the compute component 316 to determine a pose of the XR system 320 (also called head pose) and / or a pose of the image sensor 318 (or other camera of the XR system 320). In some cases, the pose of the XR system 320 and the pose of the image sensor 318 (or other camera) can be the same. The pose of the image sensor 318 refers to the position and orientation of the image sensor 318 relative to a reference frame (e.g., with respect to the object 302). In some implementations, the pose of the camera can be determined in 6-Degrees Of Freedom (6DOF), which refers to three translational components (e.g., which can be given in X (horizontal), Y (vertical), and Z (depth) coordinates relative to a reference frame such as an image plane) and three angular components (e.g., roll, pitch, and yaw relative to the same reference frame).
[0074]
[0084] In some aspects, the pose of the image sensor 318 and / or the XR system 320 can be determined and / or tracked by the compute component 316 using a visual tracking solution based on images captured by the image sensor 318 (and / or other cameras of the XR system 320). In some examples, the compute component 316 can perform tracking using computer vision-based tracking, model-based tracking, and / or simultaneous localization and mapping (SLAM) techniques. For example, the compute component 316 can perform SLAM or can communicate (wired or wireless) with a SLAM engine (not shown). SLAM refers to a class of techniques in which a map of an environment (e.g., a map of an environment modeled by the XR system 320) is created while simultaneously tracking the pose of the camera (e.g., the image sensor 318) and / or the XR system 320 relative to the map. The map may be referred to as a SLAM map and may be three-dimensional (3D). SLAM techniques may be implemented using color or grayscale image data captured by the image sensor 318 (and / or other cameras of the XR system 320) and may be used to generate estimates of 6DOF pose measurements of the image sensor 318 and / or the XR system 320. Such SLAM techniques configured to implement 6DOF tracking may be referred to as 6DOF SLAM. In some cases, the output of one or more sensors may be used to estimate, correct, and / or otherwise adjust the estimated pose.
[0075]
[0085] In some cases, 6DOF SLAM (e.g., 6DOF tracking) can associate features observed from a particular input image from the image sensor 318 (and / or other camera) to a SLAM map. 6DOF SLAM can use feature point associations from an input image to determine a pose (position and orientation) of the image sensor 318 and / or the XR system 320 relative to the input image. 6DOF mapping can also be performed to update the SLAM map. In some cases, a SLAM map maintained using 6DOF SLAM can include 3D feature points triangulated from two or more images. For example, key frames can be selected from the input images or video stream to represent the observed scene. For every key frame, a respective 6DOF camera pose associated with the image can be determined. The pose of the image sensor 318 and / or the XR system 320 can be determined by projecting features from the 3D SLAM map onto the image or video frame and updating the camera pose from the verified 2D-3D correspondence.
[0076]
[0086] In one illustrative example, the compute component 316 may extract feature points from all input images or from each key frame. As used herein, feature points (also referred to as alignment points) are distinctive or identifiable parts of an image, such as a part of a hand, an edge of a table, among others. Features extracted from a captured image may represent distinctive feature points along a three-dimensional space (e.g., coordinates on X, Y, and Z axes), and every feature point may have an associated feature location. Feature points in a key frame may either match (be the same or correspond) or not match feature points of a previously captured input image or key frame. Feature detection may be used to detect feature points. Feature detection may include image processing operations that are used to examine one or more pixels of an image to determine if a feature is present at a particular pixel. Feature detection may be used to process the entire captured image or specific portions of an image. For each image or key frame, once a feature is detected, a local image patch near the feature may be extracted. Features may be extracted using any suitable technique, such as the Scale Invariant Feature Transform (SIFT) (which locates features and generates their descriptions), Speed Up Robust Features (SURF), Gradient Location-Orientation histogram (GLOH), Normalized Cross Correlation (NCC), or other suitable techniques.
[0077]
[0087] In some examples, the AR (or virtual) object can be aligned or anchored (e.g., positioned relative to) the detected feature points in the scene. For example, the user 300 may be looking at a restaurant across the street from where the user 300 is standing. In response to recognizing the restaurant and AR content associated with the restaurant, the compute component 316 can generate an AR object that provides information about the restaurant. The compute component 316 can also detect feature points from a portion of the image that includes a sign for the restaurant, and can align the AR object to the feature points of the sign such that the AR object is displayed relative to the sign (e.g., above the sign so that it is easily identifiable by the user 300 as relating to the restaurant). In addition, the interventions described herein (e.g., interventions 106, 108, 110, 112, 124, 126, 128, 142 shown in Figures 1B-1G) can be used to generate an AR object that provides information about the restaurant.
[0078]
[0088] The XR system 320 can generate and display various AR objects for the user 300 to view. For example, the XR system 320 can generate and display a virtual interface, such as a virtual keyboard, as an AR object for the user 300 to input text and / or other characters as needed. The virtual interface can be aligned to one or more physical objects in the real world. However, in many cases, there may be a lack of real-world objects with well-defined features that can be used as a reference for alignment purposes. For example, if a user is staring at a blank whiteboard, the whiteboard may not have well-defined features to which the virtual keyboard can be aligned. An outdoor environment may provide even less well-defined points that can be used to align the virtual interface, for example, based on, among other things, a lack of points in the real world, well-defined objects being farther apart in the real world than when the user is indoors, the presence of many moving points in the real world, and points being far apart from one another.
[0079]
[0089] 4 shows a block diagram of a behavioral intervention management system 400 for detecting behaviors and generating an intervention. The behavioral intervention management system 400 can obtain user parameters 401, which may include goals and / or plans for behavior change. Examples of user parameters 401 are provided below. The behavioral intervention management system 400 can detect that an undesirable behavior is more likely to occur or that an intervention on a desired behavior is more likely to be successful. Based on the determination, the behavioral intervention management system 400 can generate an intervention to affect the user's behavior to discourage the user from performing the undesirable behavior or to encourage the user to perform the desired behavior.
[0080]
[0090] In this illustrative example of Figure 4, behavioral intervention management system 400 includes a behavioral prescription engine 410, an intervention engine 420, and a multi-user engine 430. It should be noted that components 410-430 shown in Figure 4 are non-limiting examples provided for purposes of illustration and explanation, and other examples may include more, fewer, or different components than those shown in Figure 4 without departing from the scope of the present disclosure. In one illustrative example, one or more of the components of the behavioral intervention management system may be included in or may include the behavioral intervention management engine 222 shown in Figure 2.
[0081]
[0091] The behavioral intervention management system 400 can be part of or implemented in a single computing device or multiple computing devices. In some implementations, the behavioral instruction engine 410, the intervention engine 420, and the multi-user engine 430 can be part of the same computing device. For example, the behavioral intervention management system 400 can be part of an electronic device or devices, such as a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smartphone, cellular phone, conferencing system, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a smart television, a display device, a game console, a video streaming device, an IoT device, a smart wearable device (e.g., HMD, smart glasses, etc.), or any other suitable electronic device. However, in some implementations, the behavioral instruction engine 410, the intervention engine 420, and the multi-user engine 430 can be part of two or more separate computing devices. For example, in some cases, some of the components 410-430 may be part of or implemented by one computing device, and the remaining components may be part of or implemented by one or more other computing devices. In one illustrative example, the multi-user engine 430 may be part of or implemented in a server that receives behavioral accuracy instructions and intervention effect instructions from multiple different user devices, and the behavioral instruction engine 410 and the intervention engine 420 may be implemented in the user devices.
[0082]
[0092] The behavioral intervention management system 400 can obtain one or more user parameters 401. The user parameters 401 can include, but are not limited to, user goals and plans, user characteristics, user behavior, user motivation for change (e.g., a desire to be able to play with grandchildren), one or more health conditions (e.g., high blood pressure, weight issues, diabetes, heart disease, hypertension, etc.), one or more dietary restrictions of the user (e.g., lactose intolerance, high fiber diet, low calorie diet, etc.), one or more physical capabilities of the user, environmental factors that may support or restrict certain behaviors, and any other information that may be potentially relevant to predicting the user's behavior and / or successfully generating an intervention that influences the user's behavior. In some cases, the user parameters 401 can be obtained directly from the user, such as user input via a user interface. In some cases, the user parameters 401 can be input from one or more sources in addition to or as an alternative to user-entered parameters. In one illustrative example, the user parameters 401 may be obtained during an onboarding process, such as when a user purchases an XR system (e.g., the XR system 200 shown in FIG. 2). In another illustrative example, the user parameters 401 may be input by a physician or insurance company. In another illustrative example, the user parameters 401 may be obtained from one or more other devices belonging to the user and / or one or more services in which the user participates, such as profile settings, health and fitness tracking data, etc. In some examples, standard user parameters may be specified for a particular type of user. For example, parameters may be defined for a user having a particular health condition (e.g., a user having diabetes, heart disease, hypertension, COPD, etc.), such that multiple users having similar health conditions may be associated with similar interventions.
[0083]
[0093] The behavioral intervention management system 400 can also obtain data from one or more sensors 402 to enable detection of user behavior information. For example, the one or more sensors can include one or more image sensors (e.g., image sensor 202 shown in FIG. 2), microphones, accelerometers (e.g., accelerometer 204), location sensors (e.g., GPS sensors), eye-tracking sensors, contact tracing sensors that can detect the proximity of a person to another person or group of people (e.g., Bluetooth™ sensors, etc.), or any other sensors that can be used to detect a user's behavior and / or environment (e.g., to identify that person A is with person B when ordering food at a restaurant). In some cases, one or more of the sensors 402 can be included in the same device (e.g., XR system 200) as the behavioral intervention management system 400. In some cases, one or more of the sensors 402 can be included in other devices, such as a fitness tracker, a mobile phone, an IoT device, etc.
[0084]
[0094] The behavioral instruction engine 410 can obtain data from one or more sensors 402 to identify behavioral information associated with the user. In the illustrative example of FIG. 4, the behavioral instruction engine 410 includes a behavioral trigger monitor 412, a pre-behavior monitor 414, and a behavioral artifact monitor 416. Each of the behavioral trigger monitor 412, the pre-behavior monitor 414, and the behavioral artifact monitor 416 can process data from one or more of the sensors 402 to detect corresponding behavioral information.
[0085]
[0095] The behavioral trigger monitor 412 can process data from one or more sensors to identify behavioral triggers (e.g., stress level, location, environment, time, nearby people, nearby objects, etc.) For example, the behavioral trigger monitor 412 can process data from one or more of a heart rate monitor, a galvanic skin response sensor, a blood pressure sensor, and / or other sensors to detect a user's stress response.
[0086]
[0096] The pre-action monitor 414 can process data from one or more sensors 402 to identify pre-actions, which can be actions that indicate that the user is likely to take an action (or is likely not to take an action) or has decided to take an action (or not to take an action). For example, the pre-action monitor 414 can obtain image sensor data that indicates that the user is walking toward an alcohol cabinet, reaching for a pack of cigarettes, or performing another type of pre-action that indicates that the user is likely to take / not take (or has taken / did not take) a particular action.
[0087]
[0097] The behavior artifact monitor 416 can process data from one or more sensors to identify behavior artifacts. For example, the behavior artifact monitor 416 and / or the behavior instruction engine 410 can perform feature detection (e.g., by ML models such as one or more neural networks) on images received from an image sensor of the sensors 402 and assign classes (e.g., table, child, car, cabinet, bottle, cigarette, etc.) to different features detected in the images. In one illustrative example, the behavior artifact monitor 416 can process the assigned classes to determine whether any of the images include behavior artifacts (e.g., cigarette pack, ice cream container).
[0088]
[0098] The behavioral instruction engine 410 can provide behavioral information from one or more of the behavioral trigger monitor 412, the pre-behavior monitor 414, and the behavioral artifact monitor 416 to the intervention engine 420. In some cases, the behavioral instruction engine 410 can also optionally provide behavioral information from one or more of the behavioral trigger monitor 412, the pre-behavior monitor 414, and the behavioral artifact monitor 416 to the multi-user engine 430.
[0089]
[0099] As shown, the intervention engine 420 includes a likelihood engine 422, an intervention engine 424, an intervention effectiveness engine 426, and an adjustment engine 428. The intervention engine 420 can obtain behavioral information from the behavior prescription engine 410 and process the behavioral information. The intervention engine 420 can determine whether the target behavior is likely to occur (or not occur) and, based on the likelihood determination, generate an intervention to influence the user away from the undesired behavior (or toward the desired behavior).
[0090]
[0100] The likelihood engine 422 can determine the likelihood that a behavior will or will not occur. In one exemplary implementation, the likelihood engine 422 can determine the likelihood that a behavior will occur based on applying different weights to individual components of the behavior information obtained from the behavior instruction engine 410. For example, the likelihood engine can assign a high weighting to a prior behavior because the prior behavior indicates an intent to perform the behavior by the user, while assigning a low weighting to a behavior artifact that is in view because the behavior of the particular user is not strongly influenced by seeing the behavior artifact. In some cases, the likelihood that a behavior will (or will not) occur determined by the likelihood engine 422 can be compared to a threshold. In some cases, if the likelihood exceeds a threshold, the intervention engine 420 can generate an intervention for the user to attempt to influence the user's behavior. In some cases, the behavioral intervention management system 400 can monitor the user's behavior after the prediction to determine if the prediction from the likelihood engine 422 was accurate. In some implementations, the likelihood engine 422 can be trained during a training period before any intervention is applied by the behavioral intervention management system 400. In some aspects, the likelihood engine 422 may be implemented by ML models, such as one or more neural networks. The likelihood engine 422 may continuously monitor the behavior information from the behavior instruction engine 410 to determine the likelihood that a behavior will occur (or not occur) before the behavior occurs.
[0091]
[0101] The intervention engine 424 can obtain the likelihood determined by the likelihood engine 422. If the intervention engine 424 determines that the likelihood indicates that an intervention is needed (e.g., the likelihood excludes a threshold), the intervention engine 424 can generate an intervention. In some cases, the intervention engine 424 can include a library of interventions to select from to affect the user's behavior. For example, the intervention library can include any of the interventions described herein, including the interventions 106, 108, 110, 112, 124, 126, 128 shown in Figures 1B-1G, audio interventions, contacting a support person, etc.
[0092]
[0102] The effectiveness of an intervention may be context-dependent. For example, for a particular individual, an intervention may be effective in some situations (e.g., in the morning, or when the person is alone) and ineffective in other situations (e.g., in the evening, or around other people). The intervention effectiveness engine 426 may determine the effectiveness of an intervention after the intervention engine 420 generates the intervention. In one illustrative example, the intervention effectiveness engine 426 may monitor a user's behavior after the intervention is generated to determine whether the intervention effectively influenced the user's behavior. For example, if the user engages in an undesirable behavior after the intervention, the intervention effectiveness engine 426 may determine that the intervention was ineffective for a particular user. In some cases, the intervention effectiveness engine 426 may determine that the intervention was partially effective for a user, such as when the user engages in a behavior, but to a reduced extent relative to a previous time the user engaged in the same behavior. In one illustrative example, a user may smoke one cigarette when they normally smoke two cigarettes.
[0093]
[0103] The adjustment engine 428 can determine one or more adjustments for the intervention engine 424 based on the accuracy of the behavior prediction by the likelihood engine 422 and / or the effectiveness of the intervention generated by the intervention engine 420. For example, the adjustment engine 428 can adjust one or more parameters of the intervention engine 424 to increase the likelihood that an effective intervention will be generated and decrease the likelihood that an ineffective intervention will be generated. Similarly, the adjustment engine 428 can adjust one or more parameters of the likelihood engine (e.g., weights applied to components of the behavioral information) based on whether the likelihood engine accuracy predicted the behavior.
[0094]
[0104] The multi-user engine 430 can obtain an indication of intervention effectiveness and / or accuracy of the behavior prediction from the intervention engine 420. In the illustrated example of FIG. 4, the multi-user engine 430 includes a multi-user intervention engine 432 and a multi-user behavior engine 434. In some cases, the multi-user engine 430 can obtain characteristics of the user (e.g., user parameters 401) and / or contextual information associated with the intervention. The contextual information can include, but is not limited to, a time of day, one or more actions by the user prior to the intervention, behavioral information associated with the user and / or the intervention, a location and / or environment associated with the location of the user (and / or XR device) prior to and / or during the intervention, the user's proximity to one or more individuals, and a combination of technologies / sensors being used by the user at the time of the intervention. For example, the environment associated with the user's location can include whether there are stairs available for the person to use, whether the user is in a kitchen, whether there is space available for the user to exercise, the weather at the user's location (e.g., hot, cold, rainy, windy, etc.), the presence of behavioral artifacts (e.g., cigarettes, running shoes), etc. Characteristics may include, but are not limited to, gender, age of the user, family status, target behavior, severity of the problem, country / culture, personality type (e.g., fear response, disgust response, etc.), one or more health conditions (e.g., diabetes, cardiac arrhythmia, etc.), one or more dietary restrictions (e.g., low calorie diet, vegetarian, etc.), one or more physical abilities (e.g., weak or injured hip, paralysis, etc.), and / or other characteristics.
[0095]
[0105] The multi-user intervention engine 432 can obtain indications of effectiveness of the intervention for multiple users. The multi-user intervention engine 432 can process the indications of effectiveness of the intervention, the characteristics of the users associated with the intervention, and / or the context of the intervention to generate a multi-user intervention library. In some cases, the interventions in the intervention library can include a score based on effectiveness. In some cases, the interventions can include a separate score for each type or category of intervention and for the context associated with the intervention, and for any combination thereof. In some cases, the multi-user intervention engine 432 can be implemented by ML models, such as one or more neural networks. In some cases, the multi-user intervention engine 432 can send the indications of effectiveness stored for the multi-user interventions, corresponding to the characteristics of the users and / or the context frequently experienced by the users, to the intervention engine 420 and / or the coordination engine 428 to provide an intervention that is likely to be effective for the user based on proven effectiveness for similar users in similar contexts. In one illustrative example, the multi-user intervention engine may determine that overlaying a photo of a user holding a child onto a pack of cigarettes (e.g., intervention 110 shown in FIG. 1D) is effective for users with children under the age of 14, while messages indicating progress (e.g., intervention 106 shown in FIG. 1B, intervention 112 shown in FIG. 1E) are more effective for users with children over the age of 14.
[0096]
[0106] The multi-user behavior engine 434 can similarly generate a multi-user behavior library. In some cases, the multi-user behavior library can include precursor information (e.g., behavior triggers, prior actions, and / or behavior artifacts) that can be associated with user characteristics and / or contexts frequently experienced by the user. In addition to or as an alternative to including precursor information in the multi-user behavior library, the multi-user behavior engine 434 can include behavior likelihood determination information that can be associated with user characteristics and / or contexts frequently experienced by the user. In an illustrative example, the likelihood determination information can include weightings for different types of behavior information that are most likely to accurately predict behavior based on the characteristics and / or contexts frequently experienced. In some cases, the multi-user behavior engine 434 can be implemented by ML models such as one or more neural networks. In some cases, the multi-user behavior engine 434 can send precursor information corresponding to the user characteristics and / or contexts frequently experienced by the user to the behavior instruction engine 410. In some examples, the precursor information can be used to train the behavior instruction engine 410 (e.g., when implemented by ML models such as neural networks). Additionally or alternatively, the multi-user behavior engine 434 may send likelihood determination information corresponding to characteristics of the user and / or contexts frequently experienced by the user to one or more of the intervention engine 420, the likelihood engine 422, and the intervention effectiveness engine 426.
[0097]
[0107] 5 is a flow diagram illustrating an example of a process 500 for generating one or more interventions. At block 502, the process 500 includes obtaining a behavioral instruction associated with a user. In some cases, the behavioral instruction may include one or more of a behavioral trigger, a pre-action, and a behavioral artifact (e.g., a physical object associated with the behavior). For example, the behavioral instruction may be obtained from a behavioral instruction engine system (e.g., the behavioral instruction engine 410 shown in FIG. 4).
[0098]
[0108] In block 504, the process 500 determines whether an undesirable behavior is likely to occur or a desired behavior is unlikely to occur based on the behavioral instruction received in block 502. For example, if a user has a goal to avoid eating sugary foods, the process 500 can determine whether the behavioral instruction at a particular moment indicates that the user is likely to eat sugary foods. For example, if the process 500 determines that the user is opening the freezer and reaching for the ice cream, the process 500 can indicate that the user is likely to eat the ice cream. In some cases, the process 500 can determine whether the determined likelihood exceeds a predetermined threshold. Similarly, the process 500 can determine a likelihood that a desired behavior is unlikely to occur and determine whether the likelihood exceeds a predetermined threshold. In some implementations, the process 500 can proceed to block 510 regardless of whether the behavior is determined to be likely in block 504. In some aspects, the process 500 can perform block 510 in parallel with block 506 and / or block 508. In some aspects, process 500 may proceed to block 506 if process 500 determines that the likelihood that a behavior will occur (or not occur) exceeds a threshold. In some cases, process 500 may determine the likelihood effectiveness (e.g., the likelihood that the intervention will prevent an undesirable behavior or promote a desired behavior in a given context or environment) of an intervention associated with a behavior that is likely to occur or not occur. For example, process 500 may determine that an intervention encouraging a user to go for a walk is likely to be effective if it is presented when the user stands up. This is because the likelihood that a user will go for a walk when the user stands up is greater than if the intervention is presented when the user is still sitting down.
[0099]
[0109] At block 506, the process generates an intervention to influence the user's behavior toward the user's stated goal. For example, if the process 500 determines at block 504 that the undesirable behavior is likely to occur, the process 500 can generate an intervention that is likely to discourage the user from engaging in the undesirable behavior. In another example, if the process 500 determines at block 504 that the user is likely to avoid the desired behavior, the process 500 can determine at block 506 an intervention that is likely to encourage the user to engage in the behavior. In some cases, in addition to or as an alternative to generating an intervention based on the determined likelihood, the process 500 can generate an intervention based on previous success or failure of available interventions in deterring (or encouraging) the user's target behavior. In some cases, the determined intervention can be selected from a library of intervention options for the user (e.g., obtained from the intervention engine 424 shown in FIG. 4). In some cases, the intervention options can be ranked based on the effectiveness of the interventions on previous occasions. In some cases, the determined intervention may be selected from a library of intervention options (e.g., obtained from the multi-user intervention engine 432 shown in FIG. 4) determined based on intervention effects determined from multiple users.
[0100]
[0110] In some cases, the particular intervention generated by the process 500 can be based on the determined likelihood that the behavior will occur (or not occur). For example, if the process 500 determines in block 504 that the likelihood that the particular behavior will occur (or not occur) slightly exceeds a likelihood threshold, the process 500 can determine that a small intervention is likely to deter (or encourage) the user from taking the behavior. In an illustrative example, the small intervention can include presenting a progress intervention (e.g., intervention 106 shown in FIG. 1B, intervention 112 shown in FIG. 1E, etc.) that reminds the user how long the user has successfully refrained from the undesirable behavior. In some aspects, if the process 500 determines in block 504 that the likelihood that the behavior will occur (or not occur) greatly exceeds a likelihood threshold, the process 500 can determine that only a large intervention is likely to deter (or encourage) the user from taking the behavior. In an illustrative example, the large intervention can include notifying a designated support person, playing an audio message, etc. In some cases, the use of mini- or macro-interventions may only be utilized for certain behaviors (e.g., a mini-intervention may be useful to prevent the consumption of certain amounts of food) and not for other types of behaviors (e.g., a mini-intervention may not deter a person with a drinking problem from drinking alcohol).
[0101]
[0111] In some cases, the process 500 can determine an intervention based on an indication of success of the intervention for individuals other than the user, but who share one or more characteristics with the user. In some cases, the process 500 can retrieve one or more interventions from a multi-user intervention library located on a server (e.g., retrieved from the multi-user intervention engine 432 shown in FIG. 4). In some cases, the multi-user intervention library can include an indication of intervention effectiveness associated with users who have common user characteristics (e.g., family status, personality type, any other characteristics described herein, and / or any other characteristics that may be common to different users). In some cases, in addition to or as an alternative to associating intervention effectiveness with user characteristics, the multi-user intervention library can include an indication of intervention effectiveness associated with users who commonly experience a similar context. In some cases, the process 500 can retrieve one or more interventions retrieved from the multi-user intervention library located on a server that were previously effective for other individuals who share similar characteristics with the user and / or who frequently experience similar contexts with the user.
[0102]
[0112] At block 508, the process 500 presents to the user the intervention determined at block 508. The intervention presented to the user may include any of the interventions described in this disclosure, such as, but not limited to, the interventions 106, 108, 110, 112, 124, 126, 128, and 142 shown in Figures 1B-1G, audio interventions, contacting a designated support person, etc. In some cases, the process 500 may generate the intervention and transmit the intervention to another device for presenting the intervention to the user.
[0103]
[0113] At block 510, process 500 monitors whether the behavior for which the likelihood was determined at block 504 occurs or does not occur. For example, if process 500 determines at block 504 that an undesirable behavior is likely to occur, determines an intervention at block 506, and presents the intervention at block 508, process 500 may determine at block 510 whether the undesirable behavior occurs after presenting the intervention. In another example, if process 500 determines at block 504 that the undesirable behavior is unlikely to occur, process 500 may monitor the undesirable behavior to determine whether the undesirable behavior occurs.
[0104]
[0114] At block 512, the process 500 analyzes the effectiveness of the intervention presented to the user at block 508. For example, at block 512, the process 500 can determine whether the target behavior occurred (or did not occur) after presenting the intervention at block 508. In some cases, the process 500 can determine an additional impact of the intervention. In one illustrative example, the process 500 can determine whether the prior behavior predicted to occur based on the likelihood determination at 504 was prevented by the intervention presented at block 508. In another illustrative example, the process 500 can determine whether the intervention was at least partially successful. For example, if the user engaged in the undesirable behavior to a lesser extent than in previous instances (e.g., smoked one cigarette instead of two), the process 500 can determine that the intervention was partially effective.
[0105]
[0115] At block 514, process 500 may record the intervention effects analyzed at block 512. In some examples, user feedback 522 may be utilized by process 500 at block 514. For example, the user may provide input indicating how effective the intervention was in the user's opinion, indicate whether the user took the action, and / or provide other feedback. In some cases, one or more additional parameters may be recorded along with the intervention effect to provide context to the recorded intervention effect. For example, one or more of the behavioral instructions evaluated at block 504 may be stored along with the recorded intervention effect to provide additional context to the stored intervention effect.
[0106]
[0116] At block 516, the process 500 may adjust the intervention for the user based on the intervention effectiveness determined at block 512. In one illustrative example, adjusting the intervention may include adjusting a score associated with the intervention. In some cases, a higher score may indicate a higher likelihood that the intervention will be successful in the future. In some cases, each intervention may be associated with multiple different scores, each of which may be associated with a different context. For example, a particular intervention may be effective in a first context (e.g., before the user decides to take the target behavior) but ineffective in a second context (e.g., after the user has begun the pre-behavior and / or decided to take the behavior). In such an example, the particular intervention may have a high score for the first context and a low score for the second context. In some cases, the process 500 may adjust the intervention based on one or more of the particular context evaluated at block 504 and the intervention effectiveness determined at block 512. In some cases, the process 500 may provide an indication of the intervention effectiveness to a system (e.g., the multi-user engine 430 shown in FIG. 4) that determines the effectiveness of the intervention for multiple different users. In some cases, the process 500 can tailor an intervention for the user based on one or more interventions retrieved from a multi-user intervention library located on a server that have been previously effective for other users who share similar characteristics as the user and / or who frequently experience similar contexts as the user. In some cases, the intervention tailoring can include incrementally tailoring the intervention (e.g., in real-time) based on effectiveness. For example, the system (e.g., an XR system) can present a first level of intervention that includes an image of the user's family on a pack of cigarettes. If the user reaches for a cigarette and picks it up, the system can present a stronger intervention that includes a pre-recorded message from the family member.
[0107]
[0117] At block 518, the process 500 may record the behavior prediction accuracy. For example, if the process 500 determines that the behavior is unlikely to occur at block 504 and does not provide an intervention, but the process 500 detects that the behavior has occurred at block 510, the process 500 may record a failure of the behavior prediction at block 518. In some cases, the process 500 may record a success of the behavior prediction when the process 500 determines that the behavior is likely at block 504 and detects that the behavior has occurred at block 510. In some cases, one or more additional parameters may be recorded with the behavior prediction to provide context to the recorded behavior prediction. For example, one or more of the behavioral indications evaluated at block 504 may be stored with the recorded behavior prediction to provide additional context to the stored behavior prediction. In some examples, user feedback 522 may be utilized by the process 500 at block 518. For example, the user may provide an input indicating how accurate the behavior prediction was.
[0108]
[0118] At block 520, the process 500 may adjust the behavioral prediction for the user (e.g., by adjusting the likelihood algorithm) based on the behavioral prediction accuracy recorded at block 518. In an illustrative example, adjusting the behavioral prediction may include adjusting one or more weights associated with the input behavioral instructions obtained at block 502. In some cases, the process 500 may indicate the failure or success of the behavioral prediction to the behavioral instruction engine (e.g., the behavioral instruction engine 410 shown in FIG. 4) for use in refining the behavioral triggers, prior actions, and / or behavioral artifacts that the behavioral instruction engine is monitoring. In some cases, the process 500 may provide a behavioral instruction to a server that determines the accuracy of the behavioral prediction and / or behavioral information (e.g., the behavioral triggers, prior actions, and / or behavioral artifacts) for multiple different users (e.g., obtained from the multi-user engine 430 shown in FIG. 4). In some cases, at block 520, process 500 may adjust behavior predictions for users having certain characteristics and / or experiencing certain contexts based on the indication of accuracy in determining the likelihood that other users will take the same or similar behavior, where the other users share similar characteristics as the user and / or the other users have previously experienced a similar context as the user. In some cases, process 500 may continue to block 504 after the adjustment is performed at block 520.
[0109]
[0119] As mentioned above, the behavioral intervention management system 400 and related techniques described herein can enable the system to detect the likelihood that a behavior will or will not occur and provide an intervention to encourage or discourage the user from taking that behavior according to the user's goals. The behavioral intervention management system 400 can learn user-specific behavioral instructions that are likely to result in the target behavior occurring (or not occurring) in the future and user-specific interventions that are likely to change the user's behavior in a desired way (e.g., encouraging the user to take a desired behavior or discouraging the user from taking an undesired behavior). Providing an intervention before a behavior occurs or does not occur can increase the chances that the user will choose to behave in a manner consistent with the user's goals.
[0110]
[0120] The behavioral intervention management system 400 can also learn behavioral indicators and interventions for multiple users and determine behavioral indicators and interventions that are applicable to a subset of the user population and are more likely to be successful. For example, the behavioral intervention management system 400 can associate user characteristics and / or goals with the likelihood of success of a particular intervention. Exemplary characteristics include, but are not limited to, gender, age, family status, target behavior, problem severity, country, culture, personality type, one or more health conditions, one or more dietary restrictions, one or more physical abilities, and / or other characteristics. In addition to or as an alternative to associating user characteristics and / or goals with the likelihood of success of a particular intervention, the behavioral intervention management system 400 can associate context (e.g., as represented by context information) with the likelihood of success of a particular intervention. Exemplary context information includes, but is not limited to, time of day, location of the user (e.g., work, home, near a bar), combination of technologies and / or sensors in use (e.g., whether devices near the user can provide the indicated intervention), stress level, other people in the person's vicinity, etc. In some cases, the behavioral intervention management system 400 can similarly associate characteristics and / or context with behavioral information and the success or failure of behavioral likelihood prediction (e.g., in block 504 of process 500 shown in FIG. 5 ) to learn predictive behaviors of a subset of the user population.
[0111]
[0121] FIG. 6 is a flow diagram illustrating a process 600 for generating one or more interventions. In block 602, the process 600 includes acquiring, by an extended reality (XR) device, behavioral information associated with a user of the XR device. For example, to acquire behavioral information associated with a user of the XR device, the process 600 can determine one or more behavioral triggers that are precursors to a behavior. The one or more behavioral triggers can include a stress level of the user, a heart rate of the user, an object in the field of view of the XR device, a location and / or environment in which the user is located (e.g., objects, people, etc. in the environment), a time at which the behavioral information is acquired, one or more people in proximity to the user, an activity performed by the user, any combination thereof, and / or other triggers. In some cases, to acquire behavioral information associated with a user of the XR device, the process 600 can determine one or more prior behaviors that indicate a likelihood that the user will take the behavior, as described herein. In some examples, to obtain activity information associated with a user of an XR device, process 600 may include detecting one or more activity artifacts associated with the activity in one or more images acquired by the XR device.
[0112]
[0122] At block 604, the process 600 includes determining, by the XR device, a likelihood that the user will take a certain behavior based on the behavior information. In some cases, the process 500 can determine a likely effectiveness (e.g., the likelihood that the intervention will prevent an undesirable behavior or promote a desired behavior in a given context or environment) of an intervention associated with a behavior that is likely to occur or not occur. At block 606, the process 600 includes determining, by the XR device, an intervention associated with the behavior based on the determined likelihood exceeding a likelihood threshold. In some examples, the process 600 can include determining, by the XR device, to refrain from generating a particular intervention based on the determined likelihood falling below a likelihood threshold. In some cases, the process 600 can determine, at block 606, an intervention or to refrain from generating a particular intervention based on the likely effectiveness of the intervention. For example, the process 500 can determine that an intervention encouraging the user to go for a walk is likely to be effective if presented when the user is standing, but the intervention is not effective when the user is sitting or lying down.
[0113]
[0123] At block 608, the process 600 includes generating, by the XR device, the intervention. For example, to generate the intervention, the process 600 can display the virtual content on a display of the XR device. As the virtual content is displayed by the display, the real-world environment can be viewed through the display of the XR device. In another example, the process 600 can output audio associated with the intervention (e.g., using a speaker). Generating the intervention can include any other type of output.
[0114]
[0124] At block 610, the process 600 includes determining whether the user took the action after outputting the intervention. Whether the user took the action may be based on an analysis of one or more images captured by the XR device, user input, and / or other information. At block 612, the process 600 includes determining effectiveness of the intervention based on whether the user took the action. For example, the process 600 may determine that the intervention is effective if the intervention prevents the user from performing an undesirable behavior (e.g., eating a bag of chips) or results in the user performing a desirable behavior (e.g., exercising).
[0115]
[0125] At block 614, the process 600 includes sending an indication of the effectiveness of the intervention to a server for use in determining the intervention for the one or more additional users. In some cases, the process 600 includes sending an indication of the accuracy with which the likelihood of the user taking the action is determined to the server (e.g., for use in determining the likelihood of the action for one or more additional users). In some examples, the process 600 includes sending contextual information associated with the intervention to the server. For example, as described herein, the contextual information associated with the intervention can include a time of day, one or more actions by the user of the XR device prior to the intervention, behavioral information associated with the user of the XR device, a location of the user of the XR device, a proximity of the user of the XR device to one or more individuals, any combination thereof, and / or other information. In some examples, the process 600 includes sending one or more characteristics associated with the user of the XR device to the server. For example, as described herein, one or more characteristics associated with a user of an XR device may include gender, age, family status, target behavior, country, culture, location, personality type, one or more health conditions, one or more dietary restrictions, one or more physical abilities, any combination thereof, and / or other characteristics. Exemplary operations of the server are described herein, including below with respect to FIG.
[0116]
[0126] FIG. 7 is a flow diagram illustrating a process 700 for predicting one or more behaviors. At block 702, the process 700 includes obtaining, by a server, first intervention information associated with the first user and the first intervention. In some examples, the first intervention information associated with the first user includes an intervention type, an indication of the effectiveness of the first intervention, an intervention context associated with the first intervention, one or more characteristics associated with the first user, any combination thereof, and / or other information. In some cases, the first intervention information includes contextual information associated with the first intervention. For example, the contextual information associated with the first intervention can include a time of day, one or more actions by the first user prior to the first intervention, the first intervention information associated with the first user, a location of the first user, a proximity of the first user to one or more individuals, any combination thereof, and / or other contextual information. In some aspects, the first intervention information includes one or more characteristics associated with the first user. For example, the one or more characteristics associated with the first user may include gender, age, family status, target behavior, country, culture, location, personality type, any combination thereof, and / or other characteristics.
[0117]
[0127] At block 704, the process 700 includes updating one or more parameters of the intervention library based on the first intervention information. The one or more parameters of the intervention library are based at least in part on the second intervention information associated with the second user and the second intervention. For example, the parameters of the intervention library may be updated based on multiple interventions for multiple users. The intervention library may then be used to determine or recommend an intervention for an additional user. For example, at block 706, the process 700 includes determining a third intervention for a third user based on the updated one or more parameters of the intervention library.
[0118]
[0128] In some examples, to determine a third intervention for the third user based on the updated one or more parameters of the intervention library, the process 700 includes obtaining, by a server, third intervention information associated with the third user. The process 700 may include determining a correlation between the third intervention information associated with the third user and fourth intervention information associated with the intervention library. The process 700 may further include determining the third intervention based on the correlation between the third intervention information and the fourth intervention information exceeding a correlation threshold. The process 700 may include transmitting the third intervention to a device associated with the third user. The device associated with the third user may then present (e.g., display, output via a speaker, etc.) the third intervention to the third user.
[0119]
[0129] In some aspects, the process 700 includes obtaining, by the server, fifth behavioral information associated with the fifth user and the fifth behavior. The process 700 may include updating one or more parameters of the behavioral library based on the fifth behavioral information. The one or more parameters of the behavioral library may be based at least in part on the sixth behavioral information associated with the sixth user. The process 700 may further include determining one or more behavioral parameters for the sixth user based on the updated one or more parameters of the intervention library. In some cases, the one or more behavioral parameters for the sixth user include a behavioral trigger, a prior behavior, a behavioral artifact associated with the fifth behavior, any combination thereof, and / or other parameters. In some aspects, the one or more behavioral parameters for the sixth user include one or more weightings associated with determining the likelihood that the sixth user will or will not perform the fifth behavior. In some examples, the fifth behavioral information includes one or more characteristics associated with the fifth user. In some cases, the fifth behavioral information includes contextual information associated with the fifth behavior.
[0120]
[0130] In some examples, the processes described herein (e.g., processes 500, 600, 700 and / or other processes described herein) may be performed by a computing device or apparatus. In one example, one or more of the processes may be performed by the XR system 200 shown in FIG. 2. In another example, one or more of the processes may be performed by the XR system 320 shown in FIG. 3. In another example, one or more of the processes may be performed by the computing system 1000 shown in FIG. 10. For example, a computing device with the computing system 1000 shown in FIG. 10 may include components of the XR system 200 and may implement operations of the process 500 of FIG. 5, the process 600 of FIG. 6, the process 700 of FIG. 7, and / or other processes described herein.
[0121]
[0131] The computing device may include any suitable device, such as a vehicle or a vehicular computing device, a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a network-connected watch or smartwatch, or other wearable device), a server computer, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein, including processes 500, 600, 700, and / or other processes described herein. In some cases, a computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.
[0122]
[0132] Components of a computing device may be implemented in circuitry. For example, components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein.
[0123]
[0133] Processes 500, 600, and 700 are illustrated as logical flow diagrams, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.
[0124]
[0134] Additionally, processes 500, 600, and 700, and / or other processes described herein, may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.
[0125]
[0135] As mentioned above, various aspects of the present disclosure may use machine learning models or systems. FIG. 8 is an illustrative example of a deep learning neural network 800 that may be used to implement the machine learning based feature extraction and / or activity recognition (or classification) described above. The input layer 820 includes input data. In one illustrative example, the input layer 820 may include data representing pixels of an input video frame. The neural network 800 includes multiple hidden layers 822a, 822b-822n. The hidden layers 822a, 822b-822n include "n" hidden layers, where "n" is an integer equal to or greater than 1. The number of hidden layers may be as many layers as required for a given application. The neural network 800 further includes an output layer 821 that provides an output resulting from the processing performed by the hidden layers 822a, 822b-822n. In one illustrative example, the output layer 821 may provide a classification for an object in the input video frame. The classification may include classes that identify types of activities (eg, looking up, looking down, closing eyes, yawning, etc.).
[0126]
[0136] Neural network 800 is a multi-layered neural network of interconnected nodes. Each node can represent information. The information associated with the nodes is shared between different layers, with each layer retaining the information as it is processed. In some cases, neural network 800 can include a feed-forward network, in which there are no feedback connections where the output of the network is fed back to itself. In some cases, neural network 800 can include a recurrent neural network, which can have loops that allow information to be carried across nodes while reading at the input.
[0127]
[0137] Through node-to-node interconnections between the various layers, information may be exchanged between the nodes. The nodes of the input layer 820 may activate a set of nodes in the first hidden layer 822a. For example, as shown, each of the input nodes of the input layer 820 is connected to each of the nodes of the first hidden layer 822a. The nodes of the first hidden layer 822a may transform the information of each input node by applying an activation function to the input node information. The information derived from the transformation may then be passed to the nodes of the next hidden layer 822b, which may activate such nodes, which may perform their own designated functions. Exemplary functions include convolution, upsampling, data transformation, and / or any other suitable functions. The output of the hidden layer 822b may then activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 822n may activate one or more nodes of the output layer 821, where an output is provided. In some cases, a node in neural network 800 (e.g., node 826) is shown as having multiple output lines; however, the node has a single output, and all lines shown as outputting from the node represent the same output value.
[0128]
[0138] In some cases, each node or interconnections between nodes can have weights, which are a set of parameters derived from training of the neural network 800. Once the neural network 800 is trained, it may be referred to as a trained neural network and may be used to classify one or more activities. For example, the interconnections between nodes may represent information learned about the interconnected nodes. The interconnections may have adjustable numerical weights that may be adjusted (e.g., based on a training data set), allowing the neural network 800 to be adaptive to the input and to learn as more and more data is processed.
[0129]
[0139] The neural network 800 is pre-trained to process features from data in the input layer 820 using different hidden layers 822a, 822b-822n to provide output through the output layer 821. In an example where the neural network 800 is used to identify activities being performed by a driver in frames, the neural network 800 may be trained using training data including both frames and labels, as described above. For example, training frames may be input to the network, with each training frame having a label indicating the feature in the frame (for a feature extraction machine learning system) or the class of activity in each frame. In an example using object classification for illustrative purposes, the training frames may include images of the number 2, in which case the label for the image may be [0 0 1 0 0 0 0 0 0 0].
[0130]
[0140] In some cases, neural network 800 may use a training process called backpropagation to adjust node weights. As described above, the backpropagation process may include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed during one training iteration. The process may be repeated for a number of iterations for each set of training images until neural network 800 is trained well enough that the layer weights are accurately adjusted.
[0131]
[0141] For the example of identifying objects in a frame, the forward pass may include passing a training frame through neural network 800. Before neural network 800 is trained, the weights are first randomized. As an illustrative example, the frame may include an array of numbers representing pixels of the image. Each number in the array may include a value from 0 to 255 representing the pixel intensity at that location in the array. In one example, the array may include a 28×28×3 array of numbers, with 28 rows and 28 columns of pixels and three color components (such as a red component, a green component, and a blue component, or a luma component and two chroma components).
[0132]
[0142] As mentioned above, in the first training iteration for the neural network 800, due to the weights being randomly selected at initialization, the output may contain values that do not give a preference to any particular class. For example, if the output is a vector with the probabilities that the object contains different classes, the probability values for each of the different classes may be equal or at least very similar (e.g., for 10 possible classes, each class may have a probability value of 0.1). With the initial weights, the neural network 800 is unable to make an accurate determination of what the classification of the object is likely to be, since it is unable to determine low-level features. A loss function may be used to analyze the error in the output. Any suitable loss function definition may be used, such as cross-entropy loss. Another example loss function includes the mean squared error (MSE), which is
[0133]
number
[0134] The loss is defined as E 合計 may be set equal to the value of
[0135]
[0143] The loss (or error) is large for the first training image because the actual values are very different from the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 800 can perform a backward pass by determining which inputs (weights) contributed most to the network's loss, and the weights can be adjusted so that the loss is small and eventually minimized. To determine which weights contributed most to the network's loss, the derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight at a particular layer) can be calculated. After the derivatives are calculated, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated such that the weights change in the opposite direction of the gradient. The weight update can be
[0136]
number
[0137] where w represents the weight and w i represents the initial weights and η represents the learning rate, which can be set to any suitable value, with a higher learning rate inducing larger weight updates and a smaller value indicating smaller weight updates.
[0138]
[0144] Neural network 800 may include any suitable deep network. An example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layers of a CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. Neural network 800 may include any other deep network other than a CNN, such as autoencoders, deep belief nets (DBNs), and Recurrent Neural Networks (RNNs), among others.
[0139]
[0145] FIG. 9 is an illustrative example of a convolutional neural network (CNN) 900. An input layer 920 of the CNN 900 includes data representing an image or frame. For example, the data may include an array of numbers representing pixels of the image, with each number in the array including a value from 0 to 255 representing the pixel intensity at that location in the array. Using the previous example from above, the array may include a 28×28×3 array of numbers, with 28 rows and 28 columns of pixels and three color components (e.g., a red component, a green component, and a blue component, or a luma component and two chroma components, etc.). The image is passed through a convolutional hidden layer 922a, an optional nonlinear activation layer, a pooling hidden layer 922b, and a fully connected hidden layer 922c to obtain an output at an output layer 924. Although only one of each hidden layer is shown in FIG. 9, one skilled in the art will appreciate that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in the CNN 900. As explained previously, the output may indicate a single class of object, or may include a probability of the class that best represents the object in the image.
[0140]
[0146] The first layer of the CNN 900 is the convolutional hidden layer 922a. The convolutional hidden layer 922a analyzes the image data of the input layer 920. Each node of the convolutional hidden layer 922a is connected to a region of nodes (pixels) of the input image called the receptive field. The convolutional hidden layer 922a can be thought of as one or more filters (each filter corresponds to a different activation or feature map), and each convolutional iteration of a filter is a node or neuron of the convolutional hidden layer 922a. For example, the region of the input image that the filter covers in each convolutional iteration will be the receptive field of the filter. In one illustrative example, if the input image includes a 28x28 array and each filter (and corresponding receptive field) is a 5x5 array, there will be 24x24 nodes in the convolutional hidden layer 922a. Each connection between a node and the receptive field for that node learns a weight, and possibly an overall bias, so that each node learns to analyze its particular local receptive field in the input image. Each node in the hidden layer 922a has the same weights and biases (called shared weights and shared biases). For example, a filter has an array of weights (numbers) and the same depth as the input. The filter has a depth of 3 in the example video frame (according to the three color components of the input image). An illustrative example size of the filter array is 5x5x3, corresponding to the size of the receptive fields of the nodes.
[0141]
[0147] The convolutional nature of the convolutional hidden layer 922a results from each node of the convolutional layer being applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 922a may start at the upper left corner of the input image array and convolve around the input image. As mentioned above, each convolutional iteration of the filter may be considered a node or neuron of the convolutional hidden layer 922a. In each convolutional iteration, the value of the filter is multiplied with a corresponding number of original pixel values of the image (e.g., a 5×5 filter array is multiplied with a 5×5 array of input pixel values at the upper left corner of the input image array). The multiplications from each convolutional iteration may be added together to obtain a total for that iteration or node. The process then continues at the next location in the input image according to the receptive field of the next node in the convolutional hidden layer 922a. For example, the filter may be moved to the next receptive field by a certain step amount (called a stride). The stride may be set to 1 or other suitable amount. For example, if the stride is set to 1, the filter is moved one pixel to the right in each convolution iteration. Processing the filter at each unique location of the input volume produces a number representing the filter result for that location, resulting in an aggregate value being determined for each node in the convolutional hidden layer 922a.
[0142]
[0148] The mapping from the input layer to the convolutional hidden layer 922a is called an activation map (or feature map). The activation map contains the value of each node that represents the filter result at each location of the input volume. The activation map can include an array containing various aggregate values resulting from each iteration of the filter on the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (a stride of 1), the activation map will include a 24×24 array. The convolutional hidden layer 922a can include several activation maps to identify multiple features in an image. The example shown in FIG. 9 includes three activation maps. Using the three activation maps, the convolutional hidden layer 922a can detect three different types of features, each detectable throughout the image.
[0143]
[0149] In some examples, a nonlinear hidden layer may be applied after the convolutional hidden layer 922a. A nonlinear layer may be used to introduce nonlinearity into a system that previously computed linear operations. One illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. A ReLU layer may apply a function f(x)=max(0,x) to all of the values in the input volume, which changes all negative activations to 0. Thus, ReLU can enhance the nonlinear nature of the CNN 900 without affecting the receptive field of the convolutional hidden layer 922a.
[0144]
[0150] After the convolutional hidden layer 922a (and after the nonlinear hidden layer, when used), a pooling hidden layer 922b may be applied. The pooling hidden layer 922b is used to simplify the information in the output from the convolutional hidden layer 922a. For example, the pooling hidden layer 922b may take each activation map output from the convolutional hidden layer 922a and use a pooling function to generate a condensed activation map (or feature map). Max pooling is one example of a function implemented by the pooling hidden layer. Other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions, may be used by the pooling hidden layer 922a. A pooling function (e.g., a max pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 922a. In the example shown in FIG. 9, three pooling filters are used for the three activation maps in the convolutional hidden layer 922a.
[0145]
[0151] In some examples, max pooling may be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimensions of the filter, such as a stride of 2) to the activation map output from the convolutional hidden layer 922a. The output from the max pooling filter contains the maximum number of any subregion around which the filter convolves. Using a 2×2 filter as an example, each unit in the pooling layer may summarize a region of 2×2 nodes (each node is a value in the activation map) in the previous layer. For example, four values (nodes) in the activation map are analyzed by a 2×2 max pooling filter at each iteration of the filter, and the maximum value from the four values is output as the “max” value. If such a max pooling filter is applied to an activation filter from the convolutional hidden layer 922a with dimensions of 24×24 nodes, the output from the pooling hidden layer 922b is an array of 12×12 nodes.
[0146]
[0152] In some examples, an L2 norm pooling filter may also be used, which involves computing the square root of the sum of the squares of the values in a 2×2 region (or other suitable region) of the activation map (rather than computing the maximum value as is done in max pooling) and using the computed value as the output.
[0147]
[0153] Intuitively, a pooling function (e.g., max pooling function, L2 norm pooling function, or other pooling function) determines whether a given feature is found anywhere in a region of the image and discards the exact location information. This can be done without affecting the outcome of feature detection because once a feature has been found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (as well as other pooling methods) offers the advantage that there are many fewer features to pool, thus reducing the number of parameters required in later layers of CNN 900.
[0148]
[0154] The final layer of connections in the network is a fully connected layer that connects every node from the pooling hidden layer 922b to every one of the output nodes in the output layer 924. Using the above example, the input layer includes 28×28 nodes that encode pixel intensities of the input image, the convolutional hidden layer 922a includes 3×24×24 hidden feature nodes based on application of 5×5 local receptive fields (for the filters) to three activation maps, and the pooling hidden layer 922b includes a layer of 3×12×12 hidden feature nodes based on application of max pooling filters to 2×2 regions across each of the three feature maps. Extending this example, the output layer 924 can include 10 output nodes. In such an example, every node of the 3×12×12 pooling hidden layer 922b is connected to every node of the output layer 924.
[0149]
[0155] The fully connected layer 922c can take the output of the previous pooling hidden layer 922b (which should represent the activation map of high-level features) and determine the features that are most correlated to a particular class. For example, the fully connected layer 922c can determine the high-level features that are most strongly correlated to a particular class and can include weights (nodes) for the high-level features. A product can be calculated between the weights of the fully connected layer 922c and the pooling hidden layer 922b to obtain probabilities for different classes. For example, if the CNN 900 is being used to predict that an object in a video frame is a person, there will be high values in the activation map that represent high-level features of a person (e.g., having two legs, having a face on the top of the object, having two eyes on the top left and top right of the face, a nose in the middle of the face, a mouth on the bottom of the face, and / or other features that are common to people).
[0150]
[0156] In some examples, the output from the output layer 924 may include an M-dimensional vector (in the previous example, M=10). M indicates the number of classes the CNN 900 must choose from when classifying objects in an image. Other example outputs may also be provided. Each number in the M-dimensional vector may represent a probability that the object is of a certain class. In one illustrative example, if a 10-dimensional output vector represents 10 different classes of objects as [0 0 0.05 0.8 0 0.15 0 0 0 0], the vector indicates that there is a 5% probability that the image is of the third class of object (e.g., a dog), an 80% probability that the image is of the fourth class of object (e.g., a human), and a 15% probability that the image is of the sixth class of object (e.g., a kangaroo). The probability of a class may be considered a confidence level that the object is part of that class.
[0151]
[0157] Fig. 10 illustrates an example of a system for implementing certain aspects of the present technology. In particular, Fig. 10 illustrates an example of a computing system 1000, which may be any computing device, such as an internal computing system, a remote computing system, a camera, or any components thereof, in which the components of the system communicate with each other using a connection 1005. The connection 1005 may be a physical connection using a bus, or a direct connection to a processor 1010, such as in a chipset architecture. The connection 1005 may also be a virtual connection, a network connection, or a logical connection.
[0152]
[0158] In some embodiments, computing system 1000 is a distributed system in which the functionality described in this disclosure may be distributed across a data center, multiple data centers, a peer network, etc. In some embodiments, one or more of the described system components represent many components, each performing some or all of the functionality that is the subject of the component description. In some embodiments, the components may be physical or virtual devices.
[0153]
[0159] The exemplary computing system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 coupling various system components to the processor 1010, including system memory 1015, such as read-only memory (ROM) 1020 and random access memory (RAM) 1025. The computing system 1000 may include a cache of high-speed memory 1012, either directly connected to the processor 1010, in close proximity to the processor 1010, or integrated as part of the processor 1010.
[0154]
[0160] The processor 1010 may include any general purpose processor, as well as hardware or software services, such as services 1032, 1034, and 1036 stored in a storage device 1030, configured to control the processor 1010, and special purpose processors where the software instructions are embedded in the actual processor design. The processor 1010 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.
[0155]
[0161] To enable user interaction, computing system 1000 includes input devices 1045, which may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 1000 may also include output devices 1035, which may be one or more of several output mechanisms. In some cases, a multi-modal system may enable a user to provide multiple types of input / output to communicate with computing system 1000. Computing system 1000 may include a communication interface 1040, which may generally govern and manage user input and system output.The communications interface may be any of the following: audio jack / plug, microphone jack / plug, universal serial bus (USB) port / plug, Apple® Lightning® port / plug, Ethernet port / plug, fiber optic port / plug, proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® low energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio-frequency identification (RFID) wireless signal transmission, near-field communications (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WLAN), and Bluetooth® wireless signal transmission. The wireless communication device may perform or facilitate receiving and / or transmitting wired or wireless communications using wired and / or wireless transceivers, including those utilizing WiMAX (Wireless Access), infrared (IR) communications wireless signal transmission, Public Switched Telephone Network (PSTN) signal transmission, Integrated Services Digital Network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof.The communication interface 1040 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing system 1000 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States Global Positioning System (GPS), the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS. There is no constraint to operate on any particular hardware configuration, and therefore the basic features herein may be easily substituted for improved hardware or firmware configurations as they are developed.
[0156]
[0162] The storage device 1030 may be a non-volatile and / or non-transitory and / or computer readable memory device, such as a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disc read only memory (CD-ROM) optical disk, a rewritable compact disc (CD) optical disk, a digital video disk (DVD) optical disk, a blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (ICC), a circuit, IC chips / cards, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random-access memory (RRAM),The memory may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a memory, RRAM / ReRAM, phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.
[0157]
[0163] The storage device 1030 may include software services, servers, services, etc. that cause the system to perform functions when code defining such software is executed by the processor 1010. In some embodiments, hardware services that perform particular functions may include software components stored in a computer-readable medium in conjunction with the necessary hardware components, such as the processor 1010, connections 1005, output devices 1035, etc., to perform the functions.
[0158]
[0164] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. Computer-readable media may include non-transitory media on which data may be stored and which do not include carrier waves and / or transitory electronic signals propagating wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.
[0159]
[0165] In some embodiments, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when talking about non-transitory computer readable storage media, non-transitory computer readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.
[0160]
[0166] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks, including devices, device components, steps or routines in a method embodied in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.
[0161]
[0167] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.
[0162]
[0168] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions, or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, network-attached storage devices, etc.
[0163]
[0169] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) to perform the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented on a circuit board among different chips, or on different processes executing within a single device, as further examples.
[0164]
[0170] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.
[0165]
[0171] In the above description, aspects of the present application are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. The various features and aspects of the present application described above may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from that described.
[0166]
[0172] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of the present specification.
[0167]
[0173] When a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.
[0168]
[0174] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).
[0169]
[0175] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, a claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. As another example, a claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and can additionally include unrecited items in the set of A and B.
[0170]
[0176] The various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
[0171]
[0177] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.
[0172]
[0178] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor. Alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.
[0173]
[0179] Exemplary aspects of the present disclosure include the following.
[0174]
[0180] Aspect 1: A method of generating one or more interventions, the method including: acquiring, by an extended reality (XR) device, behavioral information associated with a user of the XR device; determining, by the XR device, a likelihood that the user will take an action based on the behavioral information; determining, by the XR device, the intervention based on the determined likelihood exceeding a likelihood threshold; generating, by the XR device, the intervention; determining, after generating the intervention, whether the user took the action; determining effectiveness of the intervention based on whether the user took the action; and transmitting an indication of the effectiveness of the intervention to a server for use in determining the intervention for one or more additional users.
[0175]
[0181] Aspect 2: The method of aspect 1, further comprising transmitting contextual information associated with the intervention to a server, wherein the contextual information associated with the intervention includes at least one of a time of day, one or more actions by a user of the XR device prior to the intervention, behavioral information associated with the user of the XR device, a location of the user of the XR device, and a proximity of the user of the XR device to one or more individuals.
[0176]
[0182] Aspect 3: The method of aspect 1 or 2, further comprising transmitting to a server one or more characteristics associated with the user of the XR device, wherein the one or more characteristics associated with the user of the XR device include at least one of: gender, age, family status, target behavior, country, culture, location, personality type, one or more health conditions, one or more dietary restrictions, and one or more physical abilities.
[0177]
[0183] Aspect 4: The method of any of aspects 1-3, wherein obtaining behavioral information associated with a user of the XR device includes determining one or more behavioral triggers that are precursors to the behavior, the one or more behavioral triggers including at least one of the user's stress level, the user's heart rate, objects within a field of view of the XR device, a location where the user is located, a time at which the behavioral information is obtained, one or more people in proximity to the user, and an activity performed by the user.
[0178]
[0184] Aspect 5: The method of any of aspects 1-4, wherein obtaining behavioral information associated with a user of the XR device includes determining one or more prior actions indicative of a likelihood that the user will take that action.
[0179]
[0185] Aspect 6: The method of any of aspects 1 to 5, wherein acquiring behavioral information associated with a user of the XR device includes detecting one or more behavioral artifacts associated with the behavior in one or more images acquired by the XR device.
[0180]
[0186] Aspect 7: Any of the methods of aspects 1-6, wherein generating the intervention includes displaying virtual content on a display of the XR device, and wherein the real-world environment is viewable through the display of the XR device as the virtual content is displayed by the display.
[0181]
[0187] Aspect 8: An apparatus for generating one or more interventions, the apparatus comprising at least one memory and at least one processor coupled to the at least one memory, the at least one processor configured to obtain behavioral information associated with a user of the apparatus, determine a likelihood that the user will take a certain action based on the behavioral information, determine an intervention associated with the action based on the determined likelihood exceeding a likelihood threshold, generate the intervention, and after outputting the intervention, determine whether the user took the action, determine effectiveness of the intervention based on whether the user took the action, and transmit an indication of the effectiveness of the intervention to a server for use in determining the intervention for one or more additional users.
[0182]
[0188] Aspect 9: The device of aspect 8, wherein at least one processor is configured to transmit contextual information associated with the intervention to a server, the contextual information associated with the intervention including at least one of a time of day, one or more actions by a user of the device prior to the intervention, behavioral information associated with the user of the device, a location of the user of the device, and a proximity of the user of the device to one or more individuals.
[0183]
[0189] Aspect 10: The device of aspect 8 or 9, wherein at least one processor is configured to transmit one or more characteristics associated with a user of the device to the server, the one or more characteristics associated with the user of the device including at least one of gender, age, family status, target behavior, country, culture, location, personality type, one or more health conditions, one or more dietary restrictions, and one or more physical abilities.
[0184]
[0190] Aspect 11: A device of any of aspects 8 to 10, wherein to obtain behavioral information associated with a user of the device, at least one processor is configured to determine one or more behavioral triggers that are precursors to the behavior, the one or more behavioral triggers including at least one of the user's stress level, the user's heart rate, objects within a field of view of the device, a location where the user is located, a time at which the behavioral information is obtained, one or more people in proximity to the user, and an activity performed by the user.
[0185]
[0191] Aspect 12: A device of any of aspects 8 to 11, wherein to obtain behavioral information associated with a user of the device, at least one processor is configured to determine one or more prior actions indicative of the likelihood that the user will take that action.
[0186]
[0192] Aspect 13: An apparatus of any of aspects 8 to 12, wherein at least one processor is configured to detect one or more behavioral artifacts associated with the behavior in one or more images acquired by the apparatus in order to obtain behavioral information associated with a user of the apparatus.
[0187]
[0193] Aspect 14: A device of any of aspects 8 to 13, wherein to generate an intervention, at least one processor is configured to display virtual content on a display of the device, and when the virtual content is displayed by the display, it is possible to view the real-world environment through the display of the device.
[0188]
[0194] Embodiment 15: The device of any one of embodiments 8 to 14, wherein the device is an extended reality (XR) device.
[0189]
[0195] Aspect 16: A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform any of the operations of aspects 1-14.
[0190]
[0196] Embodiment 17: An apparatus comprising means for performing any of the operations of embodiments 1 to 15.
[0191]
[0197] Aspect 18: A method for generating one or more interventions, the method including: acquiring, by a server, first intervention information associated with a first user and the first intervention; updating one or more parameters of an intervention library based at least in part on second intervention information associated with a second user and the second intervention based on the first intervention information; and determining a third intervention for a third user based on the updated one or more parameters of the intervention library.
[0192]
[0198] Aspect 19: The method of aspect 18, wherein the first intervention information associated with the first user includes at least one of an intervention type, an indication of effectiveness of the first intervention, an intervention context associated with the first intervention, and one or more characteristics associated with the first user.
[0193]
[0199] Example 20: The method of example 18 or 19, wherein determining a third intervention for the third user based on the updated one or more parameters of the intervention library includes obtaining, by a server, third intervention information associated with the third user; determining a correlation between the third intervention information associated with the third user and fourth intervention information associated with the intervention library; determining the third intervention based on the correlation between the third intervention information and the fourth intervention information exceeding a correlation threshold; and transmitting the third intervention to a device associated with the third user.
[0194]
[0200] Example 21: The method of any of Examples 18-20, wherein the first intervention information includes context information associated with the first intervention.
[0195]
[0201] Aspect 22: The method of any of aspects 18-21, wherein the contextual information associated with the first intervention includes at least one of a time of day, one or more actions by the first user prior to the first intervention, first intervention information associated with the first user, a location of the first user, and a proximity of the first user to one or more individuals.
[0196]
[0202] Example 23: The method of any of Examples 18-22, wherein the first intervention information includes one or more characteristics associated with the first user.
[0197]
[0203] Aspect 24: The method of any of aspects 18-23, wherein the one or more characteristics associated with the first user include at least one of gender, age, family status, target behavior, country, culture, location, and personality type.
[0198]
[0204] Aspect 25: The method of any of aspects 18 to 24, wherein the method further includes acquiring, by a server, fifth behavioral information associated with a fifth user and a fifth behavior, updating one or more parameters of a behavioral library based at least in part on sixth behavioral information associated with a sixth user based on the fifth behavioral information, and determining one or more behavioral parameters for a seventh user based on the updated one or more parameters of the intervention library.
[0199]
[0205] Aspect 26: The method of any of aspects 18 to 25, wherein the one or more behavioral parameters for the seventh user include at least one of a behavioral trigger, a pre-action, and a behavioral artifact associated with the fifth action.
[0200]
[0206] Aspect 27: The method of any of aspects 18-26, wherein the one or more behavioral parameters for the seventh user include one or more weightings associated with determining the likelihood that the seventh user will perform or not perform the fifth behavior.
[0201]
[0207] Example 28: The method of any of Examples 18-27, wherein the fifth behavioral information includes one or more characteristics associated with the fifth user.
[0202]
[0208] Example 29: The method of any of Examples 18 to 28, wherein the fifth behavioral information includes context information associated with the fifth behavior.
[0203]
[0209] Aspect 30: A system for generating one or more interventions, the system comprising at least one memory and at least one processor coupled to the at least one memory, the at least one processor configured to obtain first intervention information associated with a first user and the first intervention, update one or more parameters of an intervention library based at least in part on second intervention information associated with a second user and the second intervention based on the first intervention information, and determine a third intervention for a third user based on the updated one or more parameters of the intervention library.
[0204]
[0210] Aspect 31: The system of aspect 30, wherein the first intervention information associated with the first user includes at least one of an intervention type, an indication of effectiveness of the first intervention, an intervention context associated with the first intervention, and one or more characteristics associated with the first user.
[0205]
[0211] Aspect 32: The system of aspect 30 or 31, wherein to determine a third intervention for a third user based on one or more updated parameters of the intervention library, at least one processor is configured to obtain third intervention information associated with the third user, determine a correlation between the third intervention information associated with the third user and fourth intervention information associated with the intervention library, determine the third intervention based on the correlation between the third intervention information and the fourth intervention information exceeding a correlation threshold, and send the third intervention to a device associated with the third user.
[0206]
[0212] Example 33: The system of any of Examples 30 to 32, wherein the first intervention information includes context information associated with the first intervention.
[0207]
[0213] Aspect 34: The system of any of aspects 30-33, wherein the contextual information associated with the first intervention includes at least one of a time of day, one or more actions by the first user prior to the first intervention, first intervention information associated with the first user, a location of the first user, and a proximity of the first user to one or more individuals.
[0208]
[0214] Example 35: The system of any of Examples 30-34, wherein the first intervention information includes one or more characteristics associated with the first user.
[0209]
[0215] Aspect 36: The system of any of aspects 30 to 35, wherein the one or more characteristics associated with the first user include at least one of gender, age, family status, target behavior, country, culture, location, and personality type.
[0210]
[0216] Aspect 37: The system of any of aspects 30-36, wherein at least one processor is configured to obtain fifth behavioral information associated with a fifth user and a fifth behavior, update one or more parameters of a behavioral library based at least in part on sixth behavioral information associated with the sixth user based on the fifth behavioral information, and determine one or more behavioral parameters for a seventh user based on the updated one or more parameters of the intervention library.
[0211]
[0217] Aspect 38: The system of any of aspects 30 to 37, wherein one or more behavioral parameters for the seventh user include at least one of a behavioral trigger, a pre-action, and a behavioral artifact associated with the fifth action.
[0212]
[0218] Aspect 39: The system of any of aspects 30-38, wherein the one or more behavioral parameters for the seventh user include one or more weightings associated with determining the likelihood that the sixth user will perform or not perform the fifth behavior.
[0213]
[0219] Aspect 40: The system of any of aspects 30 to 39, wherein the fifth behavioral information includes one or more characteristics associated with the fifth user.
[0214]
[0220] Example 41: The system of any of examples 30 to 40, wherein the fifth behavioral information includes context information associated with the fifth behavior.
[0215]
[0221] Aspect 42: The system of any of aspects 30 to 41, wherein the system includes at least one server.
[0216]
[0222] Aspect 43: A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform any of the operations of aspects 18-42.
[0217]
[0223] Example 44: An apparatus comprising means for performing any of the operations of Examples 18 to 42.
[0218]
[0224] Example 45: A method comprising an operation according to any one of examples 1 to 7 and an operation according to any one of examples 18 to 29.
[0219]
[0225] Aspect 46: An apparatus for performing temporal blending on one or more frames. The apparatus includes a memory (e.g., implemented in a circuit) configured to store one or more frames, and one or more processors (e.g., one processor or multiple processors) coupled to the memory. The one or more processors are configured to perform operations according to any of aspects 1-7 and any of aspects 18-29.
[0220]
[0226] Aspect 47: A computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform an operation according to any of aspects 1-7 and an operation according to any of aspects 18-29.
[0221]
[0227] Embodiment 48: An apparatus comprising means for performing the operations according to any one of embodiments 1 to 7 and any one of embodiments 18 to 29.
Claims
1. 1. A method of generating one or more interventions, comprising: acquiring, by an extended reality (XR) device, behavioral information associated with a user of the XR device; determining, by the XR device, a likelihood that the user will take an action based on the action information; determining, by the XR device, an intervention based on the determined likelihood exceeding a likelihood threshold; generating said intervention by said XR device; determining whether the user took the action after generating the intervention; determining an effectiveness of the intervention based on whether the user took the action; and transmitting an indication of the validity of the intervention to a server for use in determining an intervention for one or more additional users; A method comprising:
2. 2. The method of claim 1, further comprising transmitting contextual information associated with the intervention to the server, wherein the contextual information associated with the intervention includes at least one of a time of day, one or more actions by the user of the XR device prior to the intervention, the behavioral information associated with the user of the XR device, a location of the user of the XR device, and a proximity of the user of the XR device to one or more individuals.
3. 10. The method of claim 1, further comprising transmitting to the server one or more characteristics associated with the user of the XR device, wherein the one or more characteristics associated with the user of the XR device include at least one of: gender, age, family status, target behavior, country, culture, location, personality type, one or more health conditions, one or more dietary restrictions, and one or more physical abilities.
4. 2. The method of claim 1, wherein obtaining the behavioral information associated with the user of the XR device includes determining one or more behavioral triggers that are predictive of the behavior, the one or more behavioral triggers including at least one of a stress level of the user, a heart rate of the user, an object in a field of view of the XR device, a location where the user is located, a time when the behavioral information is obtained, one or more people in proximity to the user, and an activity performed by the user.
5. 5. The method of claim 4, wherein obtaining the behavioral information associated with the user of the XR device comprises determining one or more prior behaviors indicative of a likelihood that the user will take the behavior.
6. 2. The method of claim 1 , wherein acquiring the activity information associated with the user of the XR device comprises detecting one or more activity artifacts associated with the activity in one or more images acquired by the XR device.
7. 2. The method of claim 1, wherein generating the intervention includes displaying virtual content on a display of the XR device, wherein a real-world environment is viewable through the display of the XR device as the virtual content is displayed by the display.
8. 1. An apparatus for generating one or more interventions, comprising: At least one memory; at least one processor coupled to the at least one memory; wherein the at least one processor: Obtaining behavioral information associated with a user of the device; determining a likelihood that the user will take an action based on the behavioral information; determining an intervention associated with the behavior based on the determined likelihood exceeding a likelihood threshold; generating said intervention; determining whether the user took the action after outputting the intervention; determining the effectiveness of the intervention based on whether the user took the action; sending an indication of the validity of the intervention to a server for use in determining an intervention for one or more additional users. The apparatus is configured to:
9. The at least one processor:
10. The device of claim 8, further configured to transmit contextual information associated with the intervention to the server, the contextual information associated with the intervention including at least one of a time of day, one or more actions by the user of the device prior to the intervention, the behavioral information associated with the user of the device, a location of the user of the device, and a proximity of the user of the device to one or more individuals.
10. The at least one processor:
10. The device of claim 8, configured to transmit one or more characteristics associated with the user of the device to the server, the one or more characteristics associated with the user of the device including at least one of: gender, age, family status, target behavior, country, culture, location, personality type, one or more health conditions, one or more dietary restrictions, and one or more physical abilities.
11. 10. The device of claim 8, wherein to obtain the behavioral information associated with the user of the device, the at least one processor is configured to determine one or more behavioral triggers that are predictive of the behavior, the one or more behavioral triggers including at least one of the user's stress level, the user's heart rate, objects within a field of view of the device, a location where the user is located, a time at which the behavioral information is obtained, one or more people in proximity to the user, and an activity performed by the user.
12. 12. The device of claim 11, wherein to obtain the behavioral information associated with the user of the device, the at least one processor is configured to determine one or more prior behaviors indicative of a likelihood that the user will take the behavior.
13. 9. The device of claim 8, wherein to obtain the behavioral information associated with the user of the device, the at least one processor is configured to detect one or more behavioral artifacts associated with the behavior in one or more images acquired by the device.
14. 9. The device of claim 8, wherein to generate the intervention, the at least one processor is configured to display virtual content on a display of the device, and a real-world environment can be viewed through the display of the device when the virtual content is displayed by the display.
15. The apparatus of claim 8 , wherein the apparatus is an extended reality (XR) device.
16. 1. A method of generating one or more interventions, comprising: obtaining, by a server, first intervention information associated with a first user and a first intervention; updating one or more parameters of an intervention library based at least in part on second intervention information associated with a second user and a second intervention based on the first intervention information; determining a third intervention for a third user based on the updated one or more parameters of the intervention library; A method comprising:
17. 17. The method of claim 16, wherein the first intervention information associated with the first user includes at least one of an intervention type, an indication of effectiveness of the first intervention, an intervention context associated with the first intervention, and one or more characteristics associated with the first user.
18. determining the third intervention for the third user based on the updated one or more parameters of the intervention library; obtaining, by the server, third intervention information associated with the third user; determining a correlation between the third intervention information associated with the third user and fourth intervention information associated with the intervention library; determining the third intervention based on the correlation between the third intervention information and the fourth intervention information exceeding a correlation threshold; transmitting the third intervention to a device associated with the third user; 17. The method of claim 16, comprising:
19. The method of claim 16 , wherein the first intervention information includes contextual information associated with the first intervention.
20. 20. The method of claim 19, wherein the contextual information associated with the first intervention includes at least one of a time of day, one or more actions by the first user prior to the first intervention, the first intervention information associated with the first user, a location of the first user, and a proximity of the first user to one or more individuals.
21. The method of claim 16 , wherein the first intervention information comprises one or more characteristics associated with the first user.
22. 22. The method of claim 21, wherein the one or more characteristics associated with the first user include at least one of gender, age, family status, target behavior, country, culture, location, and personality type.
23. acquiring, by the server, fifth behavior information associated with a fifth user and a fifth behavior; updating one or more parameters of a behavioral library based at least in part on sixth behavioral information associated with a sixth user based on the fifth behavioral information; determining one or more behavioral parameters for a seventh user based on the updated one or more parameters of the intervention library; The method of claim 16 further comprising:
24. 24. The method of claim 23, wherein the one or more behavioral parameters for the seventh user include at least one of a behavioral trigger, a pre-action, and a behavioral artifact associated with the fifth behavior.
25. 24. The method of claim 23, wherein the one or more behavioral parameters for the seventh user include one or more weightings associated with determining the likelihood that the seventh user will perform or not perform the fifth behavior.
26. The method of claim 23 , wherein the fifth behavioral information includes one or more characteristics associated with the fifth user.
27. The method of claim 23 , wherein the fifth behavioral information includes contextual information associated with the fifth behavior.
28. 1. A system for generating one or more interventions, comprising: At least one memory; at least one processor coupled to the at least one memory; wherein the at least one processor: Obtaining first intervention information associated with the first user and the first intervention; updating one or more parameters of an intervention library based at least in part on second intervention information associated with a second user and a second intervention based on the first intervention information; determining a third intervention for a third user based on the updated one or more parameters of the intervention library; The system is configured as follows:
29. 30. The system of claim 28, wherein the first intervention information associated with the first user includes at least one of an intervention type, an indication of effectiveness of the first intervention, an intervention context associated with the first intervention, and one or more characteristics associated with the first user.
30. determining the third intervention for the third user based on the updated one or more parameters of the intervention library; obtaining third intervention information associated with the third user; determining a correlation between the third intervention information associated with the third user and fourth intervention information associated with the intervention library; determining the third intervention based on the correlation between the third intervention information and the fourth intervention information exceeding a correlation threshold; transmitting the third intervention to a device associated with the third user; 30. The system of claim 28 configured to:
31. 30. The system of claim 28, wherein the first intervention information includes contextual information associated with the first intervention.
32. 32. The system of claim 31 , wherein the contextual information associated with the first intervention includes at least one of a time of day, one or more actions by the first user prior to the first intervention, the first intervention information associated with the first user, a location of the first user, and a proximity of the first user to one or more individuals.
33. 30. The system of claim 28, wherein the first intervention information comprises one or more characteristics associated with the first user.
34. 34. The system of claim 33, wherein the one or more characteristics associated with the first user include at least one of gender, age, family status, target behavior, country, culture, location, and personality type.
35. The at least one processor: Obtaining fifth behavioral information associated with a fifth user and a fifth behavior; updating one or more parameters of a behavioral library based at least in part on sixth behavioral information associated with a sixth user based on the fifth behavioral information; determining one or more behavioral parameters for a seventh user based on the updated one or more parameters of the intervention library; 30. The system of claim 28 configured to:
36. 36. The system of claim 35, wherein the one or more behavioral parameters for the seventh user include at least one of a behavioral trigger, a pre-action, and a behavioral artifact associated with the fifth behavior.
37. 36. The system of claim 35, wherein the one or more behavioral parameters for the seventh user include one or more weightings associated with determining the likelihood that the sixth user will perform or not perform the fifth behavior.
38. 36. The system of claim 35, wherein the fifth behavioral information comprises one or more characteristics associated with the fifth user.
39. The system of claim 35 , wherein the fifth behavioral information includes contextual information associated with the fifth behavior.
40. 30. The system of claim 28, wherein the system includes at least one server.
Citation Information
Patent Citations
Behavior modification system, behavior modification device, behavior modification method and behavior modification program
JP2020140360A
Granular binarization for extended reality
JP2021012686A
