Metric-driven improvements of virtual reality experiences

Abstract

To provide systems and methods for obtaining metrics relating to an extended reality experience and using the obtained metrics to perform remedial actions, such as managing user motion sickness, determining user performance relating to a designed game difficulty, and performing home automation.SOLUTION: Data from a plurality of users as the users navigate between determined checkpoints is obtained and used to determine a metric, such as a median, an average, or other representative data. The current user's navigation through the same checkpoints is monitored and compared with the metric. The results from the comparison are used to enhance extended reality experience, which includes customizing the experience for the motion sickness, game difficulty level and home automation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] Embodiments of the present disclosure relate to improving virtual reality gaming experiences, such as modifying game difficulty, managing motion sickness, and implementing home automation, based on data related to a user's gameplay. [Background technology]

[0002] Virtual reality (VR) experiences are used in several fields such as gaming, education, and manufacturing, to name a few. The experiences range from less immersive to the most immersive, with the goal of giving the user a sense of a real-world environment. Virtual reality is also an experimental medium in its early stages compared to many other technologies, and has plenty of room to grow and address current demands. For example, storytelling using this medium is still in its infancy. Antoine Ribordy gives a great explanation of how VR as a medium differs from regular storytelling. Ribordy states that "In VR storytelling, the environment plays a central role. Special attention must be paid to it. In fact, the whole experience is an act of balancing the story and the environment, providing the player with the answers he is looking for."

[0003] Players or observers in a VR experience (whether together or separately in time) see different assets based on their perspective. To build viewer engagement, content creators must observe patterns of viewer behavior and determine whether their intended design matches the viewer's observations and actions. A well-designed experience is one in which viewers spend time, feel engaged, and ultimately enjoy the experience, perhaps revisiting it frequently.

[0004] To understand the impact of different decisions in VR and to determine how their designed experiences are perceived by users, content developers may use their own judgment, conduct focus groups, or gather feedback from users through informal means such as via chat groups, Discord, user surveys, or directly in gaming applications. These methods of obtaining user feedback have several problems.

[0005] One such issue is delayed feedback from users, which may result from the use of older methods such as email, chat, or surveys to determine user experience. Many users may not provide feedback at all because such methods of collecting user feedback are not convenient for the user and require the user to actively spend time on other platforms outside the game to send feedback to the game developer.

[0006] Another such problem with these feedback methods is that the feedback may be inaccurate: when a user is not immersed in the VR experience, the user may forget bits and pieces of what happened during the VR game or may only remember certain instances, making the feedback much more difficult to match to exact timestamps in the virtual game.

[0007] On the other hand, if the feedback method queries the user during a VR game, such queries may be annoying to the user because the user is more interested in immersing themselves in the game and such queries may have the same effect as a pop-up commercial during the game, which is usually ignored. Even if the user provides feedback to the system when queried during a game, the user's feedback may be limited because the user would rather rush back to the game instead of responding to the query.

[0008] Some VR experiences display certain assets and require the user to interact with the assets in order to score in a game. In such experiences, VR experience creators may want to know how the user interacted with the designed assets. However, as mentioned above, currently used methods for obtaining such feedback are ineffective and result in inaccurate, incomplete, or delayed feedback. Thus, VR experience developers cannot know exactly why a user did not interact with a designed asset in a VR experience or why the user interacted with the asset in the way that the user did.

[0009] Finally, motion sickness is a big concern when users are playing virtual games, especially in games where there is a lot of movement in the virtual world or games that require movement in the real world from the user wearing the headset. Due to the lack of feedback means, it is a challenge to gather accurate, complete, and granular data. Therefore, game developers are not able to fully address the problem of motion sickness or determine why and why a user may get motion sickness at a specific timestamp in the game.

[0010] Thus, there is a need for better systems and methods for obtaining user feedback related to a user's virtual experience within a virtual environment, such that the data can be used to enhance the user's experience. The present invention provides, for example, the following items. (Item 1) 1. A method for enhancing a virtual reality experience, comprising: generating a virtual game on a display of an electronic device, wherein a predetermined difficulty level is determined for a portion of the virtual game; collecting performance data from a plurality of users who played a portion of the virtual game, the performance data relating to the performance of the plurality of users based on the predetermined difficulty level; calculating a performance metric based on the performance data collected from the plurality of users; comparing current user performance data with said calculated performance metrics; executing an augmentation of the virtual game in response to determining that the current user performance data is not within a threshold of the performance metric; A method comprising: (Item 2) The method according to any preceding claim, wherein the performance metric thresholds include a lower limit and an upper limit. (Item 3) 2. The method of claim 1, wherein the lower limit is associated with a minimum number of interactions with a virtual object in the virtual game and the upper limit is associated with a maximum number of interactions with a virtual object in the virtual game. (Item 4) determining that the current user performance data is below a lower threshold of the performance metric; modifying the predetermined difficulty level for a portion of the virtual game to a lower predetermined difficulty level in response to the determination; The method according to any one of the preceding items, further comprising: (Item 5) determining that the current user performance data exceeds an upper limit of a threshold of the performance metric; modifying the predetermined difficulty level for a portion of the virtual game to a higher predetermined difficulty level in response to the determination; The method according to any one of the preceding items, further comprising: (Item 6) 2. The method of claim 1, wherein the enhancement of the virtual game is either an increase or a decrease in the predetermined difficulty level. (Item 7) 2. The method of claim 1, wherein the enhancement of the virtual game is a modification of a path that may be taken by the current user within the virtual game. (Item 8) 5. The method of claim 1, wherein modifying the path is associated with not allowing navigation by the current user on a path within the virtual game that was previously allowed for navigation. (Item 9) 2. The method of claim 1, wherein the portion of the virtual game includes a start checkpoint and an end checkpoint, the start checkpoint and the end checkpoint being determined based on a type of the virtual game. (Item 10) The method of any one of the preceding claims, further comprising selecting the start and end checkpoints at a shorter time span for a faster paced virtual game and at a relatively longer time span for a slower paced virtual game. (Item 11) monitoring a path of the current user within the virtual game; determining a total number of available interaction opportunities based on the path taken by the current user within the virtual game; determining a number of interaction opportunities utilized by the current user from the total number of available interaction opportunities; The method according to any one of the preceding items, further comprising: (Item 12) 5. The method of claim 1, wherein the performance of the current user is based on the number of interaction opportunities executed by the current user from the total number of interaction opportunities available to the current user. (Item 13) monitoring the current user's gaze as the user navigates a path within the virtual game; determining objects within a field of view (FOV) of the current user based on the path navigated by the current user within the virtual game; determining whether the user has interacted with an object within the FOV; The method according to any one of the preceding items, further comprising: (Item 14) The method of any one of the preceding items, further comprising prompting the user to interact with the object within the FOV in response to determining that the user has not interacted with the object within the FOV. (Item 15) 2. The method of any one of the preceding claims, wherein the prompt is either a visual alert displayed on a display of the electronic device or an audible prompt. (Item 16) 2. The method of claim 1, further comprising: highlighting the object within the FOV in response to determining that the user has not interacted with the object within the FOV. (Item 17) The method of any one of the preceding items, wherein the performance metric is selected from the group consisting of median, mean, standard deviation, or variance. (Item 18) 1. A method comprising: generating a virtual game on a display of an electronic device; determining a level of difficulty between a start checkpoint and an end checkpoint for a portion of said virtual game; monitoring a user's progress within the virtual game to determine whether the user a) meets the determined level of difficulty, b) is unable to meet the level of difficulty due to not being able to do so, or c) is on track to exceed the level of difficulty; In response to said determination, maintaining, decreasing, or increasing said level of difficulty. A method comprising: (Item 19) Monitoring the progress of said user determining coordinates of a path taken by the user within the virtual game; determining an orientation of a headset worn by the current user for playing the virtual game based on the coordinates; determining an object within a field of view of the headset based on the determined orientation; and determining whether the current user has interacted with the determined object; The method according to any one of the above items, comprising: (Item 20) 1. A system for augmenting a virtual reality experience, comprising: a communications network configured to access the electronic device; 1. A control circuit network comprising: generating a virtual game on a display of the electronic device, wherein a predetermined difficulty level is determined for a portion of the virtual game; collecting performance data from a plurality of users who played a portion of the virtual game, the performance data relating to the performance of the plurality of users based on the predetermined difficulty level; calculating a performance metric based on the performance data collected from the plurality of users; comparing current user performance data with said calculated performance metrics; executing an augmentation of the virtual game in response to determining that the current user performance data is not within a threshold of the performance metric; and a control circuit configured to: A system comprising: (Item 21) The system of any one of the preceding claims, wherein the performance metric thresholds include a lower limit and an upper limit. (Item 22) The system of any one of the preceding claims, wherein the lower limit is associated with a minimum number of interactions with virtual objects in the virtual game, and the upper limit is associated with a maximum number of interactions with virtual objects in the virtual game. (Item 23) determining that the current user performance data is below a lower threshold of the performance metric; modifying the predetermined difficulty level for a portion of the virtual game to a lower predetermined difficulty level in response to the determination; the control circuitry configured to 2. The system of claim 1, further comprising: (Item 24) determining that the current user performance data exceeds an upper limit of a threshold of the performance metric; modifying the predetermined difficulty level for a portion of the virtual game to a higher predetermined difficulty level in response to the determination; the control circuitry configured to 2. The system of claim 1, further comprising: (Item 25) 4. The system of claim 1, wherein the enhancement of the virtual game is implemented by the control circuitry configured to either increase or decrease the predetermined difficulty level. (Item 26) 2. The system of claim 1, wherein the enhancement of the virtual game is implemented by the control circuitry configured to modify paths that may be taken by the current user within the virtual game. (Item 27) The system of any one of the preceding claims, wherein the modification of the route is associated with not allowing navigation by the current user on a route within the virtual game that was previously allowed for navigation. (Item 28) The system of any one of the preceding claims, wherein a portion of the virtual game includes a start checkpoint and an end checkpoint, the start checkpoint and the end checkpoint being determined by the control circuitry based on the type of the virtual game. (Item 29) The system of any one of the preceding claims, further comprising the control circuitry configured to select the start checkpoint and the end checkpoint at a shorter time span for a faster-paced virtual game and at a relatively longer time span for a slower-paced virtual game. (Item 30) monitoring a path of the current user within the virtual game; determining a total number of available interaction opportunities based on the path taken by the current user within the virtual game; determining a number of interaction opportunities utilized by the current user from the total number of available interaction opportunities; 2. The system of claim 1, further comprising control circuitry configured to: (Item 31) The system of any one of the preceding claims, wherein the performance of the current user is based on the number of interaction opportunities executed by the current user from the total number of interaction opportunities available to the current user. (Item 32) monitoring the current user's gaze as the user navigates a path within the virtual game; determining objects within a field of view (FOV) of the current user based on the path navigated by the current user within the virtual game; determining whether the user has interacted with an object within the FOV; the control circuitry configured to 2. The system of claim 1, further comprising: (Item 33) The system of any one of the preceding items, wherein in response to determining that the user has not interacted with the object within the FOV, the control circuitry is configured to prompt the user to interact with the object within the FOV. (Item 34) The system of any one of the preceding claims, wherein the prompt is either a visual alert or an audible prompt displayed on a display of the electronic device. (Item 35) The system of any one of the preceding items, wherein in response to determining that the user has not interacted with the object within the FOV, the control circuitry is configured to highlight the object within the FOV. (Item 36) 2. The system of claim 1, wherein the electronic device is selected from the group consisting of a virtual reality, an augmented reality, or a mixed reality device. (Summary) A system and method are disclosed for obtaining metrics associated with an extended reality experience and using the obtained metrics to perform remedial actions such as managing a user's motion sickness, determining user performance associated with a designed game difficulty, and performing home automation. The method includes determining start and end checkpoints in the extended reality experience. Data from multiple users as they navigate between the determined checkpoints is obtained and used to determine metrics such as median, representative, or other representative data. A current user's navigation through the same checkpoints is monitored and compared to the metrics. Results from the comparison are used to enhance the extended reality experience, including customizing the experience with respect to motion sickness, game difficulty level, and home automation. [Brief description of the drawings]

[0011] Various objects and advantages of the present disclosure will become apparent upon consideration of the following detailed description, when taken in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout.

[0012] [Figure 1]FIG. 1 is a block diagram of a process for improving a virtual reality gaming experience by collecting and using game data to take actions or provide recommendations, according to some embodiments of the present disclosure.

[0013] [Diagram 2] FIG. 2 is a block diagram depicting selection of checkpoints based on the type of virtual game, according to some embodiments of the present disclosure.

[0014] [Diagram 3] FIG. 3 is a block diagram of data collected from multiple users as they navigate through checkpoints in a virtual game, according to some embodiments of the present disclosure.

[0015] [Figure 4] FIG. 4 is a block diagram of an example system for enhancing a virtual reality gaming experience according to some embodiments of the disclosure.

[0016] [Diagram 5] FIG. 5 is a block diagram of an electronic device for use in a gaming environment according to some embodiments of the disclosure.

[0017] [Figure 6] FIG. 6 is a block diagram of an extended reality headset for use in a gaming environment according to some embodiments of the disclosure.

[0018] [Figure 7] FIG. 7 is a flowchart of a process for collecting data related to motion sickness in a virtual gaming environment according to some embodiments of the disclosure.

[0019] [Figure 8]FIG. 8 is an example of a chart depicting various user background factors that may be taken into account to determine motion sickness associated with playing a virtual game, according to some embodiments of the present disclosure.

[0020] [Figure 9] FIG. 9 is a flowchart of a process for providing recommendations or activating home automation to manage motion sickness caused by a virtual gaming environment, according to some embodiments of the disclosure.

[0021] [Figure 10] FIG. 10 is a block diagram of an example of comparing current user data with median data to manage motion sickness according to some embodiments of the present disclosure.

[0022] [Figure 11] FIG. 11 is a block diagram of categories of recommendations that may be made to manage motion sickness in a virtual gaming environment according to some embodiments of the disclosure.

[0023] [Figure 12] FIG. 12 is a block diagram of categories of devices and machines that may be activated via an API to manage motion sickness in a virtual gaming environment according to some embodiments of the disclosure.

[0024] [Figure 13A] FIG. 13A is a flowchart of a process for collecting data related to a user's handling of a difficulty level in a virtual game, and using the data to enhance or manage the difficulty level, according to some embodiments of the present disclosure.

[0025] [Figure 13B] FIG. 13B is a flowchart of a process for comparing current user data to median user data to manage a level of difficulty in a virtual game according to some embodiments of the present disclosure.

[0026] [Figure 14] FIG. 14 is a navigation tree depicting routes taken by multiple users between given checkpoints in a virtual game, according to some embodiments of the present disclosure.

[0027] [Figure 15] FIG. 15 is a block diagram depicting a user's field of view and objects / assets encountered within the field of view, according to some embodiments of the present disclosure.

[0028] [Figure 16] FIG. 16 is a block diagram of a room layout and multiple paths taken to reach a destination, according to some embodiments of the present disclosure.

[0029] [Figure 17] FIG. 17 is a block diagram of a layout recommendation according to some embodiments of the present disclosure.

[0030] [Figure 18] FIG. 18 is a table of user navigation coordinates of user interactions as the user navigates from checkpoint C1 to checkpoint C2, according to some embodiments of the present disclosure.

[0031] [Figure 19] FIG. 19 is a table of user interactions and missed opportunities according to some embodiments of the present disclosure.

[0032] [Figure 20] FIG. 20 is a table of content refinement suggestions based on a user's navigation in an extended reality experience, according to some embodiments of the disclosure.

[0033] [Figure 21]FIG. 21 is a block diagram depicting locally rendered and cloud rendered extended reality experiences according to some embodiments of the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Detailed Description According to some embodiments disclosed herein, some of the limitations mentioned above are overcome by obtaining user metrics from a plurality of users during an extended reality experience such as a virtual game, generating median data, comparing the median data with the current data of a current user, and determining whether the current user is experiencing or will experience a similar outcome in the extended reality experience. Some of the limitations mentioned above are also overcome by determining start and end checkpoints for monitoring the current user, and providing remedial actions in response to user navigation data collected during the start and end checkpoints. For example, such remedial actions include managing motion sickness, managing game difficulty levels and adjusting the difficulty as necessary, blocking paths in the virtual game that are deemed difficult, and implementing home automation to manage the user's motion sickness, all based on the path taken by the user between the start and end checkpoints.

[0035] In one embodiment, an extended reality experience is rendered to multiple users. The extended reality experience may be a virtual reality game linked to real-life education, an augmented reality skill learning tool, or any other extended reality experience. A virtual reality game is used to illustrate the processes and embodiments herein, however, the extended reality experience may be any other extended reality experience as well.

[0036] In some embodiments, multiple checkpoints C1 and C2 are determined. The checkpoints are determined based on the type of extended reality experience used. In one embodiment, in an experience involving a slow-paced game for children, data collection every millisecond is not required, and therefore checkpoints C1 and C2 that are far apart in time may be selected. In other embodiments, in a fast-paced game such as a battle between multiple players or a car racing game, where each turn or every millisecond can affect the outcome of the game, then checkpoints C1 and C2 that are closer in time may be selected to collect more data than a slower moving game.

[0037] In one embodiment, once checkpoints are selected, data for each of a plurality of users is monitored and logged as the user navigates from checkpoint 1 to checkpoint 2. Such data may include the route taken by each of the plurality of users, the amount of time spent on the route, the challenges and obstacles overcome by the user along the route, the user's headset field of view as the user navigates from checkpoint 1 to checkpoint 2, the assets the user interacted with along the route and the time they interacted, CPU / GPU usage along the route, and the user's motion sickness data along the route.

[0038] Once data from multiple users is collected and logged, a median is calculated. Although median data is described, the embodiment is not so limited and a representative value, an average, a standard deviation, a variance, or some other calculation based on an equation, or another statistical measure may also be used. The calculated data may also be referred to herein as a performance metric. The median is then used to compare the current data of a current user currently navigating the same route in the extended reality experience. The comparison allows the system to make enhancements and modifications to the extended reality experience. This also allows the system to predict motion sickness or manage it through home automation techniques once it occurs. The comparison also allows the system to make adjustments to the game difficulty level, such as increasing or decreasing the difficulty, for a better user experience.

[0039] FIG. 1 is a block diagram of a process for improving a VR gaming experience by collecting and using game data to take actions or provide recommendations, according to some embodiments of the present disclosure.

[0040] In some embodiments, a gaming environment is displayed on a display device at block 101. The gaming environment may be an extended reality environment, including virtual reality, augmented reality, mixed reality, or any other virtual type of reality, including within the metaverse.

[0041] In some embodiments, the systems described herein, such as those of Figures 4 and 5, provide extended reality environments that include virtual simulations of both real-world experiences and other experiences that are fictional. For example, the system may simulate a virtual layout of a city, such as San Francisco or Paris, and allow a user to perform activities within the virtually simulated environment, such as walking through the streets of San Francisco or playing a game of car racing through the streets of Paris. In other examples, fictional virtual environments, such as outer space or some other imaginary environment, may be created.

[0042] In some embodiments, these systems utilize an extended reality device, such as a VR headset, VR glasses, or a mobile phone that can act as a VR device. The extended reality device may include a display screen for displaying the simulated virtual environment. The extended reality device may be worn on the user's head such that the display of the extended reality device is in front of the user's eyes, allowing the user to view an extended reality 3D simulated environment depicted on the display of the extended reality device. In some embodiments, when reference is made to navigation along a path, the reference is associated with a virtual reality device, such as a virtual reality headset or glasses. In other embodiments, when reference is made to an augmented reality embodiment in which a real world view is used, the reference is associated with an augmented reality device, such as an augmented reality headset or glasses, through which the real world environment and a virtual overlay on the real world environment can be visualized via the headset or glasses.

[0043] In some embodiments, the extended reality device may also include any one or more of a camera facing inwards to track the user's gaze, a speaker for sound effects, and a motion generating component such as a vibration module to give the user the sensation of feeling effects displayed in the virtual world, such as an earthquake. It may also include an accelerometer, a gyroscope, and a proximity sensor. It may include a processor and memory, such as a system on a chip (SoC).

[0044] The extended reality device may be capable of connecting to the Internet and downloading or accessing various applications that provide the user with an extended reality experience such as games, factory settings, medical procedures, training to drive a vehicle, exercise, etc. In some embodiments, the extended reality device may include more than one connectivity option for communicating with other devices, body sensors, or electronic devices for downloading applications, and such connectivity options may include connectivity via an API, connectivity using a SoC featuring Wi-Fi, Bluetooth, and / or other radio frequency connectivity in addition to available USB connectivity (e.g., USB Type-C).

[0045] As mentioned above, extended reality devices may connect to the Internet to download gaming or other types of applications. These applications may include assets within virtual worlds that can move in three-dimensional (3D) space, e.g., in three translational x, y, z axes and three rotational axes, commonly referred to as six degrees of freedom (6DOF).

[0046] In some embodiments, using an extended reality device, the system creates an image for each eye of the user that allows the user to play a virtual game. In some cases, during the virtual game, an avatar of a portion of the user's body may be used, the user's entire body represented as an avatar, or no avatar may be used at all. When an avatar is used, the system creates the illusion that the user is truly within the virtual environment being displayed.

[0047] In some embodiments, an extended reality device (such as an augmented reality or virtual reality headset, or a headset with dual functionality, or virtual glasses) uses head tracking technology to track the movement of a user's head while the user is wearing the device on their head. Such tracking captures the user's head movement as a means of steering the camera and viewing things in the virtual world. For example, if the user orients their head to the left, the object or asset that should be on the left side will be visible to the user. Thus, the image changes with respect to the user according to how the user orients their head, i.e., the extended reality headset.

[0048] In some embodiments, some extended reality headsets, attachments, and accessories attached directly to the headset or associated and paired with the headset additionally include gaming controllers that can act as hands for purposes of interaction within a virtual game. Using such controllers, a user may be able to perform functions within the virtual environment. For example, a user may be able to drive a car, hit another avatar, displace objects, perform a golf swing, etc.

[0049] In addition to the controller, in some embodiments, the system may also include a wearable device that can be worn on the user's body to provide a full-body VR experience. For example, embodiments may include combining an extended reality headset with sensors placed on the user's body to simulate and mimic all body movements performed in the real world as movements on an avatar of the body used in the virtual world.

[0050] In some embodiments, the extended reality environment provided may be stationary, while in other embodiments, the extended reality environment provided may be dynamically changing or may require the user to navigate from one location to another, for example, the user may be required to navigate from one virtual room to another within the extended reality environment, or to drive from one location to another within the extended reality environment.

[0051] In stationary extended reality environments, in some embodiments, a user may be able to look around using their extended reality headset without the need to move within the virtual world. For example, a stationary experience may include sitting in a virtual vehicle or playing a chess game, which may involve only limited movement of a sit-and-watch experience.

[0052] In some embodiments, other experiences may require the user to perform several movements or transition from one location to another. For example, the user may need to look around 360° to avoid being hit in a virtual environment where objects are thrown at the user from all directions. In such situations, head tracking is important and needs to be performed in real time to reduce any lag between the user movement in the real world and the image that should be relayed to the user in the virtual world based on the real world movement. For example, if the user quickly turns 180° to fight a virtual animal jumping on him from behind, the image in the virtual world should show the animal in a jumping motion on the user quickly and with little lag to provide a real world-like effect. Thus, such dynamically moving experiences are more immersive and require tracking of headset movement with greater accuracy.

[0053] Following user movements and matching them with virtual images with little lag is also important in reducing motion sickness among users because, among other factors, the brain senses movement by getting signals from the inner ear, eyes, muscles, and joints, and when movement is not accompanied by a visual image that matches the movement, this can induce motion sickness.

[0054] Referring again to block 101, in some embodiments, an extended reality environment of a room may be displayed. As depicted in block 101, an exemplary virtual room includes multiple bookshelves, chairs, a table, a lamp, and a couch to simulate a real room setting. The VR game may provide the user with challenges to score or to advance to the next level of the game, which may include rooms with higher level challenges. The challenges may require the user to find hidden objects or rearrange the layout of a room, or this may be a game in which the user chases avatars and catches them as they run from room to room.

[0055] At block 102, in some embodiments, a portion of the virtual game is selected. The portion selected may include a beginning checkpoint, also referred to as checkpoint 1 or "C1," and an ending checkpoint, also referred to as checkpoint 2 or "C2." A checkpoint, in some embodiments, is an experience or portion of the game state that is saved for rollback or restart. Typically, a checkpoint is before anything new can be developed in the experience or game. For example, in a game with multiple levels, there is typically a checkpoint after each level is surpassed. If a character dies in the game, it will be revived from a previous checkpoint.

[0056] Data between checkpoints may be collected by the system of FIG. 1 and used to provide an augmented extended reality experience. For example, because there may be multiple tasks and challenges between checkpoints, data related to the time it takes to complete a set of tasks or a sequence of tasks or challenges may represent the total time spent between checkpoints C1 and C2. When the user reaches checkpoint C2, a time-stamped entry is registered to indicate completion. If the user exits the experience before reaching checkpoint C2, the absence of this entry in the log file indicates that the user did not complete the task to reach checkpoint C2. Aggregation of such data between checkpoints helps users and programmers evaluate and tune the virtual game for an augmented experience.

[0057] In some embodiments, the checkpoints are selected based on the type of extended reality game and its environment. As depicted in FIG. 2, in block 210, in one embodiment, the extended reality game may be a slow game for children. The game may not have fast moving components or rapidly changing graphics. It may not have many assets. It may not allow for many degrees of rotation because it does not have images to show for every orientation of the user's headset. For example, the extended reality game may be a simple game like tic-tac-toe or an educational game that requires a child to identify shapes, etc.

[0058] In this embodiment, it may not be necessary to collect data at shorter intervals, since in a slower moving game it may be possible to skip frames and collect data at longer intervals without missing important components of the game. Thus, the start checkpoint C1 and the end checkpoint C2 may be selected 300 seconds apart, and data may be captured at 15 or 30 second intervals within the 300 second total time.

[0059] As depicted in FIG. 2, in block 220, in one embodiment, the extended reality game may be a medium paced virtual game. The game may have moderately fast moving components that require rendering at one or two second intervals. It may have more assets than a slower moving game. It may also allow more degrees of rotation and at each rotation have figures in the virtual world to match the headset movement. For example, the extended reality game may be a game such as playing tennis, chess, or walking through a maze. A game such as tennis may depict the same tennis court and the tracking of a tennis ball as it is hit from side to side, so the pace of rendering figures in the virtual world may be medium and track a tennis ball at some interval, such as every one to two seconds.

[0060] In this embodiment, collecting data at shorter intervals may be required to track where the tennis ball is located after each shot, so the start checkpoint C1 and end checkpoint C2 may be selected 90 seconds apart and data may be captured at intervals of every 1-2 seconds within the 90 second total time.

[0061] As depicted in FIG. 2, in blocks 230 and 240, in one embodiment, the extended reality game may be a fast-paced game or a game that is at a level that requires faster response. The game may have fast moving objects or rapidly changing graphics. It may have several objects / assets, such as a car racing game that depicts several other cars moving at different speeds in the field of view and several obstacles on the path. It may also allow all degrees of rotation and have rapidly changing images to show for each orientation of the user's headset. For example, the extended reality game may be a fast car race, a game where fighter jets chase each other through mountains, a fast roller coaster, or a shooter game where several enemies attack at the same time.

[0062] In this embodiment, it may be necessary to collect data at much shorter intervals, because in a fast moving game, if proper maneuvers are not performed, it may not be possible to skip frames without missing important parts of the game, such as a fighter jet crashing into a mountain, or a user being hit by a bullet in a game where the user is being attacked from all angles. Thus, the start checkpoint C1 and the end checkpoint C2 may be selected 30 seconds apart, and data may be captured every ½ or ⅕ of a second.

[0063] Although certain time intervals are described above in connection with blocks 210-240, embodiments are not so limited and other beginning and ending checkpoints and data collection intervals within the checkpoints are also contemplated.

[0064] Referring again to block 102, once the start and end checkpoints have been identified, the system may collect data from multiple users who played the game and navigated from the start checkpoint C1 to the end checkpoint C2. One example of such data in a graph format is depicted in FIG. 3. Although a graph chart is used as an example, the data may be in any other format, such as a table. As depicted in FIG. 3, two users, namely user 1 and user 2, navigated from the start checkpoint C1 to the end checkpoint C2. Their data was captured in all 6 DOF (only 3 DOF is shown). The data is captured from time T=t0 to time T=t1. As this can be seen, the path navigated by user 1 is different from the path navigated by user 2. The paths and other data from user 1 and user 2 can be used to calculate a median path, such as the median path depicted in FIG. 14.

[0065] The data collected relates to different experiences encountered by different users from a plurality of users, for example, different users spend different amounts of time between two checkpoints, encounter different challenges, for example, one user may choose an easier path than another user, one user may lose a life while another may get through without losing a life.

[0066] Different users may also have different experiences in discovering assets in the game. Some users may discover assets that are key to completing a level faster or with fewer attempts than others. Some users may choose to stay longer to enjoy the experience more and explore their surroundings. All such user experiences and their data may be captured while the users navigate from the start checkpoint C1 to the end checkpoint C2. As depicted in block 102, three users, namely, User 1, User 2, and User 3, were monitored by the system as the users navigated through checkpoints C1 to C2.

[0067] User engagement and interaction in an experience depends on many factors. Aggregate user behavior measured as the time spent between two checkpoints and the percentage of sessions where the user exits without reaching the next checkpoint can be considered as high-level indicators. The method and system for logging and analyzing user behavior using the described metrics shall offer content creators deep insights on how to improve engagement. Examples of such data related to behavior are further described in block 103.

[0068] Block 103 depicts some examples of the types of data that may be collected while multiple users navigate through checkpoints C1 and C2. As explained above, the intervals at which data is collected may vary based on the type of application, e.g., a slow, medium, or fast paced virtual game.

[0069] In one embodiment, data related to path coordinates may be collected as multiple users navigate through checkpoints C1 and C2. The path coordinates may be x, y, z coordinates and pitch, roll, and yaw orientations as depicted in FIG. 6. In this embodiment, the system may collect, at predetermined intervals, each movement along a path selected by a user, including all coordinates relative to a frame of reference. For example, the frame of reference may be a location within the virtual world, such as the origin of an object within the virtual world, or it may be the origin of the headset prior to the start of the game.

[0070] In another embodiment, data related to path timing may be collected as multiple users navigate through checkpoints C1 and C2. The path timing may indicate the amount of time it took the user to travel from checkpoint C1 to checkpoint C2 along a path selected by the user. In this embodiment, the system may collect data related to the amount of time it took the user to pass through certain locations along the selected path at a predetermined interval. For example, if the path is to travel from the couch to a further ramp in block 101, the system may determine the total amount of time taken from the couch to the further ramp and the amount of time for intermediate points such as from the couch to the chair, then from the chair to the further ramp, if that is the path selected by the user.

[0071] In another embodiment, data related to assets encountered may be collected as multiple users navigate through checkpoints C1 and C2. The assets encountered along the path may be determined by determining the path selected by the user and what is in the user's field of view as the user navigates along the path. In this embodiment, the system may collect data related to the user's field of view as the user navigates along the path at predetermined intervals. For example, if the selected path is to walk down the left side of the room to the nearest lamp and the user is facing straight toward the lamp, the system may determine that the only asset in the field of view encountered is the nearest lamp because the user did not swivel his or her headset to the right to view the chair and table.

[0072] In another embodiment, data related to interacted assets may be collected as multiple users navigate through checkpoints C1 and C2. The assets interacted along the path may be determined by monitoring the user's actions in the virtual game as the user navigates along his or her path. For example, a user may encounter certain assets in his or her view, but the user may not have interacted with those assets and may have simply passed them by. An example of an interaction may be moving a chair, sitting on a chair, tripping over a chair, or placing something on a chair. The system may collect data related to assets in the field of view at predetermined intervals and determine whether the user has interacted with them, which, if applicable, may record the type of interaction.

[0073] In another embodiment, data related to CPU and GPU usage may be collected as multiple users navigate through checkpoints C1 and C2. As user headset movement requires rendering and re-rendering of images in the virtual world, the amount of rendering required based on the amount of movement, the pace of movement, and the amount of imagery and detail depicted uses up CPU and GPU memory. The system may collect data related to the amount of CPU and GPU used based on the path taken by the user and the amount and quality of rendering and re-rendering performed at a given interval. Such data may be used to manage CPU and GPU usage and also to forecast usage and perform resource management for future users.

[0074] In yet another embodiment, data related to levels of difficulty may be collected as multiple users navigate through checkpoints C1 and C2. The levels of difficulty experienced in going from C1 to C2 and all intermediate points may be collected at predetermined intervals. For example, a programmer may have programmed a level of difficulty at each intermediate point between C1 and C2, preventing users from easily completing the game level. The system monitors the user's path and navigation and determines whether the users experienced the same level of difficulty or were within a programmed difficulty threshold while navigating from C1 to C2 and the intermediate points. For example, some users may encounter a higher level of difficulty, while other users may be able to complete a level more easily and faster.

[0075] In another embodiment, data related to motion sickness may be collected as multiple users navigate through checkpoints C1 and C2. Because visual effects depicted in the virtual game, such as user headset movement or a virtual roller coaster ride, may cause nausea and motion sickness, the system monitors multiple users as they navigate through checkpoints C1 and C2 and collects data such as heart rate, blood pressure, EKG, and others from sensors and other wearables attached to the users' bodies. The system collects such data when certain fast-paced twists and turns occur in the game or when graphics are rendered at a fast pace. It may also collect such data at predetermined intervals. Such data may be used to manage motion sickness by providing suggestions to the user or programmer regarding improvement steps.

[0076] In block 104, a median path based on paths taken by multiple users may be calculated. Although a median path is described, the embodiment is not so limited and a median, average, standard deviation, variance, or some other calculation based on an equation or another statistical measure for determining a path may also be used. A median of other data besides the path data collected may also be calculated. The median may represent a representative user from the multiple users. As depicted, a graph of the median user is presented in block 104 along with a table of some median data for some of the types of data described in block 103. Although some of the types of data collected are described in blocks 103 and 104, the embodiment is not so limited and other types of data are also envisioned. For example, data such as how a user reacted once the user encountered their first difficulty, or data related to the most frequently visited assets or the amount of time spent interacting with an asset may also be collected.

[0077] In block 105, an example of one type of data collected, including multiple subcategories of data collected, is depicted. In this embodiment, motion sickness data is one of the types of data being collected. When collecting a subcategory of data related to motion sickness data, the system may access multiple devices and sensors attached to the user's body, associated with wearables worn by the user, or devices that monitor the user remotely. For example, devices such as heart rate monitors, blood pressure monitors, blood glucose monitors, ECGs, and EKGs may be accessed by the system to obtain readings of the user's vitals as the user navigates from one point to the next along the path between checkpoints C1 and C2. As described above, this data may be collected for multiple users who played the game and navigated between checkpoints.

[0078] In block 106, the system may obtain current data from the current user as the user begins playing the game and navigates from checkpoint C1 to checkpoint C2. The monitoring may be in real time or may be performed at predetermined intervals at intermediate locations between checkpoints C1 and C2. The current data obtained from the current user may be compared to the median data.

[0079] As depicted, the median data indicates that at time T01, the median user's heart rate was 72 BPM while the current user's heart rate was 68 BPM. The median data further indicates that as the median user navigated from one point to the next along their path in the game, i.e., from time T01 to time T02, their heart rate increased from 72 BPM to 81 BPM. Although median data is mentioned, other types of data based on calculating a mean, average, standard deviation, variance, or some other formula can also be used. The current data captured over the same time interval indicates that the current user's heart rate jumped from 68 BPM to 92 BPM while navigating from T01 to time T02.

[0080] In one embodiment, an increase in heart rate for the current user from 68 BPM to 92 BPM may be outside of a safe threshold range. Such threshold ranges may be predetermined by the system or obtained based on medical advice for a specific user, or generally for any person playing a virtual game.

[0081] Upon determining in block 107 that there is cause for concern about the current user's rapid increase in BPM, the system may provide a recommendation to the user. Some examples of recommendations include opening a window, turning on a fan, or pausing the game for 30 seconds. In another embodiment, instead of providing a recommendation to the user, the system may, for example, automatically turn on a fan in the room where the user is currently playing the game. The system may do so by accessing the fan via an API.

[0082] Although motion sickness was used as an example of the type of data collected and remedial steps subsequently taken to manage motion sickness, the embodiments are not so limited. For example, the data collected may be of any type, including some of the example data types described in block 103, and the collected data may be compared to the current data of the current user. In response to the comparison, the system may determine whether the current data is within a threshold of data collected for the median user, and if not, remedial action may be taken.

[0083] In addition to the above, both translation and rotation data based on the user's movements while wearing the extended reality headset may be collected. In one embodiment, different users may have different rotation vector graphs in the experience. Such vector graphs may be determined using a common tree based on the user's navigation between checkpoints C1 and C2. The difference between two successive recordings may represent a state change in the user's position in the 6DOF experience. Additional details regarding the generated trees and vector graphs are described in connection with FIG. 13.

[0084] Figure 4 is a block diagram of an example system for enhancing a virtual reality gaming experience, according to some embodiments of the present disclosure, and Figure 5 is a block diagram of an electronic device used in a gaming environment, according to some embodiments of the present disclosure. Figures 4 and 5 also illustrate example devices, systems, servers, and related hardware that may be used to implement the processes, features, and functionality described in connection with Figures 1-3 and 6-21. Additionally, Figures 4 and 5 may also be used to display an extended reality experience, set a difficulty level in an extended reality experience, obtain user metrics from multiple users during an extended reality experience, such as a virtual game, generate median data, compare the median data to current data for a current user, determine whether a current user is experiencing or will experience a similar outcome in the extended reality experience, determine start and end checkpoints for monitoring a current user, provide remedial actions based on a user or multiple user performance, manage and adjust, as necessary, the difficulty level based on user interactions, manage motion sickness, perform home automation functions, run artificial intelligence or machine learning algorithms, manage the difficulty level, motion sickness, or home automation.

[0085] In some embodiments, one or more portions or all of system 400 may be configured as a system that implements various features, processes, functionality, and components of Figures 1-3 and 6-21. Although Figure 4 shows a certain number of components, in various examples, system 400 may include fewer than the number of components shown and / or one or more multiples of the number of components shown.

[0086] The system 400 is shown to include a computing device 418, a server 402, and a communications network 414. Although a single instance of the components may be shown and described with respect to FIG. 4, it should be understood that additional instances of the components may also be employed. For example, the server 402 may include or be incorporated within more than one server. Similarly, the communications network 414 may include or be incorporated within more than one communications network. The server 402 is shown communicatively coupled to the computing device 418 through the communications network 414. Although not shown in FIG. 4, the server 402 may be communicatively coupled directly to the computing device 418, for example, in a system in which the communications network 414 is absent or bypasses it.

[0087] The communication network 414 may comprise one or more network systems, such as, but not limited to, the Internet, a LAN, WIFI, or other network systems suitable for audio processing applications. In some embodiments, the system 400 executes a server 402, and functionality that would otherwise be implemented by the server 402 is instead implemented by other components of the system 400, such as one or more components of the communication network 414. In still other embodiments, the server 402 cooperates with one or more components of the communication network 414 to implement certain functionality described herein in a distributed or cooperative manner. Similarly, in some embodiments, the system 400 executes a computing device 418, and functionality that would otherwise be implemented by the computing device 418 is instead implemented by other components of the system 400, such as one or more components of the communication network 414 or the server 402, or a combination. In yet other embodiments, the computing device 418 cooperates with one or more components of the communications network 414 or the server 402 to implement certain functionality described herein in a distributed or cooperative manner.

[0088] Computing device 418 includes control circuitry 428, display 434, and input circuitry 416. Control circuitry 428 in turn includes transceiver circuitry 462, storage 438, and processing circuitry 440. In some embodiments, computing device 418 or control circuitry 428 may be configured as media device 500 of FIG.

[0089] Server 402 includes control circuitry 420 and storage 424. Storage 424 and 438 may each be an electronic storage device. As referred to herein, the phrase "electronic storage device" or "storage device" should be understood to mean any device for storing electronic data, computer software, or firmware, such as a random access memory, a read-only memory, a hard drive, an optical drive, a digital video disc (DVD) recorder, a compact disc (CD) recorder, a BLU-RAY® Disc (BD) recorder, a BLU-RAY® 4D Disc recorder, a digital video recorder (sometimes called a personal video recorder, or DVR, PVR), a solid-state device, a quantum storage device, a gaming console, gaming media, or any other suitable fixed or removable storage device, and / or any combination thereof. Each storage device 424, 438 may be used to store various types of content, metadata, and / or other types of data (e.g., they can be used to store user or group statistics as depicted in FIG. 8, difficulty level values, user and multiple user performance data or user performance metrics, start and end checkpoints, data related to home automation devices and their settings and any user preferences, lists of recommended and improvement actions, and ML and AI algorithms). Non-volatile memory may also be used (e.g., to recall start-up routines and other instructions). Cloud-based storage may be used to supplement or in place of the storage devices 424, 438.In some embodiments, data related to displaying an extended reality experience, setting a difficulty level in an extended reality experience, obtaining user metrics from multiple users during an extended reality experience such as a virtual game, generating median data, comparing the median data to current data of a current user, determining whether a current user is experiencing or will experience a similar outcome in the extended reality experience, determining start and end checkpoints for monitoring the current user, providing remediation actions based on a user or multiple user performance, managing and adjusting, as necessary, the difficulty level based on user interactions, managing motion sickness, performing home automation functions, running artificial intelligence or machine learning algorithms, data related to managing the difficulty level, motion sickness, or home automation, and all other processes and features described herein may be recorded and stored in one or more of storage devices 412, 438.

[0090] In some embodiments, the control circuitry 420 and / or 428 executes instructions for an application stored in memory (e.g., storage 424 and / or storage 438). Specifically, the control circuitry 420 and / or 428 may be instructed by the application to perform the functions discussed herein. In some implementations, any actions performed by the control circuitry 420 and / or 428 may be based on instructions received from the application. For example, the application may be implemented as a set of software or executable instructions stored in storage 424 and / or storage 438 and executed by the control circuitry 420 and / or 428. In some embodiments, the application may be a client / server application where only the client application resides on the computing device 418 and the server application resides on the server 402.

[0091] The application may be implemented using any suitable architecture. For example, it may be a standalone application implemented entirely on computing device 418. In such an approach, instructions for the application are stored locally (e.g., in storage 438) and data for use by the application is periodically downloaded (e.g., from an out-of-band feed, from an Internet resource, or using another suitable approach). Control circuitry 428 may read the instructions for the application from storage 438, process the instructions, and perform the functionality described herein. Based on the processed instructions, control circuitry 428 may determine the type of action to perform in response to input received from input circuitry 416 or from communications network 414. For example, if a predetermined difficulty level is set in a manner that expects a user to take at least five attempts before completing a portion of a game, and the current user easily exceeds expectations by completing it in one attempt, such input is used by control circuitry 428 to perform an improvement action, such as increasing the game difficulty. Other such processes are described at least in Flowcharts 1, 7, 9, and 13A, 13B depicted herein.

[0092] In a client / server based embodiment, the control circuitry 428 may include communication circuitry suitable for communicating with an application server (e.g., server 402) or other networks or servers. Instructions for performing the functionality described herein may be stored on the application server. The communication circuitry may include a cable modem, an Ethernet card, or a wireless modem for communication with other equipment, or any other suitable communication circuitry. Such communication may involve the Internet or any other suitable communication network or path (e.g., communication network 414). In another example of a client / server based application, the control circuitry 428 launches a web browser that interprets web pages provided by a remote server (e.g., server 402). For example, the remote server may store instructions for the application in a storage device. The remote server may process the stored instructions and / or generate a display using the circuitry (e.g., control circuitry 428). The computing device 418 may receive the display generated by the remote server and may display the content of the display locally via the display 434. In this manner, processing of instructions is performed remotely (e.g., by server 402), while the resulting displays, such as display windows described elsewhere herein, are provided locally on computing device 418. Computing device 418 may receive inputs from a user via input circuitry 416 and transmit those inputs to a remote server for processing and generating a corresponding display. Alternatively, computing device 418 may receive inputs from a user via input circuitry 416 and process and display the received inputs locally by control circuitry 428 and display 434, respectively.

[0093] The server 402 and the computing device 418 may transmit and receive content and data, such as objects, frames, snippets of interest, and inputs, to and from primary and secondary devices, such as AR devices. The control circuitry 420, 428 may send and receive commands, requests, and other suitable data over the communications network 414 using the transceiver circuitry 460, 462, respectively. The control circuitry 420, 428 may communicate directly with each other using the transceiver circuitry 460, 462, respectively, avoiding the communications network 414.

[0094] It should be understood that computing device 418 is not limited to the embodiments and methods shown and described herein. In non-limiting examples, computing device 418 may be a virtual, augmented, or mixed reality headset, smart glasses, or a device capable of performing functions within the metaverse, a primary device, a personal computer (PC), a laptop computer, a tablet computer, a WebTV box, a personal computer television (PC / TV), a PC media server, a PC media center, a handheld computer, a mobile phone, a smartphone, or any other device, computing appliance, or wireless device, and / or combination thereof capable of suitably displaying primary and secondary content.

[0095] Control circuitry 420 and / or 418 may be based on any suitable processing circuitry, such as processing circuitry 426 and / or 440, respectively. As referred to herein, processing circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc., and may include multi-core processors (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores). In some embodiments, the processing circuitry may be distributed across multiple separate processors, such as multiple processors of the same type (e.g., two Intel Core i9 processors) or multiple different processors (e.g., an Intel Core i7 processor and an Intel Core i9 processor). In some embodiments, control circuitry 420 and / or control circuitry 418 is configured to display an extended reality experience, set a difficulty level in the extended reality experience, obtain user metrics from multiple users during an extended reality experience, such as a virtual game, generate median data, compare the median data to current data of a current user, determine whether the current user is experiencing or will experience a similar outcome in the extended reality experience, determine start and end checkpoints for monitoring the current user, provide remediation actions based on a user or multiple user performance, manage and adjust, as necessary, the difficulty level based on user interactions, manage motion sickness, perform home automation functions, run artificial intelligence or machine learning algorithms, manage the difficulty level, motion sickness, or home automation, and perform all of the processes described and illustrated in connection with Flowcharts 1, 7, 9, and 13A, 13B.

[0096] The user input 404 may be received from a virtual, augmented, or mixed reality headset, mobile data, smart glasses. Transmission of the user input 404 to the computing device 418 may be accomplished using a wired connection, such as an audio cable, a USB cable, an Ethernet cable, or the like, attached to a corresponding input port on the local device, or may be accomplished using a wireless connection, such as Bluetooth, WIFI, WiMAX, GSM, UTMS, CDMA, TDMA, 3G, 4G, 4G LTE, or any suitable wireless transmission protocol. The input circuitry 416 may comprise a physical input port, such as a 3.5 mm audio jack, an RCA audio jack, a USB port, an Ethernet port, or any other suitable connection for receiving audio via a wired connection, or may comprise a wireless receiver configured to receive data via Bluetooth, WIFI, WiMAX, GSM, UTMS, CDMA, TDMA, 3G, 4G, 4G LTE, or other wireless transmission protocol.

[0097] Processing circuitry 440 may receive input 404 from input circuitry 416. Processing circuitry 440 may convert or translate received user input 404, which may be in the form of voice input into a microphone or movements or gestures, to a digital signal. In some embodiments, input circuitry 416 performs the translation to a digital signal. In some embodiments, processing circuitry 440 (or processing circuitry 426, as the case may be) performs the disclosed processes and methods. For example, processing circuitry 440 or processing circuitry 426 may perform processes such as those described in Flowcharts 1, 7, 9, and 13A, 13B and other figures described herein.

[0098] FIG. 5 illustrates a generalized embodiment of an electronic device 500, according to one embodiment. In an embodiment, the appliance device 500 is identical to the appliance device 402 of FIG. 4. The appliance device 500 may receive content and data via input / output (I / O) paths 502. The I / O paths 502 may provide audio content (e.g., broadcast programming, on-demand programming, Internet content, content available over a local area network (LAN) or wide area network (WAN), and / or other content) and data to control circuitry 504, which includes processing circuitry 506 and storage device 508. The control circuitry 504 may be used to send and receive commands, requests, and other suitable data using the I / O paths 502. The I / O paths 502 may connect the control circuitry 504 (and specifically the processing circuitry 506) to one or more communication paths. An I / O function may be provided by one or more of these communication paths, but is shown as a single path in FIG. 5 to avoid overcomplicating the drawing.

[0099] Control circuitry 504 may be based on any suitable processing circuitry, such as processing circuitry 506. As referred to herein, processing circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc., and may include multi-core processors (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputers. In some embodiments, the processing circuitry may be distributed across multiple separate processors or processing units, for example multiple processing units of the same type (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor).

[0100] Communications between two separate user devices, such as an extended reality headset and a gaming module or receiving server for sending messages or sending the FOV or coordinates of the head-mounted VR device to a server, displaying an extended reality experience, setting a difficulty level in an extended reality experience, obtaining user metrics from multiple users during an extended reality experience, such as a virtual game, generating median data, comparing the median data with current data of a current user to determine whether the current user is experiencing or will experience a similar outcome in the extended reality experience, determining start and end checkpoints for monitoring a current user, providing improvement actions based on a user or multiple user performance, managing and adjusting the difficulty level as needed based on user interactions, managing motion sickness, performing home automation functions, running artificial intelligence or machine learning algorithms and data related to difficulty levels, motion sickness, or home automation, and all other processes and features described herein, can be implemented, at least in part, using control circuitry 504. The processes as described herein may be implemented in or supported by any suitable software, hardware, or combination thereof. They may also be implemented on user equipment, on a remote server, or across both.

[0101] In a client / server based embodiment, the control circuitry 504 may include suitable communications circuitry to enable communication between two separate user devices for displaying an extended reality experience, setting a difficulty level in the extended reality experience, obtaining user metrics from multiple users during an extended reality experience such as a virtual game, generating median data, comparing the median data to current data of a current user, determining whether a current user is experiencing or will experience a similar outcome in the extended reality experience, determining start and end checkpoints for monitoring a current user, providing remedial actions based on a user or multiple user performances, managing and optionally adjusting the difficulty level based on user interactions, managing motion sickness, implementing home automation functions, executing artificial intelligence or machine learning algorithms, managing the difficulty level, motion sickness, or home automation, and all related functions and processes as described herein. Instructions for performing the above mentioned functionality may be stored on one or more servers. The communications circuitry may include a cable modem, an integrated services digital network (ISDN) modem, a digital subscriber line (DSL) modem, a telephone modem, an Ethernet card, or a wireless modem for communicating with other equipment, or any other suitable communications circuitry. Such communications may involve the Internet or any other suitable communications network or path. In addition, the communications circuitry may include circuitry that enables peer-to-peer communication of primary equipment devices or communication of primary equipment devices at locations remote from one another (described in more detail below).

[0102] The memory may be an electronic storage device provided as storage device 508 that is part of control circuitry 504. As referred to herein, the phrase "electronic storage device" or "storage device" should be understood to mean any device for storing electronic data, computer software, or firmware, such as a random access memory, a read-only memory, a hard drive, an optical drive, a digital video disc (DVD) recorder, a compact disc (CD) recorder, a BLU-RAY® Disc (BD) recorder, a BLU-RAY® 3D Disc recorder, a digital video recorder (sometimes called a personal video recorder or DVR, PVR), a solid-state device, a quantum storage device, a gaming console, gaming media, or any other suitable fixed or removable storage device, and / or any combination thereof. Storage 508 may be used to store user or group statistics, difficulty level values, user and multiple user performance data or metrics, start and end checkpoints, data related to home automation devices and their settings and any user preferences, lists of recommended and improvement actions, and ML and AI algorithms, and all functionality and processes discussed herein, as depicted in Figure 8. Cloud-based storage, as described in connection with Figure 5, may be used to supplement or in place of storage 508.

[0103] The control circuitry 504 may include audio generation circuitry, such as one or more analog tuners, audio generation circuitry, filters, or any other suitable tuning or audio circuitry or combination of such circuits, and tuning circuitry. The control circuitry 504 may also include scaler circuitry for upconverting and downconverting content to a preferred output format of the electronic device 500. The control circuitry 504 may also include digital-to-analog converter circuitry and analog-to-digital converter circuitry for converting between digital and analog signals. The tuning and encoding circuitry may be used by the electronic device 500 to receive and display, play, or record content. For example, the circuitry described herein, including tuning, audio generation, encoding, decoding, encryption, decryption, scaler, and analog-to-digital circuitry, may be implemented using software running on one or more general-purpose or specialized processors. If storage 508 is provided as a separate device from electronic device 500 , tuning and encoding circuitry (including multiple tuners) may be associated with storage 508 .

[0104] A user may issue commands to control circuitry 504, which are received by microphone 516. Microphone 516 may be any microphone (or multiple microphones) capable of detecting human speech. Microphone 516 is connected to processing circuitry 506 and transmits the detected voice commands and other utterances thereto for processing. In some embodiments, a voice assistant (e.g., Siri, Alexa, Google Home, and similar such voice assistants) receives and processes the voice commands and other utterances.

[0105] The electronic device 500 may include an interface 510. The interface 510 may be any suitable user interface, such as a remote control, a mouse, a trackball, a keypad, a keyboard, a touch screen, a touch pad, a stylus input, a joystick, or other user input interface. The display 512 may be provided as a standalone device or integrated with other elements of the electronic device 500. For example, the display 512 may be a touch screen or a touch sensitive display. In such a situation, the interface 510 may be integrated with or combined with a microphone 516. When the interface 510 is configured with a screen, such a screen may be one or more monitors, a television, a liquid crystal display (LCD) for a mobile device, an active matrix display, a cathode ray tube display, a light emitting diode display, an organic light emitting diode display, a quantum dot display, or any other suitable equipment for displaying visual images. In some embodiments, the interface 510 may be HDTV compatible. In some embodiments, the display 512 may be a 3D display. The speaker(s) 514 may be provided as integrated with other elements of the electronic device 500 or may be a stand-alone unit. In some embodiments, the display 512 may be output through the speaker 514.

[0106] The equipment device 500 of FIG. 5 can be implemented in the system 400 of FIG. 4 as the primary equipment device 402, although any other type of user equipment is suitable for enabling communication between two separate user devices for performing functions related to implementing machine learning (ML) and artificial intelligence (AI) algorithms, as well as all functionality discussed in connection with the figures referred to in this application.

[0107] Any other type of suitable user equipment electronic device 500 may also be used to implement the ML and AI algorithms and related functions and processes as described herein. For example, an extended reality headset or smart glasses or similar such devices may be used. The electronic device may be part of a network of devices. Various network configurations of devices may be implemented and are discussed in more detail below.

[0108] FIG. 6 is a block diagram of an extended reality headset for use in a gaming environment according to some embodiments of the disclosure.

[0109] In some embodiments, an extended reality device, such as a headset, glasses, a mobile phone, or other wearable device, may be used to perform the processes described herein. The extended reality device may be non-immersive, fully immersive, semi-immersive, or have some other combination of virtual, augmented, or mixed reality. For example, an extended reality device may be an Oculus Rift. TM , Valve TM , Sandbox VR TM , or Sony PlayStation VR TM This can also be used with Iristick Z1 TM , Epson Moverio BT-200 TM , or Sony SmartEyeglass TMThe non-immersive experience may be a smart / virtual glasses such as a smartphone, a tablet, a tablet computer, a tablet PC ...

[0110] In some embodiments, the headset may be a virtual reality headset, where a purely virtual experience may be visualized. In other embodiments, the headset may be an augmented reality headset, where the real world environment, including any virtual overlays, may be visualized. In still other embodiments, the headset may include dual functionality, having both VR and AR capabilities and being able to switch between AR and VR.

[0111] In some embodiments, the extended reality device may include a complete system with the processor and components required to provide a complete extended reality experience. In other embodiments, the extended reality device may rely on external devices, such as smartphones, computers, and servers, to perform all processing. For example, a headset may be a plastic, metal, or cardboard holding case that allows viewing, which may be connected via wires, wirelessly, or via an API to a smartphone, using its screen as a lens for viewing. A headset may also be connected to a gaming console, where the headset has its own display screen, which is powered by the computer or game console.

[0112] In one embodiment, the extended reality headset used includes 6DOF, as depicted in Figure 6. Because the headset works by immersing the user in a virtual environment with all directions, for a fully immersive experience in which the user's entire field of view is utilized, including peripheral vision, an extended reality headset that offers a full 6DOF is preferred (although extended reality headsets with 3DOF can also be used).

[0113] Having 6DOF allows a user to move in all directions and experience objects and environments in a virtual world from all directions, for example, a user can move an object in 3D space. These 6DOF correspond to rotational movements about the x, y, and z axes, commonly referred to as pitch, yaw, and roll, and translational movements along those axes, such as moving laterally along any one direction x, y, or z. Tracking all 6DOF allows the system to capture user movements and the user's field of view in both translational and rotational directions, thereby providing a full 360° view in all directions.

[0114] Although some reference is made to types of extended reality headsets, the embodiments are not so limited and any other extended reality headset available on the market may also be used in conjunction with the embodiments described herein.

[0115] FIG. 7 is a flowchart of a process for collecting data related to motion sickness in a virtual gaming environment according to some embodiments of the disclosure.

[0116] In some embodiments, at block 710, the gaming environment is displayed on the extended reality device. The gaming environment may be an extended reality environment displayed on a display screen associated with a VR headset, VR glasses, or another type of VR module. The gaming environment may display a virtual game with multiple challenges and assets and may include a virtual simulation of both the real world and imaginary surroundings. The extended reality device on which the virtual game is displayed may be connected to cameras, lenses, display screens, speakers, motion generation components, and other equipment required to provide a complete extended reality experience. The extended reality device may be capable of connecting to external applications via APIs to download or access various games. In one embodiment, the displayed virtual game may have a layout as depicted in block 101 of FIG. 1.

[0117] At block 720, a start checkpoint and an end checkpoint are selected. As explained above, a game may have several checkpoints. The start and end checkpoints in the game may be selected such that challenges or experiences are included within the checkpoints. These challenges may be promotion to the next game level, opportunities to score points, or milestones within the game.

[0118] Checkpoints also act as reference locations in the game state that are saved for rollback or restart, such as when a character dies, the game may reload or restart from the last checkpoint. In some embodiments, checkpoints C1 and C2 are selected based on the type of extended reality game and its environment. For example, the game may not have fast moving components or rapidly changing graphics, in which case checkpoints with longer durations between them may be selected and data may be acquired at longer intervals so that the amount of data does not take up too much CPU and GPU space. For example, for a 10 minute game with four levels, where the game is a slower game, 300 second checkpoints with data collection every 15-30 seconds may be appropriate.

[0119] On the other hand, for fast-paced games with several moving components or rapidly changing graphics, checkpoints with shorter durations between them may be selected and data may be acquired at shorter intervals, such as every half second or every few milliseconds, such as 100 milliseconds, to ensure that any important changes in the game are not missed.

[0120] The larger the value of the time interval for collecting data, the smaller the log file will be, but the greater the risk of missing useful information or phenomena, such as user movement that may have occurred between data collection points. The smaller the value of the time interval for collecting data, the larger the log file will be, which will also consume a lot of memory in the CPU and GPU. Therefore, some level of balancing may be implemented to ensure that the right amount of data is collected while not overloading the CPU / GPU. Such balancing attempts may take into account the type of application as described above, for example, whether this is a fast or slow moving game.

[0121] Blocks 730, 740, 750, and 760 are used to collect motion sickness data from multiple users, aggregate the data, and calculate a median value that can be used for current and future users. To accomplish the collection, aggregation, and median collection, in one embodiment, in block 730, the control circuitry monitors multiple users involved in playing the virtual game. As the multiple users progress through the path between C1 and C2, user data is collected. The data includes the translation and orientation movements (e.g., qw, qx, qy, and qz) of the users between checkpoints C1 and C2. For example, if a user moves their head to the left, thereby moving their wearable headset to the left, the field of view as the user turns left is recorded to determine the assets that were in the user's view. In addition to what is depicted in block 740, all obstacles encountered, challenges solved, assets viewed, assets interacted with, the user's vital signs during the user's movement, CPU / GPU utilization during the game, motion sickness experienced, the time it took the user to complete a certain difficulty level, the coordinates of the path taken by the user, the type of path, the time on each segment of the path, and other gaming metrics are collected.

[0122] At block 730, data related to motion sickness is also collected, among other types of data and metrics that may be collected. Such data may be obtained from wearable devices, sensors, or other attachments associated with the extended reality device, such as a smart watch that may detect heart rate, body sensors, and other types of monitors that may collect the user's vital signs. The user may also input such data into a system associated with the extended reality environment. In yet another embodiment, the control circuitry may collect demographic user information, health profile information, or other information before or during the extended reality experience. It may also obtain such information through other means (such as a profile from a content delivery storefront). The control circuitry may also obtain information such as the time of day, the time when the last meal was eaten, the user's blood glucose level, etc. Additional examples of data that may be collected are depicted in FIG. 8 below.

[0123] At block 750, a determination is made whether user data, regardless of how it is collected, may be used to determine whether the user is experiencing motion sickness. Such a determination may be made based on the user's medical profile or general medical information available on the internet. The determination may also be made based on the user's behavior in the game, for example, the user not being as careful as before, which may indicate motion sickness, or the user themselves indicating that they are experiencing motion sickness by typing information into a system associated with the extended reality environment. The system may also have an icon or selection available for the user to select whether they are experiencing motion sickness.

[0124] If motion sickness is detected in block 750, the user's vitals and any one or more of the motion sickness related data previously collected in block 730 are aggregated in block 760. Such aggregated data gives the content creator the ability to make inferences regarding when the current user may or may not experience motion sickness. The median of the aggregated data is calculated. The median data may include the median orientation (qw, qx, qy, qz) of the user at each point on the median path, providing details regarding the exact location on the path where motion sickness was experienced. An example of a median path is depicted in FIG. 14. The utilization of median data obtained from multiple users and its application to the current user is further described in the description of FIG. 9.

[0125] FIG. 8 is an example of a chart depicting various user background factors that may be taken into account to determine motion sickness associated with playing a virtual game, according to some embodiments of the present disclosure.

[0126] In some embodiments, the user's ethnicity, age, field dependency, illness, interpupillary distance, genetics, investigated past susceptibility (to various stimuli) and any prior medical history or symptoms such as causes of motion sickness due to certain environments, lack of sleep, etc., the severity of the user's motion sickness, and other types of stimuli that cause motion sickness may be taken into account in determining whether any of such user background information contributes to the user's motion sickness.

[0127] Whether this is when collecting data of multiple users, such as in block 102 of FIG. 1, or collecting current data of a current user, the data may be distorted based on the user's background information. For example, during the data collection stage, such as in block 102 of FIG. 1, one user from the multiple users may experience significant motion sickness at a particular point in the game, while another user from the multiple users experiences little or no motion sickness. In response to accessing the user's profile and based on further investigation of the user's background, it may be discovered that the user has certain medical problems due to his / her age or past susceptibility or some other factor, such as interpupillary distance disorder. In one embodiment, the system may discard such user's data and consider it to be an outlier. In another embodiment, the system may log the data in a record and apply it when a current user of a similar background plays the virtual game.

[0128] In some embodiments, ethnicity, age, field dependency, illness, or genetics may also play a role in motion sickness, and such background information may be considered when evaluating the current user or when aggregating data from previous users. In some embodiments, the system may group data into groups based on the ethnicity of the user. Such data may then be used if a determination is made that the current user is also of the same ethnicity. Similarly, age and other background data may also be grouped and used accordingly. The system may also generate multiple subgroups with combinations of background information, such as generating groups for ethnicity and subgroups for different age ranges within the same ethnicity.

[0129] In some embodiments, based on the collected data, the system may predict the likelihood and probability that the current user will experience motion sickness at a particular stage of the virtual game. For example, the system may determine (as part of determining the user's background) that 72% of users who played the game have prescription glasses. If the system encounters the current user and determines, based on the current user's profile or through facial attribute detection, that the user is wearing prescription glasses with a similar prescription, the system may determine the probability that the current user will experience motion sickness. Thus, the system may either change the game screen, provide an alert to the user to take a different route in the game, or simply warn the user that the user will likely experience motion sickness.

[0130] In some embodiments, the user may also review the user's past motion sickness data prior to or during the virtual game to determine his or her likelihood of becoming motion sick. The system may also provide a manuscript or appearance of virtual game points where the user is likely to become motion sick, and the user may avoid those points if the user so chooses.

[0131] While user background such as the user's ethnicity, age, field dependency, illness, interpupillary distance, genetics, and surveyed past susceptibility (and various other stimuli) are described above in relation to determining or predicting motion sickness, such factors can also be used to determine their application to other virtual gaming and augmentation options. For example, such factors may be applied to determine a user's likely response to a level of game difficulty or to a certain situation within a virtual game. For example, a younger user may be more likely to take a particular path in a virtual game or perform poorly at a certain game level based on their own experience than an older other. Similarly, an older user may not be able to perform well in a certain situation compared to a younger other. In addition, other background factors such as the amount of virtual gaming experience, education, or motor skills may also be used to determine whether a user is likely to perform better in a certain virtual gaming situation.

[0132] 9 is a flow chart of a process for providing recommendations or activating home automation to manage motion sickness caused by a virtual gaming environment, according to some embodiments of the present disclosure. As described above, FIG. 8 illustrates some embodiments of collecting data from multiple users, and FIG. 9 illustrates some embodiments of how such collected data may be applied to a current user to take remedial actions to manage motion sickness or provide other alternatives.

[0133] At block 910, median data is calculated based on the motion sickness data collected from multiple users as they navigate from checkpoint C1 to C2. Although calculation of median data is described, the embodiment is not so limited and calculations may also be performed to determine a representative value, a mean, a standard deviation, a variance, or some other calculation based on an equation.

[0134] Once the median data is calculated, in some embodiments the system may follow the path of Trajectory 1, while in other embodiments it may follow the path of Trajectory 2. Trajectory 1 is associated with determining remedial action after discovering that the current user is experiencing motion sickness, while Trajectory 2 is associated with anticipating motion sickness and taking remedial action prior to the user becoming motion sick.

[0135] In one embodiment, either trajectory 1 or trajectory 2 may be selected by the system. In another embodiment, the system may select either trajectory 1 or trajectory 2 based on user preferences, such as preferences contained in the user's profile. In another embodiment, the system may provide options for the user to select, which may be considered to determine whether trajectory 1 or trajectory 2 should be selected. In yet another embodiment, the system may analyze the user's prior gaming data, which may be obtained based on the execution of a machine learning algorithm, to determine the likelihood that the user will experience motion sickness. If a determination is made that the user is likely to experience motion sickness, the system may select trajectory 2, where remedial actions are applied proactively and prior to the user experiencing actual motion sickness.

[0136] In some embodiments, in trajectory 1, at block 920, the system may monitor the current user's motion sickness factor at various times. These times may be t=0, t=1, ... t=n, and the monitoring time "t" may be a predefined interval. The system may monitor the user's motion sickness factor either periodically or all the time during gameplay. In another embodiment, the system may monitor the user's motion sickness factor after a milestone or challenge in the virtual game, such as a steep drop on a virtual roller coaster in the virtual game.

[0137] At block 930, the control circuitry 420 or 428 determines whether the user's current data or factor is within the median data threshold. Some examples of user's current data and median data are provided in FIG.

[0138] In some embodiments, as depicted in FIG. 10, the control circuitry 420 or 428 may monitor user data at any one or more times during a virtual session, such as a virtual game or experience. It may also monitor user data at milestones within the game or at various gaming levels. It may also monitor user data periodically or continuously throughout the entire virtual session. The monitored factors may be any or more of heart rate, body temperature, sweat levels, dizziness, loss of concentration, lack of responsiveness to game challenges, headaches, nausea, yawning, sighing, increased salivation, belching, blurred vision, drowsiness, spatial disorientation, or any other similar factors typically associated with motion sickness. Monitoring may be through cameras and sensors associated with the extended reality device. Monitoring may also be through data obtained from wearable devices, sensors, or other attachments connected to the extended reality device via an API.

[0139] One example of monitoring a user's current data and comparing it to median data is depicted at 1010 in FIG. 10. In this example, data regarding a virtual experience, such as a virtual game, a virtual room, a virtual education, or any other type of virtual experience, may be collected from multiple users. A median may be calculated for the data collected from multiple users. As mentioned above, instead of a median calculation, other calculations may also be used. One of the monitored attributes may be to monitor acceleration depicted in a virtual game consisting of a roller coaster ride. The acceleration may be measured at curve 1, which may be a sharp curve during the virtual roller coaster ride. A system administrator or game developer may have selected the acceleration at curve 1 to be monitored. For example, a system administrator or game developer may have selected to monitor curve 1 because the curve is designed in the virtual game to be a sharp curve where users will experience some gravity, and the system administrator or game developer may anticipate potential motion sickness for some people at this curve. Whatever the reason for monitoring acceleration at curve 1, the calculated median data may indicate that the median user from the multiple users had a median heart rate of 72 BPM. Current user data on the same curve 1 with the same acceleration may also be monitored and the results may show that the current user's BPM is 75. Control circuitry 420 or 428 may determine whether the current user's BPM is within a threshold range of + / - 5 BPM of the median BPM of 72.

[0140] Another example of monitoring a user's current data and comparing it to the median data is depicted at 1020 in FIG. Block 1020 is similar to block 1010, except that the data is monitored with respect to acceleration on curve 2. The median data calculated on curve 2 may indicate that a median user from a plurality of users had a median heart rate of 77 BPM, while the current user data on the same curve 2 with the same acceleration is at 84 BPM, i.e., higher than the tolerance threshold of + / - 5 of the median BPM of 77.

[0141] In yet another embodiment, data regarding a virtual experience may be collected from multiple users at time t=t1, as depicted at 1030 in FIG. 10. In this embodiment, at time t=t1, the virtual experience may comprise a rotation in a virtual room. The rotation may be a visual rotation visible in the user's headset, or it may be that the user physically rotates in a real-world environment to carry out instructions provided in the virtual world. A median value may be calculated at time t=t1 for data collected from multiple users while the users experience a real-world or virtual rotation in the virtual room. A system administrator or game developer may have selected a specific rotation acceleration based on its expected motion sickness potential. As depicted, the median data calculated based on data collected from multiple users regarding the rotation at time t=t1 indicates a median body temperature of 98°F. The control circuitry 420 or 428 may determine that the user's body temperature is 100°F while undergoing the same virtual experience with multiple users at time t=t1 while monitoring the user's current data. Control circuitry 420 or 428 may also determine that a user's temperature of 100°F is outside a threshold of + / - 1°F from the median body temperature of 98°F.

[0142] In another example, as depicted at 1040 in FIG. 10, data regarding a virtual experience, which in this case may be a virtual game, may be collected after 30 minutes of gameplay. A system administrator or game developer may have selected the 30-minute mark to determine the amount of fatigue a game player may experience after playing a virtual game for 30 minutes and its causal effect on motion sickness. The 30-minute mark may also have been selected based on data obtained from machine learning or AI algorithms that indicate that users typically experience fatigue, eye strain, or other symptoms that cause the user to experience motion sickness. Whatever the reason for monitoring a user's responsiveness after 30 minutes of gameplay, median data calculated based on data collected from multiple users may indicate that the median user has a scoring rate of 67%. The level of responsiveness in this example may be measured based on the user's ability to score at a predetermined percentage. The level of responsiveness in this example may also be measured by determining the scoring rate that a user performs after 30 minutes of gameplay and whether it is within a threshold of the user's starting scoring rate. As depicted, the control circuitry 420 or 428, while monitoring the user's current score percentage, may determine that the user's current score percentage is 53%. The control circuitry 420 or 428 may determine that the user's score percentage is outside of + / - 10% of the median score percentage.

[0143] Although some examples of monitoring are described in connection with FIG. 10, embodiments are not so limited and other types of monitoring are also contemplated. For example, the control circuitry may also monitor the user's facial expressions to determine whether the user is experiencing motion sickness. In this example, the control circuitry may compare the user's baseline facial expressions to the facial expressions when points in the virtual experience, such as a high speed curve, descent, or acceleration, are depicted. The control circuitry may detect changes in facial expressions and determine whether the user is experiencing motion sickness.

[0144] Referring again to block 930 of FIG. 9, if a determination is made in block 930 that the user data does not exceed the median data threshold, the process returns to block 920 and repeats until the user reaches checkpoint C2 or the end of the game.

[0145] If a determination is made at block 930 that the user data exceeds the median data threshold, ie, the tolerance threshold, and deviates from the median data, the process moves to block 940 where remedial action is taken.

[0146] In one embodiment, the remedial action may be to make recommendations to reduce motion sickness. These recommendations may be provided to a user, a system administrator, a virtual experience creator, or any combination thereof. Some examples of motion sickness recommendations are provided below in connection with the discussion of FIG.

[0147] In another embodiment, the remediation action may be to automatically implement home automation via the API. Some examples of the types of home automation motion devices that may be controlled are provided below in conjunction with the discussion of FIG.

[0148] Referring again to block 910, as explained above, another path that may be selected may be the path of trajectory 2. In one embodiment, in block 950, the control circuitry 420 or 428 tracks the user's current path, which is the path taken by the user within checkpoints C1 and C2, such as checkpoints C1 and C2 depicted in FIG.

[0149] Tracking the user's current path may be accomplished by periodically or continuously determining the user's coordinates as the user navigates from checkpoint C1 to C2. For example, the user may select a path while navigating from the couch to bookshelf 1 as depicted in Figure 16. Such coordinates may be recorded or logged as shown in Figures 14 and 18.

[0150] At block 960, a determination is made whether the current route is aligned with the median route. In one embodiment, if the median user experiences motion sickness along the route and the current user's route is aligned with the median user, the control circuitry 420 or 428 may predict that the current user will also experience motion sickness. On the other hand, if the median user does not experience motion sickness along the route and the current user's route is aligned with the median user, the control circuitry 420 or 428 may predict that the current user will not experience motion sickness. The control circuitry 420 or 428 may also apply current user's background data, such as from FIG. 8, to adjust or deviate from the median route and determine whether the user will experience motion sickness based on the user's background.

[0151] If, in block 960, a determination is made that the current user's path does not match the median path, the process returns to block 950 and the current user's path continues to be tracked until it matches the median path or until checkpoint C2 or the end of the virtual experience.

[0152] If, at block 960, a determination is made that the current user's route matches the median route, the process proceeds to block 940 where remedial action is taken.

[0153] In one embodiment, the remedial action may be to make recommendations to reduce motion sickness. These recommendations may be provided to a user, a system administrator, a virtual experience creator, or any combination thereof. Some examples of motion sickness recommendations are provided below in connection with the discussion of FIG.

[0154] In another embodiment, the remediation action may be to automatically implement home automation via the API. Some examples of the types of home automation motion devices that may be controlled are provided below in conjunction with the discussion of FIG.

[0155] As depicted in block 1105 of FIG. 11, one type of recommendation made to the current user may be to reduce rotations. In this embodiment, the control circuitry 420 or 428 may determine that the amount of rotations in the virtual experience is causing the user motion sickness. Thus, the control circuitry 420 or 428 may provide an alert to the user to reduce the number of rotations in either the virtual or real environment to minimize the degree of motion sickness. If the prior rotations are not the cause for motion sickness, the control circuitry 420 or 428 may still provide an alert to the user to reduce the number of future rotations in the game, i.e., in either the virtual or real environment, to minimize the degree of motion sickness.

[0156] In another embodiment, the control circuitry 420 or 428 may provide a recommendation to the user to change their current route, which may be causing motion sickness, to a new route that is less likely to cause motion sickness. For example, the control circuitry 420 or 428 may associate a route with a certain motion sickness value based on the amount of twisting, turning, orientation, speed, acceleration, and other factors that may cause motion sickness. The control circuitry 420 or 428 may determine that a different route with a lower motion sickness value is suitable for the current user, and thus will recommend to the current user to change to the new route. In another embodiment, the control circuitry 420 or 428 may close or block a route with a high motion sickness value when such is needed.

[0157] In another embodiment, control circuitry 420 or 428 may provide recommendations to a system administrator or creator of a virtual game to reduce rotations. Such recommendations may enable the system administrator or game creator to either provide an alternate screen in the game with less rotation or to remove any instructions in the game that require the current user to perform rotations.

[0158] As depicted in block 1110, another type of recommendation made to the current user may be to reduce the user's speed or acceleration in the virtual experience. In this embodiment, control circuitry 420 or 428 may determine that the amount of speed or acceleration in the virtual experience is a contributing factor to the user's motion sickness. Thus, control circuitry 420 or 428 may provide an alert to the user to reduce the amount of speed or acceleration.

[0159] For example, a user may be using his or her virtual headset to play a virtual car racing game in which the user controls the speed and amount of acceleration as the virtual car gamely speeds across a roadway or makes a sharp turn. The control circuitry 420 or 428 may alert the user to reduce the user's speed or acceleration as the user is making his or her turn. The control circuitry 420 or 428 may also suggest to the system administrator or game developer that they should limit either the amount of speed and acceleration or the curvature of the curves the car may make turns in order to minimize the degree of motion sickness.

[0160] The recommendation may be in the form of instructions provided to the user as a pop-up screen or by other visual or audible means, as depicted in block 1115. For example, control circuitry 420 or 428 may send an audio signal to the user's headphones instructing the user not to perform a quick 360° turn.

[0161] As depicted in block 1120, one type of recommendation may be made to a system administrator or game creator to rearrange the layout of a virtual space, such as the layout of a virtual room or the layout of a virtual roller coaster or a virtual car racing game, to reduce motion sickness. For example, based on the motion sickness experienced by the median user and the current user, the circuitry 420 or 428 may provide recommendations to the system administrator or game creator to reduce the amount of climbs and descents in a virtual roller coaster, change the number of turns or the angle of turns in a roller coaster or car racing game, or change the layout of a virtual mountain climbing experience with deep drops to minimize or reduce motion sickness. Recommendations may be general or specific outlining challenges or obstacles to be removed or reduced to reduce motion sickness.

[0162] As depicted in block 1125, the control circuitry 420 or 428 may predict motion sickness based on the median user data. As explained above, if the median user data suggests that the median user experiences motion sickness at a particular time or during a particular segment of the virtual experience, the control circuitry 420 or 428 may predict a similar degree of motion sickness for the current user if the current user follows a similar route as the median user.

[0163] As depicted in block 1130, the circuitry 420 or 428 may monitor the user's vitals as the user plays a virtual game or navigates from checkpoint C1 to C2 in another type of virtual experience. Based on the monitoring of the vitals, the circuitry 420 or 428 may make recommendations to a system administrator or game creator to take some remedial action. For example, in some embodiments, a virtual game or virtual experience may be created such that when the user comes to a certain point in the game, alternative paths may be made available to the user. The circuitry 420 or 428 may make recommendations to the system administrator or game creator to eliminate one of the alternative paths on which motion sickness may be experienced and maintain another path that has a lesser likelihood of causing motion sickness. In another embodiment, other game modification options, such as changes in graphics, orientation, rotation, speed, or acceleration, may also be suggested based on the user's vitals.

[0164] As depicted in block 1135, circuitry 420 or 428 may provide other types of alerts to the current user to minimize motion sickness. For example, a virtual alert with an arrow may point toward a path of less twisting and turning or less center of gravity shift for the user due to movement in the virtual or real world.

[0165] FIG. 12 is a block diagram of categories of devices and machines that may be activated via an API to manage motion sickness in a virtual gaming environment according to some embodiments of the disclosure.

[0166] In one embodiment, as depicted in block 1205, the physical home windows may be controlled via an API as a remedy for reducing motion sickness. In this embodiment, the windows may include an automatic mechanism that allows them to be electronically controlled so that they can be opened and closed based on the type of signal received. In this embodiment, the windows may be wirelessly connected to either the control circuitry 420 or 428 associated with the headset, or the gaming server, or other intermediate home hub that is connected to both the headset and the gaming server and automated windows.

[0167] In one embodiment, if the control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as in block 930 of FIG. 9, the control circuitry 420 or 428 may send a signal to open a window, thereby allowing light and fresh air to enter the user's location to reduce the user's motion sickness.

[0168] In another embodiment, if the control circuitry 420 or 428 predicts that the user will likely experience motion sickness, such as in block 970 of FIG. 9, the control circuitry 420 or 428 may send a signal to open a window in anticipation of the user becoming motion sick and to prevent such an occurrence. The control circuitry 420 or 428 may also determine the weather outside the house and take that into account when opening the window. For example, if it is the middle of winter and it is cold outside, such as 32° F., the control circuitry 420 or 428 may open the window only for a short duration, such as 2-5 minutes, to prevent the area from getting too cold.

[0169] In another embodiment where home automation is applied, the temperature of the room in which the current user is playing a virtual game or experiencing a virtual environment through his or her headset may be changed as a remedy to reduce motion sickness, as depicted in block 1210. In this embodiment, the temperature controller or thermostat may include Wi-Fi or API capabilities that allow the controller to be controlled electronically, such that the temperature can be increased or decreased based on the type of signal received.

[0170] For example, Google Nest TM , Honeywell TM Thermostat or Ecobee TM A thermostat such as the Home is a type of thermostat that includes Wi-Fi control capabilities. In this embodiment, such exemplary thermostats and other types of temperature controllers may be wirelessly connected to either the control circuitry 420 or 428 associated with the headset, or the gaming server, or other intermediate home hub that is connected to both the headset and the gaming server and the Wi-Fi and API enabled thermostat.

[0171] In one embodiment, if the control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as in block 930 of Figure 9, the control circuitry 420 or 428 may send a signal to a thermostat. The signal may be to turn on an air conditioner or turn on or increase the speed of a fan, thereby allowing air movement and a decrease in temperature, etc., to reduce the user's motion sickness.

[0172] In another embodiment, if the control circuitry 420 or 428 predicts that the user will likely experience motion sickness, such as in block 970 of Figure 9, the control circuitry 420 or 428 may send a signal to lower the temperature, such as by turning on an air conditioner or turning on or increasing the speed of an electric fan, thereby allowing air movement and a decrease in temperature, in anticipation of the user becoming motion sick, to prevent such an occurrence. In some environments, the control circuitry 420 or 428 may also remember the user's preferred temperature setting and automatically set a thermostat to reach the preferred temperature either when motion sickness is experienced or when motion sickness is anticipated.

[0173] In another embodiment where home automation is applied, the lighting in a room where a current user is playing a virtual game or experiencing a virtual environment through his or her headset may be changed as a remedy to reduce motion sickness, as depicted in block 1215. In this embodiment, the lighting module and controller may include Wi-Fi or API capabilities that allow the lighting module and controller to be electronically controlled such that lights can be turned on / off or the intensity of the lighting can be changed.

[0174] For example, Lutron TM , Insteon TM , Wemo TM , and Phillips TM Smart home lighting controllers, switches, and modules, such as smart switches from may include Wi-Fi control capabilities. In this embodiment, smart home devices with such example smart lighting switches and lights may be wirelessly connected to either the control circuitry 420 or 428 associated with the headset, or the gaming server, or other intermediate home hub that is connected to both the headset and the gaming server and the Wi-Fi and API enabled thermostat.

[0175] In one embodiment, if the control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as in block 930 of FIG. 9, the control circuitry 420 or 428 may send a signal to the smart light switch. The signal may turn on lights in the user's location, brighten the lights, or, in the case of movable lights, move the lighting direction to face the user, thereby allowing the user's brain to respond positively to brighter lights and illuminating the user's area in an attempt to reduce the user's motion sickness. Such lighting may help reduce motion sickness, since poor peripheral vision or lack of peripheral awareness may be a contributing factor for motion sickness.

[0176] In another embodiment, if the control circuitry 420 or 428 predicts that the user will likely experience motion sickness, such as in block 970 of FIG. 9, the control circuitry 420 or 428 may send a signal to the smart light switch. The signal may turn on lights in the user's location, brighten the lights, or, in the case of movable lights, move the lighting direction to face the user, thereby allowing the user's brain to respond positively to brighter lights and illuminating the user's area in an attempt to reduce the user's motion sickness.

[0177] In yet another embodiment, as depicted in block 1220, devices such as refrigerators, water coolers, or other devices that store food may be illuminated or emit an audible sound to suggest to the user that the user should eat some food or drink something as a remedial measure to reduce motion sickness. In this embodiment, refrigerators that have a touchscreen display panel on their exterior, have Wi-Fi connectivity, or are API-enabled may be electronically connected and controlled to light up their touchscreen displays, or wake up their touchscreens, or illuminate their screens in a certain color. Such refrigerators may also provide a text alert on the touchscreen, such as a glass of orange juice, or emit an audio signal or speech that alerts the user to eat some food or drink some liquid to reduce the user's motion sickness. Such remedial measures may be taken if the user is experiencing motion sickness, or in anticipation that the user may experience this based on the path the user is currently taking in the virtual experience.

[0178] In another embodiment, as depicted in block 1225, control circuitry 420 or 428 may automatically turn on several other Wi-Fi or API enabled devices, such as a fan, music player, door, recliner chair, etc., to change the environment in the room as a remedy to reduce motion sickness. In this embodiment, the Wi-Fi or API enabled devices may allow them to be electronically controlled by control circuitry 420 or 428.

[0179] In one embodiment, if the control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as in block 930 of FIG. 9, or predicts that the user will likely experience motion sickness, such as in block 970, the control circuitry 420 or 428 may send a signal to the Wi-Fi or API enabled device. The signal may be to perform a combination of actions to change the user's environment to reduce the user's motion sickness. For example, if the user is playing a virtual game on a Wi-Fi enabled recliner, the control circuitry 420 or 428 may move the user from a reclined position to sit up straight, or vice versa, to change the user's environment.

[0180] In another embodiment where home automation is applied, as depicted in block 1230, exercise equipment located in or near a room where a current user is playing a virtual game or experiencing a virtual environment through his or her headset may be turned on and the user may be alerted to use the exercise machine that was turned on in an attempt to reduce motion sickness. In this embodiment, the exercise machine may include a controller that is Wi-Fi or API enabled. The exercise machine controller may be electronically controlled so that it can be turned on and its speed adjusted.

[0181] For example, Peloton TM , NordicTrack TM Exercise bikes and machines such as the Tonal TM Smart home gym machines such as, and other similar smart exercise equipment, may include Wi-Fi control capabilities. In this embodiment, such smart exercise equipment may be wirelessly connected to either the control circuitry 420 or 428 associated with a headset, or a gaming server, or other intermediate home hub that is connected to both the headset or gaming server and a Wi-Fi and API enabled thermostat.

[0182] In one embodiment, if the control circuitry 420 or 428 determines that the user is experiencing motion sickness, such as in block 930 of FIG. 9, the control circuitry 420 or 428 may send a signal to the smart exercise device. The signal may be to turn on the smart exercise device. The signal may also be to illuminate the smart exercise device, such as one with a display screen, or to generate an audio alert for the user to begin exercising to reduce the user's motion sickness.

[0183] In another embodiment, if the control circuitry 420 or 428 predicts that the user will likely experience motion sickness, such as in block 970 of Figure 9, the control circuitry 420 or 428 may send a signal to illuminate a smart exercise device, such as one having a display screen, or generate an audio alert for the user to begin exercising to reduce the user's motion sickness. Because certain exercises may reduce motion sickness, the smart exercise system may also be automatically controlled to turn on programs associated with such motion sickness reduction exercises.

[0184] Although several examples of types of home automation and smart devices are described above, the embodiments are not so limited and other types of home automation devices are also envisioned. For example, in some embodiments, the control circuitry 420 or 428 implements smart home automation functions to minimize or reduce the current user's motion sickness. The functions are implemented either after determining that the current user is experiencing motion sickness or proactively before the current user experiences motion sickness.

[0185] The embodiment includes control circuitry 420 or 428 that generates an extended reality experience for display on an extended reality device such as a VR headset. 420 or 428 determines whether a current user is experiencing or will experience motion sickness based on a route taken by a current user in the extended reality experience. Because some routes have more motion sickness elements than other routes, or some points or locations along the same route have more motion sickness elements than other points and locations, the control circuitry 420 or 428 analyzes the routes and determines which one is more suitable for the current user. Some examples of such elements may be a sharp turn or orientation of the user in either the virtual or real world, a fast speed, acceleration into negative gravity, a steep drop on a roller coaster, a shift in center of gravity in the virtual or real world, exposing the user to a high altitude, etc.

[0186] In some embodiments, the control circuitry 420 or 428 activates smart home devices to manage the current user's motion sickness in response to determining that the current user is experiencing or will experience motion sickness based on a route taken in the extended reality experience. In these embodiments, the control circuitry 420 or 428 determines which smart home devices are in the vicinity of the user, i.e., within a threshold distance. To do so, the control circuitry 420 or 428 scans for smart home devices, such as by using Bluetooth, and pairs the discovered smart home devices with the extended reality device, such as via an API.

[0187] A threshold distance between the extended reality device and a discovered smart home device is determined by the control circuitry 420 or 428. If the distance is within the threshold set by the control circuitry 420 or 428, the smart home device may be used by the control circuitry 420 or 428. Devices that are outside the room, outside the current user's FOV, or a certain distance away, such as 30 feet away, may be considered to be outside the threshold distance.

[0188] In some embodiments, the control circuitry 420 or 428 and smart home devices are turned on once a determination is made that the current user is likely to experience motion sickness. To make such a determination, the control circuitry 420 or 428 obtains machine learning data from multiple users who have taken the same route as the current user. The control circuitry 420 or 428 then compares the current user's route to a representative, average, or median route from the multiple users, or another representative route. The control circuitry 420 or 428 determines locations on the route where the multiple users experienced motion sickness, and if the current user is on the same route or approaching the same location on the same route where a representative user from the multiple users experiences motion sickness, the control circuitry 420 or 428 determines that the current user is also likely to experience motion sickness on the same route or at the same location along the route.

[0189] Some examples of smart home devices that may be activated by control circuitry 420 or 428 include fans, air conditioners, lights, refrigerators, exercise equipment, temperature controllers, window opening devices, and chair reclining devices.

[0190] In some embodiments, the control circuitry 420 or 428 accesses a user profile of the current user. The control circuitry 420 or 428 obtains the user's preferred settings for the smart home devices from the user profile. For example, the current user may prefer the air conditioner to be at 70° F., the fan to be at a third speed, or the lights to be at a certain brightness. The control circuitry 420 or 428 uses such preferences and turns on the smart home devices to the preferred user settings.

[0191] In another embodiment, the control circuitry 420 or 428 determines whether the activated smart device has resulted in a reduction in the current user's motion sickness. It may do so based on the current user's performance in the extended reality experience, such as the users not scoring at the same percentage. It may also receive input from the current user indicating that the user continues to experience motion sickness. It may also conclude that the current user is experiencing motion sickness if the current user's vitals match those associated with motion sickness, such as sweating or elevated body temperature. In response to determining that the activated smart device has not resulted in a reduction in the current user's motion sickness, the control circuitry 420 or 428 may increase the settings of the smart home devices to higher levels, such as a lower temperature for the air conditioner, a higher speed for the electric fan, and wider open windows. The control circuitry 420 or 428 may also activate more than one smart home device simultaneously to increase efforts to reduce motion sickness.

[0192] 13A and 13B are flow charts of a process for collecting data related to a user's handling of a pre-determined difficulty level for a virtual experience or a virtual game. In one embodiment, FIG. 13A relates to managing a pre-determined difficulty level and adjusting the difficulty level based on a comparison between the current user's performance in the virtual experience or virtual game and the difficulty level pre-determined by the control circuitry or the content creator. In other words, it determines whether the current user handled the system-set difficulty level easily, with difficulty, or as expected. On the other hand, FIG. 13B compares the performance data of the current user in the virtual experience or virtual game with a median, representative, or representative user. The median user data is calculated based on data, also referred to as performance metrics, that are aggregated by the control circuitry from multiple users who played the virtual game or experienced the virtual experience.

[0193] Specifically, according to some embodiments, FIG. 13 is a flowchart of a process for collecting data related to a user's treatment of a difficulty level in a virtual game and using the data to enhance or manage the difficulty level.

[0194] In some embodiments, in block 1310, the control circuitry 420 or 428 may generate a virtual game environment with a predetermined level of difficulty between checkpoint 1 and checkpoint 2 of the virtual game. In another environment, if this is not a virtual game, this may be some other virtual experience that may require the user to navigate from checkpoint C1 to checkpoint C2, with obstacles in between.

[0195] In one embodiment, a given level of difficulty may involve certain types of obstacles or challenges, and the control circuitry 420 or 428 may predict that the user will take a certain amount of time or a certain number of steps before solving those obstacles and challenges.

[0196] In one embodiment, a predetermined level of difficulty may involve completing or passing a gaming level after a period of time. For example, in a virtual game at a higher or more advanced level in a game, such as level 6 in an eight level virtual game, the game designer may have designed level 6 such that it takes a user at least three minutes and a minimum of five attempts to complete a challenge at level 6 of the virtual game. Thus, control circuitry 420 or 428 may be configured to determine whether the designed level of difficulty at level 6 has been successfully overcome by the user before the three minute threshold time.

[0197] In one embodiment, a predetermined difficulty level may be accompanied by a certain speed rating designed by a game designer. For example, in a virtual car racing game, there may be curves that are designed such that a user cannot go around the curve faster than 70 mph without hitting a wall or losing control of the car and spinning his / her car around in a circle. In this embodiment, control circuitry 420 or 428 may be configured to determine whether a user can drive his / her virtual car in the virtual car racing game above the difficulty threshold of 70 mph without hitting a wall or spinning around uncontrollably.

[0198] In another embodiment, a predetermined level of difficulty may involve traversing a path within a threshold number of attempts. For example, in a virtual game, a path between checkpoints C1 and C2 may be designed at a difficulty level such that it may not be possible for a user to successfully reach the end of the path in fewer than three attempts. In this embodiment, control circuitry 420 or 428 may be configured to determine whether the user can reach the end of the path in fewer attempts, such as one or two attempts.

[0199] In yet another embodiment, the predetermined difficulty level may be designed to require the user to solve a certain challenge in order to uncover a certain number of assets that are hidden from the user in order for the hidden assets to be revealed. The difficulty level in this embodiment may be designed such that the user is unable to discover all the assets without being killed at least once or within a certain predetermined amount of time. In this embodiment, the control circuitry 420 or 428 may be configured to determine whether the user is able to discover more than a threshold number of assets or pass the level before the expected time.

[0200] In block 1315, the control circuitry 420 or 428 may monitor the user's navigation through the checkpoints C1 and C2 of the virtual game. The monitoring may be performed by a camera, a sensor, a heart rate monitor, or any other wearable device connected to the extended reality headset via an API. For example, as mentioned in some of the above examples, the control circuitry 420 or 428 may monitor the user's navigation to determine whether the user is able to successfully complete an obstacle or challenge or reach the end of a level or path in fewer attempts or in less time than designed for the difficulty level. The control circuitry 420 or 428 may then compare the user's performance with the difficulty level set by the control circuitry 420 or 428 at that level of the game and determine whether the user was able to easily beat the difficulty level.

[0201] At block 1325, in one embodiment, a determination is made that A) the user is able to successfully reach checkpoint C2 and that the user is able to do so within a predetermined difficulty level threshold. In other words, the user performs as expected or at the difficulty level for which the difficulty level was designed, e.g., is able to reach the end of the path in four attempts when the difficulty level was designed such that the user would need at least three attempts to reach the end of the path. In another example, the user is able to complete the difficulty level in three minutes and the level was designed such that the user would not be able to reach checkpoint C2 in less than two minutes. In response to such a determination, at block 1330, the system may continue to monitor the user through the remainder of the game, such as through checkpoints C2 and C3, to determine whether the user meets or exceeds the difficulty level set for the path between those two checkpoints.

[0202] At block 1325, in one embodiment, a determination is made that B) the user may succeed in reaching checkpoint 2 before reaching the predefined difficulty level threshold. In other words, the user may easily surpass the difficulty level set for this stage, such as by beating the level in a shorter amount of time or with fewer attempts than the game level was designed for, i.e., before reaching the threshold level. In response to such a determination, at block 1335, the system may either automatically increase the game difficulty level or provide a suggestion to the game designer to increase the difficulty level for future users.

[0203] In block 1325, in one embodiment, a determination is made that C) the user is able to reach checkpoint C2 after a predetermined difficulty level threshold. In other words, the user is not able to easily surpass the difficulty level set for this stage, e.g., the user takes too much time or too many attempts to reach checkpoint C2. In response to such a determination, in block 1340, the system may automatically decrease the level of game difficulty. It may also suggest to the game designer to decrease the level of difficulty and provide data on the time or number of attempts the current user has gone through to complete the difficulty level.

[0204] Referring now to FIG. 13B, the figure depicts a flowchart of a process for comparing a current user's performance to a median user's performance based on a predetermined difficulty level in a virtual game, according to some embodiments of the present disclosure.

[0205] In some embodiments, the control circuitry 420 or 428 determines the type of augmentations and modifications to implement for the virtual game or experience based on how the current user is performing in the virtual game or experience. For purposes of illustration, a virtual game will be referenced, however, the process of FIG. 13A and 13B applies to both virtual games and any other type of virtual reality or extended reality experience. The control circuitry 420 or 428 generates a virtual game on a display of an electronic device, such as an extended reality headset, and predetermines a level of difficulty level for a portion of the virtual game. The control circuitry 420 or 428 collects performance data from multiple users who played a portion of the virtual game. For example, the performance data may include scores, levels cleared, obstacles overcome, objects interacted with, time taken to accomplish a task or game level, and other game-related metrics that can be used to determine how the player performed and whether the player performed well during a segment of the game. This performance data relates to the performance of multiple users based on a predetermined difficulty level. For example, the number of points scored by multiple users at a given difficulty level, whether a user performed well or exceeded an expected performance range by scoring more than the content creator could have anticipated for that difficulty level, etc. The control circuitry 420 or 428 calculates a median performance based on the performance data collected from multiple users and compares it to the performance data of the current user. In response to determining that the performance data of the current user is not within a threshold of the median performance data, the control circuitry 420 or 428 performs enhancements to the virtual game. These enhancements may include maintaining the same difficulty level, or increasing or decreasing the difficulty level. To implement these embodiments, the control circuitry 420 or 428 executes the process of FIG. 13B as described below.

[0206] In some embodiments, in block 1310, the control circuitry 420 or 428 may generate a virtual game environment with a predetermined level of difficulty between checkpoint 1 and checkpoint 2 of the virtual game. The predetermined level of difficulty is associated with certain types of obstacles or challenges that a player of the game must experience in order to score or reach the next level of the virtual game. The predetermined level of difficulty may be designed by the content creator or the control circuitry 420 or 428 such that the player experiences certain hardships to make the game challenging, for example, a user may need to spend a certain amount of time, take a certain amount of steps, lose a certain amount of lives, take a certain path that is not easily discovered, etc., to successfully pass a game level with a predetermined level of difficulty.

[0207] One example of a predetermined level of difficulty in a game includes making it difficult to complete or pass a gaming level within a certain period of time, such as in a car racing game where the curves are designed with sharp angles, thus requiring the player to slow down on the curves and thereby unable to reach the finish line within a certain amount of time. In yet another example, the game may require a certain number of attempts by the player to reach the end. In another example, the player may need to discover a certain number of assets / objects that are hidden from the player, which may be revealed after the player successfully navigates to a certain area in the game.

[0208] Whatever challenges or obstacles may be designed into the game to induce a certain level of difficulty, the control circuitry 420 or 428 obtains performance data, such as performance metrics, from a number of users who played the game between checkpoints 1 and 2 with the predetermined level of difficulty in block 1345. By aggregating the performance data, the control circuitry 420 or 428 may determine whether a number of users experienced the predetermined level of difficulty designed into the game to be experienced by the users.

[0209] In one embodiment, the control circuitry 420 or 428 may include a threshold value for a predetermined level of difficulty. For example, if the predetermined level of difficulty for a player in a car racing game is to finish the race at an average speed of 75 MPH, the threshold value may be set at + / - 15 MPH, which results in a threshold or performance window of lower performance at 60 MPH and higher performance at 90 MPH. In other embodiments, the threshold value may relate to a time or number of attempts. For example, if a user should navigate from checkpoint 1 to 2 after five attempts, the threshold value may be set at + / - 2 attempts.

[0210] At block 1350, in one embodiment, the control circuitry 420 or 428 calculates a median performance based on performance data, such as a performance metric, obtained for the multiple users. The median performance data may be a representative value of all the multiple users, an average value of all the multiple users, a standard deviation of all the multiple users, or some other equation or statistical measure for calculating the median performance.

[0211] At block 1355, a determination is made whether a median user, which in one embodiment is a hypothetical user determined based on calculating a median or other type of statistical data from a plurality of users, would be able to successfully reach checkpoint C2 and would be able to do so within the threshold of the given difficulty level. In other words, would the median user be able to perform at the difficulty level as expected for which the difficulty level was designed. From the above example, in a car racing game where the speed of a given difficulty level is set to have an average speed of 75 MPH throughout the game with a threshold of + / - 15 MPH, the determination by the control circuitry would be to determine whether the median user was within + / - 15 MPH of 70 MPH, in other words, between a lower performance at 60 MPH and a higher performance at 90 MPH.

[0212] If a determination is made that the median user has completed the task, i.e., reached checkpoint 2, or in the car racing example, reached the end of the race outside the threshold, then in block 1365 the difficulty threshold may be increased or decreased based on the performance. In one embodiment, if the performance of the median user falls below 60 MPH, the lower end of the threshold, the control circuitry 420 or 428 may determine that the current predetermined difficulty level is too high for the median user and needs to be decreased. In the car racing example, the control circuitry 420 or 428 may decrease the difficulty level of a 75 MPH average speed to 65 MPH and provide some other threshold, such as a + / - 15 threshold or + / - 20, and adjust the difficulty level, thereby making it easier for the median user to reach the game level. In other embodiments, decreasing the difficulty level may be implemented by blocking certain paths in the virtual game that have a higher difficulty level. In another embodiment, if the median user's performance falls above the upper threshold of 90 MPH, the control circuitry 420 or 428 may determine that the current predetermined difficulty level is low for the median user and needs to be increased. In the car racing example, the control circuitry 420 or 428 may increase the difficulty level for a 75 MPH average speed to 100 MPH and provide some other threshold, such as a + / - 10 threshold or + / - 5, and adjust the difficulty level, thereby making it more difficult for the median user to reach the game level.

[0213] If a determination is made that the median user has completed the task, i.e., reached checkpoint 2 or, in the car racing example, reached the end of the race within the predetermined threshold of 75 MPH, then the difficulty threshold may be maintained in block 1365. This is because control circuitry 420 or 428 may determine that the currently set predetermined threshold is the correct degree of difficulty for the median user.

[0214] In block 1370, the current user's performance is compared to the updated difficulty level in block 1365 or the maintained difficulty level in block 1360, based on the decision made in block 1355. Blocks 1375, 1385, and 1380 follow a similar process as blocks 1355, 1365, and 1360 to determine whether to increase, decrease, or maintain the same difficulty level, but based on the comparison to the updated difficulty level in block 1365 or the maintained difficulty level in block 1360, i.e., how the median user would have performed.

[0215] In one embodiment, as described through blocks 1345-1385, the control circuitry 420 or 428 may perform at least two iterations of checks on the predefined level to determine whether it should be increased, decreased, or maintained. The first iteration may be a check based on how the median user performs based on the predefined difficulty level, and if the level needs to be adjusted for the median user, the second iteration may be for the specific current user to determine whether further adjustments from those made for the median user are needed to customize for the current user. In addition to the above, statistics specific to the current user, such as those depicted in FIG. 8, may also be considered in further adjusting the predefined difficulty level. Other factors, such as whether the current user has played similar games or is an expert in the field of such games, may also be factored in to adjust the difficulty level. For example, if the control circuitry 420 or 428 determines that the user is an expert in such games, the difficulty level may be increased to make it more challenging for the current user.

[0216] In another embodiment, the control circuitry 420 or 428 may manage motion sickness in an extended reality experience by generating the extended reality experience on a display of an electronic device, such as an extended reality headset. The control circuitry 420 or 428 may determine a first checkpoint and a second checkpoint in the extended reality experience, such as the checkpoints depicted in FIG. 3. The control circuitry 420 or 428 may determine a median path (or some other representative path) based on data from a plurality of users who navigated between the first and second checkpoints. This may be done by calculating a representative value, an average, or some other calculation based on an equation or some other statistical measure to obtain the median data. The control circuitry 420 or 428 may determine a location in the median path where a threshold number of users from the plurality of users experienced motion sickness. For example, the median path may include a number of location points along the path, such as locations 1-n, as the users navigate from the start checkpoint to the end checkpoint. The control circuitry 420 or 428 may determine whether the route taken by the current user is aligned with the median route. For example, the control circuitry 420 or 428 may determine whether the current route is along the same location points as routes selected by multiple users.

[0217] In response to determining that the current user's route is aligned with the median route, the control circuitry 420 or 428 may then take remedial action prior to the user reaching a location in the median route where a threshold number of users experienced motion sickness. For example, if a route has location points 1, 2, 3, 4, and 5, and the median user experienced motion sickness at location point 4, such as due to a descent on a roller coaster, such data may be logged by the control circuitry 420 or 428.

[0218] Control circuitry 420 or 428 may determine that the current user is also taking the route taken by the median user. It may also determine the distance traveled by the median user, e.g., the distance the median user could have traveled thus far by traversing location points 1, 2, and 3. Control circuitry 420 or 428, determining that the current user is on the same route and that location point 4 is a location where motion sickness is experienced, may proactively implement remedial action prior to the user reaching location point 4, as a preventative action, so that motion sickness may be minimized or avoided.

[0219] In some embodiments, the control circuitry 420 or 428 may determine whether the path taken by the current user is aligned with the median path by determining a first set of coordinates of the median path and then comparing the first set of coordinates to a current set of coordinates associated with the current path taken by the user. If the control circuitry 420 or 428 makes a determination that the current set of coordinates is within a threshold distance of the first set of coordinates, the control circuitry 420 or 428 may take remedial action. The threshold distance may be set by a content creator of the control circuitry 420 or 428.

[0220] FIG. 14 is a navigation tree depicting routes taken by multiple users between given checkpoints in a virtual game, according to some embodiments of the present disclosure.

[0221] The median path followed by the user is represented in x, y, z coordinates. The median path may also include orientation in addition to translations of the x, y, and z coordinates, however, for simplicity and illustration only the x, y, z coordinates are depicted.

[0222] In some embodiments, the user's path between checkpoints C1 and C2 is determined using a general tree. For example, between checkpoints C1 and C2, the difference between two successive records may represent a state change of the user's position in the 6DOF virtual experience. Starting from a root node initialized to x, y, and z coordinates of 0, 0, 0, i.e., the initial position or origin where the virtual experience begins, the records are checked against the children of the root node. In some embodiments, if a child representing a state change in x, y, and z already exists, this is selected as the next starting point, and a count representing the traversal of the edge between the root node and its child is incremented. If a child representing a state change in x, y, and z does not exist, a new child node is created and the edge is attributed a count of 1. This process is performed for each new record until checkpoint C2 is reached.

[0223] In one embodiment, a path tree may be generated based on navigation between checkpoints C1 and C2 by 12 users. The leaf node with the highest traversal count represents the final state change of the median path of the users. Traversing the tree from the root node to the leaf node with the highest traversal count determines the median path taken by users in the experience.

[0224] The leaf node traversal count (-1,0,0) is the highest at n=5. Therefore, the median path for a user is modeled as a state change of (-3,-2,0) followed by a state change of (-1,0,0). Hence, the total state change is the sum of the two resulting in a move to (-4,-2,0).

[0225] Although discrete changes in x, y, and z coordinates are depicted, in reality each discrete state change, i.e., node in a typical tree, may include ranges / errors of movement in the x, y, and z directions. For example, errors of x±6x, y±6y, and z±6z may be experienced, effectively creating "areas" of user movement (like a chessboard).

[0226] FIG. 15 is a block diagram depicting a user's field of view and objects / assets encountered within the field of view, according to some embodiments of the present disclosure. In some embodiments, the field of view (FOV) from the user's eyes is depicted. In some embodiments, content creators must observe the patterns of the user's FOV to build a virtual experience with a good level of user engagement. By determining and monitoring the user's FOV, content creators can obtain data regarding objects appearing within the user's FOV. Such FOV data provides insight into whether the content creator's intended design matches the user's observations and actions, since a well-designed extended reality experience is one in which the user spends time, feels engaged, and interacts with objects within their FOV.

[0227] In one embodiment, the user's FOV is depicted as including the entire projection surface from the left vanishing point to the right vanishing point. The vanishing point 1540 is where the user's field of view is smallest, and the center of the pyramid of view 1530 is where the field of view is largest and toward which the user's eyes may be directed. As depicted, in this example, the user's FOV includes an object 1510. The control circuitry may determine that the user's gaze is directed toward the object 1510 if the pyramid of view, which is a narrower section of the user's eye than the entire FOV, is directed toward the object. To make such a determination, in one embodiment, the control circuitry may utilize an inward-facing camera associated with the extended reality headset to monitor the user's eye movements. In another embodiment, the control circuitry may determine the orientation and coordinates of the extended reality headset and then calculate the pyramid of view, thereby determining whether the user's gaze is directed toward the object 1510.

[0228] In another embodiment, in addition to the user's pyramid of view 1530 being oriented in a particular direction, the control circuitry may also determine whether the user is viewing the near field of view 1545 or the far field of view 1550. Such data may enable the control circuitry to determine the depth of the user's view and whether the user is viewing closer or more distant objects within the pyramid of view. Such type of gaze tracking, in conjunction with position and rotation vectors, may reduce the set of assets in the view and thus improve the information available to the user and content creators. For example, content creators may use gaze information to determine whether to remove clutter by removing objects in the background that are not relevant to the game or virtual experience and are causing confusion or lack of focus to the user.

[0229] In some embodiments, the FOV is determined based on the coordinates of the headset. The control circuitry may obtain the coordinates of the user on the path and use them to determine the orientation of the extended reality headset. The orientation determination may enable the control circuitry to determine objects that may be within the FOV. The control circuitry may further determine whether the user has interacted with an object within the FOV.

[0230] FIG. 16 is a block diagram of a room layout and multiple paths taken to reach a destination, according to some embodiments of the present disclosure. In some embodiments, a user may use a virtual reality headset or glasses when navigating within the room to reach a destination. As depicted, the layout in one embodiment includes a couch 1610, chairs 1620, 1625, 1630 and other chairs, a table 1635, a lamp 1640 and other objects such as another lamp, and multiple bookshelves. The diagram also depicts a path 1650 taken by a first user and a path 1660 taken by a second user. The diagram further depicts certain locations along the path, such as locations 1670, 1680, and 1690 through which the users pass. Checkpoint C1 may be the starting location of the path, and the goal of the virtual experience, in one embodiment, may be to reach checkpoint C2.

[0231] Different users may take different paths from checkpoint C1 to checkpoint C2, so that the users may spend different amounts of time between the two checkpoints and encounter different challenges. For example, one user may choose an easier or shorter path than another user, one user may lose a life while the other may pass through without losing a life, or one user may encounter more assets than another user and be able to interact with them based on the path selected. Different users may also have different experiences in discovering assets in a game. Some users may discover assets that are important to clearing a level faster or with fewer attempts than others. Some users may enjoy the experience more and choose to stay in their surroundings longer to explore them. If the experience is a game, one user may get more points based on the path selected and the number of assets seen and interacted with, compared to a user who did not interact with as many assets. Data collected for each route and by different users may be used by content creators to enhance future virtual experiences, adjust difficulty levels, manage motion sickness, or determine home automation options.

[0232] In one embodiment, as depicted, the path navigated by User 1 differs from the path navigated by User 2. The path navigated by User 1 passes through location 1690 and reaches a final destination at checkpoint C2 where ramp 1640 is located. As User 1 navigates along path 1, at each location along the path, the user's FOV allows the user to view different assets, such as a table, chair, lamp, and bookshelf, based on the orientation of the extended reality headset User 1 is wearing.

[0233] In one embodiment, at location 1690, the extended reality headset of user 1 may be oriented in the direction of the arrow. Based on the orientation, the user's FOV 1655 at location 1690 may enable user 1 to see table 1635 and chair 1625 through his headset. User 1 may also partially see chair 1630 within his FOV. All other assets, including chair 1620, may not be visible to user 1 at location 1690 when the headset orientation is pointing towards the arrow. Such data may be captured by control circuitry 420 or 428 and used for various purposes, such as enhancing future virtual experiences, adjusting difficulty levels, managing motion sickness, determining home automation options, and for other use cases.

[0234] Based on the orientation in the direction of the arrow at location 1690, user 1 may be able to interact with assets 1635, 1625, and 1630. If the extended reality experience is a virtual game, the user may score based on the user's interactions with these assets. Data related to all user interactions with the assets, including assets viewed and interacted with, assets viewed but not interacted with, and extended reality display coordinates, translation, and orientation, may be captured by the control circuitry as depicted in the examples of Figures 18 and 19. Data related to the time spent at each asset and the time taken to navigate from one location to another along path 1 may also be captured by control circuitry 420 or 428.

[0235] The path navigated by user 2 passes through locations 1670 and 1680 to reach a final destination at checkpoint C2 where ramp 1640 is located. Locations 1670, 1680, and 1690 were chosen arbitrarily along the paths of users 1 and 2 to describe the users' FOV and the assets that may potentially be interacted with. Any other location along the path may have been chosen instead because of a similar description of the FOV.

[0236] As User 2 navigates along Route 2, at each location along the route, the user's FOV allows the user to see different assets, such as tables, chairs, lamps, and bookshelves, based on the orientation of the extended reality headset that User 2 is wearing.

[0237] In one embodiment, at location 1670, the extended reality headset of User 2 may be oriented in the direction of the arrow. Based on the orientation, the FOV 1665 of the user at location 1670 may allow User 2 to view Bookshelf 2 and partially view Bookshelf 3, thereby creating an opportunity for User 2 to interact with them. All other assets, including Bookshelf 1, may not be visible to User 2 at location 1670 when the headset orientation is pointing towards the arrow.

[0238] As User 2 progresses along a path selected by User 2 in the extended reality experience, in one embodiment, at location 1680, User 2's extended reality headset may be oriented in the direction of the arrow, which may enable User 2 to view only bookshelf 2 within his or her FOV. Thus, User 2 in this orientation may have the opportunity to interact only with bookshelf 2 (if the user so chooses).

[0239] User 2's navigation data, including what User 2 saw within his or her FOV at every location and the opportunities that were available to User 2 based on what the user saw within his or her FOV, may be captured by control circuitry 420 or 428 and used for a variety of purposes, such as enhancing future virtual experiences, adjusting difficulty levels, managing motion sickness, determining home automation options, and for other use cases.

[0240] In some embodiments, User 2 may have taken a shorter path than User 1, however, User 1 may have had the opportunity to interact with more assets than User 2. If asset interactions are captured solely based on selected location points 1670, 1680, and 1690, in one embodiment, User 1 may have interacted with more assets (three chairs and a table), while User 2 may have interacted with only two items (bookshelves 2 and 3). Depending on how the content creator designed the extended reality game, if equal points were allocated per interacted asset, User 1 may obtain more points than User 2 based on his number of interactions. However, if a greater number of points were allocated for bookshelves than other objects, in one embodiment, User 2, who interacted with the bookshelves, may obtain more points than User 1, who selected a path where the user did not view or interact with any bookshelves. In another embodiment, if the length of the path or the time it takes to get from checkpoint C1 to C2 is given a higher value, user 2 may earn more points than user 1 for taking a shorter path and reaching the destination in a shorter amount of time. In another embodiment, each step along the path may have a different FOV, and the control circuitry may obtain data related to the assets within the FOV along each step, determine which assets were in view and which assets were interacted with, and generate tables such as those shown in Figures 18 and 19.

[0241] In one embodiment, user navigation and paths may be periodically or continuously monitored by control circuitry 420 or 428. Monitoring may be performed for any and all movements and orientations of the extended reality headset.

[0242] In another embodiment, the control circuitry 420 or 428 may modify the monitoring and logging time period based on scene complexity or interaction opportunity (based on path, orientation, assets in view, interaction opportunity, etc.). The control circuitry 420 or 428 may also log data more frequently when scene complexity is high and less frequently when it is low.

[0243] In yet another embodiment, path monitoring and data logging may be done periodically at some predefined place or after some time frame to manage CPU and GPU utilization so that they are not overloaded. The amount of data collected may also be capped by the allocated amount of memory space to be used within the CPU or GPU.

[0244] In addition to route data, the control circuitry 420 or 428 may also monitor and log data related to motion sickness, home automation opportunities, difficulty status, etc. The control circuitry 420 or 428 may also log data such as heart rate, ECG, and other user vitals to gain more insight into the content. The control circuitry 420 or 428 may also track the user's gaze, along with position and rotation vectors, to determine (and in some cases reduce) the set of assets in view. Examples of tables of logged data are depicted in FIGS. 18 and 19.

[0245] Once the median path taken by the user, the median orientation for each point along that path, the assets in view, and the CPU / GPU utilization have been determined, the control circuitry 420 or 428 may use the collected and logged data to optimize the content. Often, when there are assets rendered with very high detail (e.g., millions of polygons) that are not within the user's FOV, over a period of time the control circuitry 420 or 428 may observe peak GPU utilization and take appropriate action to reduce the level of detail of the high detail assets or remove them entirely.

[0246] In some embodiments, the control circuitry 420 or 428 may determine that CPU or GPU utilization is higher than expected or above a predetermined threshold. Having utilization at or above such a threshold may be an indication that the GPU may not be able to deliver rendered frames at a rate required to maintain a consistent frames per second in the experience, causing the GPU to skip frames, thus resulting in a negative impact on the user experience. In such a situation, the control circuitry 420 or 428 may reduce the complexity, detail, resolution, and / or number of polygons used to avoid over-exerting the GPU.

[0247] In some embodiments, in response to the control circuitry 420 or 428 determining that the CPU or GPU utilization is higher than expected or above a predetermined threshold, the control circuitry 420 or 428 may partially or completely alter the spatial arrangement of the assets or layout. For example, in FIG. 16, the control circuitry 420 or 428 may remove an additional number of chairs that were not viewed or interacted with, or relocate them to another side of the virtual room. The control circuitry 420 or 428 may also reduce the amount of detail of a lamp on the left side of the room that was not viewed or interacted with, or move its spatial layout closer to the median path so that it has an increased chance of being viewed and interacted with. Some additional layout changes that may be implemented are described below in FIG. 17.

[0248] In some embodiments, in response to the control circuitry 420 or 428 determining that CPU or GPU utilization is higher than expected or above a predetermined threshold, the control circuitry 420 or 428 may reduce the number of times data is collected and logged and collect and log data periodically, at timed intervals, or less frequently when a scene is changed or a milestone is achieved.

[0249] FIG. 17 is a block diagram of a layout recommendation according to some embodiments of the present disclosure.

[0250] In some embodiments, at block 1705, the control circuitry 420 or 428 modifies the layout in the extended reality experience by increasing the difficulty level. As mentioned above, one of the reasons for obtaining data from multiple users as they navigate from checkpoint C1 to C2 is to determine whether the multiple users experienced the level of difficulty as designed and expected by the creator of the extended reality experience, such as in a VR game. If a determination is made by the control circuitry 420 or 428 that the multiple users overcame the challenges and obstacles designed by the content creator and reached checkpoint C2 in less than a threshold time designed to experience a certain difficulty level, or scored more than designed, the control circuitry 420 or 428 may increase the difficulty level. Having the multiple users overcome the difficulty of the game as designed may be an indication that the difficulty level is easy for the users to get through, and therefore the control circuitry 420 or 428 may increase the difficulty level for a more challenging experience.

[0251] In some embodiments, in block 1710, the control circuitry 420 or 428 alters the layout in the extended reality experience by decreasing the difficulty level. As mentioned above, one of the reasons for obtaining data such as performance metrics from multiple users as they navigate from checkpoint C1 to C2 is to determine whether the multiple users experienced the level of difficulty as designed and expected by the creator of the extended reality experience, such as in a VR game. If a determination is made by the control circuitry 420 or 428 that multiple users had difficulty overcoming the challenges and obstacles designed by the content creator and reached checkpoint C2 in more than a threshold time designed to experience a certain difficulty level, or scored less than designed, or if some users abandoned the game due to the level of difficulty, the control circuitry 420 or 428 may decrease the difficulty level so that users may reach checkpoint C2 with less difficulty.

[0252] In some embodiments, at block 1715, the control circuitry 420 or 428 modifies the layout in the extended reality experience by modifying assets in the layout, such as by adding or removing assets. For example, the control circuitry 420 or 428 may determine that an asset was not of interest to multiple users and therefore the users did not interact with it. In another example, the control circuitry 420 or 428 may determine that the additional asset adds more processing for the CPU or GPU and is not essential to the extended reality experience. In yet another example, the control circuitry 420 or 428 may determine that adding more assets adds a higher level of difficulty. Whatever the reason for adding or removing assets, the control circuitry 420 or 428 may do so based on performance metric data collected from multiple users as they navigate from checkpoint C1 to C2.

[0253] Similar to adding or removing assets, the control circuitry may add or remove obstacles in block 1720. One of the reasons for adding or removing obstacles may be to increase or decrease the difficulty level. For example, more obstacles may be added if the control circuitry 420 or 428 determines that multiple users have solved the current challenge or bypassed the obstacle in less than a threshold time designed for solving such challenge or bypassing such obstacle. In another embodiment, an obstacle may be removed if the control circuitry 420 or 428 determines that multiple users have not solved the current challenge or bypassed the current obstacle, or have done so in an amount of time that exceeds a predetermined threshold time designed for solving such challenge or bypassing such obstacle. An obstacle may also be removed if a user has abandoned the game and has not solved any of the bypassed obstacles.

[0254] In some embodiments, at block 1725, the control circuitry 420 or 428 may provide a translational or directional cue to the user instead of or in addition to changing the layout in the extended reality experience. As depicted in FIG. 15, an arrow 1520 may direct the user to orient toward an object if the user has not been able to figure out where the object is located within a threshold amount of time. Having a translational cue may also be an option that the control circuitry 420 or 428 may activate if it determines that the user is about to abandon the game or is heading in the wrong direction.

[0255] In some embodiments, in block 1730, the control circuitry 420 or 428 alters the layout in the extended reality experience by modifying possible paths that users could have taken to reach the ending checkpoint C2. For example, if a determination is made by the control circuitry 420 or 428 that multiple users are traveling down a path that is an incorrect path or a path that would cause the users to exceed a threshold amount of time in which to reach checkpoint C2, the control circuitry 420 or 428 may block the incorrect path, create another detour in the path that directs the users back to the correct path, or provide an indication that the path is an incorrect path.

[0256] In some embodiments, in block 1735, the control circuitry 420 or 428 may display augmentations, including coloring, highlighting, animating, increasing size, or any other augmentation that would draw attention to objects that should be found by multiple users. Although several layout changes and path modifications and augmentations are discussed in FIG. 17, embodiments are not so limited and other augmentations that change the layout or draw attention to particular areas in the extended reality experience are also contemplated.

[0257] FIG. 18 is a table of user navigation coordinates of user interactions as the user navigates from checkpoint C1 to checkpoint C2, according to some embodiments of the present disclosure.

[0258] In one embodiment, control circuitry 420 or 428 may collect data based on multiple users' navigation from checkpoint C1 to checkpoint C2. Such data may be aggregated and averages calculated on the data, which may be used to compare with the current user's data and determine the type of extended reality experience modification and enhancement needed to enhance the current user's experience and make it more enjoyable.

[0259] In one embodiment, control circuitry 420 or 428 may use collected data based on multiple users' navigation from checkpoint C1 to checkpoint C2 to generate table 1800. Although several columns of categories are listed in table 1800, embodiments are not so limited and other columns reflecting data relating to other measurements and indications may also be populated in the table. For example, data related to acceleration, motion sickness, etc. may also be populated in table 1800.

[0260] In some embodiments, the data contains the position and orientation of system components, such as the head mounted display (HMD) (which is the extended reality device used by the user), left and right hand controllers (such as virtual reality or extended reality gaming controllers), assets in the view (such as the table and chair in FIG. 16), and a history of user activity at each recorded point in the simulation. It also includes CPU and GPU utilization based on the path taken by the user, and the amount of CPU and GPU usage required to render images and process the user's steps. In other embodiments, instead of recording data at each point, which can consume excessive memory and CPU / GPU usage, the control circuitry 420 or 428 may reduce the number of times data is collected and logged, and collect and log data periodically, at timed intervals, or less frequently when a scene is changed or a milestone is achieved.

[0261] For example, in a virtual experience, a 3D scene containing a number of boxes (some of which are "flying" boxes) may be displayed as part of a VR game. These flying boxes may appear transiently. In addition, control circuitry 420 or 428 may provide the user with the ability to open and close any of the boxes using point-and-click actions on a virtual reality or extended reality controller. When the user performs an action related to the flying boxes, such as pointing and clicking with their mouse, control circuitry 420 or 428 may extract a table from each log file, such as table 1800.

[0262] Control circuitry 420 or 428 may then aggregate logs received from multiple users, thereby providing the content creator with a global view of user behavior in the extended reality experience. As mentioned above, the global view may be used to manage a user's motion sickness, implement home automation, or measure a user's performance through the designed difficulty levels of the experience.

[0263] On a granular basis, in one embodiment, the control circuitry 420 or 428 may use aggregated data that provides a comprehensive view and determine a median path for a user. As mentioned above, such a median path is calculated based on a representative or average value for data obtained from multiple users and their taken paths.

[0264] In another embodiment, the control circuitry 420 or 428 may use the aggregated data to determine a median orientation (qw, qx, qy, qz) of the user at each point (or several points) on the median path. The orientation enables the control circuitry 420 or 428 to determine the direction the extended reality device (HMD) is pointing and, based on the pointing direction, determine the assets that are within the user's FOV.

[0265] In yet another embodiment, the control circuitry 420 or 428 may use aggregated data to determine which assets were viewed by a user and the duration of such viewing. For example, in some cases, a user may only view an asset for a short duration and not interact with the object, while in other cases, a user may spend a lot of time with an asset that ultimately leads the user to interact with the asset. The control circuitry may determine a representative value based on multiple users and determine whether an asset is in the representative value that is of interest to the user.

[0266] In another embodiment, the control circuitry 420 or 428 may use the aggregated data to determine whether multiple users have taken advantage of an opportunity to interact with an asset that is programmed to be interacted with. If the assets have not been interacted with, the control circuitry 420 or 428 may displace them to a different spatial location, remove them from the game, or reduce the amount of detail, such as polygons, on the assets.

[0267] In another embodiment, the control circuitry 420 or 428 may use the aggregated data to determine large changes in x, y, z (high translational movements) across successive time periods. Determining such large changes may enable the content provider to understand whether large translational movements, implying faster movements or jumps from one place to another, are leading to motion sickness in the user. The content provider may use such data to determine whether any enhancements related to motion sickness are required. Similarly, large changes in qw, qx, qy, qz (high angular movements) across successive time periods may also be monitored, recorded, and used to determine any actions required. In addition, data related to the time rate of positional and angular movements due to large movements may enable the content creator to understand where they can reduce events occurring in the user experience to control abnormal / high movements in the game.

[0268] In yet another embodiment, the control circuitry 420 or 428 may use the aggregated data to determine the time taken to complete a set of tasks or a sequence of tasks or the time taken between completing a task. Determining such data related to task completion times may enable the control circuitry to determine whether the user is accomplishing tasks with less difficulty than the level is designed for, or whether the difficulty is too complex for the user, leading to the user spending more than a threshold time or abandoning the game.

[0269] In some embodiments, the time taken to complete a set of tasks or a sequence of tasks may represent the total time spent between checkpoints C1 and C2. When the user reaches checkpoint C2, a time-stamped entry may be registered to indicate the accomplishment. If the user exits the experience before reaching checkpoint C2, the absence of this entry in the log file may be populated, and such an entry may indicate that the user did not complete the tasks to reach checkpoint C2. Aggregation of such data may be used to assess and adjust the difficulty level in the experience.

[0270] Similarly, data relating to the time taken between completing two individual tasks from an interaction / user activity sequence may be obtained and logged. In some embodiments, if after performing a first task, i.e., "Open (Box-7)," the user receives a cue that directs the user to perform a second task, i.e., "Open (Box-11)," the control circuitry may use such data to determine the number of user sessions overall that were successful in interpreting the cue. The timestamps recorded for the user actions may be used by the control circuitry to determine the time between completing the two tasks.

[0271] FIG. 19 is a table of user interactions and missed opportunities according to some embodiments of the present disclosure.

[0272] In one embodiment, at time t=T, the assets in the user's FOV may be Box-1, Box-2, Box-3, Box-4, and Flying Box-1. The interaction opportunities provided to the user in the game may be Open (Box-1), Open (Box-2), Open (Box-3), Open (Box-4), and Open (Flying Box-1). However, as depicted in the third column, "Interaction / User Activity," the user had zero opportunities in the first time interval.

[0273] In another embodiment, at time t=2T, the user has taken 1 of 5 possible opportunities in a second time interval, and in yet another embodiment, at time t=3T, the user has taken 3 of 5 opportunities in a third time interval. Such aggregated data may be used to make inferences regarding whether users are taking advantage of the opportunities offered, and, if applicable, which opportunities are and are not being taken advantage of.

[0274] FIG. 20 is a table of content refinement suggestions based on a user's navigation in an extended reality experience, according to some embodiments of the disclosure. In this embodiment, the control circuitry 420 or 428 provides content refinement suggestions and / or suggested actions based on metrics or data obtained from multiple users. Some exemplary content refinement suggestions include the control circuitry suggesting that the control circuitry redirect the user toward or highlight an object to allow the user to interact with it, or perform a desired action or train to perform a behavior. Some exemplary suggested actions include suggesting that the control circuitry display an arrow, highlight an object with a thick / contrasting color border to direct the user to an asset or other interaction opportunity, change the spatial layout / map of the experience, or provide behavior training text and 3D visualization.

[0275] FIG. 21 is a block diagram depicting locally rendered and cloud rendered extended reality experiences according to some embodiments of the disclosure.

[0276] Where data is logged and the mechanism for uploading the data to the content creator may depend on the system architecture. In some embodiments, in a cloud-rendered VR experience, data may be logged directly at the cloud server that receives position and orientation data to render frames for the extended reality headset. In other embodiments, during a locally rendered extended reality experience, data may be logged within the local computer or extended reality device and later uploaded to a system such as the system depicted in FIG. 4. Storage on a remote server or attachment to the extended reality device is also contemplated in embodiments.

[0277] It will be apparent to those skilled in the art that the methods involved in the embodiments described above can be embodied in a computer program product that includes a computer usable and / or readable medium. For example, such a computer usable medium may consist of a read-only memory device, such as a CD-ROM disk or a conventional ROM device, or a random access memory, such as a hard drive device or a computer diskette, having computer readable program code stored thereon. It should also be understood that the methods, techniques, and processes involved in the present disclosure can be implemented using processing circuitry.

[0278] The processes discussed above are intended to be illustrative and not limiting. Only the following claims are meant to set boundaries on what the invention encompasses. Furthermore, it should be noted that the features and limitations described in any one embodiment may be applied to any other embodiment herein, and that the flow charts or examples related to one embodiment may be combined with any other embodiment in a suitable manner, performed in a different order, or performed in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and / or methods described above may be applied to or used in accordance with other systems and / or methods.

Claims

1. A method for enhancing the virtual reality experience, The process involves generating a virtual game on the display of an electronic device, wherein a predetermined difficulty level is determined for a portion of the virtual game. The method involves collecting performance data from multiple users who have played the aforementioned part of the virtual game, wherein the performance data relates to the performance of the multiple users based on the predetermined difficulty level. Based on the performance data collected from the aforementioned multiple users, a performance metric is calculated. This involves comparing the current user performance data with the calculated performance metrics, In response to determining that the current user's performance data is not within the threshold of the performance metric, the virtual game is augmented. Methods that include...

2. The method according to claim 1, wherein the threshold of the performance metric includes a lower limit and an upper limit, the lower limit being associated with the minimum number of interactions with virtual objects in the virtual game, and the upper limit being associated with the maximum number of interactions with virtual objects in the virtual game.

3. The determination that the current user's performance data falls below the lower limit of the threshold for the performance metric, In response to the aforementioned decision, the predetermined difficulty level for the portion of the virtual game is modified to a lower predetermined difficulty level. The method according to claim 2, further comprising:

4. The determination that the current user's performance data exceeds a higher limit of the threshold for the performance metric, In response to the aforementioned decision, the predetermined difficulty level of the part of the virtual game is modified to a higher predetermined difficulty level. The method according to claim 2, further comprising:

5. The method according to claim 1, wherein the portion of the virtual game includes a start checkpoint and an end checkpoint, the start checkpoint and the end checkpoint are determined based on the type of the virtual game.

6. The method according to claim 5, further comprising selecting the start checkpoint and the end checkpoint over a shorter time span for faster-paced virtual games and over a relatively longer time span for slower-paced virtual games.

7. Monitoring the path of the current user within the virtual game, Based on the path taken by the current user within the virtual game, the total number of available interaction opportunities is determined. Determining the number of interaction opportunities used by the current user from the total number of available interaction opportunities. The method according to claim 1, further comprising:

8. The method according to claim 7, wherein the performance of the current user is based on the number of interaction opportunities performed by the current user from the total number of interaction opportunities available to the current user.

9. When the user navigates a path within the virtual game, the current user's gaze is monitored, Based on the path navigated by the current user within the virtual game, determine the objects within the current user's field of view (FOV), To determine whether the user has interacted with an object within the FOV. The method according to claim 1, further comprising:

10. It is a method, Creating a virtual game on the display of an electronic device, To determine the difficulty level between the start checkpoint and the end checkpoint for a part of the aforementioned virtual game, The system monitors the user's progress within the virtual game and determines whether the user is a) planning to meet the determined difficulty level, b) planning to be unable to achieve the plan because it is not possible to meet the difficulty level, or c) planning to exceed the difficulty level. In response to the aforementioned decision, maintain, decrease, or increase the level of difficulty. Methods that include...

11. Monitoring the progress of the aforementioned user is Determining the coordinates of the path taken by the user within the virtual game, Based on the aforementioned coordinates, the orientation of the headset worn by the current user to play the virtual game is determined, Based on the determined orientation of the headset, the objects within the field of view of the headset are determined, To determine whether the current user has interacted with the determined object. The method according to claim 10, including the method described in claim 10.

12. A system that enhances the virtual reality experience, A communication network configured to access electronic devices, A control circuit network, The process involves generating a virtual game on the display of the electronic device, wherein a predetermined difficulty level is determined for a portion of the virtual game. The method involves collecting performance data from multiple users who have played the aforementioned part of the virtual game, wherein the performance data relates to the performance of the multiple users based on the predetermined difficulty level. Based on the performance data collected from the aforementioned multiple users, a performance metric is calculated. This involves comparing the current user performance data with the calculated performance metrics, In response to determining that the current user's performance data is not within the threshold of the performance metric, the virtual game is augmented. A control network configured to perform the following: A system that includes these features.

13. The system according to claim 12, wherein the threshold of the performance metric includes a lower limit and an upper limit, the lower limit being associated with the minimum number of interactions with virtual objects in the virtual game, and the upper limit being associated with the maximum number of interactions with virtual objects in the virtual game.

14. The determination that the current user's performance data falls below the lower limit of the threshold for the performance metric, In response to the aforementioned decision, the predetermined difficulty level for the portion of the virtual game is modified to a lower predetermined difficulty level. The control network configured to perform the following actions The system according to claim 13, further comprising:

15. The determination that the current user's performance data exceeds a higher limit of the threshold for the performance metric, In response to the aforementioned decision, the predetermined difficulty level of the part of the virtual game is modified to a higher predetermined difficulty level. The control network configured to perform the following actions The system according to claim 13, further comprising:

16. The system according to claim 12, wherein the enhancement of the virtual game is carried out by the control circuit network, which consists of either increasing or decreasing the predetermined difficulty level.

17. The system according to claim 12, wherein the portion of the virtual game includes a start checkpoint and an end checkpoint, the start checkpoint and the end checkpoint are determined by the control circuit network based on the type of the virtual game.

18. The system according to claim 17, further comprising the control circuit network configured to select the start checkpoint and the end checkpoint in a shorter time span for faster-paced virtual games and in a relatively longer time span for slower-paced virtual games.

19. Monitoring the path of the current user within the virtual game, Based on the path taken by the current user within the virtual game, the total number of available interaction opportunities is determined. Determining the number of interaction opportunities used by the current user from the total number of available interaction opportunities. The system according to claim 12, further comprising the control circuit network configured to perform the following.

20. The system according to claim 19, wherein the performance of the current user is based on the number of interaction opportunities performed by the current user from the total number of interaction opportunities available to the current user.

21. When the user navigates a path within the virtual game, the current user's gaze is monitored, Based on the path navigated by the current user within the virtual game, determine the objects within the current user's field of view (FOV), To determine whether the user has interacted with an object within the FOV. The control network configured to perform the following actions The system according to claim 12, further comprising:

Images