System and method for providing haptic feedback to DHH individuals

Abstract

A system and method for providing haptic feedback in a virtual environment generates virtual instances, such as game levels, and interactions within those virtual instances. The interactions include a first interaction parameter and may be events, such as explosions, within the virtual instance. The system determines the spatial relationship between the player avatar and the interactions and identifies the environment between the player avatar and the interactions. A complete state, including the virtual instances and interactions, is generated. Based at least partially on the complete state, at least one actuator command is generated and sent to a wearable haptic rig. The at least one actuator command is applied to at least one haptic motor of the wearable haptic rig to activate at least one haptic motor and provide haptic feedback to the player.

Description

Technical Field

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 504,579, filed May 26, 2023, entitled "SYSTEMS AND METHODS FOR PROVIDING HAPTIC FEEDBACK TO DHH PERSONS", which is incorporated herein by reference.

[0002] The present disclosure generally relates to haptic feedback devices for DHH individuals.

Background Art

[0003] Current interactive technologies such as video games, movies, and TV shows do not create an immersive experience for deaf and hard-of-hearing (DHH) individuals. To date, subtitles and vibrations on controllers have been used as attempts to bridge the gap in this experience. However, these alternatives cannot replace sound in reproducing an immersive experience. Instead, they use a second medium such as touch to enhance the virtual soundscape.

[0004] In prior art systems, a sound card identified and played the location of sound files related to interactions. In the example of an explosion, a pre-recorded sound file of an explosion was played. To improve immersion, prior art systems calculated how sound would be played based on environmental factors when it reached the player character. For example, when the player character is standing in a stone corridor, the explosion sound can be reverberated to provide a 3D sound experience. Additionally, some prior art systems vibrated the game controller in addition to the reverberating sound to provide a sense of reality.

[0005] Traditional methods have relied on the situational awareness of hearing individuals, built through continuous auditory feedback, to create immersive experiences. These methods, such as head-related transfer functions (HRTFs), replicate an instance of how a hearing person's auditory system receives sound. HRTFs use the frequency and phase responses of an average hearing person (e.g., interaural distance, detectable frequency range) to compute situational awareness. However, HRTFs rely on hearing individuals continuously constructing situational awareness by combining and interpreting analog acoustic signals that clearly reach both ears. Therefore, previous attempts to construct virtual situational awareness assumed that any additional feedback, such as vibrations from a game controller, would be added to the "picture" of situational awareness already created by the auditory signals. However, these methods cannot inform individuals of changes in their surroundings when auditory signals do not create a "picture," as is the case with DHHs.

[0006] Using 3D surround sound to enhance the immersive experience will only widen the gap in the experience for DHH users. For example, current technology uses sound to give the player the auditory information that an enemy is approaching from the upper left rear, causing the player to quickly turn towards the sound and immediately prepare to engage the enemy. These experiences are currently inaccessible to DHH users. Therefore, there remains a need to provide DHH users with an immersive experience equivalent to that of audio experiences. [Overview of the project]

[0007] This disclosure provides a haptic feedback system for creating an immersive media experience by providing haptic feedback to a DHH. A media player, such as a video game console, may create a virtual instance, such as a game level. Interactions, such as explosions, may be created within the virtual instance. The haptic feedback system may calculate the position of the player avatar relative to the interaction. The haptic feedback system may adjust the haptic feedback provided to the player using a wearable haptic rig, at least in part, based on predetermined parameters defined by the environment of the virtual instance. As a result, the haptic feedback system may provide haptic feedback to the DHH player, such as tactile sensations indicating the location and intensity of the interaction. The tactile sensations may be adjusted using predetermined parameters to provide an immersive experience as an alternative to prior art sound systems.

[0008] A media player may generate environmental parameters, such as environmental effects, based on predetermined parameters defined by the virtual instance's environment. For example, environmental parameters may be distortion effects, shielding effects, or occlusion effects applied by the virtual instance's environment. A first interaction parameter generated by an interaction may be modified by environmental parameters to generate a second interaction parameter.

[0009] The haptic feedback system may include a signal acquisition unit for capturing a first interaction parameter. A data interceptor may receive the first interaction parameter and store it in memory. The first and second interaction parameters may be sent from the data interceptor to a data comparator for comparing the first and second interaction parameters. The data comparator matches identifiers within the first and second interaction parameters to compare and classify the changes made by the environment parameters. The data comparator uses these changes to create a total situational state. The total situational state creates a "picture" of how each signal created by the interaction at a particular place and time is affected by the environment parameters before reaching the player avatar.

[0010] The microprocessor receives all status conditions. The microprocessor may classify and prioritize at least one parameter of all status conditions from the signal acquisition unit. Classification may determine the importance of an interaction based on its location, intensity, and "threat" within the level or virtual instance to the player avatar. To determine which interactions should be provided to the player, the classified and prioritized parameters are filtered by a user status filter. The user status filter may be created using player-specific calibration settings. For example, a virtual instance may contain multiple interactions, such as explosions occurring simultaneously across the entire level. Providing haptic feedback from all interactions simultaneously may overburden or confuse the player.

[0011] A user state filter may be applied to remove excessive "noise" from all state states before the microprocessor compiles actuator instructions and outputs them to a wearable haptic rig to provide haptic feedback to the player.

[0012] A wireless or wired receiver may receive actuator commands and transmit them to a haptic rig processor. To create an immersive experience, at least one haptic motor may be activated to provide haptic feedback to the player.

[0013] Examples of devices, systems, and methods are shown in the accompanying drawings, which are intended to be illustrative and non-limiting, and in the drawings, similar reference numerals are intended to refer to similar or corresponding parts. [Brief explanation of the drawing]

[0014] [Figure 1] This figure shows one embodiment of the haptic feedback rig according to the present disclosure. [Figure 2] This is a block diagram of one embodiment of the haptic feedback system described herein. [Figure 3] This is a flowchart of one embodiment in which the haptic feedback system described herein generates all status states. [Figure 4A] This figure shows an exemplary embodiment of the present disclosure in which a media player processor of a haptic feedback system generates a first interaction parameter. [Figure 4B] This figure shows an exemplary embodiment of the present disclosure in which the interaction processor of a haptic feedback system generates environment-influenced interaction information based on the first interaction parameter shown in Figure 4A. [Figure 4C] This figure shows an exemplary embodiment of the present disclosure in which a data interceptor of a haptic feedback system generates original interaction information based on the first interaction parameter in Figure 4A. [Figure 4D] This figure shows an exemplary embodiment of the present disclosure in which a data comparator of a haptic feedback system generates a full state based on the environment-influenced interaction information in Figure 4B and the original interaction information in Figure 4C. [Figure 5] This block diagram shows the method for providing a haptic-based immersive experience as disclosed in this disclosure. [Modes for carrying out the invention]

[0015] The detailed descriptions of the embodiments of this disclosure as described herein refer to the accompanying drawings illustrating various embodiments as examples. While these embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, it should be understood that other embodiments are also feasible and that logical and mechanical modifications may be made without departing from the spirit and scope of this disclosure. Therefore, the detailed descriptions herein are presented for illustrative purposes only and are not limiting. For example, the steps described in any description of a method or process may be performed in any order, and are not limited to the order presented. Furthermore, a reference to a single embodiment may include multiple embodiments, and a reference to two or more components may include a single embodiment.

[0016] Figure 1 shows an embodiment of the haptic feedback rig 100 according to the present disclosure. The annular component (collar) 102 includes a microcontroller 104 for receiving actuator commands from a receiver 106. The microcontroller 104 reads the actuator commands and operates the haptic motors 108, 110, 112, and 114 to provide haptic feedback to the player.

[0017] The haptic feedback rig 100 may communicate with a haptic feedback system disclosed herein, including a media player such as a game console. The haptic feedback system generates actuator commands to operate haptic motors 108, 110, 112, and 114 and transmits them to the haptic feedback rig 100. The media player may generate a virtual instance, such as a video game level or a movie. Interactions or events may occur as part of the virtual instance. The haptic feedback system processes the interactions, generates actuator commands, and provides haptic feedback to the player. The haptic feedback provided by the haptic motors 108, 110, 112, and 114 creates an immersive experience that replaces the conventional soundscape provided using audio signals.

[0018] In exemplary embodiments, an actuator command may activate a single tactile motor among tactile motors 108, 110, 112, and 114 to provide a separate pulse of a predetermined intensity related to the interaction. Depending on the type and duration of the interaction, the actuator command may activate any combination of tactile motors 108, 110, 112, and 114 to provide a cascaded or chained pulse of a predetermined intensity. As an example, a first pulse may be provided to the player by activating tactile motor 108. A second pulse may be provided by activating tactile motor 110, either following or overlapping with the first pulse. The second pulse may follow the first pulse within a microsecond. A third pulse may follow the second pulse by activating tactile motor 112, and a fourth pulse may be provided by activating tactile motor 114. In this way, actuator commands simulate cascading effects or other immersive experiences to provide the player with haptic feedback related to the interaction. Actuator commands may cause each pulse to be of the same intensity, weak intensity, or strong intensity. For example, the first pulse may be stronger than the second pulse.

[0019] As a result, when a player is playing as a DHH or playing without sound, the player can experience immersion and awareness of their surroundings within the virtual instance.

[0020] Figure 2 shows a block diagram of one embodiment of the haptic feedback system 200 according to this disclosure. A media player 202, such as a game console, may include a media player processor 204 for generating virtual instances, such as video game levels. As part of the virtual instance, the media player 202 may generate interactions within the virtual instance. These interactions may range from conversations with non-playable characters ("NPCs") within the virtual instance to events such as explosions or combat. Each of these interactions may affect the player. Therefore, the player needs to be aware of the interactions provided as haptic feedback by the haptic feedback system 200.

[0021] The media player processor 204 may generate a first interaction parameter as a data packet for transmission to the interaction processor 206. The first interaction parameter may include information that details the generated interaction, such as the type of interaction and the location of the interaction. In one embodiment, the interaction processor 206 may include an audio card that is located on the media player processor 204 or connected to the media player processor 204, or connected to the motherboard of the media player 202. The interaction processor 206 is responsible for applying any environmental effects to the first interaction parameter based on the virtual instance environment. The environmental effects may be output as environmental parameters for modifying or enhancing the first interaction parameter to generate a second interaction parameter. For example, the environmental parameters may be distortion effects, shielding effects, or occlusion effects applied by the virtual instance environment.

[0022] In one embodiment, using the first interaction parameter, vibrations due to an explosion may be defined. As would be understood by one of ordinary skill in the art, sound is generated by vibrations traveling through a fluid such as air. The vibrations caused by an explosion may be interpreted by a DHH person as an impact / contact to the skin in some cases. A person with hearing may also interpret the vibrations as sound generated by the explosion in some cases. The vibrations may be modified by environmental parameters by applying a sound attenuation effect to the first interaction parameter to generate a second interaction parameter that includes the attenuated vibrations. The attenuated vibrations may be generated to reproduce the real-world physical phenomenon of an explosion occurring on the other side of a door as seen from the player avatar.

[0023] To generate and define environmental parameters, media player processor 204 and interaction processor 206 generate a trace of the interaction (such as by "wave tracing" or "ray tracing") as the interaction approaches the player avatar. The trace may include any reflections, occlusions, or obstructions of the interaction by the virtual environment as the interaction moves from its origin within the virtual instance towards the player avatar. These environmental parameters may create distortion or reverberation in terms of how the player experiences the interaction when it reaches the player avatar. As a result, an interaction with a clear line of sight to the player avatar can reach the player avatar more quickly and with fewer applied environmental effects than an interaction in another room or an interaction located behind an object in the environment. The trace generates environmental parameters as at least one environmental effect applied to the interaction.

[0024] One example of a scenario that requires a distortion effect may include an explosion occurring under a long stone corridor. As the vibrations from the explosion travel under the stone corridor, the vibrations may bounce off the stone walls to apply a reverberation effect. The reverberation may distort the original vibrations so that the reverberation reaches the player avatar in a different form than if the explosion had occurred in an empty space. To generate environmental parameters that include the reverberation as an environmental effect, the path of the vibrations may be traced. When the vibrations reach the player avatar, the player must be provided with feedback that simulates the reverberation in order to create a sense of immersion.

[0025] The signal acquisition unit 212 may include a data interceptor 214 for receiving a first interaction parameter (before tracing) from the media player processor 204, and a data comparator 216 for receiving a second interaction parameter and environmental parameters from the interaction processor 206. The data interceptor 214 captures and stores the first interaction parameter, which includes, but is not limited to, the nature of the interaction, such as whether the interaction is a vehicle, explosion, weather effect, or notification; the threat, i.e., how "dangerous" it is to the player avatar within the virtual instance; the intensity, such as the strength of the interaction; the location, i.e., where the interaction occurs relative to the player avatar; the velocity, i.e., the speed at which the interaction approaches the player avatar; and the acceleration, i.e., the change in velocity at which the interaction approaches the player avatar. The data interceptor may output a first interaction parameter or the original interaction parameter for comparison with a second interaction parameter by the data comparator 216.

[0026] Interaction processors, such as audio cards, create environment-influenced parameters, but may not remember the changes made to create the environmental effects. Conventional DAC methods focus on the sound being produced and therefore do not necessarily store traces and environment parameters to indicate what changes have been made to the first interaction parameter. Storing traces may require additional processing power and storage, which are unnecessary if the only concern is outputting the modified sound. Therefore, to identify these changes, the data comparator 216 receives the first interaction parameter, the original interaction parameter, from the data interceptor 214, and the second interaction parameter from the interaction processor 206. As those skilled in the art will understand, the signal acquisition unit 212 may be part of the interaction processor 206 without departing from this disclosure. For example, the interaction processor 206 may be modified to include storage for storing the first interaction parameter as a data interceptor 214 without departing from this disclosure. Similarly, the interaction processor may be modified, without departing from this disclosure, to store changes made to generate a second interaction parameter as a data comparator 216.

[0027] The data comparator 216 outputs a total status state 218 based on the comparison. The total status state 218 may include a first interaction parameter, an environment parameter identified by the data comparator 216, and a second interaction parameter. The total status state 218 may include a virtual instance and construct a 3D instance showing how the interaction is received by the player avatar at a defined location. The signal acquisition unit 212 may generate the total status state 218 without using an audio signal, such as one used by a digital-audio-converter (DAC) 208. The DAC 208 requires a large amount of processing power and can slow down the processing speed of the interaction. Furthermore, if the player is DHH or has the sound off, the processing power consumed to generate an audio signal for the DAC 208 is wasted. The DAC 208 typically only receives signals to create sound influenced by the environment generated by the interaction, or signals to convert sound back into a digital signal. The signal acquisition unit 212 improves the construction speed of the overall state 218 by eliminating the need for the DAC 208 to use the audio output 210 to create the soundscape during the process.

[0028] Continuing with the examples of explosions, one embodiment of the overall situation state 218 may be created by compiling similar explosion signals generated by the media player processor 204. Each signal may be assigned an identifier (such as "A-" indicating that the signal is part of interaction A) and a set of tags. One or more tags are output by the media player processor 204 as a first interaction parameter in a data packet, depending on the interaction. The tags may include a signal format, such as a .WAV file. Another tag may be an accessibility tag, such as whether the player has subtitles turned on. In this example, the accessibility tag may add a subtitle file to the first interaction parameter. Other tags may include location, type of environment (such as the player avatar being in a stone corridor), basic (original) intensity of the interaction, speed, etc.

[0029] Each tag in a single interaction may contain the same identifier. The first interaction parameter may contain one or more interactions, and therefore the identifier may be used to distinguish the tags from one another. If two interactions occur at the same time / place, such as an explosion and the sound of something (e.g., a barrel near the explosion) breaking, they may be transmitted as part of the first interaction parameter. The explosion interaction signal may contain the identifier "explosion-", while the barrel interaction signal may contain the identifier "barrel-". As a result, all situational states 218 may group similar signals together using the identifier included in the first interaction parameter.

[0030] An exemplary embodiment of the first interaction parameter may be as follows: [A-0001] Electron explosion [A-0002] elecexplo.wav [A-0003] "Electron explosion" [A-0004] 30 meters [A-0005] Sphere [A-0006] Stationary [A-0007] 30% reduction [A-0008] 17.15.-257

[0031] Identifier A may be used to group signals for an electron explosion within a virtual instance. The identifier may be defined as "electron explosion" in [A-0001]. The source file [A-0002] "elecexplo.wav" is tagged. If subtitles are enabled, [A-0003] "electron explosion" may be displayed on the graphical user interface connected to media player 202. The explosion may have a [A-0004] "danger" flag placed within a [A-0004] sphere that [A-0006] remains stationary, i.e., stationary. The force of the explosion [A-0007] may decrease at a rate of 30% over time. The location of the explosion [A-0008] may be x-coordinate 17, y-coordinate 15, and z-coordinate -257 in the virtual instance.

[0032] Upon receiving the first interaction parameter, the interaction processor 206 completes the trace, calculates potential strains, and applies the identified strains, shielding, and occlusion to generate the second interaction parameter. The signal is "bounced" from all available surfaces, with reflections, shielding, occlusion, and partial absorption by soft surfaces all applied as environmental effects. The second interaction parameter may contain the same identifier as the first interaction parameter, but with tags modified by the environmental effects. Similar or matching identifiers are grouped for comparison between the first and second interaction parameters. The comparison may be used to identify any changes made to the interaction by the virtual environment.

[0033] As mentioned above, the DAC system for creating the soundscape does not require detailed information about environmental effects such as the extent to which distortion, shielding, or occlusion is applied. The DAC 208 and audio output 210 are only concerned with the final result of a second interaction parameter that may already include environmental effects. Detailed information about environmental effects is required to generate the full state 218. To capture this information, a data interceptor 214 stores the first interaction parameter, and a data comparator 216 determines the environmental effects. The data interceptor 214 and data comparator 216 may be implemented as elements of a signal acquisition unit 212 to hold detailed information about environmental effects and generate the full state 218. The signal acquisition unit 212 may be part of a media player 202 or a microprocessor unit 220 without departing from this disclosure.

[0034] The total status state 218 may be received by the microprocessor unit 220. The microprocessor unit 220 may apply a classification and prioritization table 222 to evaluate the interactions in the total status state 218 based on their importance to the player. The classification and prioritization table 222 may rank the interaction signals using tags included in the first and second interaction parameters. The classification and prioritization table 222 may determine whether the player can detect the interaction or whether the interaction is dangerous enough to require notification to the player. To avoid placing an excessive burden on the player, it may not be desirable to output haptic feedback for all interactions.

[0035] Therefore, for example, when multiple interactions are occurring simultaneously on the battlefield, providing haptic feedback to the player for all of them—explosions, vehicle movements, gunfire, NPCs, and weather effects—may overburden and confuse the player rather than create an immersive experience. Polyphony, the number of interactions a system can generate at once, varies greatly depending on the media system. A media player 202 may be able to output 50 or 5,000 interactions at once. However, depending on the intensity of the haptic feedback, the player may not be able to organize and understand the haptic feedback from all 50 or all 5,000 interactions. As an illustrative example, in a crowded room with 100 people shouting, it may be impossible to determine what each individual is saying, whereas if only 20 people are shouting, it may be possible. Interactions occurring within a virtual instance may be ranked and filtered by a classification and prioritization table 222 to prevent overburdening the player. In a crowded room scenario, classification and prioritization table 222 identifies approximately 80 voices that are inaudible or unimportant, filters them out, and leaves only 20 voices that the player can respond to.

[0036] The classified and prioritized parameters of all situation states 218 are filtered using a user situation filter 224 that tracks which interactions the player is aware of. The user situation filter 224 may be partially based on the player's specific calibration profile. Each individual has their own unique level of spatial awareness, reaction speed, etc. By applying the user situation filter 224 to all classified and prioritized situation states 218, a player-specific cutoff may be applied. In this way, the player is not bothered by a large number of interactions that are outside of their awareness. For example, if an interaction takes place behind a closed door and the player is determined not to be aware of that interaction, the classification and prioritization table 222 may rank that interaction lower than other interactions occurring simultaneously. The user situation filter 224 may filter interactions based on the player's calibrated awareness level until the player becomes aware of that interaction.

[0037] A player with a high level of spatial awareness may notice an interaction on the other side of a closed door, even at a distance, provided that no other interactions are occurring simultaneously. The microprocessor unit 220 translates the classified and prioritized parameters of all situation states 218 into actuator instructions 226 to be sent to the wearable haptic rig 228. As a result, the player is not burdened by haptic feedback drowning out "important" interactions occurring within the virtual instance. Continuing with the crowded room example above, if a player's individual awareness level is such that they can only distinguish 15 voices, the user situation filter 224 may filter out the five lowest-ranked voices out of the remaining 20.

[0038] The user situation filter 224 may include a player profile that is at least partially based on a player-specific calibration profile. DHH players can perceive and utilize situational awareness information in their own characteristic and partially unique ways. In addition, continuous use of sensory-alteration devices (such as eyeglasses or hearing aids) can train the user's brain, and as a result, the effectiveness of the device for that user may change over time (for example, a person with mild hearing loss using a hearing aid must periodically recalibrate the hearing aid as their ears and brain adapt to the changed soundscape). The user situation filter 224 may be updated using the calibration profile during a calibration session or a play session. To fine-tune the calibration profile and update the user situation filter 224, the player may monitor various parameters, such as reaction time during play, using haptic feedback, for example. The user situation filter 224 may take into account the individual situational awareness of the DHH player when translating the overall situational states 218 into actuator commands 226.

[0039] The actuator instruction 226 is sent to a wearable haptic rig 228 for providing haptic feedback to the player. The wearable haptic rig 228 may include a receiver 230, such as a wireless or Bluetooth® receiver or a wired connection, for receiving the actuator instruction 226 from the microprocessor unit 220. The rig processor 232 applies the actuator instruction 226 to activate at least one haptic motor 234 for providing haptic feedback to the player. The haptic feedback may complement any environmental effects applied to the interaction by the environment of the virtual instance to provide the player with an immersive experience. In one embodiment, the microprocessor unit may be mounted on the wearable haptic rig 228 and send the actuator instruction 226 directly to at least one haptic motor 234. As a result, the player can experience an immersive interaction provided by haptic feedback, including, in particular, the nature, intensity, and location of the interaction in the virtual instance.

[0040] In an exemplary embodiment of the haptic feedback system 200, the media player 202 may generate a game level with the player avatar standing in a stone corridor. The explosion may occur within the virtual instance, for example, about 50 meters along the stone corridor directly in front of the player. The impact / force of the explosion is analyzed along with environmental factors such as any distortion or reverberation caused by the stone corridor in response to the shock wave of the explosion. The signal acquisition unit 212 processes the virtual instance of the stone corridor, the explosion, and the player's position to create an overall situational state 218.

[0041] The microprocessor unit 220 receives all situational states 218, classifies and prioritizes the explosion and any sub-interactions such as the destruction of objects at the explosion site or the enemy avatar's reaction, so that the interaction is appropriately prioritized and addressed. For example, the explosion may be more important to the player than the destruction of objects at the explosion site. A user situation filter 224 is applied based in part on the calibration profile to filter out interactions that the player is unaware of or that place an excessive burden on the player. For example, if the player's situational awareness is determined to be low because the player is paying attention to another interaction or the player's reaction speed is determined to be below a predetermined threshold, the player may not recognize or be able to react to the enemy avatar. Therefore, the enemy avatar's reaction does not need to be provided to the player as haptic feedback. The microprocessor unit 220 converts filtered interactions from all state conditions 218 into actuator instructions 226 and sends the actuator instructions 226 to the wearable haptic rig 228.

[0042] The wearable haptic rig 228 receives an actuator command 226 related to the explosion using the receiver 230. The rig processor 232 processes the actuator command 226 and issues a command to activate at least one haptic motor 234 of the wearable haptic rig 228 to provide haptic feedback to the player. The haptic feedback may indicate that the explosion occurred in a direction relative to the player avatar. The strength of the haptic feedback may indicate the intensity of the explosion. A distortion effect may be simulated by at least one haptic motor 234. In an exemplary example, the haptic feedback may be strongest at the player's chest to indicate that the explosion occurred in front of the player avatar. Weaker haptic feedback may be applied to both sides of the player to simulate the distortion or reverberation of the shock wave as it bounces off the walls of a stone corridor.

[0043] Figure 3 shows a flowchart of one embodiment of the present disclosure in which the haptic feedback system 300 generates all state conditions 326. A media player creates a virtual instance such as a game level (302). An interaction occurs within the virtual instance (304), and the media player generates a first interaction parameter (306). The media player processor sends the first interaction parameter to the interaction processor and the data interceptor (308).

[0044] The interaction processor receives the first interaction parameter (310) and determines the environmental effects on the interaction (312). As previously stated, the interaction processor may trace the interaction across the entire virtual environment until the interaction reaches the player and, if there are environmental effects, determine what environmental effects should be applied to the first interaction parameter. The interaction processor applies the environmental effects to the first interaction parameter to generate environmentally influenced interaction information (314). The environmentally influenced interaction information is sent to the data comparator (316).

[0045] The data interceptor receives the first interaction parameter from the media player processor and generates the original interaction information (318). The original interaction information is sent to the data comparator (320).

[0046] The data comparator receives the original interaction information and the environment-affected interaction information (322). The original interaction information and the environment-affected interaction information are compared (324), and the environmental effects applied to the first interaction parameter are identified as environment parameters. In an alternative embodiment, the interaction processor may store the environmental effects applied to the first interaction parameter as environment parameters. The data comparator uses the original interaction information, environment parameters, and environment-affected interaction information to generate a total state (326).

[0047] Figure 4A is an exemplary embodiment of the present disclosure in which the media player processor of the haptic feedback system 400 generates first interaction parameters. The media player may create a virtual instance (402). An explosion interaction is generated within the virtual instance (404). The media player processor determines that an explosion has occurred (406) and retrieves the stored explosion data set (408). As described with respect to Figure 2, the stored explosion data set may include at least one tag having a unique identifier for the explosion interaction. The media player processor may use a spatial coordinate system, such as a Cartesian coordinate system, to locate the explosion (410). The media player processor may use the same coordinate system to locate the player avatar within the virtual instance (412). The Cartesian coordinates of the explosion may be identified as x1, y1, and z1, and the coordinates of the player avatar may be identified as x2, y2, and z2. The Cartesian coordinates are grouped with the stored explosion data set and exported as explosion information to the interaction processor and data interceptor (414).

[0048] Figure 4B is an exemplary embodiment of the present disclosure in which the interaction processor of the haptic feedback system 400 generates environment-influenced interaction information based on the first interaction parameters of Figure 4A. The interaction processor receives explosion information from the media player processor (416). The interaction processor may determine the spatial relationship between the explosion and the player avatar by comparing the x1, y1, z1 coordinates with the x2, y2, z2 coordinates (418). The interaction processor may also determine the decay rate of the explosion (420), for example, in Figure 2, an exemplary decay rate of 30% depending on the distance. That is, as the shock wave moves away from the epicenter of the explosion, the intensity of the shock wave may decrease at a rate of 30% per meter. Once the interaction processor has determined the distance between the explosion and the player avatar, it can calculate the time it takes for the shock wave to reach the player. The interaction processor applies a decay rate to determine the force of the explosion when it reaches the player avatar's location (422).

[0049] The interaction processor may map the environment between the virtual instance's explosion and the player avatar (424). The interaction processor may identify at least one surface or object in the environment between the explosion and the player avatar, such as a stone corridor or door (426). The interaction processor "bounces" the explosion off at least one surface and traces the path of the explosion's shock wave to the player avatar (428). The explosion processor may modify the explosion information according to the surface effects of at least one surface (430). For example, a stone corridor may create distortion effects such as reverberation. Similarly, the intensity of the shock wave, determined using the attenuation rate, may be reduced by the shielding effect caused by a door if there is a door between the player avatar and the explosion. The interaction processor generates environment-influenced explosion information from the modified explosion information (432) and sends the environment-influenced explosion information to the data comparator (434). In one embodiment, the interaction processor may store the modifications made to the explosion information as environmental parameters, without departing from the present disclosure.

[0050] Figure 4C is an exemplary embodiment of the present disclosure in which a data interceptor of a haptic feedback system 400 generates original interaction information based on the first interaction parameters of Figure 4A. The data interceptor receives explosion information from the media player processor (436), stores the explosion information, and generates original explosion information (438). The data interceptor sends the original explosion information to a data comparator (440). In one embodiment, the data interceptor may be implemented as a memory connected to the media player in order to receive and store the explosion information from the media player processor.

[0051] Figure 4D is an exemplary embodiment of the present disclosure in which a data comparator of a haptic feedback system 400 generates a full state based on the environment-influenced interaction information in Figure 4B and the original interaction information in Figure 4C. The data comparator receives the original explosion information from a data interceptor and the environment-influenced explosion information from an interaction processor (442). The data comparator compares the original explosion information with the environment-influenced explosion information to create environment parameters (444). In one embodiment, the data comparator may identify similar interactions generated by a media processor (446). For example, similar interactions may be of the same type or occur in the same place / time period. The data comparator compiles environment parameters for each similar interaction (448). The data comparator generates a full state including the original explosion information, environment parameters, and environment-influenced explosion information (450).

[0052] Figure 5 is a block diagram illustrating a method (500) for providing a haptic-based immersive experience according to the present disclosure. A virtual instance is created (502), an interaction occurs within the virtual instance, and a first interaction parameter is generated (504). A spatial relationship between the interaction and the player avatar is determined to define the 3D space between the interaction and the player (506). An environment parameter is determined by analyzing the 3D space between the interaction and the player for any objects or effects that may alter the interaction before it reaches the player avatar (508). The effects of the environment parameter are applied to the first interaction parameter to generate a second interaction parameter (510). The entire state is determined (512) based at least in part on the first interaction parameter, the environment parameter, and the second interaction parameter. The overall state may include a virtual instance and construct a 3D space around the player avatar and between the player avatar and the interaction. At least one parameter of the overall state is classified and prioritized (514). A player-specific user profile may be applied to the overall state (516).

[0053] An actuator command is generated (518) and sent to the wearable haptic rig (520). At least one haptic motor of the wearable haptic rig is activated to provide haptic feedback to the player (522). The player receives the haptic feedback generated by the interaction and responds to the haptic feedback (524). In one embodiment, the player may respond by turning their face in the direction of the interaction or by completing the action within the virtual instance. The player may update their awareness of their situation within the virtual instance based on the haptic feedback from the wearable haptic rig (526). As a result, the DHH player can experience an immersive experience and feel as if they are inside the virtual instance.

[0054] A user profile may be generated based in part on a calibration profile associated with the player (528). The user profile may be applied to all situational states (516) and may act as a filter to prevent the player from being provided with excessive “noise” as haptic feedback. For example, the classification and prioritization in step 514 may remove signals that the player does not notice. However, if multiple interactions are occurring simultaneously around the player avatar, the player may become overloaded and their senses may become dulled if they receive all the haptic feedback at the same time. To determine the number of signals or the amount of haptic feedback that may be provided, the user profile may be calibrated to include the player’s sensitivity. Furthermore, the user profile may include user settings regarding how the user prefers to receive haptic feedback, such as the duration of the haptic feedback. The user may set preferences for haptic feedback so that it is provided in long pulses, short bursts, etc. The calibration profile may be set using a calibration scenario, or the user profile may be continuously or at regular intervals updated by monitoring the player during play. For example, the player's response time may be monitored to update the calibration profile. If the response time changes, this may indicate that the player can handle more or fewer signals / haptic feedback simultaneously. The calibration profile may be updated accordingly, and the generated user profile (528) may be filtered to filter more or fewer signals from the total situation (516).

[0055] In another aspect of this disclosure, a non-transient computer-readable medium on which non-transient program code is recorded is disclosed. The program code includes program code for being executed by a processor to generate a virtual instance (502). The program code further includes program code for generating an interaction having a first interaction parameter within the virtual instance (504). The program code also includes program code for determining the spatial relationship between a player avatar and the interaction (506). The program code further includes program code for analyzing the environment of the virtual instance and determining at least one environment parameter that affects the first interaction parameter (508). The program code further includes program code for applying the environment parameter to the first interaction parameter and generating a second interaction parameter (510). The program code also includes program code for generating an entire state state including the virtual instance, the first interaction parameter, and the second interaction parameter (512). The program code further includes program code for classifying and prioritizing (514) at least one parameter of all state states. The program code further includes program code for applying a user profile to all state states (516). The program code also includes program code for generating actuator commands and sending the actuator commands to the wearable haptic rig (518). The program code further includes program code for applying the actuator commands to at least one haptic motor of the wearable haptic rig (520). The program code further includes program code for activating at least one haptic motor to provide haptic feedback (522).

[0056] The processor's memory may be internal memory and / or external memory, and may be implemented in the form of firmware and / or software implementations. Firmware and / or software implementations may be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described herein. Implementations of the methods described herein may use machine-readable media that tangibly embody instructions. For example, software code may be stored in memory and executed by a processor unit (e.g., media player processor 204). Memory may be implemented within or outside the processor unit. As used herein, the term “memory” refers to long-term memory, short-term memory, volatile memory, non-volatile memory, or other types of memory, and is not limited to a specific type of memory, a specific number of memories, or the type of medium in which the memory is stored.

[0057] When implemented in firmware and / or software, the functionality may be stored as one or more instructions or codes on a computer-readable medium. Examples include computer-readable medium encoded in data structures and computer-readable medium encoded in computer programs. Computer-readable medium includes physical computer storage medium. The storage medium may be an available medium that can be accessed by a computer. For example, but not limited to, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or other media that can be used to store desired program code in the form of instructions or data structures and are accessible by a computer. As used herein, “disk” and “disc” include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where a disk typically reproduces data magnetically, while a disc reproduces data optically using a laser. Any combination of the above should also be included within the scope of computer-readable media.

[0058] Instructions and / or data may be provided as signals on a transmission medium included in a communication device, in addition to being stored in a computer-readable medium. For example, the communication device may include a transceiver having signals indicating instructions and data. The instructions and data are configured to cause one or more processors to perform the functions outlined in the claims.

[0059] While the present disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions, and modifications may be made herein without departing from the technology of the present disclosure as defined by the appended claims. For example, relative terms such as “top” and “bottom” are used with respect to components. Naturally, if a component is inverted, “top” becomes “bottom,” and vice versa. Furthermore, if oriented laterally, “top” and “bottom” may refer to the sides of a component. Moreover, the scope of this application is not intended to be limited to any particular configuration of the processes, machines, products, compositions, means, methods, and steps described herein. As will be readily apparent to those skilled in the art from this disclosure, existing or future-developed processes, machines, products, compositions, means, methods, or steps that perform substantially the same function or achieve substantially the same results as the corresponding configurations described herein may be used in accordance with this disclosure. Accordingly, the appended claims are intended to include such processes, machines, products, compositions, means, methods, or steps within their scope.

[0060] Those skilled in the art will further understand that various exemplary logic blocks, modules, circuits, and algorithmic steps described herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly demonstrate this hardware and software compatibility, various exemplary components, blocks, modules, circuits, and steps have been described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the design constraints imposed on the particular application and the overall system. Those skilled in the art may implement the described functionality in various ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this disclosure.

[0061] The various exemplary logic blocks, modules, and circuits described in connection with the disclosure herein may be implemented or run by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, individual gate or transistor logic, individual hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors coupled with a DSP core, or any other such configuration.

[0062] Steps of the methods or algorithms described in connection with this disclosure may be performed directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor and storage medium may reside as separate components within a user terminal.

[0063] In one or more exemplary designs, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable mediums include both computer storage media and communication media, including any medium that facilitates the transfer of computer programs from one location to another. Storage media may be any available medium accessible by a general-purpose computer or a dedicated computer. Examples, but not limited to, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to carry or store specific program code means in the form of instructions or data structures, and is accessible by a general-purpose computer or a dedicated computer, or a general-purpose processor or a dedicated processor. Any connection is also appropriately referred to as computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, “disk” and “disc” include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where a disk typically reproduces data magnetically, while a disc reproduces data optically using a laser. The above combinations should also be included within the scope of computer-readable media.

[0064] When an element is described as being "connected" or "joined" to another element (or a variation thereof), understand that the element may be directly connected or joined to the other element, or there may be an intervening element. In contrast, when an element is described as being "directly connected" or "directly joined" to another element (or a variation thereof), there is no intervening element.

[0065] In this specification, benefits, other advantages, and solutions to problems have been described with respect to specific embodiments. However, benefits, advantages, solutions to problems, and elements that may give rise to or make more pronounced benefits, advantages, or solutions should not be construed as essential, required, or necessary features or elements of this disclosure. In the appended claims, references to singular elements should be understood as "one or more" and not "one and only one" unless expressly stated.

[0066] Examples of the present disclosure are described herein with reference to the accompanying drawings. However, the present disclosure should not be construed as being limited to the examples described herein. Rather, these examples are provided to make the present disclosure thorough and complete and to fully convey the scope of the present disclosure to those skilled in the art. The singular forms “a,” “an,” and “the” as used herein are intended to include the plural form unless the context explicitly specifies otherwise. Furthermore, it will be further understood that the terms “comprises,” “comprising,” “have,” “includes,” and “including,” and / or variations thereof, when used herein, specify the presence of the described features, steps, actions, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, actions, elements, components, and / or groups thereof.

[0067] The descriptions in this disclosure are provided to enable those skilled in the art to implement or use this disclosure. Various modifications to this disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other modifications without departing from the spirit or scope of this disclosure. Accordingly, this disclosure is not intended to be limited to the examples and designs described herein, but should be given the broadest scope consistent with the principles and novel features disclosed herein.

[0068] While exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, it should be understood that the present disclosure is not limited to these detailed embodiments, and various other changes and modifications may be made by those skilled in the art without departing from the scope or spirit of the present disclosure.

Claims

1. A method for providing haptic feedback in a virtual environment, The steps to create a virtual instance, Within the virtual instance, the steps include generating an interaction that includes a first interaction parameter, The steps include determining the spatial relationship between the player avatar and the interaction, The steps include analyzing the environment of the virtual instance and determining the environmental parameters that affect the interaction, The steps include applying the aforementioned environmental parameters to the first interaction parameters to generate a second interaction parameter, The steps include generating a complete state state including the virtual instance, the first interaction parameter, and the second interaction parameter, A step of classifying and prioritizing at least one parameter of the aforementioned state of all conditions, The steps include applying the user profile to all the aforementioned conditions, The steps include generating at least one actuator command based at least partially on the overall state and sending the at least one actuator command to a wearable haptic rig, The steps include applying the at least one actuator command to at least one tactile motor of the wearable tactile rig, The steps include: activating at least one tactile motor to provide tactile feedback to the player; A method that includes this.

2. A method for providing haptic feedback in a virtual environment according to claim 1, further comprising the step of calibrating the user profile based at least in part on the player's situational awareness parameters.

3. A method for providing haptic feedback in a virtual environment according to claim 1, further comprising the step of generating a classification and prioritization table.

4. A method for providing haptic feedback in a virtual environment according to claim 1, further comprising the step of ranking the interaction by matching it with at least one other interaction within the virtual instance.

5. A method for providing haptic feedback in a virtual environment according to claim 1, further comprising the step of generating a calibration profile using a calibration scenario.

6. A method for providing haptic feedback in a virtual environment according to claim 5, further comprising the step of filtering the interaction using a user situation filter generated using the calibration profile.

7. A method for providing haptic feedback in a virtual environment according to claim 5, further comprising the step of updating the user status filter based in part on the player variables monitored during gameplay.

8. A device for providing haptic feedback in a virtual environment, Memory and At least one processor coupled to the memory, Create a virtual instance, Within the virtual instance, an interaction is generated that includes the first interaction parameter. Determine the spatial relationship between the player avatar and the aforementioned interaction. The environment of the virtual instance is analyzed, and at least one environmental parameter that affects the interaction is determined. The aforementioned environmental parameters are applied to the first interaction parameters to generate the second interaction parameters. Generate the entire state, including the virtual instance, the first interaction parameter, and the second interaction parameter. Classify and prioritize at least one parameter of the aforementioned state of all conditions, Apply the user profile to all of the aforementioned conditions, Generate at least one actuator command, and send the at least one actuator command to the wearable haptic rig. The actuator command is applied to at least one tactile motor of the wearable tactile rig, The at least one tactile motor is activated to provide tactile feedback. A system comprising at least one processor configured as follows: A device equipped with the following features.

9. Apparatus for providing haptic feedback in a virtual environment according to claim 8, wherein the at least one processor is further configured to calibrate the user profile based at least in part on the player's situational awareness parameters.

10. The apparatus for providing haptic feedback in a virtual environment according to claim 8, wherein the at least one processor is further configured to generate a classification and prioritization table.

11. An apparatus for providing haptic feedback in a virtual environment according to claim 8, wherein the at least one processor is further configured to rank at least one other interaction within the virtual instance.

12. Apparatus for providing haptic feedback in a virtual environment according to claim 8, wherein the at least one processor is further configured to generate a calibration profile using a calibration scenario.

13. Apparatus for providing haptic feedback in a virtual environment according to claim 12, wherein the at least one processor is further configured to filter the interaction using a user situation filter generated using the calibration profile.

14. The apparatus for providing haptic feedback in a virtual environment according to claim 12, wherein the at least one processor is further configured to update the user status filter in part based on the player variables monitored during play.

15. Apparatus for providing haptic feedback in a virtual environment according to claim 8, wherein the at least one processor is further configured to apply the actuator instructions to the at least one haptic motor of the wearable haptic rig according to the preference for haptic feedback.

16. A non-transient computer-readable medium on which program code is recorded, wherein the program code is executed by a processor, Program code for creating virtual instances, Within the virtual instance, program code for generating an interaction including a first interaction parameter, Program code for determining the spatial relationship between the player avatar and the interaction, Program code for analyzing the environment of the virtual instance and determining at least one environmental parameter that affects the interaction, Program code for applying the aforementioned environmental parameters to the first interaction parameters and generating the second interaction parameters, Program code for generating the entire state state, including the virtual instance, the first interaction parameter, and the second interaction parameter, A program code for classifying and prioritizing at least one parameter of the aforementioned total state, Program code for applying the user profile to all the aforementioned conditions, Program code for generating at least one actuator command and sending the at least one actuator command to a wearable haptic rig, Program code for applying the at least one actuator command to at least one tactile motor of the wearable tactile rig, Program code for operating the at least one tactile motor to provide tactile feedback, Non-transient computer-readable media, including [specific examples of such media].

17. The non-transient computer-readable medium according to claim 16, wherein the program code includes program code for calibrating the user profile based at least in part on the player's situational awareness parameters.

18. The non-transient computer-readable medium according to claim 16, wherein the program code includes program code for generating classification and prioritization tables and determining whether to provide haptic feedback to the player, partly based on the ranking of the interactions in the classification and prioritization tables.

19. The non-transient computer-readable medium according to claim 16, wherein the program code includes program code for generating a calibration profile using a calibration scenario.

20. The non-transient computer-readable medium according to claim 16, wherein the program code includes program code for updating a user status filter generated using the calibration scenario, in part based on the player's variables monitored during play.

Images