Method for adjusting earphone denoising based on real-world activity or game context

Abstract

A method is provided that includes executing an interactive application, where executing the interactive application includes rendering video and audio of a virtual environment, the video being presented on a display viewed by a user, and the audio being presented through a headset worn by the user, wherein executing the interactive application is a user input generated in response to a user's interaction with the presented video and audio; receiving an environment input from at least one sensor sensing a local environment in which the user is located; analyzing the environment input to identify an activity occurring in the local environment; in response to identifying the activity, an active denoising level applied by the headphone is then adjusted.

Description

1. Technical Field

[0001] The present disclosure generally relates to adjusting headphone noise cancellation based on real-world activity or gaming context. background

[0002] 2. Related technical description

[0003] The video game industry has gone through many changes over the years. As technology advances, video games tend to achieve a greater sense of immersion through sophisticated graphics, realistic sounds, engaging soundtracks, haptics, etc. Players are able to enjoy an immersive gaming experience in which they participate and engage in a virtual environment rather than their local physical environment.

[0004] It is in this context that the implementation of the present disclosure is proposed. Summary of the invention

[0005] Implementations of the present disclosure include methods, systems, and apparatuses directed to adjusting headphone noise cancellation based on real-world activities or gaming contexts.

[0006] In some embodiments, a method is provided, the method comprising the following method operations: executing an interactive application, wherein executing the interactive application includes rendering video and audio of a virtual environment, the video being presented on a display viewed by a user, and the audio being presented through headphones worn by the user, wherein executing the interactive application is in response to user input generated by the user's interaction with the presented video and audio; receiving environmental input from at least one sensor that senses a local environment in which the user is located; analyzing the environmental input to identify activities occurring in the local environment; and in response to identifying the activities, then adjusting a level of active noise cancellation applied by the headphones.

[0007] In some embodiments, active noise cancellation is configured to reduce the volume of ambient sounds from the local environment during presentation of audio of the virtual environment through headphones.

[0008] In some embodiments, the at least one sensor includes a camera that captures images of the local environment, and wherein analyzing the environmental input includes analyzing the captured images to identify activities occurring in the local environment.

[0009] In some embodiments, identifying the activity includes identifying additional persons in the local environment.

[0010] In some embodiments, the adjustment to the active noise cancellation level is configured to decrease the active noise cancellation level in response to identifying an additional person.

[0011] In some embodiments, identifying the activity includes identifying a proximity of an object in the local environment to the user.

[0012] In some embodiments, the adjustment to the active noise cancellation level is configured to reduce the active noise cancellation level in response to identifying the proximity of the object to the user.

[0013] In some embodiments, the interactive application is a video game.

[0014] In some embodiments, a method is provided, the method comprising the following method operations: executing a video game, wherein executing the video game generates a game state, the game state is processed to render video and audio of a virtual environment, the video is presented on a display viewed by a player, and the audio is presented through headphones worn by the player, and wherein executing the video game is responsive to player input generated by the player's interaction with the presented video and audio; determining a level of active noise cancellation to be applied by the headphones based on the game state; applying the determined level of active noise cancellation by the headphones, wherein the active noise cancellation is configured to reduce the volume of ambient sounds from the local environment during the presentation of audio of the virtual environment through the headphones.

[0015] In some embodiments, determining the level of active noise cancellation to be applied by the headphones is based on the current game progress position of the video game.

[0016] In some embodiments, determining the level of active noise cancellation to be applied by the headset is based on the current gameplay location in the virtual environment of the video game.

[0017] In some embodiments, determining the level of active noise cancellation to be applied by the headset is based on the proximity of the player's avatar to the virtual objects in the virtual environment.

[0018] In some embodiments, determining a level of active noise cancellation to apply includes receiving a parameter value defined by a game state.

[0019] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present disclosure may be better understood with reference to the following description in conjunction with the accompanying drawings, in which:

[0021] Figure 1 A user participating in a game of a video game according to an implementation of the present disclosure is shown.

[0022] Figure 2 A user in accordance with implementations of the present disclosure is shown engaging in gameplay of a video game and encountering a situation where ANC may be adjusted.

[0023] Figure 3Game progression and scenes in a video game according to implementations of the present disclosure are conceptually illustrated.

[0024] Figure 4 Conceptually illustrated is the use of a machine learning (ML) model to determine active noise cancellation settings for headphones in accordance with implementations of the present disclosure.

[0025] Figure 5 A method for controlling active noise cancellation during execution of a video game according to an implementation of the present disclosure is conceptually illustrated.

[0026] Figure 6 Components of an example apparatus that may be used to perform aspects according to various embodiments of the present disclosure are shown. DETAILED DESCRIPTION

[0027] The following implementations of the present disclosure provide methods, systems, and apparatus for adjusting headphone noise cancellation based on real-world activities or gaming contexts.

[0028] Implementations of the present disclosure relate to methods and systems for automatically adjusting noise cancellation features on headphones. The method includes detecting the user's environment by one or more devices. One or more cameras and / or motion detectors can be used to monitor the user's environment. The detection is used to identify when another person approaches the user or the user is unaware that he or she is moving toward an object (and may be in danger of injury). In one embodiment, the detection is dynamic and can be ongoing during a session and will be triggered in an "anticipatory" manner using machine learning to begin reducing noise cancellation before a safety issue occurs. In one embodiment, noise cancellation can be gradually reduced and can be gradually reduced when the user approaches an unsafe location or something approaches the user. When the object leaves the user's area, the noise cancellation can be gradually increased. In some cases, noise cancellation can be instantaneous, for example, in the case where someone is detected trying to communicate with the player. Then, the sound cancellation can be stopped immediately and then restarted at the end of the conversation. In addition to reducing or increasing noise cancellation, the game sound can also be reduced or increased along with the noise cancellation. In some cases, the video (e.g., game) that the user is watching can also provide visual cues indicating that it may be necessary to talk to someone or pay attention to and avoid obstacles. In some game scenarios, noise cancellation will also be dynamically increased. For example, if the user is approaching a critical game section or scene, noise cancellation can be automatically increased to help the user focus. In some cases, the automatic increase / decrease of noise cancellation can be built into the game by the developer, or can be a setting controlled by the game engine. These features can avoid immersion breaks during highly relevant content.

[0029] With the above overview in mind, several example diagrams are provided below to aid in the understanding of example embodiments.

[0030] Figure 1 A user participating in a game of a video game according to an implementation of the present disclosure is shown.

[0031] In the illustrated implementation, user 100 participates in an interactive game play of a video game executed by computing device 102. By way of example and not limitation, computing device 102 may be a game console, a personal computer, a laptop, a set-top box, or other general or special purpose computer having a processor and memory and capable of executing program instructions of a video game. Furthermore, while user interaction with a video game is described for purposes of illustrating embodiments of the present disclosure, it should be appreciated that the principles of the present disclosure may be applicable to other types of interactive applications that may be executed by computing device 102 and with which user 100 may interact. Furthermore, it should be appreciated that in other implementations, at least some of the functionality attributed to computing device 102 may be performed by a cloud of remote computing resources.

[0032] Execution of the video game by the computing device 102 includes rendering video and audio of the virtual environment of the video game. The video is presented on a display 104 (e.g., a television, monitor, LED / LCD display, projector screen, etc.) viewed by the user 100. When the user 100 participates in the interaction with the video game, the user 100 can operate a user input device (such as a controller 106 in the illustrated implementation) that generates user input for the video game. It should be understood that the execution of the video game is responsive to the user input generated due to the user's interaction, and based on the user input, the game state is continuously updated to drive the execution of the video game.

[0033] The audio generated by the video game is presented through headphones 108 worn by user 100. Headphones 108 may be wired or wireless, and may be connected directly or indirectly to computing device 102. For example, in some implementations, headphones 108 are connected to controller 106, and controller 106 communicates with computing device 102 to receive audio, which is then passed to headphones 108 for presentation.

[0034] The earphones 108 are equipped with active noise cancellation (ANC), which, broadly speaking, is capable of reducing the volume of ambient sounds of the local environment 112 in which the user 100 is located. To implement active noise cancellation, the earphones 108 include a microphone 120 for detecting ambient sounds, and an ANC circuit 124 that receives the signal from the microphone and generates an anti-phase signal, which, when fed to a speaker 122, produces a sound wave that cancels at least a portion of the ambient sounds reaching the user's ears. The ANC controller 126 is configured to set or adjust the level / amount of active noise cancellation applied at any given moment. As discussed in more detail below, the ANC level can be adjusted at any time based on various factors.

[0035] While ANC is used to enhance the immersion of a gaming experience by blocking ambient sounds that may interfere with the user's experience, it can sometimes be used to reduce immersion to allow the user to be more aware of their surroundings. For example, another person may be trying to get the user's attention, or the user may be unknowingly moving toward a potentially harmful situation, such as a possible collision with an object in the local environment. In addition, ANC can be selectively utilized in the context of a video game to selectively enhance the immersion of certain parts of the game. Therefore, while ANC is typically used in an on / off binary manner operated by the user, implementations of the present disclosure automatically and dynamically adjust the ANC level in response to situational factors and the video game itself.

[0036] In the illustrated implementation, the environment detection device 110 includes at least one sensor capable of sensing the local environment 112, such as by detecting an event or object in the local environment 112. By way of example and not limitation, the sensors of the environment detection device 110 may include one or more image capture devices or cameras, motion sensors, microphones, depth sensors, etc. Broadly speaking, the sensors of the environment detection device 110 generate environmental inputs that are received and analyzed by the computing device 102 to identify activities occurring in the local environment 112. In some implementations, the analysis may involve using a machine learning model or other recognition model to identify an object or activity. And, the ANC level applied by the headphones 108 is adjusted based on the identified activity.

[0037] In some implementations, the computing device 102 generates ANC control data that is transmitted to the headphones 108 and applied by the headphones 108, and more specifically by the ANC controller 126, to control the level of ANC implemented by the ANC circuit 124. In some implementations, the ANC is controlled in predefined levels or increments, where adjustments are made in a step-wise manner. In other implementations, the ANC is controlled in a substantially continuous manner, where adjustments are made in a substantially continuous and progressive manner. It should be appreciated that the specified amount or level of ANC set at a given moment will control the amplitude of the anti-phase sound waves generated by the speakers 122 of the headphones 108, and thereby control the degree to which the volume of ambient sounds heard by the user is reduced by the ANC.

[0038] In some implementations, the ANC level is controlled in response to gaming conditions. For example, in some implementations, the ANC level is a programmable parameter that can be defined by the video game for a specific scene or location (space or time) within the game or for a specific gaming state. For example, in this way, ANC can be increased to enhance user immersion during high-relevance content.

[0039] Figure 2A user in accordance with implementations of the present disclosure is shown engaging in gameplay of a video game and encountering a situation where ANC may be adjusted.

[0040] In the illustrated implementation, the user 108 is engaging in gameplay of a video game in accordance with the implementations described above. The local environment 112 is being sensed, and certain events and objects may be identified, and the ANC of the headphones 108 may be adjusted based on the identification.

[0041] For example, in some implementations, when another person 200 is detected near the user 100 or is identified as moving toward the user 100, the ANC level is reduced or turned off to allow the user 100 to more easily hear or be aware of the other person 200. In some implementations, if the person 200 attempts to speak to the user 100, the person's 200 voice is recognized and triggers the ANC to be immediately turned off. In some implementations, other types of recognized objects can trigger similar adjustments to the ANC, such as the detection of a pet.

[0042] In some implementations, the ANC level can be adjusted for the safety of the user 100. For example, if the user is detected to be too close to an object (such as the lamp 202) and there may be a risk of collision with the object, the ANC can be reduced or turned off. For example, in some implementations, as the user 100 moves toward the object, the ANC can be reduced to a greater extent as the user 100 gets closer to the object.

[0043] In some implementations, a proximity zone 204 is defined around the user 100, for example, at a predefined distance from the user 100, such that when an object crosses into the proximity zone 204, the ANC is reduced or turned off. In other implementations, there may be multiple proximity zones 204 defined at various distances from the user 100, where the proximity zones are associated with various ANC levels, such that the ANC reduction increases as the distance of the detected object from the user 100 decreases. In another implementation, when an object enters the proximity zone 204, the ANC begins to decrease, wherein as the distance of the object from the user 100 decreases, a further reduction in ANC occurs. In some implementations, when the distance of the object from the user 100 reaches another predefined distance, the ANC is completely turned off.

[0044] In some implementations, ambient sounds may be recognized and may trigger adjustments to the ANC. For example, in addition to human speech, other recognized sounds may also trigger a reduction or turn-off of the ANC, such as other human sounds (e.g., a baby crying, a person in distress), ringtones of mobile devices, pet sounds (e.g., a dog barking), alarm sounds, emergency vehicle sounds, timer sounds (e.g., an oven timer), doorbell sounds, sounds / notifications from smart speakers, etc.

[0045] It will be appreciated that after ANC has been reduced under conditions such as those described above, the ANC level may be increased upon detecting that such conditions have reversed or no longer exist in the local environment 112 of the user 100. For example, the ANC level may increase as the distance of an object from the user 100 increases.

[0046] Figure 3 Game progression and scenes in a video game according to implementations of the present disclosure are conceptually illustrated.

[0047] In the illustrated implementation, progression in the video game is conceptually illustrated by storyline 300, where movement along storyline 300 from left to right represents progression in the video game, e.g., as defined by achievement of goals, movement to predefined locations in the virtual environment of the video game, temporal movement / achievement along a timeline of the video game's narrative or story, etc.

[0048] In some implementations, the ANC level may be set or adjusted based on the current progress position of the user playing the video game. For example, at progress point A along the storyline 300, the ANC level may be set to a first predefined level, and at progress point B along the storyline 300, the ANC level may be set to a second predefined level. In some implementations, the video game developer may set parameters associated with the storyline of the game that control or affect the ANC level. And thus, as the user progresses through the storyline of the game, the ANC level is adjusted according to the parameters set for the specific progress point of the user's current game process. In this way, in some implementations, ANC may be increased at certain points to selectively enhance the immersion of the game experience at a specific time along the storyline. This can help the user better concentrate or focus in certain situations of the video game. For example, the ANC level may be increased in a game scene with dialogue so that the user can better hear and understand the dialogue. Or more generally, when the game process approaches high focus, high valence / intensity, or a critical moment, ANC may be increased to help reduce real-world distractions and enhance immersion.

[0049] In addition, in some implementations, the parameters controlling or affecting the ANC level may be associated with a spatial location within the video game virtual environment. For example, at progress point A, the user may be participating in a game process in scene A of a video game, which has a virtual environment conceptually displayed by map 310. In some implementations, various areas of the virtual environment may have different ANC levels assigned to them, or different parameter values ​​assigned to them, and these parameter values ​​control or affect the ANC level when the user's position (e.g., the position of a character controlled by the user or other entity controlled by the user during the game) is located in a specific area of ​​the virtual environment. For example, area 312 may have a first ANC level associated with it, while area 314 may have a second ANC level associated with it. Therefore, as the user moves in the virtual environment of scene A, the ANC level is dynamically adjusted in response to the user's position in the virtual environment of scene A.

[0050] It should be appreciated that different scenes of the video game, such as scene B reached at progression point B, will have different arrangements of ANC levels or parameter sets associated with the corresponding virtual environment of scene B. Also, as the user moves through various areas of the virtual environment, the ANC may be dynamically adjusted based on the user's location in the virtual environment. Likewise, such ANC adjustments / settings may be used to selectively enhance immersion in certain locations within the virtual environment of the video game.

[0051] In a related implementation, ANC may be adjusted relative to a user's proximity to a given virtual object in a virtual environment of a video game (e.g., the user's avatar / character's proximity to the given virtual object). For example, ANC may be increased when the user approaches another character in the game because a conversation with the other character may be about to occur, such that increasing ANC will help the user hear the conversation better.

[0052] In some implementations, ANC may be increased during quieter scenes of a video game to allow the user to better hear the quieter sounds of such scenes, or to amplify the effect of silence / quiet, whereas during louder scenes, ANC may be decreased because it may not be as necessary to hear the louder sounds of such scenes.

[0053] In some implementations, ANC parameters or settings are defined as part of a game engine or game platform (e.g., a console game platform), and thus, video games developed using the game engine or for the game platform can utilize the ANC parameters to selectively apply ANC in specific contexts of the video game and increase the impact of the game.

[0054] Figure 4Conceptually illustrated is the use of a machine learning (ML) model to determine active noise cancellation settings for headphones in accordance with implementations of the present disclosure.

[0055] In the illustrated implementation, a machine learning (ML) model 400 is configured and trained to recognize / identify objects, situations, or events in a video game and / or local environment, and thus determine the ANC settings of the headset. For example, the ML model 400 may be trained using labeled training data. The ML model 400 may be executed by the aforementioned computing device 102 in some implementations, or executed in the cloud in other implementations.

[0056] The ML model 400 may be configured to perform game recognition 402 based on game state data 406 of the executed video game. That is, the ML model 400 may recognize game objects and situations / events occurring in the video game. By way of example and not limitation, the game state data 406 may include data items such as user / controller input, game progress / scene information, virtual object / character information, virtual environment location information, and the like.

[0057] The ML model 400 may also be configured to perform environmental recognition 404 to recognize objects / people and events occurring in the local environment using environmental data 408. Examples of such environmental data 408 include captured images or videos of the local environment, captured audio, or other types of data from sensors that sense the local environment.

[0058] Based on events occurring in the executed video game and the local environment in which the user is located, the ML model 400 may dynamically determine the ANC settings 410. Although the ML model 400 is depicted as a single entity, in some implementations, the ML model 400 may be composed of multiple machine learning models.

[0059] It will be appreciated that the analysis of the ML model 400 may be predictive, such that future activities may be predicted, and the ANC settings 410 may be predictively determined / adjusted in anticipation of future activities. For example, a potential collision or other potentially unsafe situation for a user may be predicted, and the ANC settings 410 may be predictively reduced accordingly.

[0060] Although adjustments to the ANC have been described, in some implementations, in addition to reducing or increasing noise cancellation, the game sounds can also be reduced or increased, as well as the noise cancellation adjustments. In this way, the game sounds can be reduced while increasing the ambient volume for the user (by lowering the ANC), or the game sounds can be increased while reducing the ambient sound volume (by increasing the ANC). In addition, in some implementations, the video (e.g., a game) that the user is watching can also provide visual cues indicating that it may be necessary to talk to someone or pay attention to and avoid obstacles. In some implementations, this visual cue can take the form of a written notification message, a graphic overlay, a pointer or arrow indicating that the user should look in a specific direction.

[0061] Figure 5 A method for controlling active noise cancellation during execution of a video game according to an implementation of the present disclosure is conceptually illustrated.

[0062] At method operation 500, execution of a session of a video game is initiated. At method operation 502, execution of the video game is implemented by processing a game state of the video game. This may include processing user input to drive execution, and continually updating the game state over time and in response to user interaction. At method operation 504, game video and audio (e.g., from a virtual environment of the video game) are rendered.

[0063] At method operation 506, a baseline ANC level is determined based on the game state of the video game. As discussed above, this can be determined based on the specific context of the game process, the player's progress, the game scene, the location, etc.

[0064] At method operation 508, the local environment in which the player is located is detected, such as by a sensor device (such as a camera, microphone, motion sensor, etc.). At method operation 510, activities in the local environment are identified based on the detection of the local environment. This may involve identifying / recognizing objects and events occurring in the local environment.

[0065] At method operation 512, the baseline ANC level is adjusted based on the activity identified in the local environment. In this way, the amount of ANC applied at any given moment during the game session is determined by both the video game and the activity state in the player's local environment.

[0066] Figure 6Components of an example device 600 that can be used to perform various aspects of various embodiments of the present disclosure are shown. The block diagram shows a device 600 that can be combined with or can be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for practicing the embodiments of the present disclosure. The device 600 includes a central processing unit (CPU) 602 for running software applications and optionally an operating system. The CPU 602 can be composed of one or more homogeneous or heterogeneous processing cores. For example, the CPU 602 is one or more general-purpose microprocessors with one or more processing cores. Other embodiments can be implemented using one or more CPUs with a microprocessor architecture that is particularly suitable for highly parallel and computationally intensive applications, such as processing interpretation queries, identifying context-related resources, and immediately implementing and rendering context-related resources in a video game. The device 600 can be local to a player (e.g., a game console) playing a game clip, or away from the player (e.g., a back-end server processor), or one of many servers that use virtualization to remotely stream a game process to a client in a game cloud system.

[0067] The memory 604 stores applications and data for use by the CPU 602. The storage device 606 provides non-volatile storage devices and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD or other optical storage devices, as well as signal transmission and storage media. The user input device 608 transmits user input from one or more users to the device 600, examples of which may include a keyboard, mouse, joystick, touch pad, touch screen, static or video recorder / camera, tracking device for identifying gestures, and / or microphone. The network interface 614 allows the device 600 to communicate with other computer systems via an electronic communication network, and may include wired or wireless communications through local area networks and wide area networks (such as the Internet). The audio processor 612 is suitable for generating analog or digital audio output from instructions and / or data provided by the CPU 602, the memory 604 and / or the storage device 606. The components of device 600 , including CPU 602 , memory 604 , data storage device 606 , user input device 608 , network interface 610 , and audio processor 612 , are connected via one or more data buses 622 .

[0068] The graphics subsystem 620 is also connected to the data bus 622 and the components of the device 600. The graphics subsystem 620 includes a graphics processing unit (GPU) 616 and a graphics memory 618. The graphics memory 618 includes a display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. The graphics memory 618 may be integrated into the same device as the GPU 608, connected to the GPU 616 as a separate device, and / or implemented in the memory 604. The pixel data may be provided directly from the CPU 602 to the graphics memory 618. Alternatively, the CPU 602 provides the GPU 616 with data and / or instructions defining the desired output image, and the GPU 616 generates pixel data for one or more output images based on the data and / or instructions. The data and / or instructions defining the desired output image may be stored in the memory 604 and / or the graphics memory 618. In one embodiment, GPU 616 includes 3D rendering capabilities that generate pixel data for output images from instructions and data that define the geometry, lighting, shading, textures, motion, and / or camera parameters of a scene. GPU 616 may also include one or more programmable execution units capable of executing shader programs.

[0069] Graphics subsystem 614 periodically outputs pixel data of an image from graphics memory 618 for display on display device 610. Display device 610 may be any device capable of displaying visual information in response to a signal from device 600, including CRT, LCD, plasma, and OLED displays. For example, device 600 may provide analog or digital signals to display device 610.

[0070] It should be noted that access services delivered over a wide geographic area (such as providing access to games of the current embodiment) often use cloud computing. Cloud computing is a computing method in which dynamically scalable and usually virtualized resources are provided as a service over the Internet. Users do not need to be experts in the technical infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services typically provide common applications (such as video games) accessed online from a web browser, while the software and data are stored on servers in the cloud. Based on the way the Internet is depicted in computer network diagrams, the term cloud is used as a metaphor for the Internet and is an abstraction of the complex infrastructure that it hides.

[0071] In some embodiments, a game server can be used to perform the operation of a duration information platform for video game players. Most video games played over the Internet are operated by connection with a game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. In other embodiments, video games can be performed by a distributed game engine. In these embodiments, a distributed game engine can be performed on multiple processing entities (PEs) so that each PE performs a functional fragment of a given game engine on which a video game runs. The game engine simply regards each processing entity as a computing node. The game engine typically performs a series of operations with diverse functions to perform video game applications and additional services for user experience. For example, a game engine implements game logic, performs game calculations, physical effects, geometric transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, audio communications, game play replay functions, help functions, etc. Although a game engine can sometimes be performed on an operating system virtualized by a hypervisor of a specific server, in other embodiments, the game engine itself is distributed between multiple processing entities, each of which can reside on different server units in a data center.

[0072] According to this embodiment, the corresponding processing entity for performing the operation can be a server unit, a virtual machine, or a container, depending on the needs of each game engine fragment. For example, if a game engine fragment is responsible for camera transformations, a virtual machine associated with a graphics processing unit (GPU) can be provided to this particular game engine fragment, because it will perform a large number of relatively simple mathematical operations (e.g., matrix transformations). Other game engine fragments that require fewer but more complex operations can be provided with processing entities associated with one or more higher-powered central processing units (CPUs).

[0073] By distributing the game engine, the game engine is provided with elastic computing properties that are not constrained by the capabilities of the physical server units. Instead, more or fewer computing nodes are provided to the game engine as needed to meet the needs of the video game. From the perspective of the video game and the video game player, a game engine distributed across multiple computing nodes is no different than a non-distributed game engine executed on a single processing entity, because the game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output components to the end user.

[0074] The user uses a client device to access the remote service, and the client device includes at least a CPU, a display, and an I / O. The client device can be a PC, a mobile phone, a notebook computer, a PDA, etc. In one embodiment, the network identification client used on the game server performs the device type and adjusts the communication method adopted. In other cases, the client device uses a standard communication method such as html to access the application on the game server through the Internet. It should be understood that a given video game or game application can be developed for a specific platform and a specifically associated controller device. However, when such a game is made available via a game cloud system as presented herein, the user may access the video game with different controller devices. For example, a game may have been developed for a game console and its associated controller, and the user may use a keyboard and mouse to access the cloud-based version of the game from a personal computer. In this scenario, the input parameter configuration can define the mapping of the input generated from the user's available controller device (in this case, i.e., keyboard and mouse) to the execution of the video game that is acceptable.

[0075] In another example, a user may access the cloud gaming system via a tablet computing device, a touch screen smart phone, or other touch screen driven device. In this case, the client device and the controller device are integrally formed in the same device, wherein input is provided by means of a detected touch screen input / gesture. For such a device, the input parameter configuration may define a specific touch screen input corresponding to the game input of the video game. For example, during the running of a video game, a button, a directional pad, or other type of input element may be displayed or overlaid to indicate a position on the touch screen where the user can touch to generate a game input. Gestures (such as swipes or specific touch movements in a particular direction) may also be detected as game inputs. In one embodiment, guidance may be provided to the user, indicating, for example, how to provide input for gameplay via a touch screen before starting to play a video game, so as to adapt the user to the operation of the controls on the touch screen.

[0076] In some embodiments, the client device is used as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device can process these inputs in turn, and then transmit the input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller itself can be a networked device with the ability to directly communicate input to the cloud game server via a network without first communicating such input through a client device. For example, the controller can be connected to a local networking device (such as the above-mentioned router) to send data to and receive data from the cloud game server. Therefore, although the client device may still be required to receive video output from a cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send input directly to the game cloud server over the network to bypass the client device.

[0077] In one embodiment, the networked controller and client device may be configured to send certain types of input directly from the controller to the cloud gaming server, and to send other types of input via the client device. For example, inputs whose detection does not depend on any additional hardware or processing other than the controller itself can be sent directly from the controller to the cloud gaming server via the network, thereby bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs that utilize additional hardware or require client devices to process can be sent by client devices to the cloud gaming server. These may include video or audio captured from the gaming environment, which can be processed by the client device before being sent to the cloud gaming server. In addition, inputs from the motion detection hardware of the controller can be processed by the client device in conjunction with the captured video to detect the position and motion of the controller, which is then communicated to the cloud gaming server by the client device. It should be understood that the controller device according to various embodiments may also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0078] In one embodiment, various technical examples can be implemented using a virtual environment through a head-mounted display (HMD). The HMD may also be referred to as a virtual reality (VR) head-mounted device. As used herein, the term "virtual reality" (VR) generally refers to the interaction of a user with a virtual space / environment, which involves viewing the virtual space in a real-time response to the movement of the HMD (controlled by the user) through the HMD (or VR headset) to provide the user with a feeling in a virtual space or metaverse. For example, a user can see a three-dimensional (3D) view of the virtual space when facing a given direction, and when the user turns to one side and thus turns the HMD in the same way, the view of the side in the virtual space is rendered on the HMD. The HMD can be worn in a manner similar to glasses, goggles, or a helmet, and is configured to display video games or other metaverse content to the user. The HMD can provide a very immersive experience to the user by means of its display mechanism being close to the user's eyes. Therefore, the HMD can provide a display area that occupies most of the user's or even the entire field of view to each eye of the user, and can also provide viewing with three-dimensional depth and perspective.

[0079] In one embodiment, the HMD may include a gaze tracking camera configured to capture images of the user's eyes as the user interacts with the VR scene. The gaze information captured by the gaze tracking camera may include information related to the user's gaze direction and specific virtual objects and content items in the VR scene that the user is paying attention to or interested in interacting with. Thus, based on the user's gaze direction, the system can detect specific virtual objects and content items that may be of potential interest to the user (where the user is interested in interacting with and engaging with them), such as game characters, game objects, game props, etc.

[0080] In some embodiments, the HMD may include an outward-facing camera configured to capture images of the user's real-world space, such as the user's body movements and any real-world objects that may be located in the real-world space. In some embodiments, the images captured by the external camera may be analyzed to determine the position / orientation of the real-world objects relative to the HMD. Using the known position / orientation of the HMD, real-world objects, and inertial sensor data from the user's posture and movement, the user's posture and movement during the user interaction with the VR scene can be continuously monitored and tracked. For example, when interacting with a scene in a game, a user may make various gestures, such as pointing and walking toward a specific content item in the scene. In one embodiment, the system may track and process gestures to generate predictions of interactions with specific content items in the game scene. In some embodiments, machine learning may be used to facilitate or assist the predictions.

[0081] During the use of the HMD, various single-hand and two-hand controllers can be used. In some implementations, the controller itself can be tracked by tracking the lights included in the controller or tracking the shapes, sensors and inertial data associated with the controller. Using these different types of controllers or even just using gestures made and captured by one or more cameras, the virtual reality environment or metaverse rendered on the HMD can be docked, controlled, manipulated, interacted with and participated in the virtual reality environment or metaverse. In some cases, the HMD can be wirelessly connected to the cloud computing and game system through a network. In one embodiment, the cloud computing and game system maintain and execute the video game played by the user. In some embodiments, the cloud computing and game system are configured to receive input from the HMD and interface objects through the network. The cloud computing and game system are configured to process input to affect the game state of the video game being executed. Output (such as video data, audio data and tactile feedback data) from the video game being executed is transmitted to the HMD and the interface object. In other implementations, the HMD can communicate wirelessly with the cloud computing and game system through alternative mechanisms or channels such as cellular networks.

[0082] In addition, although the implementations in the present disclosure may be described with reference to head-mounted displays, it should be understood that in other implementations, non-head-mounted displays may be substituted, including but not limited to portable device screens (e.g., tablets, smart phones, laptops, etc.) or any other type of display that can be configured to render video and / or provide interactive scenes or virtual environments according to the present implementations. It should be understood that the various embodiments defined herein can be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that can define more implementations by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0083] Embodiments of the present disclosure may be practiced with various computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through wired or wireless networks.

[0084] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted so that they occur at slightly different times, or the operations may be distributed in a system that allows processing operations to occur at various intervals associated with the processing as long as the processing of telemetry and game state data used to generate the modified game state is performed in the desired manner.

[0085] One or more embodiments may also be manufactured as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage devices (NAS), read-only memories, random access memories, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. A computer readable medium may include a computer readable tangible medium distributed on a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

[0086] In one embodiment, the video game is executed locally on a game console, a personal computer, or on a server. In some cases, the video game is executed by one or more servers in a data center. When executing the video game, some instances of the video game may be simulations of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. In some embodiments, the simulation is an instance of the video game. In other embodiments, the simulation may be generated by an emulator. In either case, if the video game is represented as a simulation, the simulation can be executed to render interactive content that can be interactively streamed, executed, and / or controlled by a user input.

[0087] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it is apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the present embodiments are to be considered illustrative rather than restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method comprising: executing an interactive application, wherein executing the interactive application includes rendering video and audio of a virtual environment, the video being presented on a display viewed by a user and the audio being presented through headphones worn by the user, and wherein executing the interactive application is responsive to user input generated by the user's interaction with the presented video and audio; receiving environmental input from at least one sensor that senses a local environment in which the user is located; analyzing the environmental input to identify activities occurring in the local environment; In response to identifying the activity, a level of active noise cancellation applied by the headphones is then adjusted.

2. The method of claim 1, wherein the active noise cancellation is configured to reduce the volume of ambient sounds from the local environment during presentation of the audio of the virtual environment through the headphones.

3. The method of claim 1, wherein the at least one sensor comprises a camera that captures images of the local environment, and wherein analyzing the environmental input comprises analyzing the captured images to identify the activity occurring in the local environment. The method of claim 1 , wherein identifying the activity comprises identifying additional persons in the local environment. 5 . The method of claim 1 , wherein the adjustment of the active noise cancellation level is configured to decrease the active noise cancellation level in response to identifying the additional person. The method of claim 1 , wherein identifying the activity comprises identifying a proximity of an object in the local environment to the user.

7. The method of claim 1, wherein the adjustment of the active noise cancellation level is configured to decrease the active noise cancellation level in response to identifying the proximity of the object to the user.

8. The method of claim 1, wherein the interactive application is a video game.

9. A method comprising: executing a video game, wherein executing the video game generates a game state, the game state is processed to render video and audio of a virtual environment, the video being presented on a display viewed by a player and the audio being presented through headphones worn by the player, and wherein executing the video game is responsive to player input generated by the player's interaction with the presented video and audio; determining a level of active noise cancellation to be applied by the headset based on the game state; A level of active noise cancellation determined by the headset application, wherein the active noise cancellation is configured to reduce a volume of ambient sounds from the local environment during presentation of the audio of the virtual environment through the headset.

10. The method of claim 9, wherein determining the level of active noise cancellation to be applied by the headset is based on a current game progress position of the video game.

11. The method of claim 9, wherein determining the level of active noise cancellation to be applied by the headset is based on a current gameplay location in the virtual environment of the video game.

12. The method of claim 9, wherein determining the level of active noise cancellation to be applied by the headset is based on proximity of a player avatar to a virtual object in the virtual environment.

13. The method of claim 9, wherein determining the level of active noise cancellation to apply comprises receiving a parameter value defined by the game state.

14. A non-transitory computer readable medium having program instructions embodied thereon, the program instructions, when executed by at least one computing device, causing the at least one computing device to perform a method comprising: executing an interactive application, wherein executing the interactive application includes rendering video and audio of a virtual environment, the video being presented on a display viewed by a user and the audio being presented through headphones worn by the user, and wherein executing the interactive application is responsive to user input generated by the user's interaction with the presented video and audio; receiving environmental input from at least one sensor that senses a local environment in which the user is located; analyzing the environmental input to identify activities occurring in the local environment; In response to identifying the activity, a level of active noise cancellation applied by the headphones is then adjusted.

15. The non-transitory computer-readable medium of claim 14, wherein the active noise cancellation is configured to reduce the volume of ambient sounds from the local environment during presentation of the audio of the virtual environment through the headphones.

16. The non-transitory computer-readable medium of claim 14, wherein the at least one sensor comprises a camera that captures images of the local environment, and wherein analyzing the environmental input comprises analyzing the captured images to identify the activity occurring in the local environment.

17. The non-transitory computer-readable medium of claim 14, wherein identifying the activity comprises identifying additional persons in the local environment.

18. The non-transitory computer-readable medium of claim 14, wherein the adjustment of the active noise cancellation level is configured to decrease the active noise cancellation level in response to identifying the additional person.

19. The non-transitory computer-readable medium of claim 14, wherein identifying the activity comprises identifying a proximity of an object in the local environment to the user.

20. The non-transitory computer medium of claim 14, wherein the adjustment of the active noise cancellation level is configured to decrease the active noise cancellation level in response to identifying the proximity of the object to the user.

Images