Head-up displays in virtual reality
The use of a head-mounted display with gaze tracking and AI-enhanced HUD provides timely and relevant in-game information, addressing the distraction issue of character-mounted displays and enhancing gameplay interaction.
Patent Information
- Application Number
- JP2024529185
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-16
- Filing Date
- 2022-11-03
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-11-03
AI Technical Summary
Existing video games that display information on character-mounted displays divert user attention from gameplay, leading to delays in understanding and reacting to critical in-game information due to focus on the display rather than the game environment.
A method and system that utilizes a head-mounted display (HMD) to track user gaze and generate a heads-up display (HUD) interface near objects of interest, providing relevant information through gaze detection and AI-enhanced object identification.
Enhances user interaction by allowing timely access to critical game information without distracting from the game environment, improving gameplay experience and decision-making.
Smart Images

Figure 2024545002000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to computing games, and more particularly to providing information of objects to a heads-up display within a virtual environment. [Background technology]
[0002] Video games provide gameplay within a virtualized game world. These video games may be formatted in single-player or multiplayer modes. For example, video games may provide gameplay from a variety of perspectives or viewpoints, such as first-person or third-person perspectives. In a first-person perspective, the player sees the virtual game environment from the eyes of the character in a three-dimensional virtual space. In a third-person perspective, the player controls the character as if a third person were looking at the character, such as from an over-the-shoulder or behind-the-character perspective. Popular genres of first-person and / or third-person games include shooters that involve combat situations using highly desirable weapons (e.g., large, effective guns with unlimited ammo) that follow a storyline or sequence of actions.
[0003] In some configurations, information related to the video game may be displayed on a small form factor display located on a part of the body of the character being controlled, such as a wrist-, arm-, or leg-mounted display, or may be held in the character's hand. In this manner, the user may manipulate the character within the virtual environment to obtain a view of the display. For example, in the case of a display mounted on the character's wrist, much like a wristwatch, the user may raise the arm and / or wrist and / or tilt the character's head (e.g., downwards) so that the display becomes viewable. The information on the display may relate to any aspect of gameplay.
[0004] However, looking at information on a display held or worn by a character may cause some problems for a user playing a game. For example, when looking at and understanding the information presented on the display, the user's attention is entirely focused on the display, not the gameplay. That is, a situation presented during gameplay (e.g., a battle with a boss) does not stop just because the character's viewpoint is turned to the display. Therefore, the user may choose not to look at any information on the display in order to focus on the gameplay (e.g., continuing to fight the boss), even if the information could be useful (e.g., a game hint for fighting the boss). That is, the user may not be given enough time to look at the display while playing the game. Furthermore, there may be too much information presented on the display. That is, there may be too much information and the user may not be able to get any useful information from it or may need to spend a long time to understand the information. The delay in viewing and understanding the information may adversely affect any of the user's subsequent actions and / or reactions while playing a video game.
[0005] It is in this context that embodiments of the present disclosure arise. Summary of the Invention
[0006] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate to computer games, and more particularly to providing heads-up display (HUD) object information within a virtual environment displayed through a head-mounted display (HMD).
[0007] In one embodiment, a method is disclosed that renders a view of a virtual environment of a video game for display on a user's head mounted display, the virtual environment including a plurality of objects. The method tracks a user's gaze. The method detects that the user's gaze is directed at an object of the view for a period of time. The method generates a heads-up display (HUD) interface that includes information related to the object. The method displays the HUD interface proximate to the object of the view of the virtual environment.
[0008] In another embodiment, a non-transitory computer readable medium storing a computer program for a game is disclosed. The computer readable medium includes program instructions for rendering a view of a virtual environment of a video game for display on a user's head mounted display, the virtual environment including a plurality of objects. The computer readable medium includes program instructions for tracking a user's gaze. The computer readable medium includes program instructions for detecting that the user's gaze is directed toward an object of the view for a period of time. The computer readable medium includes program instructions for generating a heads up display (HUD) interface including information related to the object. The computer readable medium includes program instructions for displaying the HUD interface near the object of the view of the virtual environment.
[0009] In another embodiment, a computing system includes a processor and a memory coupled to the processor having instructions stored therein that, when executed by the computer system, cause the computer system to perform a method of rendering a view of a virtual environment of a video game for display on a user's head mounted display, the virtual environment including a plurality of objects. The method includes tracking a user's gaze. The method includes detecting that the user's gaze is directed toward an object of the view for a period of time. The method includes generating a heads-up display (HUD) interface including information related to the object. The method includes displaying the HUD interface near the object of the view of the virtual environment.
[0010] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure.
[0011] The present disclosure may be best understood by referring to the accompanying drawings in conjunction with the following description. [Brief description of the drawings]
[0012] [Figure 1] 1 illustrates components of an exemplary device that can be used to implement aspects of various embodiments of the present disclosure. [Diagram 2] 1 is a flow diagram illustrating a method for presenting information on a virtual reality head-up display according to one embodiment of the disclosure. [Figure 3A] 1 illustrates a user viewing a virtual reality scene including a head-up display providing information about objects in the scene, according to an embodiment of the present invention. [Figure 3B] 1 illustrates an object being highlighted to a user viewing a virtual reality scene according to one embodiment of the present disclosure. [Figure 3C]1 illustrates a scope mode view that provides a magnified view of a target object within a virtual reality scene and the presentation of information related to the target object in a head-up display, according to one embodiment of the present disclosure. [Figure 4] 1 illustrates the application of rules and / or artificial intelligence to determine the context of a scene in a virtual reality system, identifying objects within the scene, and generating information related to the objects based on the context that is displayable via a heads-up display located within the scene, according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Reference will now be made in detail to various embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Although the following detailed description contains many specific details for purposes of explanation, those skilled in the art will recognize that many variations and modifications to the following details are within the scope of the present disclosure. Thus, the aspects of the present disclosure described below are presented without any loss of generality to, and without imposing limitations on, the claims that follow this description.
[0014] Generally speaking, various embodiments of the present disclosure describe methods for providing information of an object in a head-up display (HUD) within a virtual environment as viewed through a head-mounted display (HMD), and methods for providing information within the HUD within a scope mode view of the virtual environment that magnifies the object of interest.
[0015] With the above general understanding of the various embodiments in mind, detailed examples of the embodiments will now be described with reference to the various drawing figures.
[0016] Throughout this specification, references to a "game" or a "video game" or a "game application" are meant to represent any type of interactive application that is directed through the execution of input commands. By way of example only, interactive applications include applications for gaming, word processing, video processing, video game processing, and the like. Additionally, the terms introduced above are interchangeable.
[0017] It should be understood that embodiments of the present disclosure may employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful machine operations. Embodiments of the present disclosure also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purposes, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings herein. Alternatively, it may be more convenient to construct a more specialized apparatus to perform the required operations.
[0018] Embodiments of the present disclosure may be practiced with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.
[0019] Some of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In this application, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps utilize physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as transactions, bits, values, elements, symbols, characters, samples, or pixels, or the like.
[0020] It should be noted, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. As will become apparent from the discussion that follows, unless specifically stated otherwise, discussions utilizing terms such as "rendering," "tracking," "detecting," "generating," "surfacing," "removing," "performing," and the like throughout this disclosure are understood to refer to the actions and processes (e.g., flow diagram 200 of FIG. 2) of a computer system or similar electronic computing device or processor (e.g., system 100 of FIG. 1). A computer system or similar electronic computing device manipulates and transforms data, which are represented as physical (electronic) quantities in the memory, registers, or other such information storage, transmission, or display devices of the computer system.
[0021] The present disclosure may also be embodied as computer-readable code on a computer-readable medium, such as a program module, executed by one or more computers or other devices. Additionally, communication media may embody computer-executable instructions, data structures, and program modules and includes any information delivery media. By way of example, and not limitation, communication media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media. Combinations of any of the above may also be included within the scope of computer-readable media.
[0022] FIG. 1 illustrates components of an exemplary device 100 that may be used to implement aspects of various embodiments of the present disclosure. For example, FIG. 1 illustrates an exemplary hardware system suitable for executing a video game configured to provide and / or present information of objects in a head-up display (HUD) within a virtual environment and / or virtual reality (generated by execution of an application) viewed through a head-mounted display (HMD) according to one embodiment of the present disclosure. The block diagram illustrates device 100, which broadly represents any single or multi-processor computing device or system capable of executing computer-readable instructions. Examples of device 100 may incorporate or be, but are not limited to, a personal computer, a workstation, a server computer, a game console, a client-side terminal, a laptop, a mobile device, a distributed computing system, or other digital device, each of which is suitable for practicing embodiments of the present disclosure.
[0023] Both a central processing unit (CPU) 102 and a graphics processing unit (GPU) 116 are coupled to the memory 104. Generally, the CPU 102 and / or the GPU 116 may represent any type or form of processor and / or computing device capable of processing data or interpreting and executing instructions. In a particular embodiment, the CPU 102 and / or the GPU 116 may receive instructions from a software application or a hardware module. These instructions may cause the CPU 102 and / or the GPU 116 to perform one or more functions of the exemplary embodiments described and / or illustrated herein. The CPU 102 and / or the GPU 116 may also perform and / or be a means for performing any other steps, methods, or processes described and / or illustrated herein.
[0024] In particular, the CPU 102 is configured to run software applications and optionally an operating system. The CPU 102 may be configured with one or more homogeneous or heterogeneous processing cores. According to various embodiments, the CPU 702 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments may be implemented using one or more CPUs with a microprocessor architecture specifically adapted for highly parallel, computationally intensive applications (e.g., media and interactive entertainment applications) for applications configured to perform graphics processing during game execution.
[0025] For example, CPU 102 may provide data and / or instructions defining a desired output image to GPU 116, from which GPU 116 generates pixel data for one or more output images. The data and / or instructions defining the desired output images may be stored in memory 104 and / or a dedicated graphics memory (not shown). In one embodiment, GPU 116 includes 3D rendering capabilities to generate pixel data for output images from instructions and data defining scene geometry, lighting, shading, texture, motion, and / or camera parameters. GPU 116 may further include one or more programmable execution units capable of executing shader programs.
[0026] Memory 104 stores applications and data for use by CPU 102 and GPU 116. Memory 104 generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and / or other computer-readable instructions for applications, and may include, but is not limited to, fixed disk drives, removable disk drives, flash memory devices, RAM, ROM, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. In the example of FIG. 1, memory 104 is a shared memory, whereby the memory stores instructions and data for both CPU 102 and GPU 116. Alternatively, CPU 102 and GPU 116 may each have their own dedicated separate memory.
[0027] GPU 116 periodically outputs pixel data for an image from memory (e.g., graphics memory, frame buffer, etc.) to be displayed on display device 110. Display device 110 may be any device capable of displaying visual information in response to signals from device 100, including CRT, LCD, plasma, and OLED displays. Device 100 may provide analog or digital signals to display device 110, for example. In one embodiment, display device 110 includes an HMD (e.g., displaying virtual reality or VR content).
[0028] The user input devices 108 communicate user input from one or more users to the device 100, and examples of which may include a keyboard, a mouse, a joystick, a touchpad, a touch screen, a still or video recorder / camera, and / or a microphone. Additionally, a network interface (not shown) may provide communication with other systems (e.g., communication between game consoles, HMDs, controllers, etc.) over an electronic communication network, which may include wired or wireless communication over local area networks and wide area networks such as the Internet. The components of the device 100, including the CPU 102, the GPU 116, the memory 104, the user input devices 108, the network interface, etc., are connected via one or more data buses 122.
[0029] In some embodiments, a computer readable medium containing a computer program can be loaded into the computing system 100. All or a portion of the computer program stored on the computer readable medium can then be stored in various portions of the system memory 104 and / or storage devices. When executed by a processor (e.g., the CPU 102 and / or the GPU 116), the computer program loaded into the computing system 100 can cause and / or be a means for the CPU 102 and / or the GPU 116 to perform the functions of the exemplary embodiments described and / or illustrated herein. Additionally or alternatively, the exemplary embodiments described and / or illustrated herein can be implemented in firmware and / or hardware.
[0030] In one embodiment, CPU 102 and / or GPU 116 are configured to provide an emulation of a computing platform (e.g., an operating system) configured to execute software applications. The emulation of a computing platform can be implemented in software, hardware, or a combination of the two. In this manner, CPU 102 and / or GPU 116 can be configured to execute any type of application, whether the application is a legacy application or an application designed for a future computing platform.
[0031] 2 is a flow diagram 200 illustrating a method for presenting information on a heads-up display in a virtual reality environment, according to one embodiment of the disclosure. In particular, the flow diagram 200 enhances a user's and / or player's ability to play a video game by providing information about the video game during the player's gameplay. In this manner, the user is provided with additional information related to the video game that is not typically provided during gameplay. This additional information may result in more enjoyable gameplay by allowing the user to make more informed decisions during gameplay.
[0032] At 210, the method includes rendering and displaying a view of the virtual environment and / or virtual reality on a user's head mounted display (HMD) during execution of the video game. For example, video frames for a three-dimensional field of view (FOV) into the virtual environment are generated in response to the user's input controls and other control signals. Additionally, the virtual environment may include a number of objects disposed in a scene, and the scene may be displayed on the HMD in a series of image frames.
[0033] At 220, the method includes tracking the user's gaze. In particular, a direction of the user's gaze relative to the HMD and, accordingly, a direction of the user's gaze relative to a scene of the virtual environment displayed on the HMD is determined. Gaze tracking can be performed using any of a number of techniques. For purposes of illustration only, gaze tracking can be performed within the HMD using one or more gaze tracking cameras configured to capture images of the user's eyes, which images are analyzed to determine the user's gaze direction. The user's gaze direction can be defined relative to the HMD, relative to the user's physical environment (e.g., gaze tracking can be performed by tracking the position and orientation of the HMD within the physical environment, allowing gaze direction to be defined with reference to the physical environment, such as a real-world gaze), and / or relative to the virtual environment rendered and / or displayed on the HMD.
[0034] At 230, the method includes detecting that a user's gaze is directed toward an object in the view for a period of time. In particular, a direction of the user's gaze relative to the virtual environment is detected. The direction into the scene is also used to determine whether the gaze is directed toward one or more objects in the scene that are located near the gaze direction. For example, a gaze direction toward the scene may indicate that the user's attention is directed toward an object in the scene that is aligned, tightly aligned, or loosely aligned with the gaze direction. By way of example, the gaze direction may indicate a gaze drawn into the scene, in which an object intersects the gaze or can be inferred to intersect the gaze (i.e., is near or within the gaze range).
[0035] Additionally, eye tracking can indicate that the direction of gaze is fixed on the object for a period of time, i.e., the direction of gaze is consistently (e.g., directed in the same direction or roughly the same direction) for a period of time to an area of a scene of the virtual environment that includes the object. In this manner, it can be determined that the user's gaze is fixed on the object for that amount of time, which is sufficient to indicate that the user's attention is likely focused on the object and that the user is interested in the object.
[0036] At 240, the method includes generating a heads-up display (HUD) interface that includes information related to the object. That is, if the user's gaze is determined to be directed toward the object, the user is indicated to be interested in the object, and information about the object may be presented to the user. In one embodiment, the information is displayed in a HUD interface that is entered into a scene of the virtual environment. For example, the HUD interface may be configured as any interface (e.g., a display, a window, etc.) that can be placed in the scene to display information displayed in the interface. Furthermore, the information may be in one of many forms that the user can understand, such as text, symbols, icons, graphs, bars, status bars, video, audio, etc., and the information may be used to serve one of many purposes, such as providing background information about the object, providing helpful information related to the object, etc.
[0037] At 250, the method includes displaying a HUD interface near an object in a view of the virtual environment. In particular, the HUD interface containing information related to the object is positioned in the scene near and / or adjacent to the object. In this manner, the HUD interface and the information contained therein can be easily associated with a particular object by a user.
[0038] Additionally, the HUD interface may be static for a period of time, such that the HUD interface and the information contained therein remain unchanged. In another embodiment, the HUD interface and the information contained therein may be continually updated during that period. In yet another embodiment, the HUD interface is movable throughout the scene. For example, as an object moves, the HUD interface may follow the object's movement, such that the HUD interface remains near and / or adjacent to the object.
[0039] 3A illustrates a user viewing a virtual reality scene 310 including a head-up display (HUD) interface 350 providing information about objects in the scene, according to an embodiment of the present invention. For purely illustrative purposes, a user 302 is seated and wearing a head-mounted display (HMD) 320 and playing a video game. In this manner, the scene is presented within the HMD 320, allowing the user 302 to view a 3D representation of the scene 310. Although embodiments of the present disclosure may be described in the context of a user playing a video game or game application, it will be understood that the user may interact with any type of application, including video games, service-related applications, work-related applications (e.g., word processing, spreadsheets, design applications, computer-aided design (CAD) applications, etc.).
[0040] Purely for purposes of illustration, a scene 310 may be displayed during gameplay of a user 320, and the scene may include multiple objects (e.g., characters, backgrounds, environments, etc.). For example, the scene 310 may be a jungle scene including a makeshift camp within a forest of trees, and the video game may be a first-person shooter game. The camp may include some form of protection 360, such as an awning including one or more posts. A character 340 is also presented within the scene 310.
[0041] As shown, a field of view (FOV) of a scene is presented to a user 302 within an HMD 320, with the FOV being defined between a number of lines 325. The user 302 typically determines what FOV to present depending on the orientation of the HMD 320 and / or the user's line of sight. As shown, the FOV is a portion of a scene 310, and includes a shield 360, an object 370-A, a character 340, and various other background objects (e.g., trees, etc.).
[0042] As shown, the user's gaze is determined within the FOV; that is, gaze tracking can be used to determine in which direction the user's gaze is directed. The gaze direction may indicate the direction in which the user is looking and, accordingly, the area of the scene 310 that is the focus of the user's attention. The gaze of the user 302 is determined between a number of lines 330. A particular gaze direction can be determined by a line 330-A determined by gaze tracking. An object 370-A may intersect the line 330-A, indicating that the gaze of the user 302 is directed toward that object 370-A.
[0043] A HUD interface 350 is presented within the scene 310, the HUD interface including information related to the object 370-A. For example, information may be presented to the user 302 through the HUD interface 350 upon determining that the user's attention is directed to the object. For example, the HUD interface may be configured as any interface (e.g., a display, a window, etc.) that can be placed in the scene to display the information presented within the interface. For purposes of illustration, the HUD interface may be a two-dimensional (2D) or three-dimensional (3D) panel configured to contain information. In one embodiment, the HUD interface 350 is presented as an overlay within the scene. In another embodiment, the HUD interface 350 is embedded in the virtual environment and / or the scene 310 such that the HUD interface 350 is integrated into the virtual environment through the execution of a video game, e.g., having the HUD interface 350 as another object 350 in the scene 310.
[0044] As such, a HUD interface 350 is presented to the user within the display of the HMD 320. The HUD interface 350 may be configured as any interface (e.g., a display, a window, etc.) that may be placed in a scene to display information presented within the interface. For purposes of illustration, a close-up of the HUD interface 350 is shown in FIG. 3A to show information 351 presented within the HUD interface, where the information 351 may be in any of a number of forms that a user can understand, such as text, symbols, icons, graphs, bars, status bars, video, audio, etc.
[0045] In an embodiment, the information 351 presented in the HUD interface 350 may serve one or more purposes. For example, the information 351 may be intended to be useful to the user, such as to aid the user in progressing through the gameplay (e.g., providing gameplay hints that may be triggered by or related to the object). Also, the information 351 may include hints that the user needs to pick up the object in order to progress through the gameplay. In a useful and practical form, the information 351 may inform the user of what the object is and why it is in the scene. In other implementations, the information 351 may be intended to help increase the user's awareness of the video game and / or gameplay of the video game. That is, the information 351 may be peripherally related to the object 370-A, such as setting the scene 310 or providing background information for the scene. For example, the information 351 may help convey a particular image of the scene of the virtual environment or help create a particular mood or emotion of the gameplay (e.g., nervousness, happiness, sadness, etc.). In other embodiments, information 351 may be used to provide a user with background information about the object.
[0046] By way of example only, information 351 can provide background information about the object, provide helpful information related to the object, and the like. For example, information 351 may be related to object 370-A and may identify the object as an assault weapon and provide one or more characteristics of an assault weapon (e.g., mid-range capability, weapon accuracy, weapon capacity, weapon effectiveness rated on a lethality scale, etc.). Additionally, information 351 may include notes related to object 370-A. As previously discussed, in one embodiment, the notes may be hints that can help advance the user's gameplay. For example, the notes may indicate that ownership of object 370-A can help complete a current or future task in the user's 302 gameplay.
[0047] In one embodiment, the HUD interface 350 is opaque and / or has one or more degrees of transparency such that the virtual environment, including the scene 310, is visible through the HUD interface 350. In this manner, the information 351 in the HUD interface 350 may be presented to the user without interfering with the user's gameplay. For example, the user 302 may be in the middle of an intense gameplay (e.g., a boss fight) where information may be helpful to the user (e.g., hints for defeating the boss) but may be detrimental to the user if the user's attention is drawn too far away from the scene 310 (e.g., losing the boss while accessing the information 351 from the HUD interface 350).
[0048] In one embodiment, while the user 302 is looking at the object 370-A, the HUD interface 350 is zoomed in within the display of the HMD 320. That is, a zoom in is performed within the FOV presented to the HMD towards the HUD interface 350, and the HUD interface 350 gradually grows within the FOV while the user's gaze remains directed at the object. The zoom in and out of the HUD interface 350 may be performed as a function of the user's gaze. For example, when it is detected that the user 302 is looking away from the object 370-A, a zoom out is performed within the FOV from the HUD interface 350, and the HUD interface gradually grows within the user's FOV. Similarly, when it is detected that the user 302 is looking at the object 370-A, a zoom in is performed within the FOV towards the HUD interface 350, and the HUD interface gradually grows within the user's FOV until the particular size of the HUD interface is reached.
[0049] In one embodiment, the HUD interface 350 is terminated and / or removed from the HMD 320 upon detecting that the user's gaze is directed away from the object 370-A for a certain period of time. The certain period of time can be selected to indicate that the user's 302 attention is no longer focused on the object 370-A. In particular, eye tracking can be implemented to detect that and how the gaze is directed away from the object. For example, the gaze is tracked and analyzed to determine a gaze history function of the gaze, which measures the user's attention to the object. The gaze history function can include a linear section and / or a non-linear section with respect to time. Furthermore, the HUD interface 350 can be terminated and / or removed if the gaze history function satisfies a time decay function, which can be used to measure the decay of the general user's attention to the general object and determine when the general user's attention is no longer directed to the general object, i.e., when the general user is no longer interested in the general object. The time decay function can include a linear section and a non-linear section with respect to time and can indicate the degree of the user's attention. A match between certain sections of the gaze history function and the time decay function may indicate that the user's 302 attention is no longer directed to the object 370-A and / or that the user is no longer interested in the object. In one embodiment, so long as the gaze history function satisfies the time decay function, the transparency of the HUD interface 350 displayed within the HMD 320 is increased (e.g., increased transparency) during a transition period before completely removing the HUD interface 351 displayed within the HMD 320 (e.g., the HUD interface 350 fades out). In another embodiment, if the gaze history function does not follow the time decay function, the transparency of the HUD interface 350 displayed within the HMD 320 is decreased to indicate that the user's attention may have been distracted from the object 370-A, but that the user has returned attention to the object again (e.g., the HUD interface 350 fades back in).
[0050] 3B illustrates an object being highlighted for a user viewing a virtual reality scene, according to one embodiment of the disclosure. For example, a user may be playing the video game introduced in FIG. 3A, viewing a scene 310' at a different time during gameplay. As shown, the FOV of the scene 310' is presented to the user 302 within the HMD 320, and the FOV is defined between a number of lines 325'. As shown, the FOV is a portion of the scene 310' and generally includes similar objects as the scene 310 of FIG. 3A, such as a jungle scene including a makeshift camp in a forest of trees, characters 340, protection 360, highlighted object 370-B (corresponding to object 370-A), etc.
[0051] Additionally, the line of sight of the user 302 is defined within the FOV. The line of sight direction can indicate the direction in which the user is looking and, accordingly, the area of the scene 310' that is the focus of the user's attention. The line of sight of the user 302 is defined between a number of lines 330. A particular line of sight direction can be defined by line 330-B determined by eye tracking. Line 370-B can intersect with guard 360, in which case the line of sight of the user 302 can move from right to left or left to right, such as when surveying the scene 310'.
[0052] Although the assault weapon (i.e., corresponding to object 370-A in FIG. 3A ) is not in the line of sight of the user 302, the assault weapon may be within an adjacent region of the line of sight, e.g., within a portion of the field of view defined between the lines 325′. In one embodiment, the assault weapon is then highlighted as object 370-B to draw the user's attention to the assault weapon. Thereafter, when it is determined that the user 302 has directed their gaze at the assault weapon for a period of time, a HUD interface including information related to the assault weapon may be present and / or displayed or surfaced within the scene 310′.
[0053] In one embodiment, objects are identified within the scene 310'. Further, one or more objects may be determined to be important to the user's 302 gameplay. For example, an offensive weapon may be determined to be important to the user's gameplay (e.g., necessary to accomplish a task in the gameplay). The importance of the objects may be determined through rules configured to determine the most relevant objects within the scene or through artificial intelligence (AI) that considers the context of the gameplay, objects within the scene displayed in the FOV, etc.
[0054] In one embodiment, after an object (e.g., an assault weapon) is identified and determined to be important to the user's gameplay, the object is highlighted (e.g., as highlighted object 370-B) within the virtual environment scene 310'. Highlighting may occur if the important object is determined to be within the user's line of sight direction even if the user's line of sight direction does not perfectly match the object. Highlighting of the object may direct the user's attention to the highlighted object 370-B and indicate that the object is important. Without the highlighting, the user may not notice the object and / or may take a long time to notice the object.
[0055] In one embodiment, an object is highlighted only upon a first interaction with that object, i.e., the object has not been previously encountered in the user's 302 gameplay and it may be determined that the user's gaze is directed to the object for the first time in the user's gameplay. As such, it is possible that the user was previously unaware of the object and that the highlighting drew the user's attention to the highlighted object 370-B.
[0056] 3C illustrates a scope mode view 380 providing a zoomed-in view of an object of interest (e.g., character 340) contained within a scene of a virtual environment and the presentation of information 356 related to the object in a HUD interface 355 of the scope mode view 380, according to one embodiment of the disclosure. For example, the scope mode view 380 may be presented to the user 302 when viewing the scene 310 of FIG. 3A or when viewing the scene 310' of FIG. 3B.
[0057] A zoomed-in view of the object of interest is rendered within a scope mode view 380 displayed on the HMD 320. In particular, the FOV of the scene provided to the user 302 in the HMD 320 may be replaced with a scope mode view 380 that includes a zoomed-in view of the object 340 being targeted and / or zoomed in. That is, a scope mode view 380 of the zoomed-in object is presented within the display of the user's HMD. In particular, a corresponding scene object may be targeted within the scope mode view 380, such as the character 340. In one implementation, the scope mode view automatically centers the object displayed on the HMD (e.g., around the virtual crosshairs 385 of the scope). In another implementation, the scope mode view is controlled by user input (e.g., through one or more controllers).
[0058] Scope mode view 380 may indicate that the user's 302 attention is directed to the object being magnified; that is, the object is of interest to the user, such as character 340 as the object of interest. In one embodiment, a HUD interface 355 is displayed in scope mode view 380. For example, HUD interface 355 may be provided as an overlay within scope mode view 380, or may be embedded within the portion of the scene being magnified within scope mode view 380, as described above.
[0059] Additionally, as previously discussed, information related to the object may be presented in the HUD interface 355. A close-up of the HUD interface 355 is shown in FIG. 3C, illustrating information 356 contained in the HUD interface 355. As shown, the information 356 is related to the object being zoomed in. For illustrative purposes only, the information 356 may include the name of the character 340 being zoomed in (i.e., Doko), that the character is a warrior class enemy combatant, highly skilled in close combat, and that seven hits are required to disable the character. In this manner, the user 302 may be fully aware of the situation before engaging with the character 340, Doko.
[0060] FIG. 4 illustrates application of artificial intelligence (AI) to determine the context of a scene in a virtual reality, identifying objects within the scene, and generating information related to the objects based on the context viewable through a head-up display located within the scene, in accordance with one embodiment of the present disclosure.
[0061] In one embodiment, objects within a scene may be identified through rules configured to determine the most relevant objects therein, i.e., identifying an object within a scene, determining whether the object is important to the user's gameplay, generating information related to the object, and displaying or surfacing the information within a HUD interface within the user's FOV in the HMD is performed manually, such as by implementing the rules.
[0062] In another embodiment, an AI may be implemented to identify an object within a scene, determine whether the object is important to the user's gameplay, generate information related to the object, and display or surface the information within a HUD interface of the scene within the user's FOV in the HMD. In particular, the object may be determined through application of an AI that takes into account the context of gameplay and the scene displayed in the user's FOV.
[0063] The AI model 491 is trained using training data provided as input to the machine learning engine 490, and implements a training algorithm to fine-tune the parameters of the AI model 491, enabling the trained AI model to classify and / or identify objects within the user's gameplay scene, enable the AI model to classify and / or identify the importance of objects within the user's gameplay scene, and / or enable the AI model to classify and / or identify information related to objects classified and / or identified as important. Various techniques can be used to train and / or build the AI model 491, such as artificial intelligence, supervised and / or unsupervised learning algorithms, reinforcement learning, and the like. The machine learning engine 490 that runs the AI may be implemented as a type of neural network, representing a network of interconnected nodes including a hierarchy of nodes or layers 492. Knowledge can be exchanged between the nodes, and the AI model 491 is gradually trained by activating certain nodes in the layers 492 in response to inputs. Repeated training using multiple inputs typically improves the accuracy of an AI model491 at classifying objects within a scene, determining their importance, and identifying information associated with those objects.
[0064] As shown, a scene 310 is provided as an input to a feature extractor and / or classifier 410 configured to extract features from the scene. For example, the features may be objects or parts of objects contained within the scene 310. Classification of these objects may be performed by applying manual rules, which result in classification of features and / or objects corresponding to those features. In this manner, objects within the scene are recognized through classification.
[0065] In another embodiment, the extracted features are provided as input to the machine learning engine 490, and the AI model 491 is used to classify objects within the scene 310. In this manner, objects within a scene of a virtual environment are identified and / or classified using artificial intelligence (AI). For example, the AI model 491 (implemented by the machine learning engine 490) is configured to identify and / or classify objects within a scene. That is, the AI model 491 is a machine learning model configured to apply machine learning to classify objects within a scene.
[0066] In yet another embodiment, the AI model 491 is a machine learning model configured to determine a context of a scene within a user's gameplay. That is, the extracted features and classified features are used by the AI model 491 to classify and / or determine a context of a scene within a user's gameplay playing a video game. Thus, the AI model 491 provides the scene context 425 as an output.
[0067] In another embodiment, AI model 491 is configured to determine that the identified object is important to the user's gameplay based on the previously determined context. That is, the extracted and / or classified features and the game context are used by AI model 491 applying machine learning to classify and / or determine the importance of the object within the user's gameplay. For example, AI model 491 can determine whether an object is important or unimportant, which is provided as an output from AI model 491 (e.g., important objects 440).
[0068] If an object is classified as important by the AI model, information related to the object based on the context may be generated. The information may be generated and / or identified or classified using the AI model 491, or may be generated and / or identified using rules. When using AI, the AI model 491 provides the information as output (e.g., information 445).
[0069] In one embodiment, a user may be overwhelmed by the number of objects within a scene that are classified and that capture the user's attention. Thus, a filter 430 may be implemented to reduce the number of objects that are deemed important in the user's gameplay. For example, the filter 430 may identify only the most important objects within the user's gameplay and apply an importance filter based on the classified context of the user's scene and / or gameplay. In one embodiment, the filter determines that the user's gaze is directed toward a corresponding object for the first time during the user's gameplay and passes the object through the filter 430 for presentation to the user. Other identified objects that the user has encountered do not pass through (e.g., are filtered out of) the filter 430. The classified and filtered objects 440 are provided as output from the AI model 491.
[0070] Further, a user may be overwhelmed by the amount of information about a particular object that has been classified and / or identified and classified as important. Thus, the filter 430 may be implemented to reduce the amount of information ultimately presented to the user through the HUD interface. For example, the filter 430 may determine the most relevant and / or important information based on the classified context of the user's scene and / or gameplay. The classified, generated, and / or filtered information 445 is provided as output from the AI model 491.
[0071] In one embodiment, information contained within a HUD interface for presentation to a user is shared with other users and / or players. For example, a user may be part of a team including multiple players participating in a multiplayer game session. Information displayed to the user during the user's gameplay may be presented to other team members, for example, through a HUD interface. The information may be presented to other team members even if they are not looking at the same scene and / or objects that drive the propagation of information within the HUD interface. For example, a user may encounter a sniper within a jungle and information about the sniper is presented to the user through a HUD interface in the scene. The information about the sniper may be useful to other team members even if they have not encountered the sniper during their gameplay. In this manner, all team members are aware of the sniper (e.g., location, capabilities, etc.) and can adjust their gameplay accordingly.
[0072] In another embodiment, team members participating in a multiplayer game session may utilize a team chat platform to communicate during their gameplay. That is, a communication session is established between the user and other players in the team. The chat session may be monitored (e.g., monitoring the audio and / or text of the session). More specifically, an automatic speech recognition (ASR) algorithm may be applied to the audio to understand the communication. A context for the chat may be identified based on the speech recognition. Information related to the chat may be generated based on the context of gameplay by members of the team. For example, information deemed important and disclosed during the chat session may be generated. The information may then be presented to one or more team members in a corresponding HUD interface. In this manner, important information revealed during a chat between two or more members of the team is communicated to all players on the team.
[0073] Accordingly, the present disclosure describes methods and systems for providing information of a head-up display (HUD) of an object within a virtual environment viewed through a head-mounted display (HMD). In this manner, for example, information related to the user's gameplay may be provided to the user to enhance the user's experience when playing a video game.
[0074] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations that use various features disclosed herein. Thus, the examples provided are only some of the possible examples and are not intended to limit the various implementations that are possible by combining various elements to define more implementations. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.
[0075] Embodiments of the present disclosure may be practiced with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.
[0076] With the foregoing embodiments in mind, it will be appreciated that embodiments of the present disclosure may employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful machine operations. The disclosed embodiments also relate to devices or apparatus for performing these operations. The apparatus may be specially constructed for the required purposes, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used with computer programs written in accordance with the teachings herein, or it may prove more convenient to construct a more specialized apparatus to perform the required operations.
[0077] The present disclosure may also be embodied as computer readable code on a computer readable medium, such as a program module, executed by one or more computers or other devices. A computer readable medium is any data storage device that can store data, which can then be read by a computer system. By way of example and not limitation, a computer readable storage medium may comprise a non-transitory computer storage medium and a communication medium. Examples of computer readable media include, but are not limited to, hard disks, network attached storage (NAS), read only memory, random access memory, electrically erasable programmable ROM (EEPROM), flash memory or other memory technology, compact disc ROM (CD-ROM), CD-R, CD-RW, magnetic tape and other optical and non-optical data storage devices, or any other medium that can be used to store accessible information. A computer readable medium may include computer readable tangible media distributed across networked computer systems such that the computer readable code is stored and executed in a distributed fashion.
[0078] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted to occur at slightly different times, or may be distributed within a system that allows processing operations to occur at various intervals related to processing, so long as the processing of the overlay operation is performed in the desired manner.
[0079] Although the foregoing disclosure has been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments of the present disclosure are not to be limited to the details provided herein, but may be modified within the scope and equivalents of the appended claims.
Claims
1. Rendering a view of a virtual environment of a video game for display on a user's head mounted display, the virtual environment including a plurality of objects; Tracking the user's gaze; Detecting that the user's gaze is directed at an object in the view for a period of time; generating a head-up display (HUD) interface including information related to the object; A method for displaying the HUD interface proximate to the object in the view of the virtual environment.
2. and providing the HUD interface as an overlay. The method of claim 1.
3. further comprising embedding the HUD interface within the virtual environment. The method of claim 1.
4. the HUD interface is opaque such that the virtual environment is visible through the HUD interface. The method of claim 1.
5. further detecting that the line of sight is directed away from the object; generating a gaze history function of the tracked gaze after detecting the gaze as directed away from the object, the gaze history function measuring the user's attention to the object; removing the HUD interface if the gaze history function satisfies a time decay function that measures decay of a general user's attention to a general object; The method of claim 1.
6. and further comprising: increasing the transparency of the HUD interface prior to performing the removal of the HUD interface so long as the gaze history function satisfies the time decay function. The method according to claim 5.
7. further comprising: identifying a plurality of objects within a scene of the virtual environment using artificial intelligence (AI), the plurality of objects including the object such that the object is identified; Using the AI to determine a context of the scene within the user's gameplay; determining that the object is important to the user's gameplay based on the context; generating the information related to the object based on the context; The method of claim 1.
8. further comprising: highlighting the object within the view of the virtual environment after the object is identified and the object is determined to be important to the gameplay of the user; the HUD interface is displayed after the user's gaze is detected to be directed toward the highlighted object for the period of time. The method of claim 7.
9. further comprising: zooming in on the HUD interface such that the HUD interface becomes progressively larger within the user's field of view while the user's gaze is directed at the object; detecting that the gaze of the user is directed away from the object; zooming out of the HUD interface so that the HUD interface becomes smaller and smaller within the field of view of the user; The method of claim 1.
10. further, the field of view of the user is a sniper mode view of the virtual environment.
10. The method of claim 9.
11. The method of claim 1 , wherein the information contained within the HUD interface is shared with players, the user and the players being members of a team participating in a multiplayer game session.
12. further comprising: establishing a communication session between the user and a player, the user and the player being members of a team participating in a multiplayer game session; monitoring audio of said communication session; performing speech recognition on the audio; identifying a context of the audio based on the speech recognition; generating second information related to the context; presenting the second information in the HUD interface; The method of claim 1.
13. Furthermore, it is determined that the line of sight of the user is directed toward the object for the first time during the game play of the user. The method of claim 1.
14. 1. A computer readable medium storing a computer program for a game, the computer readable medium comprising: a display device for displaying a view of a virtual environment of a video game for display on a user's head mounted display, the virtual environment including a plurality of objects; having program instructions for tracking the user's gaze, having program instructions for detecting that the user's gaze is directed at an object in the view for a period of time; having program instructions for generating a heads-up display (HUD) interface including information related to the object; having program instructions for displaying the HUD interface proximate to the object in the view of the virtual environment. Computer-readable medium.
15. further comprising program instructions for detecting that the line of sight is directed away from the object; generating a gaze history function of the tracked gaze after detecting the gaze is directed away from the object, the gaze history function measuring the user's attention to the object; and program instructions for removing the HUD interface when the gaze history function satisfies a time decay function measuring decay of a general user's attention to a general object.
15. The computer-readable medium of claim 14.
16. and further comprising program instructions for identifying a plurality of objects within a scene of the virtual environment using artificial intelligence (AI), the plurality of objects including the object as the object is identified; program instructions for using the AI to determine a context of the scene within the user's gameplay; determining that the object is important to the user's gameplay based on the context; and program instructions for generating the information associated with the object based on the context.
15. The computer-readable medium of claim 14.
17. and further comprising program instructions for highlighting the object within the view of the virtual environment after the object has been identified and the object has been determined to be important to the gameplay of the user; the HUD interface is displayed after the user's gaze is detected to be directed toward the highlighted object for the period of time.
20. The non-transitory computer readable medium of claim 16.
18. 1. A computer system comprising: A processor; and a memory coupled to the processor and storing instructions that, when executed by the computer system, cause the computer system to perform a method, the method comprising: Rendering a view of a virtual environment of a video game for display on a user's head mounted display, the virtual environment having a plurality of objects; Tracking the user's gaze; Detecting that the user's gaze is directed at an object in the view for a period of time; generating a head-up display (HUD) interface including information related to the object; displaying the HUD interface near the object in the view of the virtual environment; Computer system.
19. Furthermore, in the method, Detecting that the line of sight is directed away from the object; generating a gaze history function of the tracked gaze after detecting that the gaze is directed away from the object, the gaze history function measuring the user's attention to the object; removing the HUD interface if the gaze history function satisfies a time decay function that measures decay of a general user's attention to a general object; 20. The computer system of claim 18.
20. Furthermore, in the method, using artificial intelligence (AI) to identify a plurality of objects within a scene of the virtual environment, the plurality of objects including the object such that the object is identified; Using the AI to determine a context of the scene within the user's gameplay; determining that the object is important to the user's gameplay based on the context; generating the information related to the object based on the context; 20. The computer system of claim 18.
Citation Information
Patent Citations
Information processing device, information processing method, and program
JP2016192122A
System and method that transition between transparent mode and non-transparent mode in head mount display
JP2018010657A
Game processing program, game processing method, and game processor
JP2019193697A
Heads-up display system and method
US20200301504A1
