Head-up display in virtual reality
By tracking the user's gaze direction in a virtual environment and generating relevant information, the problem of delay and overload of information viewing in video games is solved, the user's attention and reaction speed in the game is improved, and the gaming experience is enhanced.
Patent Information
- Application Number
- CN202280088406.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2021-11-16
- Filing Date
- 2022-11-03
- Publication Date
- 2024-11-08
AI Technical Summary
In video games, users wear information viewing delays and information overloads on the monitor cause users to distract themselves in gameplay, affecting reaction time and action execution.
Rendering views of the virtual environment through a head-mounted display (HMD) and tracking the user's gaze direction, generating a head-up display (HUD) interface associated with the target object, providing useful information to improve the user's decision-making ability in the game.
Improves user's attention concentration and reaction speed in the game, provides useful information to enhance the gaming experience, while avoiding the problem of information overload.
Smart Images

Figure CN118922232A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to computer games, and more particularly to providing information of objects in a heads-up display within a virtual environment. Background Art
[0002] Video games provide gameplay within a virtualized game world. These video games may be formatted in a single-player mode or a multiplayer mode. For example, video games may provide gameplay from a variety of perspectives or viewpoints, including a first-person perspective or a third-person perspective, among others. In the first-person perspective, the player views the virtual game environment from the eyes of a character in a three-dimensional virtual space. In the third-person perspective, the player controls the character as if the third person were viewing the character, such as from a perspective over the character's shoulder or behind his back. Popular game genres viewed from a first-person and / or third-person perspective include shooter games, which involve combat situations using highly desirable weapons (e.g., large, effective guns with unlimited bullets) that follow a storyline or course of action.
[0003] In some configurations, information related to the video game may be presented on a display having a small form factor placed on a body part of the controlled character (such as a display worn on a wrist, arm, or leg, or may be held in the character's hand). In this way, the user may manipulate the character within the virtual environment to obtain a view of the display. For example, for a display worn on the character's wrist, like a watch, the user may raise the arm and / or wrist and / or may tilt the character's head in some manner (e.g., downward) so that the display may be viewed. The information on the display may relate to any aspect of the gameplay.
[0004] However, viewing information on a display held or worn by a character may cause some problems for users playing games. For example, when viewing and digesting the information presented on the display, the user's attention is completely focused on the display rather than the gameplay. That is, the situation presented in the gameplay (e.g., fighting a Boss) does not stop because the character's viewpoint is pointed at the display. Therefore, the user may choose to give up viewing any information on the display in order to focus on the gameplay (e.g., continue to fight the Boss), even if the information may be helpful (e.g., game tips for fighting the Boss). That is, there may not be enough time during the gameplay to provide the user with a view of the display. In addition, there may be too much information presented on the display. That is, there may be too much information so that the user cannot obtain any useful information from the information, or it may take a long time to digest the information. The delay in viewing and digesting the information may have an adverse effect on any subsequent actions and / or reactions of the user when playing the video game.
[0005] It is against this background that embodiments of the present disclosure are presented. Summary of the invention
[0006] Embodiments of the present disclosure relate to computer games, and more particularly to providing information of objects in a heads-up display (HUD) within a virtual environment as viewed through a head-mounted display (HMD).
[0007] In one embodiment, a method is disclosed. The method includes rendering a view of a virtual environment of a video game for display in a head mounted display of a user, wherein the virtual environment includes a plurality of objects. The method includes tracking a gaze of the user. The method includes detecting that the gaze of the user is directed toward an object in the view over a period of time. The method includes generating a heads-up display (HUD) interface including information related to the object. The method includes presenting the HUD interface near the object in the view of the virtual environment.
[0008] In another embodiment, a non-transitory computer-readable medium storing a computer program for a game is disclosed. The computer-readable medium includes program instructions for rendering a view of a virtual environment of a video game for display in a head-mounted display of a user, wherein the virtual environment includes a plurality of objects. Program instructions for tracking the user's gaze. The computer-readable medium includes program instructions for detecting that the user's gaze is directed toward an object in the view over a period of time. The computer-readable medium includes program instructions for generating a heads-up display (HUD) interface including information related to the object. The computer-readable medium includes program instructions for displaying the HUD interface near an object in the view of the virtual environment.
[0009] In another embodiment, a computing system includes a processor and a memory, the memory being coupled to the processor and having instructions stored therein, which, if executed by a computer system, causes the computer system to perform a method. The method includes rendering a view of a virtual environment of a video game for display in a head-mounted display of a user, wherein the virtual environment includes a plurality of objects. The method includes tracking the gaze of the user. The method includes detecting that the gaze of the user is directed to an object in the view over a period of time. The method includes generating a heads-up display (HUD) interface including information related to the object. The method includes presenting the HUD interface near an object in the view of the virtual environment.
[0010] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate by way of example the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The present disclosure may be best understood by referring to the following description in conjunction with the accompanying drawings, in which:
[0012] Figure 1 Components of an example device that may be used to perform aspects of various embodiments of the present disclosure are shown.
[0013] Figure 2 is a flow chart illustrating a method for presenting information in a heads-up display in virtual reality according to one embodiment of the present disclosure.
[0014] Figure 3A A user is shown viewing a scene in virtual reality including a heads-up display providing information about objects in the scene according to an embodiment of the present invention.
[0015] Figure 3B An object is shown highlighted for a user viewing a scene in virtual reality according to one embodiment of the present disclosure.
[0016] Figure 3C A scope mode view is shown according to one embodiment of the present disclosure, which provides a magnified view of a target object within a scene in virtual reality, and presentation of information related to the object in a heads-up display.
[0017] Figure 4 The application of rules and / or artificial intelligence according to one embodiment of the present disclosure is shown to determine a context of a scene in virtual reality, identify objects within the scene, and generate information related to the objects based on the context that can be viewed through a heads-up display provided within the scene. DETAILED DESCRIPTION
[0018] Reference will now be made in detail to various embodiments of the present disclosure, examples of which are shown in the accompanying drawings. Although the following detailed description contains many specific details for illustrative purposes, any person of ordinary skill in the art will appreciate that many variations and modifications of the following details are within the scope of the present disclosure. Therefore, without losing the generality of the claims attached to this specification and without imposing any limitations on the claims, the various aspects of the present disclosure described below are set forth.
[0019] Generally speaking, various embodiments of the present disclosure describe a method for providing information about an object in a heads-up display (HUD) within a virtual environment, such as viewed through a head-mounted display (HMD), and providing the information within the HUD within a scope mode view of a magnified target object in the virtual environment.
[0020] With the above general understanding of various embodiments in mind, exemplary details of the embodiments will now be described with reference to the various figures.
[0021] Throughout this specification, references to "games" or "video games" or "gaming applications" are intended to refer to any type of interactive application that is directed by executing input commands. For purposes of illustration only, interactive applications include applications for gaming, word processing, video processing, video game processing, etc. Furthermore, the terms introduced above are interchangeable.
[0022] It should be understood that the embodiments of the present disclosure may employ various computer-implemented operations involving data stored in a computer system. These operations are those operations that require physical manipulation of physical quantities. Any operation described herein that forms a part of the embodiments of the present disclosure is a useful machine operation. The embodiments of the present disclosure also relate to devices or equipment for performing these operations. The device may be specially constructed for the desired purpose, or the device may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used together with computer programs written according to the teachings of this article, or it may be more convenient to construct more specialized devices to perform the desired operations.
[0023] Embodiments of the present disclosure may be practiced with various computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked over wired or wireless networks.
[0024] Some parts of the following detailed description are presented in terms of other symbolic representations of operations of data bits in programs, logic blocks, processing, and computer memories. These descriptions and representations are the means by which technicians in the field of data processing use to most effectively communicate their work content to other technicians in the field. In this application, programs, logic blocks, processes, etc. are considered to be self-consistent sequences of steps or instructions that lead to desired results. Steps refer to steps that utilize physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has been confirmed that, mainly for common use reasons, these signals can sometimes be appropriately referred to as transactions, bits, values, elements, symbols, characters, samples, pixels, etc.
[0025] However, it should be borne in mind that all of these and similar terms are intended to be associated with appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise apparent from the following discussion, it should be understood that throughout this disclosure, discussions using terms such as "rendering," "tracking," "detecting," "generating," "visualizing," "removing," "identifying," "executing," and the like refer to computer systems or similar electronic computing devices or processors (e.g., Figure 1 100) actions and processes (e.g., Figure 2 A computer system or similar electronic computing device manipulates and transforms data represented as physical (electronic) quantities within a computer system memory, registers, or other such information storage, transmission, or display device.
[0026] The present disclosure may also be embodied as computer-readable code, such as a program module, executed by one or more computers or other devices on a computer-readable medium. In addition, communication media may embody computer-executable instructions, data structures, and program modules, and include any information transfer media. As an example and not limitation, communication media include wired media, such as a wired network or a direct wired connection, and wireless media, such as acoustic, radio frequency (RF), infrared, and other wireless media. Any combination of the above may also be included within the scope of computer-readable media.
[0027] Figure 1 Components of an example apparatus 100 that may be used to perform aspects of various embodiments of the present disclosure are shown. For example, Figure 1 An exemplary hardware system suitable for executing a video game according to one embodiment of the present disclosure is shown, the hardware system being configured to provide and / or present information of an object in a head-up display (HUD) within a virtual environment and / or virtual reality (such as generated by executing an application) as viewed through a head-mounted display (HMD). The block diagram shows a device 100, which broadly represents any single-processor or multi-processor computing device or system capable of executing computer-readable instructions. Examples of device 100 may be combined with or may be, but are not limited to, a personal computer, a workstation, a server computer, a game console, a client terminal, a laptop, a mobile device, a distributed computing system, or other digital device, each of which is suitable for practicing embodiments of the present disclosure.
[0028] Both the central processing unit (CPU) 102 and the graphics processing unit (GPU) 116 are coupled to the memory 104. In general, the CPU 102 and / or the GPU 116 can represent any type or form of processor and / or computing device capable of processing data or interpreting and executing instructions. In some embodiments, the CPU 102 and / or the GPU 116 can receive instructions from a software application or a hardware module. These instructions can cause the CPU 102 and / or the GPU 116 to perform one or more functions of the example embodiments described and / or shown herein. The CPU 102 and / or the GPU 116 can also perform any other steps, methods or processes described and / or shown herein, and / or be a device for performing any other steps, methods or processes described.
[0029] In particular, CPU 102 is configured to run software applications and an optional operating system. CPU 102 may include one or more homogeneous or heterogeneous processing cores. According to various embodiments, CPU 702 is one or more general purpose microprocessors having one or more processing cores. Additional embodiments may be implemented using one or more CPUs having a microprocessor architecture that is particularly suitable for highly parallel and computationally intensive applications such as media and interactive entertainment applications configured for graphics processing during game execution.
[0030] For example, CPU 102 provides data and / or instructions defining a desired output image to GPU 116, and GPU 116 generates pixel data of one or more output images according to the data and / or instructions. The data and / or instructions defining the desired output image may be stored in memory 104 and / or a dedicated graphics memory (not shown). In an embodiment, GPU 116 includes 3D rendering capabilities for generating pixel data of output images according to instructions and data defining the geometry, lighting, shading, texturing, motion, and / or camera parameters of a scene. GPU 116 may also include one or more programmable execution units capable of executing shader programs.
[0031] Memory 104 stores applications and data used by CPU 102 and GPU 116. Memory 104 generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data for applications and / or other computer-readable instructions, and may include, without limitation, fixed disk drives, removable disk drives, flash memory devices, RAM, ROM, and CD-ROM, DVD-ROM, Blu-ray Disc, HD-DVD or other optical storage devices, and signal transmission and storage media. Figure 1In the example of , memory 104 is a shared memory, where the memory stores instructions and data for both CPU 102 and GPU 116. Alternatively, there may be separate memories dedicated to CPU 102 and GPU 116, respectively.
[0032] GPU 116 periodically outputs pixel data of an image from a memory (e.g., a graphics processor, a frame buffer, etc.) for display on display device 110. Display device 110 may be any device capable of displaying visual information in response to a signal from device 100, including CRT, LCD, plasma, and OLED displays. Device 100 may provide, for example, an analog or digital signal to display device 110. In one embodiment, display device 110 includes an HMD (e.g., displaying virtual reality or VR content).
[0033] The user input device 108 communicates user input from one or more users to the device 100, examples of which may include a keyboard, mouse, joystick, touchpad, touch screen, still or video recorder / camera, and / or microphone. In addition, a network interface (not shown) may provide communication with other systems via an electronic communication network (e.g., communication between game consoles, HMDs, controllers, etc.), and may include wired or wireless communication over local area networks and wide area networks (such as the Internet). The components of the device 100 (including the CPU 102, GPU 116, memory 104, user input device 108, network interface, etc.) are connected via one or more data buses 122.
[0034] In some embodiments, a computer-readable medium containing a computer program may be loaded into the computing system 100. All or a portion of the computer program stored on the computer-readable medium may then be stored in the system memory 104 and / or in various portions of storage devices. When executed by a processor (e.g., CPU 102 and / or GPU 116), the computer program loaded into the computing system 100 may cause the CPU 102 and / or GPU 116 to perform the functions of the example embodiments described and / or illustrated herein, and / or be a means for performing the functions. Additionally or alternatively, the example embodiments described and / or illustrated herein may be implemented in firmware and / or hardware.
[0035] In one embodiment, CPU 102 and / or GPU 116 are configured to provide emulation of a computing platform (e.g., an operating system) that is configured to execute software applications. Emulation of a computing platform may be accomplished through software, hardware, or a combination of both. In this manner, CPU 102 and / or GPU 116 may be configured to execute any type of application, whether the application is a legacy application or an application designed for a future computing platform.
[0036] Figure 2 2 is a flow chart 200 illustrating a method for presenting information in a heads-up display in virtual reality according to one embodiment of the present disclosure. In particular, flow chart 200 improves a user's and / or player's ability to play a video game by providing information about the video game within the player's gameplay. In this way, the user is provided with additional information related to the video game that is not typically provided during gameplay. This additional information can make the gameplay more interesting because the user is able to make more informed decisions during the gameplay.
[0037] At 210, the method includes rendering a view of a virtual environment and / or virtual reality for display in a head mounted display (HMD) of a user during execution of a video game. For example, video frames are generated for a three-dimensional field of view (FOV) in a virtual game environment based on input controls and other control signals from the user. In addition, the virtual environment includes a plurality of objects provided within a scene, wherein the scene can be displayed in the HMD as a series of image frames.
[0038] At 220, the method includes tracking the gaze of the user. In particular, the gaze direction of the user relative to the HMD and, accordingly, the gaze direction relative to the scene of the virtual environment displayed in the HMD is determined. Gaze tracking can be performed using any of a variety of techniques. For illustration purposes only, gaze tracking can be performed within the HMD using one or more gaze tracking cameras configured to capture images of the user's eyes, which are analyzed to determine the gaze direction of the user. The gaze direction of the user can be defined relative to the HMD, relative to the user's physical environment (e.g., by tracking the position and orientation of the HMD within the physical environment, the ability to perform gaze tracking allows the gaze direction to be determined with reference to the physical environment, such as in real-world gaze), and / or relative to a virtual environment rendered and / or displayed in the HMD.
[0039] At 230, the method includes detecting that a user's gaze is directed toward an object in the view over a period of time. In particular, a gaze direction of the user relative to the virtual environment is detected. In addition, the direction of entry into the scene is used to determine that the gaze is directed toward one or more objects in the scene that are located near the gaze direction. For example, the gaze direction into the scene may indicate that the user's attention is directed toward an object in the scene that is aligned, closely aligned, or loosely aligned with the gaze direction. For example, the gaze direction may indicate a line of sight drawn into the scene, where an object intersects the line of sight, or may be inferred to intersect the line of sight (i.e., is close to or within the line of sight).
[0040] Additionally, gaze tracking may indicate that the gaze direction is fixed on the object for a period of time. That is, the gaze direction is always directed (e.g., in the same or approximate direction) to an area of a scene in the virtual environment for a period of time, wherein the area of the scene includes the object. In this way, it may be determined that the user's gaze is fixed on the object during that period of time. The period of time is sufficient to indicate that the user's attention may be focused on the object and that the user is interested in the object.
[0041] At 240, the method includes generating a heads-up display (HUD) interface including information related to the object. That is, once it is determined that the user's gaze is directed to the object (which indicates that the user is interested in the object), information about the object can be presented to the user. In one embodiment, the information is presented in a HUD interface that is populated in a scene of a virtual environment. For example, the HUD interface can be configured as any interface (e.g., a display, a window, etc.) that can be placed in a scene to display information presented in the interface. In addition, the information can be one of many formats understandable to the user, such as text, symbols, icons, graphics, bars, status bars, videos, audio, etc., and the information can be used for one of many purposes, such as providing background information about the object, providing useful information related to the object, etc.
[0042] At 250, the method includes presenting a HUD interface near an object in a view of a virtual environment. In particular, the HUD interface including information related to the object is placed near and / or adjacent to the object in the scene. In this way, the HUD interface and the information contained therein can be easily associated with the user's particular object.
[0043] In addition, the HUD interface can be static for a period of time, so that the HUD interface and the information contained therein remain unchanged. In another implementation, the HUD interface and the information contained therein can be continuously updated during the period of time. In other implementations, the HUD interface can move in the scene. For example, when an object moves, the HUD interface can follow the movement of the object, so that the HUD interface remains close to and / or adjacent to the object.
[0044] Figure 3A A user is shown viewing a scene 310 in virtual reality including a heads-up display (HUD) interface 350 that provides information about objects in the scene, according to an embodiment of the present invention. Purely for illustrative purposes only, a user 302 is seated and wearing a head mounted display (HMD) 320 to play a video game. In this manner, the scene is presented within the HMD 320, enabling the user 302 to view a 3D rendering of the scene 310. Although embodiments of the present disclosure may be described with respect to a user playing a video game or gaming application, it should be understood that the user may interact with any type of application, including video games, service-related applications, work-related applications (e.g., word processing, spreadsheets, design applications, computer-aided design (CAD) applications, etc.).
[0045] Purely for illustration, a scene 310 may be presented during gameplay of a user 320, wherein the scene may include a plurality of objects (e.g., characters, backgrounds, environments, etc.). For example, the scene 310 may be a jungle scene including a makeshift camp in the woods, wherein the video game may be a first-person shooter game. The camp may include some form of protection 360, such as an awning including one or more poles. A character 340 is also presented within the scene 310.
[0046] As shown, a field of view (FOV) of a scene is presented to a user 302 within an HMD 320, wherein the FOV is defined between lines 325. The user 302 typically determines what FOV is presented by the orientation of the HMD 320 and / or the user's gaze direction. As shown, the FOV is a portion of a scene 310 and includes a guard 360, an object 370-A, a character 340, and various other background objects (e.g., trees, etc.).
[0047] As shown, the user's gaze is confined within the FOV. That is, gaze tracking can be used to determine the direction of the user's gaze. The gaze direction can indicate the direction of the user's attention and, accordingly, the area of the scene 310 where the user's attention is focused. The gaze of user 302 is confined between lines 330. The specific direction of gaze can be defined using line 330-A determined by gaze tracking. Object 370-A can intersect line 370-A and indicate that the gaze of user 302 is directed toward object 370-A.
[0048] The HUD interface 350 is presented in the scene 310, wherein the HUD interface includes information related to the object 370-A. For example, once it is determined that the user's attention is directed to the object, information can be presented to the user 302 through the HUD interface 350. For example, the HUD interface can be configured as any interface (e.g., a display, a window, etc.), which can be placed in the scene to display the information presented in the interface. For illustrative purposes, the HUD interface can be a two-dimensional (2D) or three-dimensional (3D) panel configured to include information. In one embodiment, the HUD interface 350 is presented in the scene as an overlay. In another embodiment, the HUD interface 350 is embedded in the virtual environment and / or the scene 310, so that the HUD interface 350 is integrated into the virtual environment through the execution of the video game, for example, the HUD interface 350 is used as another object 350 in the scene 310.
[0049] Thus, the HUD interface 350 is presented to the user within the display of the HMD 320. The HUD interface 350 may be configured as any interface (e.g., a display, a window, etc.) that may be placed in a scene to display information presented within the interface. Figure 3A An enlarged view of the HUD interface 350 is shown to facilitate viewing of information 351 presented within the HUD interface, where the information 351 may be in one of many formats understandable to a user, such as text, symbols, icons, graphics, bars, status bars, video, audio, etc.
[0050] In an embodiment, the information 351 presented in the HUD interface 350 can be used for one or more purposes. For example, the information 351 can be intended to be useful to the user, such as helping to advance the user's gameplay (e.g., providing a prompt that can be triggered by an object or related to an object in the gameplay). In addition, the information 351 can include a prompt that the user needs to pick up the object in order to advance the gameplay. In a practical form that is helpful, the information 351 can let the user know what the object is and why the object is in the scene. In other implementations, the information 351 can be intended to help the user understand the video game and / or the gameplay of the video game. That is, the information 351 can be peripherally related to the object 370-A, such as setting the scene 310, or providing background information to the scene. For example, the information 351 can help convey a certain picture of the scene in the virtual environment, or help create a certain atmosphere or emotion (e.g., suspense, happiness, sadness, etc.) of the gameplay. In other implementations, the information 351 can be used to provide background information about the object to the user.
[0051] For illustration purposes only, information 351 may provide background information about the object, provide useful information related to the object, etc. For example, information 351 may be related to object 370-A, such as identifying the object as an offensive weapon, and providing one or more characteristics of the offensive weapon (e.g., medium-range capability, accuracy of the weapon, capacity of the weapon, effectiveness of the weapon rated by mortality level, etc.). In addition, information 351 may include annotations related to object 370-A. As previously described, in one implementation, annotations may be tips that help advance the user's gameplay. For example, annotations may indicate that ownership of object 370-A may help complete current or future tasks in the user's 302 gameplay.
[0052] In one embodiment, HUD interface 350 is opaque and / or has one or more transparencies such that the virtual environment including scene 310 is visible through HUD interface 350. In this way, information 351 in HUD interface 350 can be presented to the user without interfering with the user's gameplay. For example, user 302 may be in the middle of an intense gameplay session (e.g., fighting a boss) where information may be useful to the user (e.g., tips for defeating the boss), but diverting the user's attention too much from scene 310 may be harmful to the user (e.g., losing to the boss while accessing information 351 from HUD interface 350).
[0053] In one embodiment, the HUD interface 350 is enlarged within the display of the HMD 320 while the gaze of the user 302 is directed toward the object 370-A. That is, a zoom-in is performed within the FOV presented in the HMD toward the HUD interface 350, so that the HUD interface 350 gradually becomes larger within the FOV while the user's gaze continues to be directed toward the object. The zoom-in toward the HUD interface 350 and the zoom-out from the HUD interface can be performed based on the user's gaze. For example, when it is detected that the gaze of the user 302 is directed away from the object 370-A, a zoom-out away from the HUD interface 350 is performed within the FOV, so that the HUD interface gradually becomes smaller within the user's FOV. Similarly, when it is detected that the gaze of the user 302 is directed toward the object 370-A, a zoom-in is performed within the FOV toward the HUD interface 350, so that the HUD interface gradually becomes larger within the user's FOV until a specific size of the HUD interface.
[0054] In one embodiment, when it is detected that the user's gaze is directed away from the object 370-A for a period of time, the HUD interface 350 is terminated and / or not displayed in the HMD 320. This period of time is optional to indicate that the user 302's attention is no longer focused on the object 370-A. In particular, gaze tracking can be implemented to detect that the gaze is directed away from the object, and in what way. For example, the gaze is tracked and analyzed to determine the gaze history function of the gaze, wherein the gaze history function measures the user's attention to the object. The gaze history function may include linear and / or nonlinear parts relative to time. In addition, when the gaze history function satisfies the time decay function, the HUD interface 350 may be terminated and / or removed, wherein the time decay function measures the decay attention of the general user about the general object, and can be used to determine when the general user's attention is no longer directed to the general object, that is, the general user is no longer interested in the general object. The time decay function may include linear and nonlinear parts relative to time, and may indicate the user's attention level. The gaze history function matches a specific part of the time decay function to indicate that the user 302's attention is no longer directed to the object 370-A, and / or the user is no longer interested in the object. In one embodiment, as long as the gaze history function satisfies the time decay function (e.g., fading out the HUD interface 350), the transparency of the HUD interface 350 increases (e.g., increasing the degree of transparency) during the transition period before completely removing the HUD interface 351 displayed within the HMD 320. In another embodiment, when the gaze history function does not follow the time decay function, the transparency of the HUD interface 350 displayed within the HMD 320 decreases, indicating that although the user's attention may have been diverted from the object 370-A, the user has once again focused his attention back on the object (e.g., fading in the HUD interface 350).
[0055] Figure 3B 1 shows an object highlighted for a user viewing a scene in virtual reality according to one embodiment of the present disclosure. For example, a user may be playing Figure 3A , and viewing scene 310' at different times during gameplay. As shown, the FOV of scene 310' is presented to user 302 within HMD 320, where the FOV is defined between lines 325'. As shown, the FOV is a portion of scene 310' and generally includes Figure 3A Objects similar to the objects in scene 310 of FIG. 310 , such as a jungle scene including a makeshift camp in the woods, character 340 , guard 360 , highlighted object 370 -B (which corresponds to object 370 -A), and the like.
[0056] In addition, the gaze of user 302 is confined within the FOV. The gaze direction may indicate the direction of the user's attention, and accordingly indicate the area of the scene 310' where the user's attention is focused. The gaze of user 302 is confined between lines 330'. The specific direction of the gaze may be defined using line 330-B determined by gaze tracking. Line 370-B may intersect guard 360, where the gaze of user 302 may move from right to left, or from left to right, such as when browsing scene 310'.
[0057] Although offensive weapons (i.e., Figure 3A 370-A) is not in the direction of gaze of user 302, but the offensive weapon may be in an adjacent area of the gaze direction, such as within a portion of the FOV defined between lines 325'. In one embodiment, the offensive weapon is then highlighted as object 370-B to draw the user's attention to the offensive weapon. Thereafter, once it is determined that user 302 has directed gaze toward the offensive weapon for a period of time, the HUD interface may be populated and / or displayed within scene 310', including information related to the offensive weapon.
[0058] In one embodiment, objects are identified within the scene 310'. Additionally, one or more objects may be determined to be important in the gameplay of the user 302. For example, an offensive weapon may be determined to be important in the user's gameplay (e.g., it is required to complete a task in the gameplay). The importance of an object may be determined by a rule configured to determine the most relevant objects within the scene, or may be determined by an artificial intelligence (AI) that considers the context of the gameplay, objects within the scene displayed in the FOV, etc.
[0059] In one embodiment, after an object (e.g., an offensive weapon) is identified and determined to be important to the user's gameplay, the object is highlighted within the scene 310' of the virtual environment (e.g., as a highlighted object 370-B). When an important object is determined to be within the range of the user's gaze direction, the highlighting may occur even if the user's 302 gaze direction may not be completely aligned with the object. The highlighting of the object may direct the user's attention to the highlighted object 370-B and may indicate that the object is important. Without the highlighting, the user may not be aware of the object and / or it may take a long time to become aware of the object.
[0060] In one embodiment, an object is highlighted only upon the first interaction with the object. That is, it may be determined that the object has not been previously encountered in the user's 302 gameplay, and the user's gaze is directed toward the object for the first time in the user's gameplay. Therefore, the user may not have been previously aware of the object, and the highlighting may draw the user's attention to the highlighted object 370-B.
[0061] Figure 3C 380 is shown, according to one embodiment of the present disclosure, which provides a magnified view of a target object (e.g., character 340) included in a scene in a virtual environment, and presentation of information 356 related to the object in a HUD interface 355 in the scope mode view 380. For example, when viewing Figure 3A When viewing scene 310, or when viewing Figure 3B When the scene 310' is viewed, a scope mode view 380 may be presented to the user 302.
[0062] A magnified view of the targeted object is rendered within a scope mode view 380 displayed in the HMD 320. In particular, the FOV of the scene provided to the user 302 in the HMD 320 can be replaced with a scope mode view 380 that includes a magnified view of the targeted and / or magnified object 340. That is, the scope mode view 380 of the magnified object is presented within the display of the user's HMD. In particular, the object in the corresponding scene can be a target within the scope mode view 380, such as the character 340. In one implementation, the scope mode view can automatically center the object for display in the HMD (e.g., centered on the virtual crosshairs 385 of the scope). In another implementation, the scope mode view is controlled by user input (e.g., by one or more controllers).
[0063] The scope mode view 380 may indicate that the user 302's attention is directed to the object being magnified. That is, the object is of interest to the user, such as character 340 as a target object. In one implementation, the HUD interface 355 is populated in the scope mode view 380. For example, the HUD interface 355 may be provided as an overlay within the scope mode view 380, or may be embedded within the portion of the scene being magnified within the scope mode view 380, as previously described.
[0064] Additionally, as previously described, information related to the object may be presented in the HUD interface 355 . Figure 3CAn enlarged view of the HUD interface 355 is shown to illustrate information 356 contained in the HUD interface 355. As shown, the information 356 is related to the object being enlarged. For illustrative purposes only, the information 356 may include the name of the enlarged character 340 (i.e., Doko), which is a warrior-level enemy combatant who is very good at hand-to-hand combat and requires 7 hits to defeat. In this way, the user 302 can fully understand the situation before engaging with the character 340 Doko.
[0065] Figure 4 The application of artificial intelligence (AI) according to one embodiment of the present disclosure is shown to determine the context of a scene in virtual reality, identify objects within the scene, and generate information related to the objects based on the context that can be viewed through a heads-up display provided within the scene.
[0066] In one embodiment, objects within a scene may be identified by rules configured to determine the most relevant objects within the scene. That is, identification of objects within a scene, determination of the objects as being important to the user's gameplay, and generation of information related to the objects and presentation of the information within a HUD interface within the user's FOV in the HMD are all performed manually, such as by implementing rules.
[0067] In another embodiment, AI may be implemented to identify objects within a scene, determine that the object is important to the user's gameplay, generate information related to the object, and present the information within a HUD interface in the scene within the user's FOV in the HMD. In particular, the objects may be determined by applying AI that takes into account the context of the gameplay and the scene displayed in the user's FOV.
[0068] The AI model 491 is trained using training data provided as input to the machine learning engine 490, which implements a training algorithm to slightly adjust the parameters of the AI model 491 so that the trained AI model can classify and / or identify objects within the user's gameplay scene, so that the AI model can classify and / or identify the importance of objects within the user's gameplay scene, and / or classify and / or identify information related to objects classified and / or identified as important. Various techniques can be used to train and / or build the AI model 491, such as using artificial intelligence, supervised and / or unsupervised learning algorithms, reinforcement learning, etc. The machine learning engine 490 that performs AI can be implemented as a class of neural networks and represents an interconnected node network including a node hierarchy or layer 492. Knowledge can be exchanged between nodes, and certain nodes within the layer 492 are activated in response to input to gradually train the AI model 491. Repeated training with multiple inputs generally improves the accuracy of the AI model 491 in classifying objects within the scene, determining their importance, and identifying information related to these objects.
[0069] As shown, scene 310 is provided as input to feature extractor and / or classifier 410, which is configured to extract features from the scene. For example, features can be objects or parts of objects included in scene 310. Manual rules can be applied to perform classification on these objects so that features and / or objects corresponding to these features are classified. In this way, objects within the scene that are classified are known.
[0070] In another embodiment, the extracted features are provided as input to the machine learning engine 490, and the AI model 491 is used to classify objects within the scene 310. In this way, artificial intelligence (AI) is used to identify and / or classify multiple objects within the scene of the virtual environment. For example, the AI model 491 (implemented by the machine learning engine 490) is configured to identify and / or classify objects within the scene. That is, the AI model 491 is a machine learning model configured to apply machine learning to classify objects within the scene.
[0071] In yet another embodiment, AI model 491 is a machine learning model configured to determine the context of a scene within a user's gameplay. That is, AI model 491 uses the extracted features and the classified features to classify and / or determine the context of a scene within the gameplay of a user playing a video game. Thus, AI model 491 provides the context 425 of the scene as an output.
[0072] In another embodiment, the AI model 491 is configured to determine that the identified object is important to the user's gameplay based on the previously determined context. That is, the AI model 491 that applies machine learning uses the extracted features and / or classified features and the game context to classify and / or determine the importance of the object within the user's gameplay. For example, the AI model 491 can determine whether the object is important, which is provided as an output from the AI model 491 (e.g., important objects 440).
[0073] When the AI model classifies an object as important, context-based information about the object may be generated. The information may be generated and / or identified or classified using AI model 491, or may be generated and / or identified using rules. If AI is used, AI model 491 provides information as an output (e.g., information 445).
[0074] In one embodiment, the number of objects that are classified and attract the user's attention in the scene may overwhelm the user. Therefore, the filter 430 can be implemented to reduce the number of objects that are considered important in the user's gameplay. For example, the filter 430 can apply an importance filter that only identifies the most important objects in the user's gameplay, and the importance filter is based on the classified background of the scene and / or the user's gameplay. In one embodiment, the filter determines that the user's gaze points to the corresponding object for the first time in the user's gameplay, and passes the object through the filter 430 to be presented to the user. Other identified objects that the user has encountered are not passed through the filter 430 (e.g., filtered out). The classified and filtered objects 440 are provided as outputs from the AI model 491.
[0075] In addition, the amount of information that is classified and / or identified and classified as important for a particular object may be overwhelming to the user. Therefore, a filter 430 may be implemented to reduce the amount of information ultimately presented to the user through the HUD interface. For example, the filter 430 may determine the most relevant and / or important information based on the classified context of the scene and / or the user's gameplay. The classified, generated and / or filtered information 445 is provided as an output from the AI model 491.
[0076] In one embodiment, the information included in the HUD interface for presenting to the user is shared with other users and / or players. For example, a user may be part of a team that includes multiple players participating in a multiplayer game session. The information presented to the user in the user's gameplay (such as through the HUD interface) may be presented to other team members. Even if other team members may not view the same scene and / or cause the object of information propagation in the HUD interface, the information may also be presented to them. For example, a user may encounter a sniper in the jungle, wherein information about the sniper is presented to the user through the HUD interface in the scene. Information about the sniper may be useful to other team members, even if these team members have not yet encountered the sniper in their gameplay. In this way, all team members will be aware of the sniper (e.g., position, ability, etc.), and their gameplay can be adjusted accordingly.
[0077] In another embodiment, team members participating in a multiplayer game session can communicate using a team chat platform during their gameplay. That is, a communication session is established between the user and other players in the team. The chat session can be monitored (e.g., monitoring the audio and / or text of the session). More particularly, an automatic speech recognition (ASR) algorithm is applied to the audio in order to understand the communication. The context of the chat can be identified based on speech recognition. Information related to the chat can be generated based on the context of the team member's gameplay. For example, information that is considered important and revealed during the chat session can be generated. The information is then presented to one or more team members in the corresponding HUD interface. In this way, important information revealed during the chat between two or more members in the team is disseminated to all players in the team.
[0078] Thus, the present disclosure describes methods and systems for providing object information in a heads-up display (HUD) within a virtual environment viewed through a head-mounted display (HMD). In this way, for example, information relevant to the user's gameplay is provided to the user and the user experience when playing a video game can be enhanced.
[0079] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that may define more implementations by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.
[0080] Embodiments of the present disclosure may be practiced with various computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked over wired or wireless networks.
[0081] In view of the above embodiments, it should be understood that the embodiments of the present disclosure may employ various computer-implemented operations involving data stored in a computer system. These operations are those operations that require physical manipulation of physical quantities. Any operation described herein that forms a part of the embodiments of the present disclosure is a useful machine operation. The embodiments of the present disclosure also relate to devices or equipment for performing these operations. The device may be specially constructed for the desired purpose, or the device may be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines may be used together with computer programs written according to the teachings of this article, or it may be more convenient to construct more specialized devices to perform the desired operations.
[0082] The present disclosure may also be embodied as a computer-readable code, such as a program module, executed by one or more computers or other devices on a computer-readable medium. A computer-readable medium is any data storage device that can store data, which can then be read by a computer system. As an example and not limitation, a computer-readable storage medium may include a non-transitory computer storage medium and a communication medium. Examples of computer-readable media include, but are not limited to, a hard drive, a network attached storage device (NAS), a read-only memory, a random access memory, an electrically erasable programmable ROM (EEPROM), a flash memory or other storage technology, a compact disk ROM (CD-ROM), CD-R, CD-RW, a magnetic tape and other optical and non-optical data storage devices, or any other medium that can be used to store accessible information. The computer-readable medium may include a computer-readable tangible medium distributed on a network-coupled computer system so that the computer-readable code is stored and executed in a distributed manner.
[0083] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be coordinated so that they occur at slightly different times, or may be distributed in a system that allows the processing operations to occur at various intervals associated with the processing, so long as the processing covering the operations is performed in the desired manner.
[0084] Although the foregoing disclosure has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the embodiments of the present invention are to be considered as illustrative rather than restrictive, and the embodiments of the present disclosure are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
Claims
1. A method comprising: rendering a view of a virtual environment of a video game for display in a head mounted display of a user, wherein the virtual environment includes a plurality of objects; tracking a gaze of the user; detecting that the gaze of the user is directed toward an object in the view over a period of time; generating a heads-up display (HUD) interface including information related to the object; as well as The HUD interface is presented proximate the object in the view of the virtual environment.
2. The method according to claim 1, further comprising: The HUD interface is provided as an overlay.
3. The method according to claim 1, further comprising: The HUD interface is embedded in the virtual environment.
4. The method according to claim 1, The HUD interface is opaque, so that the virtual environment is visible through the HUD interface.
5. The method according to claim 1, further comprising: detecting that the gaze is directed away from the object; generating a gaze history function of the tracked gaze after detecting that the gaze moves away from the object, wherein the gaze history function measures the user's attention with respect to the object; as well as When the gaze history function satisfies a time decay function that measures a general user's decaying attention on a general object, the HUD interface is removed.
6. The method according to claim 5, further comprising: As long as the gaze history function satisfies the time decay function, the transparency of the HUD interface is increased before performing the removing of the HUD interface.
7. The method according to claim 1, further comprising: identifying a plurality of objects within a scene of the virtual environment using artificial intelligence (AI), wherein the plurality of objects includes the object, such that the object is identified; using the AI to determine the context of the scene within the user's gameplay; determining, based on the context, that the object is important to the gameplay of the user; as well as The information related to the object is generated based on the background.
8. The method according to claim 7, further comprising: After identifying the object and after determining that the object is important to the gameplay of the user, highlighting the object within the view of the virtual environment, After detecting that the user's gaze is directed toward the highlighted object within the period of time, the HUD interface is displayed.
9. The method according to claim 1, further comprising: Zooming in toward the HUD interface so that when the gaze of the user is directed toward the object, the HUD interface gradually becomes larger within the field of view of the user; Detecting that the gaze of the user is directed away from the object; as well as Zooming away from the HUD interface causes the HUD interface to gradually become smaller within the field of view of the user.
10. The method according to claim 9, further comprising: Wherein the field of view of the user is a sniper mode view of the virtual environment.
11. The method according to claim 1, further comprising: sharing the information included in the HUD interface with the player, Wherein the user and the player are members of a team participating in a multi-player gaming session.
12. The method according to claim 1, further comprising: establishing a communication session between the user and the player, wherein the user and the player are members of a team participating in a multi-player gaming session; monitoring audio of the communication session; performing speech recognition on the audio; identifying a context of the audio based on the speech recognition; generating second information related to the context; as well as The second information is presented in the HUD interface.
13. The method according to claim 1, further comprising: It is determined that the gaze of the user is directed toward the object for the first time during the user's gameplay.
14. A non-transitory computer-readable medium storing a computer program for a game, the computer-readable medium comprising: program instructions for rendering a view of a virtual environment of a video game for display in a head mounted display of a user, wherein the virtual environment includes a plurality of objects; program instructions for tracking the gaze of the user; program instructions for detecting that the gaze of the user is directed toward an object in the view over a period of time; program instructions for generating a heads-up display (HUD) interface including information related to the object; and Program instructions for presenting the HUD interface proximate the object in the view of the virtual environment.
15. The non-transitory computer readable medium of claim 14, further comprising: program instructions for detecting that the gaze is directed away from the object; program instructions for generating a gaze history function of the tracked gaze after detecting that the gaze moves away from the object, wherein the gaze history function measures the user's attention with respect to the object; and Program instructions for removing the HUD interface when the gaze history function satisfies a time decay function that measures a general user's decaying attention to a general object.
16. The non-transitory computer readable medium of claim 14, further comprising: program instructions for identifying a plurality of objects within a scene of the virtual environment using artificial intelligence (AI), wherein the plurality of objects includes the object, such that the object is identified; program instructions for determining, using the AI, a context for the scene within the user's gameplay; program instructions for determining, based on the context, that the object is important to the gameplay of the user; and Program instructions for generating the information related to the object based on the context.
17. The non-transitory computer readable medium of claim 16, further comprising: program instructions for highlighting the object within the view of the virtual environment after identifying the object and after determining that the object is important to the gameplay of the user, After detecting that the user's gaze is directed toward the highlighted object within the period of time, the HUD interface is displayed.
18. A computer system comprising: processor; a memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to perform a method comprising: rendering a view of a virtual environment of a video game for display in a head mounted display of a user, wherein the virtual environment includes a plurality of objects; tracking a gaze of the user; detecting that the gaze of the user is directed toward an object in the view over a period of time; generating a heads-up display (HUD) interface including information related to the object; and The HUD interface is presented proximate the object in the view of the virtual environment.
19. The computer system of claim 18, wherein the method further comprises: detecting that the gaze is directed away from the object; generating a gaze history function of the tracked gaze after detecting that the gaze moves away from the object, wherein the gaze history function measures the user's attention with respect to the object; as well as When the gaze history function satisfies a time decay function that measures a general user's decaying attention on a general object, the HUD interface is removed.
20. The computer system of claim 18, the method further comprising: identifying a plurality of objects within a scene of the virtual environment using artificial intelligence (AI), wherein the plurality of objects includes the object, such that the object is identified; using the AI to determine the context of the scene within the user's gameplay; determining, based on the context, that the object is important to the gameplay of the user; as well as The information related to the object is generated based on the background.
