Multi-line fixation mapping of objects for determining fixation targets

Abstract

The present application discloses a method for determining an object of interest that a user is looking at in a three-dimensional ("3D") scene. The method includes determining a first gaze direction of the user to the 3D scene, wherein the 3D scene includes a plurality of components. The method also includes executing a plurality of first traces in the 3D scene, wherein each trace of the plurality of first traces is close to the first gaze direction. The method also includes determining a confidence value for each component that intersects at least one trace of the plurality of first traces. The method may additionally include identifying the component having the highest confidence value among all components that intersect at least one trace of the plurality of first traces as the object of interest of the user.

Description

[0001] This application is a divisional application of the invention patent application with the application date of March 29, 2019, the national application number of 201910251739.2, and the invention name of "Multi-line gaze mapping of objects for determining gaze focus targets". Technical Field

[0002] Embodiments of the present invention relate to tracking a user's gaze while the user observes a 3D scene, and in particular to how to determine whether the user's gaze is in the direction of an area of ​​interest in the 3D scene.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application is a continuation-in-part of U.S. Patent Application No. 15 / 171,548, filed on June 2, 2016, which claims the benefit of UK Patent Application No. 1509622.5, filed on June 3, 2015, both of which are incorporated herein by reference in their entirety for all purposes as if fully set forth herein. Background Art

[0005] Detecting the eye and the direction of its gaze is well known and can be accomplished, for example, by illuminating the area in which the eye is to be sought using infrared illumination; capturing an image of the area; and detecting bright spots in the image, the bright spots in the image coming from the pupil and cornea of ​​the eye. This method exploits what is known to photographers as the bright-eye or "red-eye" effect, in which light enters the eye and is reflected through the pupil or absorbed and re-emitted through the pupil, making the pupil appear brighter than the rest of the eye and face. A separate, smaller bright spot (also called a sparkle spot) is produced by the cornea. The relative positions of the pupil and corneal sparkle spot can be used to determine the gaze direction of the eye. Further details are given in U.S. Patent No. 6,152,563, the entire contents of which are incorporated herein by reference for all purposes as if fully set forth herein.

[0006] Alternatively or additionally, a similar technique may be used wherein the infrared illuminator is spaced apart from the image sensor so that the image captured by the image sensor has an unlit pupil, also referred to as a "dark pupil" effect.

[0007] Such gaze tracking technology may be implemented, for example, in a remote gaze tracker proximate to a display, or in a wearable device (e.g., a pair of glasses, a virtual reality headset, an augmented reality headset, a helmet, or the like).

[0008] Such gaze tracking techniques can be used to determine whether a user is looking at a particular object or area on the screen (these objects or areas are collectively referred to as "areas" in this document). This can be used as part of a game, for example. This allows the user to interact with an image on the screen by looking at the image on the screen (the act of looking at the image has a predetermined result), or by combining looking at the image with another control (e.g., the user's gaze is directed toward the image while the user presses a key on a keyboard or mouse).

[0009] Typically, the image on the screen may include a gaze-interactable area and a non-gaze-interactable area.

[0010] Previous methods for determining whether a user's gaze is toward a gaze-interactive region in an image tend to be based on requiring the developer to specify an "interaction mask" to indicate the location of the interactive elements on the screen. These methods can work well within certain constraints. Those constraints include the use of a static "camera" (i.e., the viewpoint from which the image on the screen is determined), and maintaining a small number of movable objects that act as occluders (elements that need to be marked as gaze-interactive, but which are only used for the purpose of transmitting visual selection information, and are not "interactive" themselves). Furthermore, such systems typically rely on gaze-interactive objects being visible when in the visual scene.

[0011] Additionally, it has previously been possible to poll or otherwise project lines from a virtual camera to determine objects in the scene that intersect with them. However, due to the inherent lack of 100% accuracy in gaze tracking technology, it is more preferred to poll regions in the scene to account for errors or deviations in the user's determined gaze position. In practice, this requires searching the interior of a cone shape projected from a visual camera. This is a processing intensive and inefficient solution.

[0012] However, in 3D (e.g., in many computer games or in virtual reality headsets, where the camera can rotate through the scene and there is typically a large amount of geometry defining the scene that can act as occluders), previous approaches have been less successful. Even without occluders, the 3D case is problematic. Creating the necessary occlusion to cope with a 3D environment and the different positions of objects (whether they are interactive objects or occluders) observed from different viewpoints becomes very complex.

[0013] This is because, for example, the number of actors (game entities that can interact with the player in an active way) in 3D games is typically much greater than in 2D applications. In practice, this means that every object in the scene needs to be considered a potential occluder. In contrast, objects that are actually gaze-interactable (e.g., parts of other characters, where characters are actors that can be controlled by the AI ​​or the player) may constitute only 5-10% of each scene. As a result, ten times or more of the bandwidth is required for occluders rather than for gaze-interactable objects. This is inefficient to implement and difficult for game developers to implement.

[0014] Additionally, some entities in the game scene, such as world geometry (houses, mountains, etc.), do not naturally expose rendering bounds or physical bounds, which means that these meshes need to be projected on the screen to create interactive volume occluders for them. This can be computationally expensive. Furthermore, some of this world geometry is very bulky (mountains, etc.), so in order to project them in a meaningful way (to get suitable occluders), it is necessary to first employ a mesh segmentation algorithm before performing any projection. This becomes impractical.

[0015] Finally, in some scenarios, it is desirable to know whether the user's gaze is directed toward an object or area that is not visually rendered in the image displayed on the monitor or screen. For example, the object or area may not be visible or may no longer be within the boundaries of the image displayed on the screen.

[0016] Therefore, there exists the problem of how to efficiently implement gaze tracking in 3D scenes.It is an object of the present invention to at least partially overcome this problem. Summary of the invention

[0017] In one embodiment, a method for determining a focus target of a user's gaze in a three-dimensional ("3D") scene is provided. The method may include determining a first gaze direction of the user to the 3D scene, wherein the 3D scene includes a plurality of components. The method may also include executing a plurality of first traces in the 3D scene, wherein each of the plurality of first traces is close to the first gaze direction. The method may also include determining a confidence value for each component that intersects with at least one of the plurality of first traces. The method may additionally include identifying the component with the highest confidence value among all components that intersect with at least one of the plurality of first traces as the focus target of the user. The method may further include determining a second gaze direction of the user to the 3D scene. The method may further include executing a plurality of second traces in the 3D scene, wherein each of the plurality of second traces is close to the second gaze direction. The method may also include determining a confidence value for each component that intersects with at least one of the plurality of second traces and does not intersect with at least one of the plurality of first traces. The method may also include changing the user's focus target to a component with the highest confidence value among all components intersecting at least one of the plurality of second traces if the highest confidence value of all components intersecting at least one of the plurality of second traces is at least higher than the highest confidence value of all components intersecting at least one of the plurality of first traces by a threshold amount.

[0018] In another embodiment, a system for determining a target of interest that a user is looking at in a 3D scene is provided. The system may include a display device, an eye tracking device, and a processing device. The display device is used to present a 3D scene to a user, wherein the 3D scene includes a plurality of components. The eye tracking device is used to determine a user's gaze direction of the 3D scene. The processing device is configured to perform a method. The method may include receiving data from the eye tracking device, the data identifying a first gaze direction of the user to the 3D scene. The method may also include executing a plurality of first traces in the 3D scene, wherein each of the plurality of first traces is close to the first gaze direction. The method may also include determining a confidence value for each component intersecting with at least one of the plurality of first traces. The method may additionally include identifying the component having the highest confidence value among all components intersecting with at least one of the plurality of first traces as the target of interest of the user. In addition, the method may include receiving data from the eye tracking device, the data identifying a second gaze direction of the user to the 3D scene. The method may also include executing a plurality of second traces in the 3D scene, wherein each of the plurality of second traces is close to the second gaze direction. The method may also include determining a confidence value for each component that intersects at least one trace in the plurality of second traces and does not intersect at least one trace in the plurality of first traces. The method may also include, if the highest confidence value of all components that intersect at least one trace in the plurality of second traces is at least higher than the highest confidence value of all components that intersect at least one trace in the plurality of first traces by a threshold amount, changing the user's focus target to the component with the highest confidence value among all components that intersect at least one trace in the plurality of second traces.

[0019] In another embodiment, a non-transitory machine-readable medium is provided, the non-transitory machine-readable medium having instructions stored thereon for determining a focus target of a user's gaze in a 3D scene. The instructions are executable by one or more processors to perform a method. The method may include receiving data identifying a first gaze direction of a user to the 3D scene. The method may also include executing a plurality of first traces in the 3D scene, wherein each of the plurality of first traces is close to the first gaze direction. The method may also include determining a confidence value for each component intersecting with at least one of the plurality of first traces. The method may additionally include identifying the component having the highest confidence value among all components intersecting with at least one of the plurality of first traces as the focus target of the user. In addition, the method may include receiving data identifying a second gaze direction of the user to the 3D scene. The method may also include executing a plurality of second traces in the 3D scene, wherein each of the plurality of second traces is close to the second gaze direction. The method may also include determining a confidence value for each component that intersects at least one of the plurality of second traces and does not intersect at least one of the plurality of first traces. The method may also include, if the highest confidence value of all components that intersect at least one of the plurality of second traces is at least a threshold amount higher than the highest confidence value of all components that intersect at least one of the plurality of first traces, changing the user's focus to the component with the highest confidence value among all components that intersect at least one of the plurality of second traces. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention is described with reference to exemplary embodiments and the accompanying drawings, in which:

[0021] Figure 1 It is a diagram used to explain how a 3D scene is rendered as a 2D image;

[0022] Figure 2 is a schematic diagram showing the use of separate vision cameras and gaze-detection cameras;

[0023] Figure 3 are schematic diagrams showing different uses of separate vision cameras and gaze-detection cameras;

[0024] Figure 4 is a schematic diagram showing another use of separate vision cameras and gaze-detection cameras;

[0025] Figure 5 is a schematic diagram of a system for implementing gaze detection;

[0026] Figure 6is a block diagram of a method for determining a focus target of a user's gaze on a 3D scene according to the present invention;

[0027] Figure 7 yes Figure 6 illustrative examples of execution of the method; and

[0028] Figure 8-10 is related Figure 6 Additional instructions for the implementation of the method. DETAILED DESCRIPTION

[0029] The present invention relates to gaze tracking relative to a 3D scene. The term "gaze tracking" is used herein to refer to any system that detects and / or monitors the position of an eye and / or the direction of gaze of an eye. Such systems will be known and understood by those skilled in the art.

[0030] It is also important to note that the present description also refers to "gaze-interactable" objects (or subjects) or areas. This phrase is intended to indicate an object that the system can detect the user's gaze is directed towards. This may or may not result in an explicit "interaction" for the user. For example, when the system detects that the user's gaze is towards a gaze-interactable object, the user may not see the evidence. However, the fact that the object has been casually looked at has been detected and can be used by the system in a non-explicit way (e.g., for statistical analysis). Therefore, the term "gaze-interactable" should not be interpreted as requiring explicit interaction with the user.

[0031] Figure 1 An example of a 3D scene is provided. Such a scene may be a virtual 3D scene 10, for example, in a computer game or created for a virtual reality headset. The scene 10 contains 3D objects 1. Such objects 1 may move over time in the scene 10. In fact, objects 1 may also move out of the scene 10, and other objects 1 may move into the scene. When such a 3D scene is viewed on a monitor or computer screen, the monitor or screen presents a 2D image representing the view seen through a viewport 2 on the 3D scene from a particular position in space. The viewport 2 defines an image, which is seen by a camera or eye located at an abstract position relative to the 3D scene 10. The combination of the camera position and the viewport 2 sets the field of view 4. Figure 1 Also shown is a viewing frustum 3 which is the extent of the 3D scene 10 shown in the viewport 2 (ie, extends to the rearmost portion of the 3D scene 10 ).

[0032] Figure 1The schematic diagram of also applies to stereoscopic images used, for example, in virtual reality headsets. In that case, each eye of the user is presented with a slightly different viewport to the scene, and the user's brain then perceives a single 3D image. In this case, each image is typically deformed in a non-linear manner to compensate for the lens characteristics. However, the essential concept is the same as that described above with reference to Figure 1 The same is discussed for each image presented to each eye of the user.

[0033] The 3D object 1 can be an abstract, infinite, complex 3D shape constructed from polygons. The image seen through the viewport 2 is a 2D projection of the 3D object 1 on the plane of the viewport 2 when viewed from the position of the camera, as described above. However, the screen on which the 2D viewport 2 image is displayed is composed of an array of discrete elements or pixels. Therefore, the projection of the 3D object 1 must be rasterized on the display grid by sampling the projected image at each screen pixel position to produce an image that will be presented as a visual scene (i.e., the scene observed through the viewport 2). The pixel values ​​are stored in a storage area or buffer as a 2D "bitmap" that corresponds to the same dimensions as the screen or viewport 2.

[0034] It is noted that the term "bitmap" has several possible meanings. In the context of the present invention, and as is common in the field of computer graphics, "bitmap" is used to refer to the concept of a spatially mapped array of pixels, which may also be considered a "pixmap". The term bitmap is not intended to impose any limitations on the bit depth of the stored information. In the present invention, the bit depth of the stored information is variable.

[0035] The present invention uses a bitmap that specifically stores information about gaze-interactive areas in the 3D scene 10. This bitmap can be combined with the visual bitmap, or can be completely separate. According to the present invention, the use of such a bitmap is a dynamic and efficient way to generate "interactive masks", which otherwise the developer has to specify manually in the 3D scene. The efficiency is partially achieved because the development of masks is considered to be a "screen space" problem rather than a 3D scene problem. That is, compared to the problem of specifying masks as a problem specified by the developer to be solved in the 3D world as before, the starting point of the present invention is to realize that gaze tracking is performed relative to a 2D window or viewport (typically, but not necessarily, a screen). Therefore, "masks" can be created based on a 2D window, rather than in the 3D world.

[0036] When considering gaze tracking, not all 3D objects 1 will be gaze-interactive. Therefore, according to the present invention, it is possible to simplify the 3D scene 10 for gaze tracking by considering only those areas that are gaze-interactive. This can be used to generate a separate bitmap that can be stored to a buffer for use with a gaze detection system. This can be discussed in more detail later.

[0037] The bitmaps used for gaze detection will correspond to the visual scene observed by the user on the screen in a predetermined manner. For example, they may represent the same 3D scene 10 observed from the same camera position through the same viewport (but nonetheless, storing different information, since the gaze detection bitmap will only take into account the properties of the gaze-interactable objects, as discussed below). However, that relationship is not the only possible one.

[0038] For example, the cameras for the gaze detection bitmap and the visual scene may be co-existing (coincident), but the visual scene displayed to the user may be incomplete. An example of such a reason may be that the user is using the "picture-in-picture" mode on the television. Therefore, it is possible to use the gaze detection bitmap to track whether the user's gaze tracks the position of the object as the object passes through the screen area showing different, overlaid images. In another variation, the same object can appear in different forms in the visual and gaze interaction renderings of the 3D scene 10. For example, in the gaze interaction rendering, object 1 can have an increased size so that when the user's gaze is directed to the object 1, the user's gaze is more likely to notice that object 1. Other different possibilities are discussed later.

[0039] In some scenarios, the viewport used for the gaze-interactive bitmap may be different from the visual scene viewport. This may be the case, for example, if it is desired to track whether the user's gaze continues to be directed toward an object that moves outside the viewing frustum 3. In that case, the camera used for the gaze-interactive bitmap may be located at the same abstract position as the camera used for the visual scene viewport, but the size of the gaze-interactive viewport may be larger. This is in Figure 2 It should be noted that although Figure 2-4 3D concepts are shown, but for simplicity they are presented in 2D plan views.

[0040] exist Figure 2 In FIG. 5 , a 3D gaze-interactive object 1 is observed from the same position by both a visual camera 20 and a gaze detection camera 30. However, the corresponding gaze detection camera has a larger field of view, resulting in a larger gaze detection viewport 5 compared to the smaller visual scene viewport 2 (note: for clarity purposes, features 2 and 5 are shown in FIG. 5 ). Figure 25. The 3D objects 1 are shown slightly apart in the image, but for the purposes of this discussion they are considered to be coincident). In addition, there may be other differences in optical parameters between the cameras, such as depth of focus, resolution, and distortion including barrel distortion, pincushion distortion, etc. As a result, only the square 3D object 1 is seen in the visual camera 20, while the triangular 3D object 1 is partially visible to the gaze detection camera through the gaze detection viewport 5. In this way, it is possible to determine that the user's gaze is towards the triangular object 1, even if it is not visible in the visual viewport 2.

[0041] Figure 3 An alternative possibility is depicted, in which the positions of the visual scene camera 20 and the corresponding gaze tracking camera 30 are not consistent. This may, for example, take into account an offset between the position and orientation of the screen (compared to the user's eyes). In this case, Figure 3 As shown, not only are different gaze-interactable objects 1 visible through different viewports 2, 5 (the gaze detection viewport 5 shows both the triangular object and the square object 1, whereas the visual viewport 2 only shows the square object 1), but the viewports 2, 5 will also show different projections of the square object 1 when it is viewed from different angles.

[0042] As mentioned above, in another scene, not all objects in the scene can be gaze-interactable. Figure 4 In Figure 4 , only the square object 1 is gaze-interactable, and therefore only the object 1 is rendered to the viewport 5 of the gaze detection camera 30. However, both the square object and the triangular object 1 are rendered to the corresponding visual viewport 2.

[0043] It should be noted that in Figure 4 In the scene of , square object 1 is rendered to gaze detection viewport 5, even though square object 2 may not be visible (or at least partially blocked) in the visual scene due to the position of triangle object 1 between square object 1 and viewport 2. Non-gaze-interactive objects are effectively ignored in the creation of the gaze interaction bitmap.

[0044] In operation, Figure 4 In a scene of the present invention, the visual scene and the corresponding gaze-interactable scene can (for example) be rendered completely independently. Optionally, objects in a single virtual scene 10 can be provided with characteristics to identify the object as an object that is rendered or not rendered in one or the other of the visual scene or the gaze detection scene.

[0045] Alternatively, although not shown in the figure, there are elements in the scene that are actually only gaze-interactive and are not obvious in the visual scene. For example, an object can become invisible in the visual scene, but can remain gaze-interactive (i.e., so that the user can try to track its progress by inferring its previous movement). Alternatively, the 3D scene can contain sectors, zones, and ranges that are gaze-interactive, but they do not correspond to specific visual objects. For example, a scene in a game can contain a "dangerous" range that will trigger a response if the user's gaze is directed toward the range.

[0046] The above section of the description has discussed different ways in which the vision camera and the gaze interaction camera 20, 30 may correspond to each other. The following section discusses in more detail how to generate and use a gaze interaction bitmap.

[0047] Figure 5 A schematic diagram of a system 40 is shown in which gaze detection can be implemented. System 40 includes a screen or monitor 41, and a gaze detector 42. The screen can be in common or uncommon form, including a virtual screen (e.g., a retinal projector or a head-mounted display in a vehicle). Both the gaze detector 42 and the screen 41 are connected to a computer 43. In modern computers 43, for parallel computing, a graphics processing unit (GPU) 45 is often 10-100 times faster than a central processing unit (CPU) 44. Therefore, it is preferred to implement the present invention using a GPU 45. However, the present invention can be implemented using any suitable processor (e.g., CPU 44).

[0048] The computer also includes a memory 46, which includes a buffer in which the gaze detection bitmap 47 can be stored. As mentioned above, the buffer is preferably of variable bit depth. The buffer contains at least enough bit depth to encode a property representation of each gaze-interactive area 1 (as mentioned above, the term "area" is used to include both the range of 3D objects and 3D space) presented in the scene 10. In the case where the gaze interaction camera and the visual representative camera 30, 20 are co-located and have the same size of viewport 5, 2, the buffer can be implemented in conjunction with the visual buffer. Specifically, if the visual buffer is implemented in RGBA format, it is common to have unused bits in the alpha channel. In some embodiments of the present invention, those idle bits can be used to store information related to the area of ​​interest 1.

[0049] In any case, whatever the specific format used, the buffer is used to store a bitmap 47 representing the viewport 5 on the 3D scene from the gaze interaction camera 30. The pixels of the bitmap 47 onto which the gaze-interactive area 1 is projected are used to store the properties of the gaze-interactive area 1.

[0050] The property stored in the bitmap 47 may be a unique identifier (ID) that identifies a particular gaze-interactable area. However, this is not always the case. For example, the idea of ​​having a "dangerous" area in scene 10 has been discussed before. In that case, instead of identifying that the user's gaze has been directed toward (any) one of the "dangerous" areas, there is no need to distinguish between the various "dangerous" ranges. In that case, the information stored in the bitmap may simply indicate that the range has a "dangerous" property, without any unique ID.

[0051] The pixels in the bitmap 47 may be used to store information about more than one property of the region of interest. For example, in conjunction with (or in lieu of) a unique ID, it may be desirable to store other information about the gaze interaction.

[0052] Such gaze interaction information may be metadata associated with the object 1, for example, indicating the relative visual importance of the object 1. This information may be used to assist the algorithm to detect whether the user is looking at a particular object in the case where two gaze-interactable objects 1 are located close to each other: if it is difficult to determine which of the two objects the user is looking at, it may be assumed that the user is looking at the most visually "important" object.

[0053] Other useful gaze interaction information that can be stored in the bitmap 47 can also include information about the motion of the object 1, for example, its speed and / or acceleration. This information can also be related to a heuristic that measures the likelihood that the user's gaze is directed toward any particular object in a particular zone (for example, faster moving objects are more likely to attract the user's attention). Any other variables used for such a measure can also (or alternatively) be stored in the bitmap 47.

[0054] In addition to the visual scene information, creating the gaze interaction bitmap 47 adds computational requirements. In the present invention, one way to save computational cost is to (if necessary) for example downsample the size of the gaze interaction bitmap 47 compared to the visual scene (i.e., use fewer pixels for it). Another optional way is to specify the parts of the visual scene that are potentially gaze-interactive (e.g., to ignore menu bars or other visual items that are constant, which may mean that some areas of the scene will never be visually interactive). This will reduce the computational load (but at the expense of reduced accuracy in the case of downsampling).

[0055] As is apparent from the above discussion, the only items rendered to the gaze interaction bitmap 47 are the objects and regions identified as gaze-interactable areas 1. Such objects and regions are predefined in the programming / design of the 3D scene 10.

[0056] Additionally, the information stored in the bitmap 47 may include relevant properties defined when creating the scene 10 regarding the gaze-interactable area 1. In some cases, it may be desirable to store information about multiple objects in the same pixel of the bitmap (i.e., if multiple gaze-interactable objects are aligned one behind the other), and in that case, an associated index may be stored to, for example, indicate the order of approach of the objects relative to the viewport.

[0057] Furthermore, any available depth buffer and / or stencil buffer data may be used when rendering the bitmap 47 based on a projection of the 3D scene 10. In this way, it is possible to effectively provide a more efficient, integrated and supported alternative to previous methods to handle occlusion and world geometry splitting / projection, which otherwise would be cumbersome. This goal is possible by considering "interactive occlusion" in a 2D environment in the same manner as projecting a scene into a viewport, and thus utilizing existing graphics rendering capabilities in the creation of the occlusion (i.e., the bitmap 47). Once the bitmap 47 is created and stored, it can be used in conjunction with any eye tracking hardware 42 to determine whether the user's gaze is toward the gaze-interactive area 1. For example, this can be done by monitoring the user's eyes relative to the visual scene displayed on the screen 41, but it can also be done by monitoring the user's eyes relative to the viewport 5 used to gaze at the interactive bitmap 47, or even a completely separate viewport. As long as the relationship between the different viewports is known, the corresponding scenes 10 from each viewport can be correlated and a determination can be made regarding the user's gaze direction.

[0058] Once the user's gaze direction is determined, the gaze interaction bitmap 47 can be used to determine whether the gaze is towards the gaze-interactable area 1 by referring to the property information stored in the bitmap 47. Any information about the properties of the gaze-interactable area 1 towards which the user's gaze is directed can then be used to determine the result that the user is looking at a particular area 1.

[0059] As discussed above, by effectively rendering the gaze interaction scene, the present invention allows for efficient determination of whether a user is looking at a gaze-interactive object in a 3D scene. The information about the gaze-interactive scene (e.g., in conjunction with the visual scene) can be used to determine the result of the user's gaze direction.

[0060] Figure 6 A block diagram of a method 600 of the present invention for identifying an object of interest at which a user is gazing in a three-dimensional ("3D") scene is shown. Figure 7A schematic diagram 700 of a simulated 3D viewpoint of a user is shown, with the user's gaze on a 3D scene determined by an eye tracking device. The user's viewpoint 705 observes a 2D display 710 displaying a rendered 3D scene 715. In this example, the 3D scene 715 contains two objects 720A, 720B. The objects 720A, 720B are rendered as shapes 725A, 725B. Figure 8 Display 710 is shown as viewed directly from the user's point of view 705 .

[0061] In block 605, a 3D scene 715 is displayed to a user on a display 710. Display 710 may be a wearable device such as a virtual reality headset, or a conventional display such as a television, monitor, or projection device. The scene may contain a number of objects rendered in 3D space so that the system causing the scene to be displayed can identify them individually and assign / maintain different properties to each object.

[0062] In block 610, the user's gaze direction with respect to the 3D scene is detected using an eye tracking device, as discussed herein. It is desirable to determine what object in the 3D scene the user is most likely looking at, the user's most likely gaze being due to the fact that objects in the 3D scene can move relative to the user (i.e., the objects can move, or the user's position / perspective can move), objects can be in close proximity to each other, and / or obscured (in whole or in part) by other objects or items. Thus, the remainder of the method continues to determine the object being looked at by the user. First detected gaze direction 805 is in Figure 8 Shown in.

[0063] At block 615, the processing device performs a plurality of line traces in the 3D scene near the user's viewpoint. Line traces can be roughly described as imaginary lines projected into the 3D scene to determine what objects or parts of objects the user may be observing. Thus, the line traces may intersect with large objects in the immediate foreground of the scene, but small objects immediately behind and obscured by the large object in the scene do not intersect with the line traces.

[0064] Four example stitches 810 are Figure 8 805 . In fact, any number of traces may be used. The positions of the traces may be distributed in different sectors 815 as shown, with the distances of the traces from the gaze direction 805 being arranged in a Gaussian manner (i.e., the traces are more likely to be closer to the gaze direction 805 than farther away). Although only four sectors 815 are shown, any number of sectors may be used. Similarly, the size of the approximate circle 815 in which the traces 810 are distributed may vary.

[0065] At block 620, it is determined what object the traces 810 intersect. In this example, object 720A is intersected (as shown by the intersection of one of the traces 810 with the 2D rendering 725A). This object is added to the list of candidate objects that may be the user's target of attention.

[0066] At block 625, a confidence value is determined for each candidate object in the list. The confidence value may be greater than a minimum value, or a function of the number of intersecting traces (here 1) relative to the total number of projected traces (here 4).

[0067] At block 630, since this is the first detected gaze direction 805 (and first set of line traces) for the scene, the candidate object with the highest confidence value is selected as the target of interest. Therefore, object 720A (2D rendering 725A) is selected as the target of interest. This is as expected because the first gaze direction 805 is closest to object 720A.

[0068] At block 635, the confidence value for each object in the candidate object list is decayed by a first value. Thus, the confidence value associated with object 720A is decayed by a first value. Any object in the list that does not intersect a trace in the previous round of gaze direction detection / trace may be decayed by a second value that is greater than the first value.

[0069] The method is then repeated, wherein the rendered 3D scene is displayed again, with any changes in the orientation and / or position of the adjusted objects of the scene. In the examples discussed herein, the orientation and / or position are not changed, as this is not necessary for understanding the embodiments of the present invention.

[0070] Fig. 9 A second detected gaze direction 905 and corresponding trace 910 are shown. In this round of detection and traces, one trace has hit object 720A and the other trace has hit object 720B. Therefore, object 720B is added to the candidate object list and the associated confidence value is calculated as described above. Object 720A also continues in the list and the associated confidence value is calculated. Since the confidence value of object 720B does not exceed the confidence value of object 720A, which was the previously determined target of interest, by at least the threshold amount, the determined target of interest continues to be object 720A.

[0071] However, once again, the confidence values ​​of all objects on the candidate object list are decayed by a certain value. As will be apparent to one skilled in the art, decaying the confidence values ​​of candidate objects assists in removing candidate objects from the list (once the candidate objects are not hit by the trace during one or more rounds of execution of the method). The minimum possible confidence value also ensures that candidate objects are not immediately removed from the candidate object list by decay. In addition, one skilled in the art will also recognize the potential need to require that the confidence value of any potential new target of interest exceed that of previously identified targets of interest by at least a threshold amount to prevent hysteresis (i.e., repeatedly changing targets of interest between objects due to insignificant changes / differences in confidence values).

[0072] Fig.10 A third round of gaze detection 1005 and trace 1010 is shown. The above process is repeated, and due to the increasing number of traces 1010 that encounter object 720B, together with the decay of the confidence value of object 720A, the confidence value of object 720B exceeds the confidence value of object 720A by at least a threshold amount, thereby selecting a new focus target (object 720B).

[0073] In some embodiments, if an object still on the candidate object list is not hit by the new round of stitches and therefore appears to be "missed", one or more stitches are made at and / or near the last known location where the stitches last intersected the "missed" object. Because of the arbitrariness of the stitch locations that are chosen, it is possible that an object may be missed when it is actually still there, or is at least partially obscured by another object or other item.

Claims

1. A method for determining an object of interest at which a user is looking in a three-dimensional ("3D") scene, wherein the method comprises: Determining a first gaze direction of a user with respect to a 3D scene, wherein the 3D scene comprises a plurality of components; performing a plurality of first traces in the 3D scene, wherein each of the plurality of first traces is close to the first gaze direction, and wherein the plurality of first traces comprises at least two traces, wherein each of the at least two traces is at a different angular distance from the first gaze direction; determining a confidence value for each component intersected by at least one stitch in the plurality of first stitches; as well as A component having a highest confidence value among all components intersecting at least one stitch of the plurality of first stitches is identified as a target of interest of the user.

2. The method for determining a focus target that a user is looking at in a 3D scene according to claim 1, wherein the method further comprises: Determining a second gaze direction of the user with respect to the 3D scene; performing a plurality of second traces in the 3D scene, wherein each of the plurality of second traces is close to the second gaze direction; determining a confidence value for each component that intersects at least one of the plurality of second traces and does not intersect at least one of the plurality of first traces; If the highest confidence value of all components intersecting with at least one of the plurality of second traces is at least higher than the highest confidence value of all components intersecting with at least one of the plurality of first traces by a threshold amount, the user's focus target is changed to the component having the highest confidence value among all components intersecting with at least one of the plurality of second traces.

3. The method for determining a focus target that a user is looking at in a 3D scene according to claim 2, wherein the method further comprises: A confidence value for each component that intersects at least one of the first plurality of traces and at least one of the second plurality of traces decays over time.

4. The method for determining a focus target that a user is looking at in a 3D scene according to claim 2, wherein the method further comprises: decaying a confidence value of each component intersecting at least one of the plurality of first traces and at least one of the plurality of second traces at a first rate over time; as well as The confidence value of each component that intersects at least one of the first plurality of traces but does not intersect at least one of the second plurality of traces decays over time at a second rate, wherein the second rate is greater than the first rate.

5. The method for determining a focus target that a user is looking at in a 3D scene according to claim 1, wherein: The 3D scene is rendered on at least one two-dimensional display.

6. The method for determining a user's gaze target in a 3D scene according to claim 1, wherein each of the plurality of first traces approaches the first gaze direction comprises: Each of the plurality of first traces has an angle close to the first gaze direction.

7. The method for determining a focus target that a user is looking at in a 3D scene according to claim 1, wherein: The angular distance of each of the at least two stitches is smaller than a predetermined maximum angular distance.

8. The method for determining a focus target that a user is looking at in a 3D scene according to claim 7, wherein: The angular distance between the first gaze direction and any one of the at least two stitches is random.

9. The method for determining a focus target of a user's gaze in a 3D scene according to claim 1, wherein the plurality of first traces comprises: At least two stitches, wherein each of the at least two stitches is at a different angular direction with respect to the first gaze direction.

10. The method for determining a focus target that a user is looking at in a 3D scene according to claim 9, wherein: Each of the at least two traces is in a different sector of a circle centered at the first gaze direction.

11. The method for determining a focus target that a user is looking at in a 3D scene according to claim 2, wherein: If none of the second plurality of traces intersects a particular component that intersects at least one of the first plurality of traces: determining a previous position of the particular component; and At least one stitch is performed in the direction of the previous position.

12. The method for determining a focus target of a user's gaze in a 3D scene according to claim 11, wherein performing at least one line trace toward the previous position comprises: A plurality of third stitches are performed, the plurality of third stitch angles being close to the direction of the previous position.

13. The method for determining a focus target of a user's gaze in a 3D scene according to claim 1, wherein determining a confidence value for each component intersecting at least one of the plurality of first traces comprises: assigning a predetermined minimum value to the confidence value; or The confidence value is a function of: The number of traces in the plurality of traces that intersect the component; and The total number of stitches in the plurality of stitches.

14. The method for determining a focus target of a user's gaze in a 3D scene according to claim 1, wherein determining a confidence value for each component intersecting at least one of the plurality of first traces comprises: assigning a predetermined minimum value to the confidence value; or The confidence value is a function of: a number of stitches in the plurality of stitches that intersect the component; a total number of stitches in the plurality of stitches; and The specified or determined properties of the component.

15. A system for determining an object of interest that a user is looking at in a 3D scene, the system comprising: A display device, the display device is used to present a 3D scene to a user, wherein the 3D scene includes a plurality of components; An eye tracking device, the eye tracking device is used to determine the gaze direction of the user with respect to the 3D scene; as well as A processing device, the processing device being configured to at least: receiving data from the eye tracking device identifying a first gaze direction of a user with respect to the 3D scene; performing a plurality of first traces in the 3D scene, wherein each of the plurality of first traces is close to the first gaze direction, and wherein the plurality of first traces comprises at least two traces, wherein each of the at least two traces is at a different angular distance from the first gaze direction; determining a confidence value for each component intersected by at least one stitch in the plurality of first stitches; as well as A component having a highest confidence value among all components intersecting at least one stitch of the plurality of first stitches is identified as a target of interest of the user.

16. The system for determining a focus target of a user's gaze in a 3D scene according to claim 15, wherein the processing device is further configured to at least: receiving data from the eye tracking device identifying a second gaze direction of the user with respect to the 3D scene; performing a plurality of second traces in the 3D scene, wherein each of the plurality of second traces is close to the second gaze direction; determining a confidence value for each component that intersects at least one of the plurality of second traces and does not intersect at least one of the plurality of first traces; as well as If the highest confidence value of all components intersecting with at least one of the plurality of second traces is at least higher than the highest confidence value of all components intersecting with at least one of the plurality of first traces by a threshold amount, the user's focus target is changed to the component having the highest confidence value among all components intersecting with at least one of the plurality of second traces.

17. The system for determining a focus target of a user's gaze in a 3D scene according to claim 16, wherein the processing device is further configured to at least: decaying a confidence value of each component intersecting at least one of the plurality of first traces and at least one of the plurality of second traces at a first rate over time; and The confidence value of each component that intersects at least one of the first plurality of traces but does not intersect at least one of the second plurality of traces decays over time at a second rate, wherein the second rate is greater than the first rate.

18. A non-transitory machine-readable medium having stored thereon instructions for determining an object of interest at which a user is looking in a 3D scene, wherein the instructions are executable by one or more processors to at least: Receiving data identifying a first gaze direction of a user with respect to the 3D scene; performing a plurality of first traces in the 3D scene, wherein each of the plurality of first traces is close to the first gaze direction; determining a confidence value for each component intersected by at least one of the plurality of first traces, wherein the plurality of first traces comprises at least two traces, wherein each of the at least two traces is at a different angular distance from the first gaze direction; and A component having a highest confidence value among all components intersecting at least one stitch of the plurality of first stitches is identified as a target of interest of the user.

19. The non-transitory machine-readable medium of claim 18, the non-transitory machine-readable medium having stored thereon instructions for determining a target of interest at which a user is looking in a 3D scene, wherein the instructions are further executable by the one or more processors to at least: receiving data identifying a second gaze direction of the user with respect to the 3D scene; performing a plurality of second traces in the 3D scene, wherein each of the plurality of second traces is close to the second gaze direction; determining a confidence value for each component that intersects at least one of the plurality of second traces and does not intersect at least one of the plurality of first traces; as well as If the highest confidence value of all components intersecting with at least one of the plurality of second traces is at least higher than the highest confidence value of all components intersecting with at least one of the plurality of first traces by a threshold amount, the user's focus target is changed to the component having the highest confidence value among all components intersecting with at least one of the plurality of second traces.

20. A method comprising any one or any combination of the technical features of claims 1 to 14.

21. A system comprising any one or any combination of the technical features of claims 15 to 17.

22. A non-transitory machine-readable medium, comprising any one or any combination of the technical features of claims 18-19.

Images