Peripheral device tracking system and method

Abstract

To provide a system and method for tracking locations of peripheral devices operated by a user.SOLUTION: A system comprises: a tracking unit 800 configured to determine locations of peripheral devices based on images captured by a camera associated with an HMD worn by a user; a generation unit 810 configured to generate, in response to a determination by the tracking unit 800 that the peripheral devices are invisible in the images, an array of home points behind the user; an evaluation unit 820 configured to evaluate each of the invisible peripheral devices in accordance with an output of inertial measurement units associated with the peripheral devices; and a selection unit 830 configured to select one home point from among the array of home points, for each of the invisible peripheral devices. The home points are positions defined in dependence upon constraints upon the user's motion.SELECTED DRAWING: Figure 8

Description

[[Technical Field]]

[0001] The present disclosure relates to a peripheral device tracking system and method. [[Background Art]]

[0002] The purpose of describing "Background Art" in the present specification is to generally illustrate the idea of the present disclosure. Aspects of technology and disclosure attributed to the inventor in this Background Art section do not, explicitly or implicitly, mean that they are prior art to the present invention, unless it is stated that they are prior art.

[0003] In recent years, there has been increasing demand for head-mounted displays (HDMs) for use in virtual reality, augmented reality, or mixed reality. These may be used to extend a user's peripheral devices for the purpose of improving productivity (e.g., providing overlays of additional information related to tasks performed by the user), or they may be intended for the popularization of entertainment through games and video content (whether interactive or otherwise). This increase in demand has been accelerated in part by developments in display technology, and in part by improvements in processing power available in devices that generate immersive content.

[0004] Many users can interact with these devices using only their hands through hand signals associated with different functions. In addition, one or more gestures or voice commands may be used for the purpose of improving functionality. To further enhance such interactivity with content, one or more control devices for receiving input may be provided. Motion capture may be performed on these peripheral devices in some cases, which provides users with a more natural input method. Such motion capture may be camera-based, or may use one or more internal motion sensors (e.g., accelerometers or gyroscopes for identifying changes in position / orientation). [[Summary of the Invention]] [Problems that the invention aims to solve]

[0005] However, such tracking can fail under certain conditions. For example, camera-based tracking solutions only work if they can capture images of the surrounding devices. When occlusion occurs, information about the position of the surrounding devices is lost, making interaction with the content impossible and resulting in errors. This problem is particularly pronounced with inside-out tracking (the opposite of outside-in tracking). This is because inside-out tracking always has a limited tracking volume (i.e., the volume of surrounding devices that can be tracked based on the camera's field of view, etc.).

[0006] This disclosure is made in the context described above. [Means for solving the problem]

[0007] This disclosure is defined in claim 1.

[0008] Furthermore, specific aspects and features of this disclosure are defined in additional claims.

[0009] It should be understood that the above general description of the present invention and the following detailed description are typical examples and not limitations of the present invention. [Brief explanation of the drawing]

[0010] A more complete understanding of this disclosure and its many advantages can be gained by referring to the attached drawings and reading the following detailed description. [Figure 1] This is a schematic diagram of an HMD (Head-Mounted Display) worn by a user. [Figure 2] This is a schematic plan view of the HMD. [Figure 3] This is a schematic diagram illustrating the formation of a virtual image using an HMD (Head-Mounted Display). [Figure 4]This is a schematic diagram of another type of display used in HMDs. [Figure 5] This is a schematic diagram of a pair of three-dimensional images. [Figure 6] This is a schematic diagram of a tracking system according to one or more embodiments of the present disclosure. [Figure 7] This is a schematic diagram illustrating how to track the location of one or more peripheral devices operated by a user. [Figure 8] This is a schematic diagram of an operable system for tracking the position of one or more peripheral devices operated by a user. [Figure 9] This is a schematic diagram of a configuration for executing the tracking process according to the embodiment of the present disclosure. [Modes for carrying out the invention]

[0011] Embodiments of the present disclosure are described below. Throughout the drawings, identical or similar components are denoted by the same reference numerals.

[0012] Embodiments of the present disclosure are described below. Throughout the drawings, identical or similar components are denoted by the same reference numerals. In Figure 1, a user 10 is wearing an HMD 20 (for example, a conventional head-mountable device, but in other examples, audio headphones or a head-mountable light source) on his head 30. The HMD comprises a frame 40 (formed in this example by a rear strap and a top strap) and a display section 50. As described above, many eye-tracking configurations can be considered suitable for use with an HDM system. However, it should not be considered essential to use such an HDM system.

[0013] As will be discussed later with reference to other drawings, the HDM in Figure 1 may have further features. However, for the sake of clarity in this initial explanation, these are not shown in Figure 1.

[0014] The HMD in Figure 1 completely (or substantially completely) obscures the user's view of the surrounding environment. The only things the user can see are pairs of images displayed within the HMD (which, in many embodiments, are provided by an external processing device such as a game console). Of course, in some embodiments, alternatively or additionally, images may be generated by a processor or retrieved from the HMD's own memory.

[0015] The HMD has associated headphone transducers or earpieces 60 (which fit into the user's left and right ears 70). The earpieces 60 reproduce audio signals provided from an external sound source (which may be the same as the video signal source that gives the display a video signal).

[0016] The fact that the user can only see what is displayed on the HMD, and the fact that the user can only hear what is delivered to the earpieces (due to noise blocking or active noise cancellation characteristics of the earpieces or other electronic circuits), combined, allows this HMD to be considered a so-called "fully immersive" HMD. However, it should be noted that in some embodiments, the HMD is not fully immersive, and the user may be given a device that allows them to see and hear their surroundings. This can be achieved, for example, by configuring the display to be partially transparent or semi-transparent, and / or by displaying external scenery (captured using a camera mounted on the HMD, etc.) on the HMD's display, and / or by configuring ambient sounds to pass through the earpieces, and / or by providing a microphone that generates an input audio signal (to be delivered to the earpieces) independently of ambient sounds.

[0017] A forward-facing camera may be provided to capture images on the front of the HMD during use. In some embodiments, such images may be used for head tracking. Alternatively, such images may be captured to be suitable for augmented reality (AR) experiences. A Bluetooth® antenna 124 may provide a means of communication, or it may be positioned simply as a directional antenna for detecting the direction of surrounding Bluetooth transmitters.

[0018] During operation, a video signal for the display is provided by the HMD. This may be provided by an external video signal source 80 (e.g., a video game console or a data processing device such as a personal computer). In this case, the signal may be transmitted to the HMD by a wired or wireless connection 82. A suitable example of a wireless connection is a Bluetooth® connection. An audio signal for the earpiece 60 may be transmitted by the same connection. Similarly, any control signals sent from the HMD to the video (audio) signal source may be transmitted by the same connection. Furthermore, a power supply (which may include one or more batteries and / or be connected to a mains outlet) may be connected to the HMD via a cable. The power supply and video signal source 80 may be separate units or integrated into the same physical unit. Separate cables may be used for power supply and video (and audio) signal supply, or these cables may be integrated into one (e.g., separate conductors may be used, such as in a USB cable, or the signal and power supply may be balanced and transmitted as the same current over the same bundle of physical wires, such as "Power over Ethernet®"). Video and / or audio signals may be transmitted, for example, by fiber optic cables. In another embodiment, at least part of the function for generating images and / or audio signals to be presented to the user may be realized by processes that form part of the electronic circuitry and / or the HMD itself. Power supply may be provided by part of the HMD itself.

[0019] Some embodiments of the present invention can be applied to an HMD including at least one electrical and / or optical cable (such cable connects the HMD to other devices such as a power supply and / or a video (and / or audio) signal source). Accordingly, embodiments of the present invention may include the following. (a) An HMD that includes its own power supply (as part of the HMD configuration) and is cabled to a video and / or audio signal source. (b) An HMD that is cabled to a power supply and a video and / or audio signal source using one or more physical cables. (c) An HMD that includes its own video and / or audio signal source (as part of the HMD configuration) and is cabled to a power supply. (d) An HMD that is wirelessly connected to a video and / or audio signal source and cabled to a power supply.

[0020] When one or more cables are used, the physical position at which the cables enter or are connected to the HMD is not particularly important from a technical point of view. Conventionally, to prevent the cable from touching the user's face during operation, the cable has been input or connected to the side or back of the HMD (relative to the orientation of the user's head when worn during normal operation). Therefore, the position of the cable relative to the HMD in FIG. 1 should be considered as merely a schematic representation.

[0021] As such, the configuration in FIG. 1 provides an example of a head-mountable display system including a frame mounted to a viewer's head, and a display element mounted at a gaze display position. The frame defines one or two gaze display positions. The gaze display position is disposed in front of the viewer's eyes during use. The display element provides a virtual image of a video display signal from a video signal source to the viewer's eyes.

[0022] Figure 1 shows only one example of an HMD, and other forms are possible. For example, the HMD may use a frame similar to conventional eyeglasses. In that case, substantially horizontal legs would extend backward from the display to the top of the user's ears and bend downward behind the ears. In another (non-immersive) example, the user's view of the external environment does not actually have to be completely blocked. That is, the displayed image may be superimposed on the external environment (from the user's perspective). An example of such a configuration is shown in Figure 4.

[0023] In the example in Figure 1, separate displays are provided for the user's left and right eyes. Figure 2 is a schematic plan view of how this is achieved. Figure 2 shows the position of the user's eyes 100 and the relative position 110 of the user's nose. The display portion 50 schematically comprises an external shield 120 to block ambient light from the user's eyes and an internal shield 130 to prevent the other eye from seeing the display viewed by one eye. With respect to the user's face, the external shield 120 and the internal shield 130 form two compartments 140 for each eye. Within each compartment are provided a display element 150 and one or more optical elements 160. Figure 3 shows the optical path formed by the display element and optical elements (which provides the user with a display).

[0024] Referring to Figure 3, the display element 150 generates a display image. (In this example) the display image is refracted by the optical element 160 (schematically shown as a single convex lens, but may be a composite lens, etc.). As a result, a virtual image 170 is generated. To the user, the virtual image 170 appears larger and much farther away than the real image generated by the display element 150. For example, the virtual image may have an apparent image size (diagonal length) of more than 1 meter and may be positioned more than 1 meter from the user's eyes (or from the HMD frame). Generally, regardless of the purpose of the HMD, it is desirable that the virtual image be positioned far away from the user. For example, in the case of an HMD intended for watching movies, it is desirable that the user's eyes can relax while viewing. This requires a distance of at least several meters (to the virtual image). In Figure 3, solid lines (e.g., line 180) represent actual light rays, and dotted lines (e.g., line 190) represent virtual light rays.

[0025] Figure 4 shows an alternative configuration. This configuration may be used when it is desirable that the user's view of the surrounding environment is not completely obstructed. However, this configuration can also be applied to HMDs where the user's view of the outside is completely obstructed. In the configuration of Figure 4, the display element 150 and the optical element 200 work together to provide an image that is projected onto the mirror 210. The mirror 210 reflects the image toward the user's eye position 220. The user perceives the virtual image as being in front of the user at a position 230, but at a reasonable distance from the user.

[0026] In the case of an HMD that completely blocks the user's view of the surrounding environment, the mirror 210 can be a virtually 100% reflective mirror. In this case, the configuration in Figure 4 has the advantage of being able to position the display and optical elements closer to the user's head's center of gravity and next to the user's eyes. This allows for a smaller HMD for the user wearing it. Alternatively, if the HMD is designed not to completely block the user's view of the surrounding environment, the mirror 210 may be a partially reflective mirror. This allows the user to see the surrounding environment through the mirror 210, and the virtual image is superimposed on the surrounding environment.

[0027] When a user's left and right eyes are given separate displays, stereoscopic images can be displayed. Figure 5 shows an example of a pair of stereoscopic images to be displayed to the left and right eyes. These images are displaced laterally from each other. The displacement of the image features depends on the lateral (actual or simulated) spacing of the cameras that captured the images, the camera angle convergence, and the (actual or simulated) distance of each image feature from the camera position.

[0028] Note that the lateral displacement in Figure 5 may actually be reversed. That is, the image shown for the left eye may actually be the image for the right eye, and the image shown for the right eye may actually be the image for the left eye. This is because some stereoscopic displays tend to shift the object to the right in the right-eye image and to the left in the left-eye image. This can simulate the feeling that the user is looking at the scenery behind them through a stereo window. However, some HMDs use the configuration shown in Figure 5 to give the user the impression that they are looking at the scenery through binoculars. The choice between these two configurations is left to the discretion of the designer.

[0029] In some situations, HMDs may be used solely for watching movies or similar content. In this case, there is no need to change the apparent viewpoint of the displayed image when the user moves their head (for example, from one side to the other). However, in other uses such as virtual reality (VR) and augmented reality (AR) systems, the user's viewpoint needs to track the trajectory of their movement relative to the real or virtual space in which they exist.

[0030] As described above, in some uses of HMDs, such as virtual reality (VR) and augmented reality (AR) systems, the user's viewpoint needs to track the trajectory of their movement relative to the real or virtual space in which they are located.

[0031] Tracking is performed by detecting the movement of the HMD and changing the apparent viewpoint of the displayed image. As a result, the apparent viewpoint tracks the movement. Motion tracking may be performed using hardware motion detectors (e.g., accelerometers or gyroscopes) or any preferred configuration including external cameras capable of capturing images of the HMD and outward-facing cameras mounted on the HMD.

[0032] Embodiments of the present disclosure aim to improve the reliability of inside-out tracking (i.e., tracking using outward-facing cameras mounted on an HMD). In particular, embodiments of the present disclosure provide a more reliable method for tracking peripheral devices (e.g., game controllers) using cameras mounted on (or embedded in) an HMD.

[0033] Peripheral devices are equipped with internal tracking units (e.g., accelerometers and gyroscopes), but their accuracy is low, limiting their performance to, for example, position tracking. These units, in particular, can detect drift or noise, which can cause the loss of useful information. While these problems can be partially solved by processing inertial tracking data, they cannot be completely resolved and can cause delays.

[0034] Figure 6 schematically shows a tracking system according to one or more embodiments of the present disclosure.

[0035] First, a processing device 600 (e.g., a game console or computer) is provided. This device is capable of running one or more applications (e.g., computer games) and providing images for display on the HMD630 and / or other related displays. In some embodiments, this function may be integrated into the HMD630. In this case, a separate processing device is not required.

[0036] The processing device 600 may be associated with a camera 610 that captures images of the user 620. This camera may be used to capture images for use in content processed by the processing device, or it may be used to track the user if outside-in tracking is employed.

[0037] User 620 wears the HMD 630 (as described with reference to Figures 1 and 2) and can provide input for the control process (e.g., by motion or button press) using peripheral devices 640. These peripheral devices may be of any preferred form and may be worn by the user instead of being held in the hand (e.g., a wristband may be fixed to the user's body, or the user may wear a device such as a smartwatch®). In some examples, a single peripheral device may be used, or two or more peripheral devices may be used (in combination of wearable devices and / or portable devices). In some embodiments, the peripheral device 640 may include one or more buttons or other operable elements to enable user input. The peripheral device 640 may also include one or more markers to assist in position tracking of the peripheral device 640 by a camera.

[0038] In some embodiments of this disclosure, the peripheral device 640 comprises one or more inertial measuring units (IMUs), such as accelerometers and gyroscopes. These can also be used by the user 620 to determine the position and / or orientation of each peripheral device during operation.

[0039] Figure 7 schematically illustrates a method for tracking the position of one or more user-operated peripheral devices according to one or more embodiments of the present disclosure. This method relates to continuous tracking of an invisible peripheral device (which may be one or more of several peripheral devices used by the user). This method is particularly suitable for inside-out tracking configurations (i.e., where the position of the HMD is determined using images captured by a camera associated with the HMD), but can be similarly applied to outside-in tracking with minor modifications (e.g., repositioning the camera).

[0040] Step 700 includes determining the position of one or more peripheral devices based on images captured by cameras associated with the HMD worn by the user. The cameras may be mounted on the HMD or integrated in other ways. There may be multiple cameras mounted on the HMD. In some embodiments, stereo cameras may be used to assist in determining the depth of the object to be imaged.

[0041] The positioning of peripheral devices may be performed in any preferred manner based on the captured image. For example, edge detection may be used to identify peripheral devices based on a template or outline associated with them. Alternatively or additionally, active markers (e.g., illuminated spheres or patterns of LEDs) may be provided to the peripheral devices to aid in identification within the image. A further alternative or additional feature is the presence of passive markers that can be used to aid in identification within the image. In some embodiments, peripheral devices may be configured to communicate wirelessly (using a wireless connection) to provide information to aid in identification within the image. For example, the distance and orientation of the peripheral device may be evaluated based on the wireless signal strength, or wireless communication may be used to identify where in the image the peripheral device appears.

[0042] Step 710 includes the step of generating an array of home points behind the user in response to the tracking unit determining that one or more peripheral devices are not visible in the image (in other words, that the peripheral devices have moved out of the tracking volume). These home points are locations defined based on constraints on the user's movement. This array is not limited to the area behind the user and may include other areas of the user's environment. This array is generated with respect to a volume that is not visible in the image captured by the camera but where peripheral devices are expected to be present. This volume may be, for example, above the user's head. In this case, its position may be determined based on the movement of the peripheral devices and the user's head (i.e., if the camera is mounted on the HMD, based on the movement of the HMD and the associated camera movement).

[0043] Here, the term "home point" does not refer to a physical element, but rather to a location within the user's environment. These points may be represented by markers in a virtual environment, and may or may not be visible to the user, depending on the specific implementation. In other words, an array of home points can be thought of as a set of locations defined within the user's environment. These home points act as proxies for the location of peripheral devices at a given time during operation. Home points are selected as likely (or at least possible) locations where peripheral devices are likely to be at a given time.

[0044] Home points may be generated in any preferred distribution and granularity, while being consistent with the constraints of the user's movement. In other words, home points may be distributed according to the positions that peripheral devices can take, based on the constraints of the movements the user can perform. For example, home points may not be defined for portable peripheral devices that are further from the user than the sum of the user's arm length and the length of the peripheral device. Home points may be distributed according to a predetermined model (e.g., one given by a content creator such as a game developer or platform developer) or a body model of the user (or a representative user rather than a specific user).

[0045] A home point may be generated for each user and / or each peripheral device, as needed. The home point is also generated depending on which part of the body the peripheral device is associated with. For example, different home points will be generated for a wrist attachment compared to an elbow attachment.

[0046] In some embodiments, the home point is generated based on the posture the user was in before the peripheral device moved out of the tracking volume. For example, if the user had bent their arm before image-based tracking, this would restrict the position of the peripheral device; that is, in this case, the user could not immediately extend their arm. From the perspective of such features, the body model may include information about the maximum (and / or typical) rate of motion of the user's movements. This would allow for determining how long the posture assessment is valid. Such a body model may be defined for a general (representative) user, or it may be developed for a specific user based on an adjustment process (in which the user's movements are tracked to identify one or more aspects of the user's movements, e.g., maximum range of motion, maximum speed of motion, and / or trajectory associated with the movement). This body model may be limited to the user's upper body (which may suffice when using a portable peripheral device). Even when a mobile device is the peripheral device, a full-body model may be desirable, for example, because the position of the legs may represent a shift in balance, which may represent the position of the upper body.

[0047] Step 720 includes evaluating the position of each invisible peripheral device based on the output of one or more inertial measurement units for each peripheral device. This may be based on readings (outputs) of one or more IMUs associated with the peripheral device, along with, for example, the last known position or a previously evaluated position obtained from optical tracking. In some embodiments, the readings may be used, alternatively or additionally, along with the last known pose information about the user's body or part of the body (e.g., information about the user's most recent arm flexion).

[0048] Step 730 includes selecting one home point from an array of home points for each invisible peripheral device. This selection may be performed in any preferred manner, or in accordance with several considerations or a combination thereof. Examples of such considerations are given below.

[0049] The first consideration may be the relative position of the peripheral device (i.e., the evaluated position) and the relationship between each home point in the array. In some cases, it may be desirable to select the home point closest to the evaluated position of the peripheral device. However, in other cases, it may be desirable to select a home point located further away from the evaluated position (for example, the second closest or third closest position). In this case, the likelihood of switching the home point selection for a particular peripheral device during the tracking process is reduced.

[0050] The second consideration may be an evaluation of the peripheral device's velocity (e.g., the last known velocity). In this case, a home point can be selected that lies on (or at least near) the peripheral device's trajectory. This trajectory may be based on the last known velocity obtained from optical tracking, a body model (which represents constraints on the user's movement), and / or velocity evaluation.

[0051] A third consideration may be an consideration of predicted future positions based on the content in which the user interacts. For example, the context within the content might infer that the user will move the peripheral device to a specific location or in a specific direction. This could involve interacting with a specific virtual object, for instance. In a game, the user might be playing as an archer. In such a scenario, if the user moves the controller outside the tracking volume, it could be assumed that the user is drawing a bow or reaching for an arrow. Each of these actions would be linked to a specific predicted trajectory and / or final position. This could then be used to select an appropriate home point.

[0052] Step 740 includes generating a position for each invisible peripheral device so that it converges to a selected home point. Therefore, the generated position for each peripheral device does not necessarily coincide with the position evaluated in step 720. However, they may coincide (when both the evaluated position and the generated position converge to the same selected point). Generally, the generated position can be considered an improved evaluation position. Thus, the output (generated) position for each peripheral device reaches the selected home point.

[0053] This generation may be performed in any preferred manner to provide convergence. For example, a linear approach may be taken (where a fixed position modifier is applied in the selected direction), or a nonlinear approach may be taken (where the larger the change, the closer or further the evaluated position moves from the home point).

[0054] The rate at which convergence is performed can be freely chosen. For example, convergence may be performed at a predetermined number of frames, or under other time measures. The convergence rate may be chosen according to any number of factors, such as the distance between the evaluated position and the selected home point, the type of movement, the type of surrounding device, the type of content interacting with the user, and / or user preferences.

[0055] In some embodiments, steps 720-740 may be performed iteratively so that evaluation, home point selection, and position determination are performed multiple times. This may be performed at the content frame rate, the peripheral device IMU update rate, or other rates. These steps may also be performed at different rates. For example, step 740 may be repeated multiple times before the evaluation value of the updated peripheral device position is determined based on the IMU data. Alternatively, step 740 may be repeated alone, and no further evaluation may be performed after the first (or a predetermined number of) evaluations following step 720. This is advantageous in that it prevents the reliability of the evaluation from decreasing over time due to the accumulation of errors such as sensor drift. Similarly, steps 730 and 740 may be repeated (at any preferred rate for a particular implementation, not necessarily at the same frequency), with position generation and home point selection being repeated, respectively.

[0056] There are various variations of different forms of peripheral devices suitable for use in the embodiments. When multiple peripheral devices are used, they may be the same as each other or may include a range of different devices. A gamepad designed for the user to hold in both hands may be provided. Alternatively or additionally, (if the user does not hold a gamepad in both hands) the device may include a controller designed for the user to hold in one hand. These devices may include one or more buttons in addition to one or more IMUs. Further devices that are considered suitable peripheral devices may be worn by the user instead of carried by the user, such as a wristband (including a smartwatch) or other device with IMUs that the user does not hold in their hand.

[0057] These peripheral devices may have one or more additional features to assist in tracking the position of each peripheral device. Examples of such features include trackable markers that can be identified within an image of the peripheral device (e.g., illuminated elements, characteristic patterns such as AR markers). Alternatively or additionally, one or more wireless communication elements may be provided. These can communicate with and / or with other devices to assist in positioning the peripheral devices. This may be based on signal strength and / or direction, rather than providing specific location information for the peripheral devices, to assist in determining their relative position.

[0058] As described above, in some embodiments, each peripheral device is given a single IMU. However, it may be advantageous to give a single peripheral device multiple IMUs. This allows for the collection of more data, thereby improving accuracy and / or precision. This may be based on the type of multiple sensors given (e.g., accelerometer, gyroscope, and / or magnetometer), and / or on sensors given at multiple locations throughout the peripheral device.

[0059] Figure 8 schematically illustrates a system for tracking the position of one or more peripheral devices operated by a user. This system comprises a tracking unit 800, a generation unit 810, an evaluation unit 820, and a selection unit 830. This system can be associated with one or more cameras, which may be mounted on a user-worn HMD. The fields of view of these cameras can be used to define a tracking volume. Within this tracking volume, peripheral devices can be tracked based on captured images. In other words, the tracking volume is a three-dimensional region of the environment in which peripheral devices can be imaged. The tracking volume may respond to the movement of one or more cameras, as this can change the portion of the environment displayed in the captured image.

[0060] The tracking unit 800 can determine the position of one or more peripheral devices based on images captured by a camera associated with the HMD worn by the user. This may be added based on information obtained from a wireless communication protocol used to locate one or more inertial measurement units and / or devices. The inertial measurement units may include, for example, any combination of accelerometers, gyroscopes and / or magnetometers, or other units capable of determining the movement of peripheral devices.

[0061] The generation unit 810 can generate an array of home points behind the user in response to the tracking unit determining that one or more peripheral devices are not visible in the image. The home points are positions defined according to the constraints of the user's movement. In some embodiments, "not visible in the image" may mean that a peripheral device cannot be tracked if, for example, the marker is no longer in the captured image (even if part of the peripheral device is still in the image).

[0062] In some embodiments, if the tracking volume is defined with respect to a particular configuration, the generation unit 810 may be considered capable of generating an array of home points in response to one or more peripheral devices moving out of the tracking volume. In this case, the array of home points is generated with respect to a volume of the environment outside the tracking volume. This may be the entire environment outside the tracking volume, or a part of the environment outside the tracking volume that corresponds to the location where the corresponding peripheral devices are most likely to be.

[0063] The evaluation unit 820 can evaluate each invisible peripheral device in accordance with the output of one or more inertial measurement units associated with each peripheral device. In some embodiments, the evaluation unit 820 may use a body model representing the user to limit the evaluation of the position of the peripheral devices. The body model includes information about the user's range of motion and / or velocity. This body model may be specific to the user's upper body, or it may be specific to the entire body, or it may be specific to the use of a particular peripheral device. In some embodiments, the model may be a representative model of the user's body (e.g., a stock model downloaded from a server), or it may be specially created for each user (e.g., by a tuning process).

[0064] In some embodiments, the evaluation unit 820 can evaluate the position of a peripheral device in response to the content in which it interacts with the user. For example, it can infer actions the user is likely to take based on the content, and based on this, it can determine the trajectory the peripheral device is likely to follow (i.e., identify a likely trajectory that approximates the trajectory required for the inferred action). This may be based on predicted inputs (e.g., interaction with an object) or the type of predicted interaction (e.g., use of a specific virtual item).

[0065] The selection unit 830 can select one home point from an array of home points for each invisible peripheral device. In some embodiments, the selection unit 830 can select the home point closest to the evaluated location of the peripheral device. Selectively or additionally, the predicted trajectory of the peripheral device may be used to select a home point (in other words, the home point may be selected based on a prediction of the future proximity of the peripheral device location to the home point location).

[0066] Instead of selecting a home point to be used throughout the tracking process for peripheral devices, the selection unit 830 may update the selected home point after a predetermined time interval has elapsed. This may be a predetermined time or may be based on an input update rate related to content, etc. In some embodiments, the tracking unit 930 may generate a position for each of the invisible peripheral devices at a rate defined by the update rate of one or more associated inertial measurement units.

[0067] After this selection, the tracking unit 800 can generate a position for each of the invisible peripheral devices. This is done so that the position for each invisible peripheral device converges to the selected home point. The tracking unit 800 may generate the positions with a convergence rate (which depends on the distance between the evaluated position and the selected home point). For example, a larger distance (or, depending on the implementation, a smaller distance) may result in more aggressive convergence. The convergence rate may be linear or nonlinear with respect to the change in interval (e.g., exponential or proportional to the square of the distance).

[0068] The configuration in Figure 8 is an example of a processor (e.g., a GPU and / or CPU in a game console or other computing device) capable of tracking the position of one or more peripheral devices operated by the user, and is capable of operating in the following ways: The position of one or more peripheral devices is determined based on images captured by a camera associated with the HMD worn by the user. The tracking unit, in response to determining that one or more peripheral devices are not visible in the image, generates an array of home points behind the user. Home points are locations defined based on the constraints of the user's movement. The position of each invisible peripheral device is evaluated based on the output of one or more inertial measurement units for each peripheral device. For each invisible peripheral device, select one home point from the array of home points. The position of each invisible peripheral device is generated so that it converges to the selected home point.

[0069] Figure 9 schematically shows a configuration for performing a tracking process according to an embodiment of the present disclosure. This configuration includes a camera 900, a peripheral device 910, and a processing device 920. These units are not limited to a configuration in which each unit is integrated into one, and may be given in any preferred configuration. For example, the HMD may include elements corresponding to both the camera 900 and the processing device 920, and the peripheral device 910 may include one or more processing elements capable of operating as the processing device 920. Furthermore, functions may be distributed among multiple devices as needed. For example, some processing (e.g., optical tracking of the peripheral device 910) may be performed on the HMD, while position evaluation, etc., may be performed on a related game console or other computing device.

[0070] Camera 900 may include any preferred camera configuration. For example, camera 900 may include a single camera or an array of cameras. Depending on the embodiment, these cameras may be stereo cameras or infrared cameras, provided that they improve tracking of peripheral devices (e.g., acquisition of depth information or identification of infrared markers). In some embodiments, camera 900 may be fixed to (or integrated with) the HMD worn by the user. However, in other embodiments, camera 900 may be positioned away from the user and capable of capturing the user and peripheral devices.

[0071] As described above, one or more peripheral devices 910 may include any combination of portable or wearable devices equipped with inertial measurement units such as accelerometers, gyroscopes and / or magnetometers. Examples of such devices include gamepads, input devices, wristbands, armbands, and devices worn around the legs or other body parts. These peripheral devices may include one or more elements to assist tracking, such as optically recognizable markers or optical beacons (e.g., passive or active markers).

[0072] The processing device 920 may be a standalone element such as a game console or a computer, or it may be embedded in other devices such as a camera phone, HMD, and / or peripheral devices. The functions of the processing device 920 are not limited to being performed on a single device, but may be distributed and performed on any number of suitable devices. These devices may be local or remote (for example, like a server), as needed.

[0073] The above technologies can be implemented by hardware, software, or a combination thereof. If a software-controlled data processing device is used to implement one or more features of this embodiment, it is clear that such software, storage, or communication medium, such as a non-transient machine-readable storage medium to which the software is provided, is also an embodiment of the present disclosure.

[0074] The above discussion merely discloses and describes examples of embodiments of the present invention. Those skilled in the art will understand that the present invention can be realized in other specific forms without departing from the spirit and essential features of the invention. Accordingly, the disclosure of the present invention is for illustrative purposes only and is not intended to limit the scope of the invention or the claims. This disclosure, including any identifiable modifications of the above teachings, partially defines the scope of the terms of the claims. The subject matter of the invention is not dedicated to the public.

[0075] Embodiments of this disclosure can be carried out by following the items listed below.

[0076] 1. A system for tracking the position of one or more peripheral devices operated by a user, A tracking unit configured to determine the position of one or more peripheral devices based on images captured by a camera associated with the HMD worn by the user, A generation unit configured to generate an array of home points behind the user in response to the tracking unit determining that one or more peripheral devices are not visible in the image, An evaluation unit configured to evaluate each invisible peripheral device in accordance with the output of one or more inertial measurement units associated with each peripheral device, Each of the aforementioned invisible peripheral devices is configured to select one home point from the array of home points, Equipped with, The aforementioned home point is a position defined according to the constraints on the user's movement, The system is characterized in that the tracking unit generates the position of each of the invisible peripheral devices so as to converge to a selected home point.

[0077] 2. The system according to item 1, characterized in that the camera is mounted on the HMD.

[0078] 3. The camera's field of view is used to define the tracking volume on which the peripheral device is tracked based on the captured image. The system according to item 1, characterized in that the generation unit is configured to generate an array of home points in response to one or more peripheral devices moving out of the tracking volume.

[0079] 4. The system according to item 3, characterized in that the array of home points is generated with respect to the volume of the environment outside the tracking volume.

[0080] 5. The system according to item 1, characterized in that the inertial measurement unit includes an accelerometer, a gyroscope, and / or a magnetometer.

[0081] 6. The evaluation unit is configured to use a body model representing the user in order to limit the evaluation of the position of the peripheral device, The system according to item 1, characterized in that the body model includes information regarding the range and / or speed of the user's movements.

[0082] 7. The system according to item 1, characterized in that one or more peripheral devices include optically recognizable features.

[0083] 8. The system according to item 1, characterized in that the selection unit is configured to select the home point closest to the evaluated location of the peripheral device.

[0084] 9. The system according to item 1, characterized in that the selection unit is configured to update the selected home point after a predetermined time interval has elapsed.

[0085] 10. The system according to item 1, characterized in that the tracking unit is configured to generate a position with a convergence rate that depends on the distance between the evaluated position and the selected home point.

[0086] 11. The system according to item 1, characterized in that the evaluation unit evaluates the position of peripheral devices according to the content interacting with the user.

[0087] 12. The system according to item 1, characterized in that the tracking unit is configured to generate a position with respect to each of the invisible peripheral devices at a rate defined by the update rate of one or more associated inertial measurement units.

[0088] 13. A method for tracking the location of one or more peripheral devices operated by a user, The steps include determining the position of one or more peripheral devices based on images captured by a camera associated with the HMD worn by the user, The tracking unit, in response to determining that one or more peripheral devices are not visible in the image, generates an array of home points behind the user. A step of evaluating the position of each invisible peripheral device according to the output of one or more inertial measurement units for each peripheral device, With respect to each of the invisible peripheral devices, the step of selecting one home point from the array of home points, The steps include generating the position of each of the aforementioned invisible peripheral devices so that it converges to the selected home point, Includes, The method is characterized in that the home point is a location defined based on the constraints of the user's movement.

[0089] 14. Computer software that, when run on a computer, causes the computer to perform any of the actions described in item 13.

[0090] 15. A non-temporary, machine-readable storage medium on which computer software as described in item 14 is recorded.

Claims

1. A system for tracking the position of one or more peripheral devices operated by a user, A tracking unit configured to determine the position of one or more peripheral devices based on images captured by a camera associated with the HMD worn by the user, A generation unit is configured to generate an array of home points behind the user in response to the tracking unit determining that one or more peripheral devices are not visible in the image. An evaluation unit configured to evaluate each invisible peripheral device in accordance with the output of one or more inertial measurement units associated with each peripheral device, Each of the aforementioned invisible peripheral devices is configured to select one home point from the array of home points, Equipped with, The aforementioned home point is a position defined according to the constraints on the user's movement, The tracking unit generates the position of each of the invisible peripheral devices so as to converge to a selected home point. The system is characterized in that the tracking unit is configured to generate a position with a convergence rate that depends on the distance between the evaluated position and the selected home point.

2. The system according to claim 1, characterized in that the camera is mounted on the HMD.

3. The camera's field of view is used to define the tracking volume over which the peripheral device is tracked based on the captured image. The system according to claim 1, characterized in that the generation unit is configured to generate an array of home points in response to one or more peripheral devices moving out of the tracking volume.

4. The system according to claim 3, characterized in that the array of home points is generated with respect to the volume of the environment outside the tracking volume.

5. The system according to claim 1, characterized in that the inertial measurement unit includes an accelerometer, a gyroscope, and / or a magnetometer.

6. The evaluation unit is configured to use a body model representing the user in order to limit the evaluation of the position of the peripheral device. The system according to claim 1, characterized in that the body model includes information regarding the range and / or speed of the user's movements.

7. The system according to claim 1, characterized in that one or more peripheral devices include optically recognizable features.

8. The system according to claim 1, characterized in that the selection unit is configured to select the home point closest to the evaluated location of the peripheral device.

9. The system according to claim 1, characterized in that the selection unit is configured to update the selected home point after a predetermined time interval has elapsed.

10. The system according to claim 1, characterized in that the evaluation unit evaluates the position of a peripheral device in accordance with the content interacting with the user.

11. The system according to claim 1, characterized in that the tracking unit is configured to generate a position with respect to each of the invisible peripheral devices at a rate defined by the update rate of one or more associated inertial measurement units.

12. A method for tracking the location of one or more peripheral devices operated by a user, The steps include determining the position of one or more peripheral devices based on an image captured by a camera associated with the HMD worn by the user, The tracking unit, in response to determining that one or more peripheral devices are not visible in the image, generates an array of home points behind the user. A step of evaluating the position of each invisible peripheral device according to the output of one or more inertial measurement units for each peripheral device, With respect to each of the invisible peripheral devices, the step of selecting one home point from the array of home points, The steps include generating the position of each of the aforementioned invisible peripheral devices so that it converges to the selected home point, Includes, The aforementioned home point is a location defined based on the constraints of the user's movement. The method is characterized in that the tracking unit is configured to generate a position with a convergence rate that depends on the distance between the evaluated position and a selected home point.

13. A non-temporary, machine-readable storage medium on which computer software is recorded, The aforementioned computer software, when executed on a computer, causes the computer to perform a method for tracking the location of one or more peripheral devices operated by the user. The aforementioned method, The steps include determining the position of one or more peripheral devices based on an image captured by a camera associated with the HMD worn by the user, The tracking unit, in response to determining that one or more peripheral devices are not visible in the image, generates an array of home points behind the user. A step of evaluating the position of each invisible peripheral device according to the output of one or more inertial measurement units for each peripheral device, With respect to each of the invisible peripheral devices, the step of selecting one home point from the array of home points, The steps include generating the position of each of the aforementioned invisible peripheral devices so that it converges to the selected home point, Includes, The aforementioned home point is a location defined based on the constraints of the user's movement. The storage medium is characterized in that the tracking unit is configured to generate a position with a convergence rate that depends on the distance between the evaluated position and the selected home point.

Images