Image-based finger tracking and controller tracking

Abstract

Images from a camera (208) on, for example, a VR HMD (200) of a hand grasping a computer game controller (212) are identified. The images can be cropped (306) to a region containing the controller and the hand to simplify processing, and a virtual representation of the hand is presented on a display (202) such as the HMD's display, where the virtual representation is generated based on both image analysis of the region and touch signals (302) from the controller.

Description

Technical Field

[0001] This application generally relates to image-based finger tracking and controller tracking in computer simulations such as computer games. Background Technology

[0002] Hand tracking is desirable for applications such as virtual reality (VR) computer games, to present a virtual representation of the user's hand on a display such as a VR head-mounted display (HMD) during gameplay, for example, when the user has to grab virtual objects.

[0003] It is possible to track hands using, for example, a camera on an HMD, but as understood in this paper, accurately identifying the hand and subsequently presenting a virtual depiction of its posture is complex when the user's hand is partially obscured, for example, by a computer game controller held by the user. Tracking the hand based on sensors on the controller can introduce "dead zones" for parts of the hand not located near the sensors and for parts of the hand that can have a wide degree of freedom of movement, such as the thumb. Summary of the Invention

[0004] Therefore, the device includes at least one processor programmed with instructions to recognize images from at least one camera of a hand gripping a computer game controller. The instructions can be executed to crop the image to include an area containing the controller and the hand, and to present a virtual representation of the hand on a computerized display, generated at least in part based on image analysis of the area and at least one touch signal from the controller.

[0005] In some implementations, the camera is mounted on a head-mounted display (HMD). The device may include an HMD. In an exemplary embodiment, touch signals may be generated by manipulation of control key elements on the controller and / or generated from sensors on the controller instead of from the control key elements on the controller.

[0006] Instructions can be executed to generate a virtual representation of a portion of a hand obscured by the camera using touch signals. In some implementations, instructions can be executed to generate a virtual representation using controller recognition without using hand recognition, in response to the recognition of touch signals from the controller.

[0007] In the example, instructions can be executed to perform a machine learning (ML) module to generate a virtual representation by modifying key points of a template image based on images in the region and touch signals from a controller. The ML model may include at least one neural network (NN) and at least one heatmap.

[0008] In another aspect, the method includes recognizing an image of a hand grasping a computer simulation controller. The method also includes receiving at least one touch signal from the computer simulation controller and generating an image of a virtual hand for display based on both the image of the hand and the touch signal.

[0009] On the other hand, the device includes at least one computer storage device that is not a transient signal and sequentially includes instructions executable by at least one processor to receive at least one image of a human hand grasping a computer game controller, receive at least one touch signal from the controller, and generate a virtual hand representing the human hand based on both the image and the touch signal, so as to present the virtual hand on at least one computerized display.

[0010] The details of both the structure and operation of this application can be best understood with reference to the accompanying drawings, in which the same reference numerals refer to the same parts, and in the drawings: Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating some or all of the computer components that can be used in various implementations;

[0012] Figure 2 An exemplary system is shown, illustrating components including a head-mounted display, a computer simulation controller, and a virtual image processor;

[0013] Figure 3 An exemplary flowchart is shown to illustrate exemplary logic for generating an image of a virtual hand based on both a camera image and a touch signal from a controller.

[0014] Figures 5 to 9 Various exemplary postures of a human hand grasping an exemplary computer game controller are shown;

[0015] Figure 10 The image shows a hand holding the controller and an associated virtual image of the full hand;

[0016] Figure 11 An exemplary machine learning (ML) module is shown that can be used to generate images of virtual hands; and

[0017] Figure 12 An exemplary system flow is shown. Detailed Implementation

[0018] This disclosure generally relates to a computer ecosystem, which includes various aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. Systems described herein may include server components and client components that can be network-connected, enabling data exchange between the client components and the server components. Client components may include one or more computing devices, including game consoles (such as Sony...). This includes game consoles made by Microsoft, Nintendo, or other manufacturers; virtual reality (VR) headsets; augmented reality (AR) headsets; portable televisions (such as smart TVs and internet-enabled TVs); portable computers (such as laptops and tablets); and other mobile devices (including smartphones and additional examples discussed below). These client devices can operate in a variety of operating environments. For example, some client computers may use operating systems such as Linux, Microsoft operating systems, or Unix operating systems, or operating systems made by Apple, Inc. or Google. These operating environments can be used to execute one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by internet servers discussed below. Furthermore, the operating environment according to the principles of the present invention can be used to execute one or more computer game programs.

[0019] The server and / or gateway may include one or more processors that execute instructions to configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may connect via a local intranet or virtual private network. The server or controller may be a game console (such as Sony). Instantiation of personal computers, etc.

[0020] Information can be exchanged between clients and servers over a network. For this purpose, and for security reasons, servers and / or clients may include firewalls, load balancers, temporary storage devices, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form a device that implements methods for providing secure communities (such as online social networking sites) to network members.

[0021] The processor can be a single-chip or multi-chip processor, which can perform logic by means of various lines (such as address lines, data lines and control lines) as well as registers and shift registers.

[0022] Components included in one embodiment can be used in any suitable combination in other embodiments. For example, any of the various components described herein and / or depicted in the figures can be combined, interchanged, or excluded from other embodiments.

[0023] "A system having at least one of A, B and C" (similarly, "a system having at least one of A, B or C" and "a system having at least one of A, B and C") includes: a system having only A; a system having only B; a system having only C; a system having both A and B; a system having both A and C; a system having both B and C; and / or a system having both A, B and C, etc.

[0024] Now for specific reference Figure 1 An exemplary system 10 is illustrated, which may include one or more of the exemplary devices mentioned above and further described below according to the principles of the invention. The first device among the exemplary devices included in system 10 is a consumer electronics (CE) device, such as an audio-visual device (AVD) 12, such as, but not limited to, an internet-enabled TV with a TV tuner (equivalently, a set-top box controlling a TV). Alternatively, the AVD 12 may also be a computerized internet-enabled (“smart”) phone, tablet computer, laptop computer, HMD, wearable computerized device, computerized internet-enabled music player, computerized internet-enabled headset, computerized internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to implement the principles of the invention (e.g., communicate with other CE devices implementing the principles of the invention, perform the logic described herein, and perform any other functions and / or operations described herein).

[0025] Therefore, in order to implement this principle, AVD 12 can be... Figure 1Some or all of the components shown are constructed. For example, AVD 12 may include one or more displays 14, which may be implemented by a high-definition or ultra-high-definition "4K" or higher flat screen and may be touch-enabled for receiving user input signals via touch on the display. AVD 12 may include one or more speakers 16 for outputting audio according to the principles of the invention, and includes at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to control AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors 24. A graphics processor 24A may also be included. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12, and other elements described herein including AVD 12, to implement the principles of the invention, such as controlling display 14 to display images thereon and to receive input from said display. Furthermore, it should be noted that network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above.

[0026] In addition to the foregoing, the AVD 12 may also include one or more input ports 26, such as a High Definition Multimedia Interface (HDMI) port or a USB port that physically connects to another CE device and / or a headphone port that connects headphones to the AVD 12 to present audio from the AVD 12 to the user. For example, input port 26 may be wired or wirelessly connected to a wired or satellite source 26a of audio-visual content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. When implemented as a game console, source 26a may include some or all of the components described below with respect to CE device 44.

[0027] AVD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices. In some cases, the one or more computer memories are embodied as stand-alone devices within the chassis of the AVD, or as personal video recording devices (PVRs) or video disk players for playing back AV programs, either inside or outside the chassis of the AVD, or as removable memory media. Furthermore, in some embodiments, AVD 12 may include a location or positioning receiver, such as, but not limited to, a cellular phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular phone base station and provide said information to processor 24 and / or in conjunction with processor 24 to determine the altitude at which the AVD 12 is set. Component 30 may also be implemented by an inertial measurement unit (IMU), which typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions.

[0028] Continuing the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras (such as webcams), and / or cameras integrated into AVD 12 and capable of being controlled by processor 24 to acquire pictures / images and / or videos according to the principles of the present invention. AVD 12 may also include a Bluetooth transceiver 34 and other near-field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0029] In addition, AVD 12 may include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, odometers, or magnetometers), infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, and gesture sensors (e.g., for sensing gesture commands) that provide input to processor 24. AVD 12 may include an over-the-air (OTA) TV broadcast port 38 for receiving OTA TV broadcasts that provide input to processor 24. In addition to the foregoing, it should be noted that AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power AVD 12, such as a kinetic energy harvester that can convert kinetic energy into electrical energy to charge the battery and / or power AVD 12.

[0030] Still referencing Figure 1In addition to AVD 12, System 10 may also include one or more other CE device types. In one example, the first CE device 44 may be a computer game console that can be used to send audio and video of a computer game to AVD 12 via commands sent directly to AVD 12 and / or via a server described below, while the second CE device 46 may include components similar to the first CE device 44. In the example shown, the second CE device 46 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player 47. In the example shown, only two CE devices 44, 46 are shown; it should be understood that fewer or more devices may be used. The devices described herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may be combined with some or all of the components shown in the case of AVD 12.

[0031] Referring now to the aforementioned at least one server 50, said at least one server includes at least one server processor 52, at least one tangible computer-readable storage medium 54 (such as a disk-based storage device or a solid-state storage device), and at least one network interface 56, said at least one network interface, under the control of the server processor 52, allowing communication via network 22 with... Figure 1 It communicates with other devices and can actually facilitate communication between the server and client devices according to the principles of the invention. It should be noted that network interface 56 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (e.g., a wireless telephone transceiver).

[0032] Therefore, in some implementations, server 50 may be an internet server or an entire server "farm," and in exemplary implementations for, for example, online gaming applications, the server may include and perform "cloud" functionality, enabling devices of system 10 to access a "cloud" environment via server 50. Alternatively, server 50 may be a combination of one or more game consoles or... Figure 1 The other devices shown are implemented in the same room or on other computers nearby.

[0033] Figure 2 A display device 200, such as a virtual reality (VR) head-mounted display (HMD), is shown. Figure 2 Like the other components shown, this display device can be combined with the above-mentioned... Figure 1 Any or all of the components discussed. Figure 2The exemplary HMD 200 shown may include one or more video displays 202 controlled by one or more processors 204, which may wirelessly communicate with other components using one or more wireless network interfaces 206. The HMD 200 may also include one or more outward-facing cameras 208 for imaging objects such as the hand of the wearer of the HMD 200.

[0034] The HMD 200 can be used to play video games executed by a source 210, such as a video game console and / or a remote server, under the control of one or more handheld controllers 212. Controllers 212 may include one or more operable control keys 214 to control the game or simulation, each control key being associated with one or more sensors 216 to generate a signal indicating manipulation or contact with the associated control key 214. Additionally, controllers 212 may include one or more non-control key sensors 218, which are not associated with control keys but are located at known locations on controllers 212 to sense hand contact or proximity with sensor 218, thereby providing a signal indicating such contact or proximity. Controllers 212 may include one or more processors 220 configured to transmit signals from sensors 216, 218, and control keys 214 to other components using one or more network interfaces 222. Controllers may also include one or more position sensors 223, such as inertial sensors, GPS sensors, accelerometers, magnetometers, gyroscopes, and combinations thereof.

[0035] One or more processors 224 (such as any of the processors described herein) can be from Figure 2 Other components in the device receive signals and can access instructions on one or more computer storage devices 226 to implement logic consistent with the principles described herein.

[0036] An example of this logic is... Figure 3 As shown herein, and can be executed by any processor or combination of processors shown herein. Starting at box 300, an image is received from camera 208 of controller 212, for example, which can be held by a human hand. Moving to decision diamond box 302, it is determined whether a signal has been received from any of the controller sensors 216, 218 indicating that the controller is actually being held. If not, the logic can move to box 304 to perform hand image recognition in free space and generate a virtual representation of the hand based solely on the image of the hand.

[0037] However, if it is determined that the controller is being held as indicated by signals from the sensors on the controller, the logic moves to box 306 to crop the image to only the area of ​​the controller and surrounding objects, which can be inferred to be a hand. In this way, only image recognition can be performed to identify the controller, rather than performing more complex image recognition to identify the hand. If image details are required, the remaining image area after cropping can be processed using super-resolution.

[0038] Moving to box 308, the cropped image can be analyzed based on both the image and signals from controller sensors 216, 218 to determine the hand's pose. Generally, a portion of the object surrounding the controller image and inferred to be part of the hand can be used to render a portion of a virtual image of the hand, where signals from the controller representing contact points are used to "fill" the invisible portion of the hand located behind the controller image.

[0039] The reference frame of the visible portion of the hand can be registered with the controller's reference frame in various ways. For example, the position of the controller 212 can be derived from the signal from the position sensor 223 and the visible portion of the hand, registered to the indicated contact positions of sensors 216, 218 using machine vision, and converted into the controller's reference frame as represented by the position sensor 223. Alternatively, machine vision can be used to establish a reference frame based on the centroid of the hand image, and the controller position can be registered to the centroid based on the indicated contact positions of sensors 216, 218.

[0040] Furthermore, when the machine learning (ML) module is used as further described below, the model can be trained on a ground truth image of the hand holding the controller, along with the accompanying ground truth sensor signals and a ground truth synthesized virtual full hand image corresponding to the local hand image of the holding controller and the corresponding sensor signals.

[0041] In fact, Figure 3 The diagram shows that at box 308, a cropped area of ​​the controller with a hand can be input to the ML module, while at box 310, corresponding touch signals from controller sensors 216, 218, generated simultaneously with image generation, are input to the ML module. The ML module uses both the sensor signals and the controller / hand image to output a virtual image of a complete hand at box 312, the posture of which is the same as when gripping the controller in the cropped area generated at box 306. At box 314, the virtual image is displayed on a display such as an HMD 200.

[0042] Figures 5 to 9 The diagram shows corresponding cropped images 500-900 of the hands 502-902 of the grasping controller 504-904. These cropped images can represent the ground truth images used for training and... Figure 3 The actual cropped image generated at frame 306 after training.

[0043] Figure 10 The image shows a cropped controller / hand image 1000 and a synthetic virtual full-hand image 1002 generated using image 1000 and sensor signals, which can represent the ground truth input during ML module training, or as... Figure 3 The illustration shows the output of the virtual hand image at box 312.

[0044] Figure 11 An exemplary ML module or engine 1100 is shown that can be used without the need for initial hand detection 1102. Instead, as previously described, the left keypoint estimation stage 1104 and the right keypoint estimation stage 1106 (details of the right stage 1104 are shown only for clarity) can receive multiple images 1108 of the hand holding the controller, cropped if necessary, and enhanced with super-resolution if required, according to principles discussed elsewhere herein. Images 1108 can be processed by a key neural network 1110 such as, but not limited to, a convolutional neural network (CNN).

[0045] The key neural network 1110 generates both 2D and 1D heatmaps 1112 and 1114, from which keypoints 1116 are derived to change the pose of the template hand 1118 based on keypoints 1116. Model parameters are learned by optimizing min E(θ). This is just one heatmap technique that can be used.

[0046] This generates an initial virtual image 1120 of the entire hand. Controller sensor signals 1122 from the controller in image 1108 are fed back to key NN 1110 as shown by line 1124, and / or controller touch input signals 1122 may be fed directly to key NN 1108 along with image 1108.

[0047] Regarding the exemplary heatmap technique discussed herein, in a non-limiting implementation, K heatmaps of size W0×H0,{H1,H2,...,Hk} can be estimated, where each heatmap Hk indicates the positional confidence of the k-th keypoint of the virtual hand to be rendered. (A total of K keypoints). (See "Efficient Object Localization Using Convolutional Networks" by Tomson et al.) arXiv:1411.4280v3This method, described in June 2015, generates heatmaps by passing images through multiple resolution libraries in parallel to simultaneously capture features at various scales. The output is a discrete heatmap, rather than a continuous regression. The heatmap predicts the probability of a joint occurring at each pixel. A multi-resolution CNN architecture (coarse heatmap model) is used to implement a sliding window detector to produce coarse heatmap outputs. This is just one exemplary heatmap technique that can be used.

[0048] refer to Figure 12 The system flow 1200, 1202, which is substantially the same as that of the other system flow, is shown in the diagram of multiple images from multiple cameras. Therefore, system flow 1200 is shown and disclosed in detail.

[0049] Image 1204 is received, wherein an analog controller 1206 is identified in a rectangular sub-region 1208 of image 1204 to allow the generation of a cropped image 1210 within the rectangle. The cropped image 1210 is input to one or more neural networks 1212, which also receive controller tracking information 1214. Tracking information 1214 may include the position of controller 1206 in space, the rotation of the controller, the speed and acceleration of the controller, and the rotational speed of the controller as indicated by one or more sensors in the controller, such as an inertial motion unit (IMU), magnetometer, accelerometer, and gyroscope. As discussed above, a ground truth image of a hand holding the controller and the accompanying controller tracking input can be used to train neural network 1212. Note that, as mentioned elsewhere herein and as... Figure 12 As shown, the neural network 1212 can also receive information from the controller input element 1215, such as face buttons and / or joysticks and / or finger sensors and / or grip buttons.

[0050] The neural network 1212 outputs a two-dimensional heatmap 1216, a one-dimensional heatmap 1218 representing depth, and the resulting three-dimensional hand joint positions 1219. These parameters are used for skeleton fitting 1220, which is applied to an image 1222 representing the hand pose in image 1204 in three dimensions. Skeleton fitting is the result of a calibration process that obtains a fixed bone length. Skeleton fitting attempts to minimize the difference (energy) between the heatmap and the joint positions projected onto the heatmap, as well as the temporal variation of the last frame, thereby obtaining the optimal joint rotation for calibrating the skeleton structure.

[0051] If needed, the controller tracking 1214 can be used to project the image 1224 onto the rectangle 1208.

[0052] It should be understood that although the principles of the invention have been described with reference to some exemplary embodiments, these embodiments are not intended to be limiting, and various alternative arrangements can be used to achieve the subject matter claimed herein.

Claims

1. A device for image processing, comprising: at least one processor programmed with instructions to: identify an image from at least one camera of a hand grasping a computer game controller; and present on a computerized display a virtual representation of the hand generated based at least in part on image analysis of a region of the image containing the controller and the hand and at least one touch signal from the controller, wherein, the touch signal is used to generate a virtual representation of a portion of the hand occluded from camera view by the controller.

2. The device of claim 1, wherein the at least one camera is mounted on a head-mounted display (HMD).

3. The device of claim 2, comprising the HMD.

4. The device of claim 1, wherein the touch signal is from a control key element of the controller.

5. The device of claim 1, wherein the touch signal is from a sensor on the controller, rather than from a control key element of the controller.

6. The device of claim 1, wherein the instructions are executable to generate the virtual representation using controller recognition without hand recognition in response to identifying a touch signal from the controller.

7. The device of claim 1, wherein the instructions are executable to: execute a machine learning (ML) model to generate the virtual representation by modifying key points of a template image based on the image in the region and the touch signal from the controller.

8. The device of claim 7, wherein the ML model comprises at least one neural network (NN) and at least one heat map.

9. A method for image processing, comprising: identifying an image of a hand grasping a computer-simulated controller; receiving at least one touch signal from the computer-simulated controller; and generating an image of a virtual hand for display of the image of the virtual hand based on both the image of the hand and the touch signal, wherein the generating step includes using the touch signal to generate a virtual representation of a portion of the hand occluded from camera view by the controller.

10. The method of claim 9, comprising presenting the image of the virtual hand on a computerized display.

11. The method of claim 10, wherein the computerized display comprises a head-mounted display (HMD).

12. The method of claim 9, comprising cropping the image of the hand grasping the computer-simulated controller to a region containing the controller and the hand, and using only the region and not other portions of the image of the hand grasping the computer-simulated controller to generate the image of the virtual hand.

13. The method of claim 9, wherein the touch signal is from a control key element of the controller.

14. The method of claim 9, wherein the touch signal is from a sensor on the controller, rather than from a control key element of the controller.

15. The method of claim 9, comprising generating the image of the virtual hand using controller recognition without using hand recognition in response to identifying a touch signal from the controller.

16. The method of claim 12, comprising executing a machine learning (ML) model to generate the virtual representation by modifying key points of a template image based on the image in the area and the touch signal from the controller.

17. The method of claim 16, wherein the ML model comprises at least one key neural network (NN) and at least one heat map.

18. An apparatus for image processing, comprising: at least one computer storage device that is not a transitory signal and comprises instructions executable by at least one processor to: receive at least one image of a human hand grasping a computer game controller; receive at least one touch signal from the controller; and generate a virtual hand representing the human hand based on both the image and the touch signal for presenting the virtual hand on at least one computerized display, wherein the touch signal is used to generate a virtual representation of a portion of the hand that is occluded from a camera field of view by the controller.

Images