Extending functionality of a game controller

Abstract

Methods and systems are provided for verifying an input provided at a controller, including detecting a finger gesture on a surface of the controller. In response to detecting a finger gesture, multi-modal data is collected from a plurality of sensors and components that track the finger gesture. The multi-modal data is used to generate a combinatorial model using a machine learning algorithm. The composite model is trained according to training rules defined for different finger gestures. An output from the combined model is recognized for a finger gesture. The output is interpreted as an input defining an interactive application selected for interaction.

Description

Technical Field

[0001] The present disclosure relates to recognizing input provided by finger gestures on a handheld controller, and more particularly to using multimodal data collected from multiple sensors and components associated with the handheld controller to verify input provided via finger gestures. Background Art

[0002] As the number of interactive applications and video games available to users on different devices continues to increase, it becomes increasingly important to accurately detect input provided via different devices. For example, input to a video game provided by a user using a handheld controller must be properly recognized and correctly interpreted in order to accurately affect the game state of the video game. Relying on only a single data mode (e.g., an image that tracks finger gestures) may lead to erroneous results in the video game.

[0003] It is against this background that embodiments of the present disclosure are presented. Summary of the invention

[0004] Implementations of the present disclosure relate to systems and methods for providing multimodal finger tracking to detect and verify finger gestures provided on an input device, such as a handheld controller. Multimodal finger tracking and verification ensures that finger gestures are properly identified and correctly interpreted, thereby reducing errors caused by relying solely on a single tracking mode. A custom finger tracking model (e.g., an ensemble model) is generated and trained using multiple data modalities captured by multiple sensors and components associated with a handheld controller (hereinafter referred to as a "controller") to improve the accuracy of detecting and interpreting finger gestures.

[0005] The traditional way of detecting input relies on a single data source model. For example, the traditional way relies on a general camera (i.e., a single data source) to detect and track the user's fingers on the controller. The tracking accuracy using a single source is less reliable and prone to errors, resulting in unsatisfactory results at the interactive application. In order to overcome the shortcomings of the traditional way, multimodal data is collected from multiple sensors and components associated with the controller, which are used to provide input and verify the finger gesture detection at the controller. The collected multimodal data is used to generate and train a multimodal data model, which is then used to correctly interpret the finger gesture. Since multiple data modes are used to generate and train the model, the multimodal data model is also referred to as a "combined model" in this article. According to the training rules defined for different finger gestures, the combined model is continuously trained using additional multimodal data collected over time. An output is selected from the combined model and used to confirm / verify the finger gesture detected at the controller. The finger gesture can correspond to pressing a real button or pressing a virtual button defined on the controller or an input provided on a touch screen interface set on the controller, and the output is identified as corresponding to the correct interpretation of the finger gesture. Virtual buttons can be recognized on any surface of the controller that does not have real buttons, and finger gestures on virtual buttons can be defined as single or double click or press or swipe in a specific direction, etc.

[0006] The model incorporates multimodal finger tracking technology in the following way: several model components are considered while generating and training the combined model, such as finger tracking using image feed from an image capture device, Inertial Measurement Unit (IMU) data from an IMU sensor provided in the controller, wireless signals from wireless devices provided in the user's environment, data from various sensors such as distance / proximity sensors, pressure sensors, etc. The combined model helps in accurately detecting finger gestures provided at the controller by tracking and validating finger gestures using data from more than one modality.

[0007] In one implementation, a method for validating input provided at a controller is disclosed. The method includes detecting a finger gesture provided by a user on a surface of the controller. The finger gesture is used to define an input of an interactive application selected by the user for interaction. Multimodal data is collected by tracking the finger gesture on the controller using multiple sensors and components associated with the controller. A combined model is generated using the multimodal data received from the multiple sensors and components. The combined model is continuously trained using additional multimodal data collected over time to generate different outputs, wherein the training is based on training rules defined for different finger gestures. A machine learning algorithm is used to generate and train the combined model to define different outputs. Outputs from the combined model are identified for the finger gestures. The outputs identified from the combined model are interpreted as inputs that define the interactive application.

[0008] In an alternative implementation, a method for defining input to an interactive application is disclosed. The method includes receiving a finger gesture provided by a user on a surface of a controller. The finger gesture is used to define an input to an interactive application selected by the user for interaction. Multimodal data capturing attributes of the finger gesture on the controller is received from a plurality of sensors and components associated with the controller. Weights are assigned to modal data corresponding to each mode included in the multimodal data captured by the plurality of sensors and components. The weight assigned to each mode indicates that the finger gesture is accurately predicted using the modal data of each mode. The finger gesture and the multimodal data are processed based on the weight assigned to each mode to identify an input to the interactive application corresponding to the finger gesture detected at the controller.

[0009] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure may be best understood by reference to the following description taken in conjunction with the accompanying drawings.

[0011] Figure 1A An abstract pipeline for building a combined model to detect gestures provided by a user on a controller is shown according to one implementation of the present disclosure.

[0012] Figure 1B A simplified block diagram illustrating various components for defining an abstract pipeline to build a combined model to detect input provided via finger gestures at a controller in accordance with one implementation of the present disclosure.

[0013] Figure 2A and Figure 2BAn example of determining real button presses and virtual button presses at a controller using modality data collected from an inertial measurement unit sensor associated with the controller in response to finger gestures detected at the controller is shown in accordance with one implementation of the present disclosure.

[0014] Figure 3A Example wireless signals and reflected signals captured using a wireless communication device to detect different finger movements of a user holding a controller according to one implementation of the present disclosure are shown.

[0015] Figure 3B Some example wireless signal graphs capturing amplitude changes caused by tracking certain user movements are shown in accordance with some implementations of the present disclosure.

[0016] Figure 4 An example microphone array for capturing the sound of activated controls (eg, button presses) on a controller to predict a virtual button press is shown in accordance with one implementation of the present disclosure.

[0017] Figure 5A and Figure 5B Various views of an image capture device coupled to a wireless controller to capture finger gestures provided at the controller are shown in accordance with one implementation of the present disclosure.

[0018] Figure 6 A simplified operational flow of a method for validating input provided at a controller for an interactive application according to one implementation of the present disclosure is shown.

[0019] Figure 7 Components of an exemplary computing device that can be used to perform aspects of various implementations of the present disclosure are shown. DETAILED DESCRIPTION

[0020] In the following description, a number of specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be appreciated by those skilled in the art that the present disclosure may be practiced without some or all of these specific details. In other cases, well-known process steps are not described in detail in order to avoid making the present disclosure unclear.

[0021] As the number of interactive applications continues to grow, it becomes increasingly important to accurately identify the manner in which inputs are provided using various devices. In particular, in interactive applications (such as high-intensity video game applications), inputs provided using a controller must be properly detected and accurately interpreted so that the game state of the video game can be updated in a timely and correct manner. To this end, inputs provided by a user at an input device (such as a handheld controller) are detected, and in response, multiple data modalities are collected from multiple sensors and components associated with the controller to track finger gestures at the controller. The multimodal data is used to generate and train a custom finger tracking model, which is then used to identify outputs corresponding to finger gestures detected at the controller. The trained custom finger tracking model improves the accuracy of predicting finger gestures better than a general-purpose camera-only finger tracking model because finger gesture prediction relies on multiple modal data sources to verify finger gestures.

[0022] Some of the modal data and multiple sensors and components associated with the controller that capture the modal data include: (a) IMU data captured using an inertial measurement unit (IMU) sensor (such as a magnetometer, gyroscope, accelerometer, etc.); (b) wireless communication devices (such as Bluetooth enabled devices installed in the environment) TM The present invention relates to a method for detecting a finger gesture in a multi-modal manner, wherein the multi-modal data collection engine is a multi-modal data collection engine that is executed on a server and is used to collect various modal data captured by multiple sensors and components at the controller and to generate and train the combined model. In an alternative implementation, the multi-modal data collection engine may be allowed to execute at the controller itself or at a processor that is co-located with the controller and coupled to the controller to reduce latency.

[0023] Each modality of data captures some attributes of the finger gesture provided at the controller, and these attributes are used to validate the finger gesture so that the correct input corresponding to the finger gesture can be identified to affect the results of the interactive application selected by the user to interact. The multiple data modalities captured by tracking the finger gesture on the controller are used to provide additional validation to correctly determine the finger gesture so that the correct input can be identified for the interactive application.

[0024] Figure 1A An example abstract pipeline for building a combined model to detect finger gestures according to some implementations is shown. The abstract pipeline uses a multimodal finger tracking method, in which multiple sensors and components associated with a controller are used to track finger gestures, and data generated from such tracking is used to identify and / or verify finger gestures on the controller. Each of the multiple sensors and components captures modal data of a specific mode. For example, the multimodal data 120-1 captured by the sensors and components from finger tracking includes camera feed 120a1, inertial measurement unit sensor (IMU) data 120b1, WiFi data (including WiFi forward signal and reflected signal) 120c1, sound data 120d1, sensor data from distance / proximity sensor 120e1, sensor data from pressure sensor 120f1, and so on. Of course, the above list of data for capturing finger gestures is provided as an example and should not be considered exhaustive or limiting, and other forms of modal data captured from different sensors and components may also be considered when identifying and / or verifying finger gestures. The modal data from each mode is analyzed to detect finger gestures. Considering modal data from a single mode may result in erroneous or unreliable predictions of finger gestures. Therefore, in order to improve the accuracy of finger gesture detection, modal data from multiple modes are considered during analysis. In some implementations, a voting module is used to vote on the modal data associated with different modes, wherein a weight is assigned to the modal data for each mode. The weights assigned to the modal data for each mode may or may not be equal, and in some implementations, the decision to assign equal or unequal weights (144) to the modal data for each mode is made based on the reliability of each mode in correctly predicting the finger gesture. The weights assigned to the different data modes are then used during analysis to correctly predict the finger gesture. After correctly predicting the finger gesture, input associated with the finger gesture is identified and used as interactive input from the user for an interactive application selected by the user to interact with. Reference will be made to Figure 1B Describe the details of the analysis.

[0025] Figure 1BVarious components of a system 100 are shown for correctly detecting finger gestures at a controller so that input to an interactive application can be properly recognized. The various components and sensors used to build a combined model represent an abstract pipeline that can be joined to correctly detect finger gestures. The combined model is generated and trained using modal data from several modal components and sensors that capture different attributes of finger gestures provided at the controller. As described above, the different modes of data captured by tracking finger gestures include image feeds, IMU data, WiFi signals (wireless signals including reflected signals), data obtained from distance / proximity sensors, pressure sensors, etc. The attribute information related to the finger gestures captured by these components and sensors is forwarded to a server computing device for further processing. The server computing device processes the multimodal data to properly identify the finger gestures at the controller.

[0026] To this end, a system 100 for determining input to an interactive application includes: a controller 110, such as a handheld controller, used by a user to provide finger gestures; a plurality of sensors and components 120 associated with the controller 110 for capturing various attributes of the finger gestures; and a server device 101 for processing the finger gestures and multimodal data capturing the various attributes of the finger gestures and for verifying the finger gestures. The finger gestures are provided by the user as input to an interactive application selected by the user for interaction. The server computing device (or hereinafter referred to as the "server") 101 engages a modal data collection engine 130 to collect various modal data forwarded by the plurality of sensors and components 120, and engages a modal data processing engine 140 to process the multimodal data to recognize the finger gestures and define the input to the interactive application. The modal data collection engine 130 and the modal data processing engine 140 can be part of a multimodal processing engine executed on the server 101.

[0027] In some implementations, the server 101 may be a game console or any other computing device co-located in the environment in which the user operates. The game console or computing device may then be coupled to other game consoles via a network as part of a multi-player setting for a video game. In some implementations, the controller 110 is a networked device, wherein the controller 110 is directly coupled to the remote server 101 via a network (not shown) such as the Internet. In the case of a networked device, the controller 110 is coupled to the network via a router embedded in the controller 110 or external to the controller 110. In other implementations, the controller 110 is coupled to the remote server 101 via a game console or another client device (not shown) over the Internet, wherein the game console or client device is co-located with the controller 110. As part of the initial setup, the controller 110 is paired with the server 101, either upon detecting the presence of the controller 110 in the vicinity of a game console, another client device, or a router (in the case where the server 101 is connected to the controller 110 via a game console, router, or other computing device co-located with the controller 110), or upon detecting a user activating the controller 110 (i.e., in the case where the server 101 is remote from the controller 110). Details of the finger gestures provided on the surface of the controller 110 are forwarded to the game console / server 101 for processing.

[0028] In response to detecting a finger gesture provided by a user on the surface of the controller 110, various sensors and components 120 associated with the controller 110 are automatically activated to capture different attributes of the user's finger gesture. Some of the sensors and components 120 that are automatically activated to collect various attributes of the finger gesture include an image capture device 120a, an IMU 120b, a WiFi device 120c, a microphone array 120d, a distance / proximity sensor 120e, and a pressure sensor 120f. In addition to the above sensors and components, other sensors and / or components may also be used to collect attributes of the finger gesture at the controller.

[0029] In some implementations, the image capture device 120a is a camera embedded within a mobile computing device (such as a mobile phone or tablet computing device, etc.). Alternatively, the camera can be a web camera, a console camera, or a camera that is part of an HMD. The camera or a device in which the camera is embedded is paired with the game console / server 101 using a pairing engine 125a. Pairing allows the image capture device (i.e., camera) to receive an activation signal from the game console / server 101 to capture images of a user's finger gestures on the controller 110. Upon detecting a finger gesture on the surface of the controller 110, the game console / server 101 generates an activation signal to the image capture device. The mobile computing device with the embedded camera is supported on a support structure disposed on the controller 110 to allow the camera in the mobile computing device to capture close-up views of various features of the finger gestures provided on the controller 110. Reference will be made to Figure 5A and Figure 5B More information is discussed regarding the holding structure. Once activated, the image capture device captures images of various attributes of the finger gesture, including the finger used to provide the gesture, the position of the finger relative to an input control on the controller 110 (e.g., a button, a touch screen surface, other interactive surface, etc.), the movement of the finger on the controller, the type of finger gesture provided (e.g., a single click, a double click, a swipe gesture, etc.), etc. The images capturing the attributes of the finger gesture are forwarded to the modality data collection engine 130 executing on the game console / server 101 as an image data camera feed.

[0030] In response to activation of various sensors and components, an inertial measurement unit (IMU) sensor 120b integrated within the controller 110 is used to capture IMU signals related to finger gestures. In some implementations, when the user holds the controller 110 in their hand, the IMU signals captured by the IMU sensor are used to distinguish different finger gestures detected at the controller 110. For example, IMU signals captured of subtle taps at a location defined on the back of controller 110 may be interpreted as meaning a first input (i.e., virtual button 1), IMU signals captured of subtle taps at a location defined on the front of controller 110 (excluding any buttons or interactive interfaces) may be interpreted as meaning a second input (i.e., virtual button 2), IMU signals captured of taps at the upper right corner of the back of controller 110 may be interpreted as meaning a third input (i.e., virtual button 3), taps at the upper left corner of the back of controller 110 may be interpreted as meaning a fourth input (i.e., virtual button 4), IMU signals captured of taps on the back of the controller (using the middle finger) may be interpreted as meaning a fifth input (i.e., virtual button 5), taps on a real button on the front of controller 110 may be interpreted as meaning a sixth input (e.g., a real button press), and so on.

[0031] In some implementations, virtual buttons defined by tracking finger gestures can be used to extend the functionality of the controller 110. The extended functionality allows the user to interact with more than one application at the same time, and this interaction can be done without having to interrupt one application to use another application. Virtual buttons can be defined by identifying the finger positions of the user when holding the controller 110 and the finger gestures provided by the user with respect to the identified finger positions. Sometimes, when a user is playing a game (e.g., executed on a game controller or a game server), the user may also be listening to music provided by a second application (e.g., a music application). Typically, when a user must interact with a music application, the user must pause the game they are currently playing, access a menu to interact with the music application, and use a button or interactive surface on the controller 110 to advance to the next song in their playlist. In order to avoid the user pausing the game and provide other ways to interact with the music application while playing the game, virtual buttons can be defined to extend the functionality of the controller 110. Virtual buttons can be defined and associated with pre-assigned commands so that the user can use the virtual buttons to interact with the music application without interrupting the user's current game.

[0032] In another implementation, finger gesture tracking can be used to allow users with certain disabilities to communicate in online games when the user holds the controller. For example, when a user with a disability holds the controller 110, finger gesture tracking can be used to detect different finger positions and gestures. The positions and gestures of these fingers can be interpreted as Morse code inputs (points and strokes representing clicks and slides) using a machine learning (ML) algorithm, and such interpretation can be performed by the ML algorithm by identifying the user's disability provided in the user profile of the user. In addition, the Morse code input can be translated into text characters or provided as game input. The text characters can be used to communicate with other players / viewers / users by converting text to speech, or provided as text responses on a chat interface. The Morse code input can be interpreted as being related to game input and used to affect the game state of the game played by the user. The above-mentioned application of using finger gesture tracking and interpretation to identify virtual buttons and / or inputs to interactive applications for users with disabilities is provided only as an example and should not be considered exhaustive or limiting, and other applications can also be envisioned.

[0033] Figure 2A and Figure 2B Some example signal amplitude changes captured in corresponding IMU signals for different finger gestures using the IMU sensor 120b are shown. Figure 2AThe positions of the user's fingers relative to different input controls (ie, buttons and touch screen interaction interface) defined on the controller 110 are shown when the user operates the controller 110 . Figure 2B Shown are some example amplitude variations captured for three different finger gestures detected at different locations on a controller in some implementations. Figure 2B The amplitude changes shown represent the amplitude changes along the X, Y and Z axes captured in the IMU signal for different finger gestures. For example, the amplitude changes shown along the X, Y and Z axes in the box 'VB1' are associated with finger gesture 1, which can be interpreted as meaning that the finger gesture 1 provided by the user at the controller 110 corresponds to the input associated with the virtual button 1. Similarly, the amplitude changes shown along the X, Y and Z axes in the box 'VB2' are associated with finger gesture 2, which can be interpreted as meaning that the finger gesture 2 provided by the user at the controller 110 corresponds to the input associated with the virtual button 2, and the amplitude changes shown along the X, Y and Z axes in the box 'RB1' correspond to finger gesture 3, which can be interpreted as meaning that the finger gesture 3 provided by the user at the controller 110 corresponds to the input associated with the real button 1. Of course, virtual buttons 1, 2, 3, etc. and real buttons 1, 2, 3, etc. can be defined as being associated with different inputs for different interactive applications. The IMU signal capturing the subtle signal is forwarded to the modality data collection engine 130 as IMU sensor data.

[0034] Reference back Figure 1B In response to the activation of various sensors and components, WiFi devices (i.e., wireless devices) 120c distributed in the user's environment begin to capture WiFi signals. The data captured in the WiFi signals can be used to detect the user's position, the relative positions of various parts of the user's body (including hands and fingers), body movements (including finger movements / finger gestures), etc. The data from the WiFi signals and the reflected signals of the WiFi signals can be used to detect different finger movements of the user when holding the game controller 110.

[0035] Figure 3A and Figure 3B Various WiFi signals captured using a WiFi device in an environment (i.e., geographic location) where a user (i.e., User 1) is located are shown. The WiFi device includes a transmitter 301 (such as a router, a laptop, other computing devices including a personal digital assistant (e.g., a voice assistant)) and a receiver device 302 (such as a second laptop or desktop computing device, etc.). The above list of devices representing transmitters 301 and receivers 302 is provided as an example only and should not be considered exhaustive or limiting. Transmitter 301 continuously transmits WiFi signals, and receiver 302 receives various WiFi signals. Figure 3AVarious WiFi signals transmitted by a transmitter 301 and WiFi signals received by a receiver 302 in a room where a user is located are shown. Some WiFi signals received by the receiver 302 include WiFi signals 303 reflected by the walls of the room, WiFi signals 304 reflected by user 1, WiFi signals 305 representing the line of sight of user 1, and WiFi signals 306 reflected by the floor of the room. The WiFi signals received by the receiver 302 are continuously monitored to determine the movement of the user within the geographical location of the room. Initially, when there are no objects or users between the transmitter 301 and the receiver 302, the WiFi signals transmitted by the transmitter 301 and received by the receiver 302 are used to first determine the channel state information (CSI) representing the channel characteristics of the communication link between the transmitter 301 and the receiver 302 for the geographical location where the user 1 is to operate. The channel characteristics are used to establish a baseline for the WiFi signal and take into account the combined effects of scattering, fading, and signal strength attenuation with distance. Then, when the user 1 appears in the geographical location (e.g., room) and when the user 1 moves in the room, the CSI is determined. The change in CSI is caused by the user's body blocking or reflecting one or more of the above WiFi signals. The change in the corresponding one or more WiFi signals collected over a period of time is used to determine the user's movement in a geographic location (e.g., a room), including the different finger movements of user 1 when user 1 holds and operates game controller 110. The channel characteristics of the WiFi signal can be used to capture snapshots of body parts (including user 1's fingers). These snapshots can then be used to reconstruct the body parts to determine which body part (e.g., finger) has been moved. WiFi signals can include signals provided by a router and Bluetooth signals provided by a controller. TM Signal.

[0036] Figure 3BThe CSI signal amplitude change of a Wifi signal (e.g., a single subcarrier) received from a WiFi device due to user movement in some implementations is shown. A graph of the signal amplitude change is plotted over time. A signal amplitude change 321 of a WiFi signal transmitted when there is no user or object between the transmitter 301 and the receiver 302 in a geographic location is shown. In some implementations, the signal amplitude change 321 establishes a baseline CSI signal amplitude change. Signal amplitude changes 322-326 capture changes due to user movement. For example, signal amplitude change 322 shows an example of a change in a WiFi signal when user 1 is detected sitting in a room (i.e., geographic location), signal amplitude change 323 captures a change in a WiFi signal when a room door is detected to be opened or closed, signal amplitude change 324 captures a change in a WiFi signal when user 1 is detected typing on an input device (such as a keyboard), signal amplitude change 325 captures a change in a WiFi signal when user 1 is detected waving, and signal amplitude change 326 captures a change in a WiFi signal when user 1 is detected walking around the room. Therefore, using WiFi signals, different finger movements when a user operates the controller 110 can be detected. The WiFi signals capturing various user actions are forwarded to the modality data collection engine 130 as WiFi signals.

[0037] Reference back Figure 1B In response to detecting a finger gesture on the controller 110, a microphone array 120d embedded in or attached to the controller 110 is activated. The finger gesture can be a real button press or a virtual button press at the controller 110, and the activated microphone array 120 is configured to capture the sound attributes of the button press. Specifically, the microphones within the microphone array 120d work together to determine the direction of the sound source and accurately determine the location. The attributes of the finger gesture (i.e., direction and location) are used to determine whether the finger gesture corresponds to a real button press or a virtual button press. In some implementations, different positions of the controller in addition to setting the position of the real button and the touch screen interface can represent different virtual buttons. For example, the upper left corner of the back of the controller 110 can be defined to represent virtual button 1, the upper right corner of the back of the controller 110 can be defined to represent virtual button 2, the top center position of the back of the controller 110 can be defined to represent virtual button 3, and so on. The attributes of the sound caused by the finger gesture can be interpreted to determine whether the finger gesture corresponds to a real button press or a virtual button press, and to determine whether a real button or a virtual button is pressed. The properties of the sound captured by the microphone array 120 d are forwarded to the modality data collection engine 130 .

[0038] Figure 4An example microphone array 120d embedded in or coupled to the controller 110 is shown for capturing sounds originating from a user's finger gestures at the surface of the controller 110. The microphone array 120d is shown to include 4 microphones (401a, 401b, 401c, and 401d). The intensity of the sound captured by each of the microphones 401 varies based on the distance of the sound from the corresponding microphone 401. The sound signal captured by each of the microphones 401a-401d is then forwarded to a digital signal processor (DSP) 402 for processing. In some implementations, the DSP 402 is configured to assign different weights to different sounds captured by the microphones. The sound and the relative weights assigned to each of the sounds captured by the different microphones in the microphone array 120d are analyzed using, for example, triangulation techniques to identify the properties of the various sounds captured, such as direction, position, duration, volume, frequency, etc., and the properties and relative weights are used to determine a specific button press, or a slide and slide direction, etc. In an alternative implementation, instead of assigning different weights to each sound, the DSP 402 can assign different weights to different attributes of each sound detected / captured by the microphones in the microphone array 120d, and use the weights and the detected attributes to correlate the sound with a specific button press or finger swipe. The weights assigned to the different attributes can be used to determine which sounds should be ignored as ambient sounds / noise, and which sounds should be focused on to determine finger gestures. The attributes of the sound and the details of the analysis are then forwarded to the modal data collection engine 130 as electrical signals.

[0039] Reference back Figure 1BIn some implementations, in response to detecting a finger gesture on the controller 110, one or more distance / proximity sensors 120e are activated to capture attributes of the finger gesture provided at the controller 110. Similar to the microphone array, the distance / proximity sensor 120e can be used to capture attributes of the finger gesture provided at the back of the controller 110. The attributes of the finger gesture provided on the back of the controller 110 captured by the distance / proximity sensor 120e can be used to independently determine a virtual button press, or can be used in conjunction with the attributes of the sound captured by the microphone array 120d to further verify the virtual button press determined from the finger gesture. The additional verification provided by the data captured by the distance / proximity sensor 120e makes the detection of the virtual button press from the finger gesture more accurate. In some implementations, the distance / proximity sensor 120e can include an ultrasonic sensor, an infrared sensor, an LED time-of-flight sensor, a capacitive sensor, etc. The above list of distance / proximity sensors 120e is provided as an example and should not be considered exhaustive or limiting. The attributes of the finger gesture captured by the distance / proximity sensor 120e are forwarded to the modal data collection engine 130. In addition to the distance / proximity sensor 120e, the pressure sensor 120f can also be activated to capture the attributes of the pressure applied at different locations of the controller 110 through the finger gesture (e.g., location, amount of pressure, time of pressure application, finger used to apply pressure, etc.). For example, if the pressure application time is less than a threshold, the attributes of the pressure captured by the pressure sensor 120f can be ignored. If the pressure application time is greater than the threshold, the attributes of the applied pressure can be forwarded as input to the modal data collection engine 130. Similarly, if the amount of pressure applied is less than the threshold amount, the data captured by the pressure sensor 120f can be ignored. However, if the amount of pressure applied is greater than the threshold amount, the data captured by the pressure sensor 120f can be considered as input to the modal data collection engine 130. The attributes of the pressure that meets or exceeds the threshold / threshold amount provided via the finger gesture are forwarded to the modal data collection engine 130 as pressure sensor data.

[0040] As previously described, in some implementations, the image capture device may be a camera embedded within a mobile computing device, such as a mobile phone or a tablet computing device. In these implementations, for example, a camera of a mobile phone may be preferred over a web camera, a console camera, or a camera embedded in an HMD. In alternative implementations, a camera embedded in a mobile phone (i.e., a mobile computing device) may be used to capture images of finger gesture attributes in addition to a web camera / console camera / HMD camera. In implementations in which a camera of a mobile computing device is used to capture images of finger gesture attributes, the mobile computing device (e.g., a mobile phone) may be coupled to the controller 110.

[0041] Figure 5A and Figure 5B One such implementation is shown in which a mobile computing device (eg, mobile phone 502) is coupled to controller 110. In some implementations, mobile phone 502 is coupled to controller 110 using a retaining structure 504. Figure 5A A front perspective view of the controller 110 is shown having a retaining structure 504 for receiving and retaining a mobile phone 502 . Figure 5B A back view of a holding structure 504 coupled to the controller 110 and configured to receive, hold, and manipulate a mobile phone 502 is shown. In some implementations, the holding structure 504 is a three-dimensional (3D) printed structure that can be attached to the controller 110. In some implementations, the 3D printed structure is equipped with a motor (not shown) to move the mobile phone 502 to different positions to allow a camera of the mobile phone 502 to capture various attributes of a finger gesture.

[0042] Also refer to Figure 1B , Figure 5A and Figure 5BIn some implementations, in order to accommodate different mobile phone models (size of the mobile phone, position of the camera, number of cameras, etc.), the game console / server 101 first performs an automatic pairing operation to pair the mobile phone 502 with the game console / server 101. A signal is sent from the pairing engine 125a to the mobile phone to initiate the pairing operation. After the mobile phone 502 is successfully paired with the game console / server 101, the mobile phone 502 is mounted on the holding structure by initiating a calibration operation. The calibration engine 125b is used to determine the brand and model of the mobile phone 502, and send a signal to the support structure to adjust the size of the support structure to accommodate the mobile phone 502. In response to the signal from the calibration engine 125b, the motor of the operating holding structure 504 adjusts the size of the holding structure 504 so that the mobile phone 502 can be safely received. In addition to automatically calibrating the size of the holding structure to accommodate the mobile phone 502, the calibration engine 125b also calibrates the angle that the mobile phone needs to be adjusted to capture images of different positions of the finger. The angle is automatically calibrated in response to detecting the presence of a user's finger and a finger gesture provided on the controller 110, and the angle is dynamically determined based on the position of the user's finger when the user provides a finger gesture on the controller 110. As part of the angle calibration operation, the calibration engine 125b tracks the user's hand and finger and sends a second signal to the controller 110 and / or the holding structure 504 to adjust the motor so as to move the mobile phone 502 so that the camera of the mobile phone 502 is aligned with the calibrated angle. In response to the second signal from the calibration engine 125b, the motor of the holding structure 504 is engaged to move and rotate the movable portion of the holding structure 504 to achieve good hand and finger tracking positioning. Once the mobile phone is moved into position, the camera of the mobile phone 502 is activated to capture images of various attributes of the finger gesture. The captured image is streamed to the game console / server 101 as an image data camera feed.

[0043] Inputs from the plurality of sensors and components 120 are collected by the modal data collection engine 130 to generate multimodal data. The multimodal data is processed to identify modes and the amount of modal data captured for each mode included in the multimodal data collected from the sensors and components. The details of the modes captured by the sensors and components 120, the amount of modal data for each mode, and the multimodal data are forwarded by the modal data collection engine 130 to the modal data processing engine 140 for further processing.

[0044] The modality data processing engine 140 analyzes the modality data of each mode included in the multimodal data to recognize and / or verify the finger gestures at the controller. Figure 1AAs described, in order to improve the accuracy of finger gesture detection, modal data from multiple modes are considered in the analysis. As part of the analysis, the modal data associated with each mode is assigned a weight. The weight assigned to the modal data of each mode can be equal or unequal. In some implementations, the decision to assign equal or unequal weights (144) to the modal data of each mode is made based on the accuracy of correctly predicting the finger gesture using the modal data of each mode. In some implementations, multimodal data can be roughly classified into a first set of modal data captured using multiple sensors and a second set of modal data captured using components. In some implementations, the weight assigned to the modal data of each mode captured by the sensor is greater than the weight assigned to the modal data of each mode captured by the component. For example, IMU sensor data captured by an IMU sensor or distance sensor data captured by a distance / proximity sensor is assigned a greater weight than an image data camera feed captured by an image capture device. In another example, IMU sensor data and sound data are assigned a greater weight than WiFi signal data. The weight assignment engine 144a is used to identify the patterns associated with each modal data included in the multimodal data and the reliability of the modal data of the corresponding pattern in predicting finger gestures. Based on the reliability of each pattern, the weight assignment engine 144a assigns weights to the modal data. Then, the weights assigned to different data patterns are used together to correctly predict finger gestures. For example, the modal data processing engine 140 generates cumulative weights using the weights assigned to the modal data of each pattern included in the multimodal data. The cumulative weights are used to correctly predict finger gestures. Since the game console / server 101 relies on more than one data pattern to identify and / or verify finger gestures, the predicted finger gestures are more accurate. Once the finger gesture is identified / verified, the input associated with the finger gesture is identified and used as a user input for an interactive application (such as a video game) to affect the game state.

[0045] In some implementations, the modal data processing engine 140 engages a machine learning (ML) algorithm 146 to analyze multimodal data captured by multiple sensors and components to identify and / or verify finger gestures provided at the controller 110. The ML algorithm 146 generates and trains an ML model 150 using the multimodal data. The ML model 150 is used to predict and / or verify finger gestures so that appropriate inputs corresponding to the predicted / verified finger gestures can be identified and used to affect the state of an interactive application (e.g., a video game) selected by the user to interact. The ML algorithm 146 engages a classification engine (i.e., a classifier) ​​148 to generate and train the ML model 150. The ML model 150 includes a network of interconnected nodes, wherein each pair of consecutive nodes is connected by an edge. The classifier 148 is used to populate various nodes within the network of interconnected nodes of the ML model 150, wherein each node is associated with one or more modalities of modal data. Interrelationships between nodes are established to understand the complexity of the modal data of different modes, and to identify outputs for identifying or verifying finger gestures and identifying inputs corresponding to finger gestures.

[0046] In some implementations, the classifier 148 is predefined for different modes to understand the complexity of the modal data of each mode when correctly predicting and / or verifying the finger gestures provided at the controller. When the finger gestures are provided at the controller 110, the classifier 148 uses the modal data captured by the sensors and components in real time to further train the ML model 150, and uses the ML model 150 to determine the amount of influence of the modal data of each mode on the correct prediction / verification of the finger gestures. The ML model 150 can be trained according to the training rules 142 to improve the accuracy of the finger gesture prediction. The training rules are defined for each finger gesture based on the finger anatomy, the way of holding the controller, the position of the finger relative to the button, etc. The machine learning (ML) algorithm 146 uses the modal data of different modes included in the multimodal data as inputs to the nodes of the ML model 150, gradually updates the nodes using additional multimodal data received over time, and adjusts the output to meet the predefined criteria for different finger gestures. The ML algorithm 146 uses reinforcement learning to strengthen the ML model 150 by building the ML model 150 using an initial multimodal data set, learning the complexity of each modality and how the modal data of each modality affects the correct prediction / validation of finger gestures, and using additional modal data received over time to enhance the learning and reinforcement of the model. The adjusted output of the ML model 150 is used to correctly predict / validate different finger gestures. Outputs from the adjusted output are selected to correspond to finger gestures, and this selection can be based on the accumulated weight of the multimodal data, which indicates an accurate prediction of the finger gesture.

[0047] In some implementations, finger gesture tracking using modal data captured for different modes may be specific to a certain user. For example, each user may operate the controller 110 in a different way. For example, a first user may hold the controller 110 in a specific way, provide input on the controller in a specific way or at a specific speed or with a specific pressure, etc. When a second user uses the controller 110 to provide input via finger gestures, the way they hold the controller 110 or use the controller 110 to provide input may be different from the first user. In order to take into account the way different users hold the controller 110, a reset switch may be set at the controller 110 to allow resetting or reprogramming of finger gesture tracking so that the finger gesture interpretation can be user-specific rather than user-unknown. The reset switch can be defined as a specific button press or a specific sequence of button presses. In another example, finger gesture tracking can be reset or reprogrammed according to the needs of the user, where, for example, such a request can be based on the specific environment in which the finger gesture is to be tracked.

[0048] Although various implementations discussed herein describe that the modality data collection engine 130 and the modality data processing engine 140 with the ML algorithm 146 to recognize / verify finger gestures are implemented on the server 101, the modality data collection engine 130, the modality data processing engine 140 with the ML algorithm 146 can be implemented locally on a disk within a computing device coupled to the controller 110 (e.g., a game console co-located with the controller) rather than on a remote server 101 to reduce latency. In such an implementation, a graphics processing unit (GPU) on the game console can be used to increase the speed of predicting finger gestures.

[0049] Various implementations discussed herein teach a multimodal finger tracking mechanism in which several modal components capture multimodal data. Each of the modal components provides information about finger gesture detection to the voting engine. The voting engine assigns appropriate weights to the modal data of each mode in turn, which indicates the accurate prediction of the finger gesture using the modal data of the corresponding mode. For example, a greater weight can be given to the modal data generated from the sensor because the sensor tends to detect gestures more accurately than the web camera feed. Since the system relies on modal data of more than one mode, the relative weight of the modal data based on the prediction accuracy leads to a reduction in the error of detecting finger gestures. The modal data processing engine 140 uses real button press state data, video features (i.e., images from an image capture device), audio features using a microphone array associated with the controller 110, sensor data, and WiFi signals to train a custom finger tracking model (i.e., ML model 150) to predict finger gestures with higher accuracy than when relying on predictions of a single data source (such as relying on a general camera finger tracking model alone).

[0050] Figure 6 An operational flow of a method for verifying input provided at a controller in some implementations is shown. The method begins at operation 610, when a finger gesture is detected on a surface of a controller operated by a user for interacting with an interactive application (such as a video game). The finger gesture can be provided on any surface of the controller, including on input controls, such as real buttons and touch screen interactive interfaces. In response to the detection of the finger gesture, multiple sensors and components are activated to capture various attributes of the finger gesture, as shown in operation 620. Each sensor or component captures modal data of a specific mode, and the modal data captured by multiple sensors and components are collected to define multimodal data. A combined model is generated using a machine learning algorithm, as shown in operation 630. The combined model is generated to include a network of interconnected nodes, each of which is populated with modal data of one or more modes. The knowledge generated at each node based on the modal data included therein is exchanged between different nodes within the network through interconnection, and the knowledge is propagated to other nodes, thereby building on the knowledge. In order to improve the accuracy of predicting / verifying the finger gesture, a weight can be given to the modal data of each mode, which indicates the accurate prediction of the finger gesture using the modal data of the corresponding mode. Based on the accumulated weights of the various modal data collected for the finger gesture, outputs are defined in a combined model (also referred to herein as "ML model 150"), where each output satisfies a certain level of prediction criteria for predicting the finger gesture. As shown in operation 640, an output from the combined model is identified for the finger gesture. The output is identified as satisfying at least the prediction criteria defined or required for the finger gesture. The identified output from the combined model is used to define an input associated with the finger gesture.

[0051] Figure 7Components of an example device 700 that can be used to perform various aspects of various embodiments of the present disclosure are shown. The block diagram shows a device 700 that can be incorporated with or can be a personal computer, a video game console, a personal digital assistant, a head-mounted display (HMD), a wearable computing device, a laptop or desktop computing device, a server, or any other digital computing device suitable for practicing the embodiments of the present disclosure. For example, in various implementations discussed herein, the device 700 represents a first device and a second device. The device 700 includes a central processing unit (CPU) 702 for running software applications and optionally an operating system. The CPU 702 can be composed of one or more homogeneous or heterogeneous processing cores. For example, the CPU 702 is one or more general-purpose microprocessors with one or more processing cores. Other embodiments can be implemented using one or more CPUs with a microprocessor architecture that is particularly suitable for highly parallel and computationally intensive applications, such as processing interpretation queries, identifying context-related resources, and immediately implementing and rendering context-related resources in video games. Device 700 can be local to the player playing the game session (e.g., a game console), remote from the player (e.g., a backend server processor), or one of many servers in a gaming cloud system that uses virtualization to remotely stream the game play to client devices.

[0052] The memory 704 stores applications and data for use by the CPU 702. The storage device 706 provides non-volatile storage devices and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD or other optical storage devices, as well as signal transmission and storage media. The user input device 708 conveys user input from one or more users to the device 700, examples of which may include a keyboard, a mouse, a joystick, a touch pad, a touch screen, a static or video recorder / camera, a tracking device for identifying gestures, and / or a microphone. The network interface 714 allows the device 700 to communicate with other computer systems via an electronic communication network, and may include wired or wireless communications through local area networks and wide area networks (such as the Internet). The audio processor 712 is suitable for generating analog or digital audio output from instructions and / or data provided by the CPU 702, the memory 704 and / or the storage device 706. The components of device 700 , including CPU 702 , memory 704 , data storage device 706 , user input device 708 , network interface 714 , and audio processor 712 , are connected via one or more data buses 722 .

[0053] The graphics subsystem 720 is also connected to the data bus 722 and the components of the device 700. The graphics subsystem 720 includes a graphics processing unit (GPU) 716 and a graphics memory 718. The graphics memory 718 includes a display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. The graphics memory 718 can be integrated with the GPU 716 in the same device, connected to the GPU 716 as a separate device, and / or implemented in the memory 704. The pixel data can be provided directly from the CPU 702 to the graphics memory 718. Alternatively, the CPU 702 provides the GPU 716 with data and / or instructions defining the desired output image, and the GPU 716 generates pixel data for one or more output images from the data and / or instructions. The data and / or instructions defining the desired output image can be stored in the memory 704 and / or the graphics memory 718. In one embodiment, the GPU 716 includes a 3D rendering capability to generate pixel data for an output image from instructions and data defining the geometry, lighting, shadows, textures, motion, and / or camera parameters of the scene. GPU 716 may also include one or more programmable execution units capable of executing shader programs.

[0054] Graphics subsystem 720 periodically outputs pixel data of an image from graphics memory 718 for display on display device 710. Display device 710 may be any device capable of displaying visual information in response to a signal from device 700, including CRT, LCD, plasma, and OLED displays. In addition to display device 710, the pixel data may also be projected onto a projection surface. For example, device 700 may provide an analog or digital signal to display device 710.

[0055] It should be noted that access services delivered over a wide geographic area (such as providing access to games of the current embodiment) often use cloud computing. Cloud computing is a computing method in which dynamically scalable and usually virtualized resources are provided as services over the Internet. Users do not need to be experts in the technical infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, such as infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Cloud computing services typically provide online common applications (such as video games) accessed from a web browser, while the software and data are stored on servers in the cloud. Based on the way the Internet is depicted in computer network diagrams, the term cloud is used as a metaphor for the Internet and is an abstraction of the complex infrastructure that it hides.

[0056] In some embodiments, a game server can be used to perform the operation of a duration information platform for video game players. Most video games played over the Internet are operated by connection with a game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. In other embodiments, video games can be performed by a distributed game engine. In these embodiments, a distributed game engine can be performed on multiple processing entities (PEs) so that each PE performs a functional fragment of a given game engine on which a video game runs. The game engine simply regards each processing entity as a computing node. The game engine typically performs a series of operations with diverse functions to perform video game applications and additional services for user experience. For example, a game engine implements game logic, performs game calculations, physical effects, geometric transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, audio communications, game play / playback functions, help functions, etc. Although a game engine can sometimes be performed on an operating system virtualized by a hypervisor of a specific server, in other embodiments, the game engine itself is distributed between multiple processing entities, each of which can reside on different server units in a data center.

[0057] According to this embodiment, the corresponding processing entity for performing the operation can be a server unit, a virtual machine, or a container, depending on the needs of each game engine fragment. For example, if a game engine fragment is responsible for camera transformations, a virtual machine associated with a graphics processing unit (GPU) can be provided to this particular game engine fragment, because it will perform a large number of relatively simple mathematical operations (e.g., matrix transformations). Other game engine fragments that require fewer but more complex operations can be provided with processing entities associated with one or more higher-powered central processing units (CPUs).

[0058] By distributing the game engine, the game engine is provided with elastic computing properties that are not constrained by the capabilities of the physical server units. Instead, more or fewer computing nodes are provided to the game engine as needed to meet the needs of the video game. From the perspective of the video game and the video game player, a game engine distributed across multiple computing nodes is no different than a non-distributed game engine executed on a single processing entity, because the game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output components to the end user.

[0059] The user accesses the remote service using a client device, which includes at least a CPU, a display, and an I / O. The client device can be a PC, a mobile phone, a notebook computer, a PDA, etc. In one embodiment, the network executed on the game server recognizes the type of device used by the client and adjusts the communication method used. In other cases, the client device uses a standard communication method such as HTML to access the application on the game server through the Internet.

[0060] It should be understood that a given video game or game application may be developed for a specific platform and a specific associated controller device. However, when such a game is made available via a game cloud system as presented herein, a user may access the video game with different controller devices. For example, a game may have been developed for a game console and its associated controller, while a user may access a cloud-based version of the game from a personal computer using a keyboard and mouse. In such a scenario, an input parameter configuration may define a mapping of inputs generated from a user's available controller device (in this case, a keyboard and mouse) to acceptable inputs for execution of the video game.

[0061] In another example, a user may access the cloud gaming system via a tablet computing device, a touch screen smart phone, or other touch screen driven device. In this case, the client device and the controller device are integrated together in the same device, where input is provided by means of detected touch screen input / gestures. For such a device, the input parameter configuration may define a specific touch screen input corresponding to the game input of the video game. For example, during the running of a video game, buttons, directional pads, or other types of input elements may be displayed or overlaid to indicate the location on the touch screen where the user can touch to generate game input. Gestures (such as swipes or specific touch movements in a particular direction) may also be detected as game inputs. In one embodiment, guidance may be provided to the user, indicating, for example, how to provide input for gameplay via a touch screen before starting to play a video game, so as to adapt the user to the operation of the controls on the touch screen.

[0062] In some embodiments, the client device is used as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to send input from the controller device to the client device. The client device can process these inputs in turn, and then send the input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller itself can be a networked device with the ability to directly communicate input to the cloud game server via a network without first communicating such input through a client device. For example, the controller can be connected to a local networking device (such as the above-mentioned router) to send data to and receive data from the cloud game server. Therefore, although the client device may still be required to receive video output from a cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send input directly to the game cloud server over the network, thereby bypassing the client device.

[0063] In one embodiment, the networked controller and client device may be configured to send certain types of input directly from the controller to the cloud gaming server, and to send other types of input via the client device. For example, inputs whose detection does not depend on any additional hardware or processing other than the controller itself can be sent directly from the controller to the cloud gaming server via the network, thereby bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs that utilize additional hardware or require client devices to process can be sent by client devices to the cloud gaming server. These may include video or audio captured from the gaming environment, which can be processed by the client device before being sent to the cloud gaming server. In addition, inputs from the motion detection hardware of the controller can be processed by the client device in conjunction with the captured video to detect the position and motion of the controller, which is then communicated to the cloud gaming server by the client device. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0064] In one embodiment, various technical examples can be implemented using a virtual environment through a head-mounted display (HMD). The HMD may also be referred to as a virtual reality (VR) head-mounted device. As used herein, the term "virtual reality" (VR) generally refers to the interaction of a user with a virtual space / environment, which involves viewing the virtual space in a real-time response to the movement of the HMD (controlled by the user) through the HMD (or VR head-mounted device) to provide the user with a feeling of being in a virtual space or metaverse. For example, a user can see a three-dimensional (3D) view of the virtual space when facing a given direction, and when the user turns to one side and thus turns the HMD in the same manner, the view of the side in the virtual space is rendered on the HMD. The HMD can be worn in a manner similar to glasses, goggles, or a helmet, and is configured to display video games or other metaverse content to the user. The HMD can provide a very immersive experience to the user by means of its display mechanism being set close to the user's eyes. Therefore, the HMD can provide a display area that occupies most of the user's or even the entire field of view to each eye of the user, and can also provide viewing with three-dimensional depth and perspective.

[0065] In one embodiment, the HMD may include a gaze tracking camera configured to capture images of the user's eyes as the user interacts with the VR scene. The gaze information captured by the gaze tracking camera may include information related to the user's gaze direction and specific virtual objects and content items in the VR scene that the user is paying attention to or interested in interacting with. Thus, based on the user's gaze direction, the system can detect specific virtual objects and content items that may be of potential interest to the user (where the user is interested in interacting with and engaging with them), such as game characters, game objects, game props, etc.

[0066] In some embodiments, the HMD may include an outward-facing camera configured to capture images of the user's real-world space, such as the user's body movements and any real-world objects that may be located in the real-world space. In some embodiments, the images captured by the outward-facing camera may be analyzed to determine the position / orientation of the real-world object relative to the HMD. Using the HMD, the known position / orientation of the real-world object, and the inertial sensor data from the inertial motion unit (IMU) sensor, the user's posture and movement may be continuously monitored and tracked during the user's interaction with the VR scene. For example, when interacting with a scene in a game, a user may make various gestures, such as pointing and walking toward a specific content item in the scene. In one embodiment, the system may track and process gestures to generate predictions of interactions with specific content items in the game scene. In some embodiments, machine learning may be used to facilitate or assist the predictions.

[0067] During the use of the HMD, various single-hand and two-hand controllers can be used. In some implementations, the controller itself can be tracked by tracking the lights included in the controller or tracking the shapes, sensors and inertial data associated with the controller. Using these different types of controllers or even just using gestures made and captured by one or more cameras, the virtual reality environment or metaverse rendered on the HMD can be docked, controlled, manipulated, interacted with and participated in the virtual reality environment or metaverse. In some cases, the HMD can be wirelessly connected to the cloud computing and gaming system through a network. In one embodiment, the cloud computing and gaming system maintain and execute the video game played by the user. In some embodiments, the cloud computing and gaming system are configured to receive input from the HMD and interface objects through a network. The cloud computing and gaming system are configured to process input to affect the game state of the video game being executed. Output (such as video data, audio data and tactile feedback data) from the video game being executed is sent to the HMD and the interface object. In other implementations, the HMD can communicate wirelessly with the cloud computing and gaming system through alternative mechanisms or channels such as cellular networks.

[0068] In addition, although the implementations in the present disclosure may be described with reference to head-mounted displays, it should be understood that in other implementations, non-head-mounted displays may be substituted, including but not limited to portable device screens (e.g., tablets, smart phones, laptops, etc.) or any other type of display that can be configured to render video and / or provide interactive scenes or virtual environments according to the present implementations. It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that can define more implementations by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0069] As noted, embodiments of the present disclosure for communicating between computing devices may be practiced using a variety of computer device configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, head-mounted displays, wearable computing devices, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked over wired or wireless networks.

[0070] In some embodiments, wireless technology can be used to facilitate communication. Such technology can include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. The 5G network is a digital cellular network in which the service area covered by the provider is divided into small geographical areas called cells. The analog signals representing sound and images are digitized in the phone, converted by an analog-to-digital converter and sent as a bit stream. Through the channels allocated by the transceiver from the frequency pool reused in other cells, all 5G wireless devices in the cell communicate with the local antenna array and low-power automatic transceiver (transmitter and receiver) in the cell through radio waves. The local antenna is connected to the telephone network and the Internet through a high-bandwidth optical fiber or wireless backhaul connection. As in other cell networks, a mobile device that crosses from one cell to another automatically goes to the new cell. It should be understood that the 5G network is only an exemplary type of communication network, and the embodiments of the present disclosure can utilize an earlier generation of wireless or wired communications, as well as a newer generation of wired or wireless technology after 5G.

[0071] In view of the above embodiments, it should be understood that the present disclosure may employ various computer-implemented operations involving data stored in a computer system. These operations are those operations that require physical manipulation of physical quantities. Any operation described herein that forms a part of the present disclosure is a useful machine operation. The present disclosure also relates to a device or apparatus for performing these operations. The apparatus may be specially constructed for the desired purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines may be used together with a computer program written according to the teachings of this article, or it may be more convenient to construct a more specialized apparatus to perform the desired operation.

[0072] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be coordinated so that they occur at slightly different times, or the operations may be distributed in a system that allows the processing operations to occur at various time intervals related to the processing, so long as the processing of telemetry and game state data used to generate the modified game state is performed in the desired manner.

[0073] One or more embodiments may also be manufactured as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage devices (NAS), read-only memories, random access memories, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium may include a computer readable tangible medium distributed on a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

[0074] Although the foregoing embodiments have been described in some detail for the purpose of clarity of understanding, it is apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the present embodiments should be considered as illustrative rather than restrictive, and the embodiments are not limited to the details given herein, but may be modified within the scope and equivalent range of the appended claims.

[0075] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that may define more implementations by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

Claims

1. A method for validating input provided at a controller, comprising: detecting a finger gesture provided by a user on a surface of the controller, the finger gesture being used to define an input for an interactive application selected by the user for interaction; receiving multimodal data collected by tracking the finger gestures on the controller, the multimodal data comprising modality data corresponding to different modes captured for the finger gestures using a plurality of sensors and components associated with the controller; generating a combined model using the multimodal data received from the plurality of sensors and components, the combined model being continuously trained using additional multimodal data collected over time to generate different outputs, the combined model being trained using a machine learning algorithm according to training rules defined for different finger gestures; as well as an output from the combined model is recognized for the finger gesture detected on the surface of the controller and the output is interpreted as defining the input of the interactive application based on the finger gesture detected at the controller, The operations of the method are performed by a processor of the server.

2. The method of claim 1, wherein identifying the output comprises: assigning weights to the modality data included in the multimodality data captured by each of the plurality of sensors and components, wherein the weight assigned to the each modality data captured by a sensor or component of the plurality of sensors and components indicates an accurate prediction of the finger gesture using the each modality data; as well as A cumulative weight of the multimodal data is calculated using the weight assigned to the modality data of each mode included in the multimodal data, the cumulative weight being used to identify the output for the finger gesture.

3. The method of claim 2, wherein the weight assigned to the modality data captured by each of the plurality of sensors and components is greater than the weight assigned to the modality data captured by each of the plurality of sensors and components, and wherein the weights assigned to the modal data captured by the plurality of sensors are equal, and the weights assigned to the modal data captured by the plurality of components are equal. 4 . The method of claim 2 , wherein the modality data captured for different modes included in the multimodal data are assigned the same weight. 5 . The method of claim 2 , wherein different weights are assigned to the modality data captured for each mode included in the multimodal data.

6. A method as claimed in claim 1, wherein the combined model is trained according to training rules defined for the different finger gestures, wherein the training rules are defined based on finger anatomy, finger positions relative to input controls on the controller, and the user's controller holding style.

7. The method of claim 1, wherein the multimodal data includes video data, audio data, image data, sensor data, and wireless signals collected from the plurality of sensors and components, and Wherein recognizing the output comprises recognizing a real button press or a virtual button press on the controller based on the finger gesture, or an input provided on a touch screen interface, and the output is interpreted as the input defining the interactive application.

8. The method of claim 1, wherein the plurality of sensors include any one or a combination of an inertial measurement unit (IMU) sensor, a pressure sensor, a proximity sensor, a distance sensor, or a capacitive sensor, and The plurality of components may include an image capture device, a wireless communication device, or a microphone array.

9. The method of claim 1 , wherein the multimodal data comprises a WiFi signal, the WiFi signal comprising a forward signal and a reflected signal captured by the one or more wireless communication devices, the forward signal and the reflected signal being interpreted as snapshots defining a body part of the user, the snapshots of the body part being used to reconstruct the movement of one or more fingers of the user when the user provides the finger gesture.

10. The method of claim 1, wherein the plurality of sensors and components include an image capture device, The multimodal data includes images of different positions held by the user's fingers captured by the image capture device when the user provides the finger gesture, the angle of the image capture device is dynamically adjusted to capture images of different positions of the fingers, and the dynamic adjustment is performed by automatically calibrating the angle of the image capture device in response to detecting the presence of the fingers and the finger gesture provided by the user on the controller.

11. The method of claim 10, wherein the image capture device is a camera integrated into a mobile computing device, or a web camera, or an image capture device of a gaming console, or an image capture device of a computing device or a camera of a head mounted display, wherein the image capture device is communicatively coupled to the controller.

12. The method of claim 11, wherein when the image capture device is the camera of the mobile computing device, the mobile computing device is disposed on a holding structure coupled to the controller, the holding structure comprising a motor configured to receive and hold the mobile computing device and dynamically adjust an angle of the camera to align with a calibrated angle, thereby enabling the capture of images of different positions held by the user's fingers when the user performs a finger gesture, and The retaining structure is a three-dimensional printed structure.

13. The method of claim 1, wherein the plurality of sensors and components include one or more inertial measurement unit sensors (IMUs) configured to detect finger gestures of the user on the surface of the controller and generate IMU signals, and Wherein the multimodal data includes the IMU signals received from the one or more IMUs, the IMU signals being interpreted to identify attributes of the finger gesture, the attributes being used to identify user input at the controller.

14. The method of claim 1, wherein the plurality of sensors and components comprises a microphone array embedded within or coupled to the controller, wherein the finger gesture is a button press on the controller, and The multimodal data includes properties of audio data captured by the microphone array, the properties of the audio data captured by multiple microphones in the microphone array are interpreted using triangulation techniques to identify the direction and position of the sound relative to each microphone in the microphone array, and the direction and the position are used to determine the pressed button.

15. A method for defining input for an interactive application, comprising: receiving a finger gesture provided by a user on a surface of a controller, wherein the finger gesture is used to define the input of the interactive application selected by the user for interaction; receiving multimodal data capturing attributes of the finger gesture on the controller, the multimodal data comprising modality data corresponding to different modes captured by a plurality of sensors and components associated with the controller; assigning a weight to modality data corresponding to each mode included in the multimodal data, the weight assigned to each mode indicating an accurate prediction of the finger gesture using the modality data of each mode; as well as processing the finger gesture and the multimodal data based on the weights assigned to each mode to identify the input for the interactive application, The operations of the method are performed by a processor of the server.

16. The method of claim 15, wherein processing the finger gesture and the multimodal data comprises: generating a combined model using the multimodal data received from the plurality of sensors and components; continuously training the combined model using additional multimodal data collected over time to generate different outputs, the combined model being trained using a machine learning algorithm according to training rules defined for different finger gestures; as well as Output from the combined model is identified, the output corresponding to the finger gesture detected on the surface of the controller, the output being interpreted as the input defining the interactive application.

17. The method of claim 16, wherein the training rules are defined based on finger anatomy, finger positions relative to input controls on the controller, and the user's controller holding style.

18. A method as claimed in claim 15, wherein the processing of the finger gesture includes calculating a cumulative weight of the multimodal data using the weight of the modal data assigned to each mode included in the multimodal data, and the cumulative weight is used to identify the input of the interactive application.

19. The method of claim 15, wherein the plurality of sensors include any one or a combination of an inertial measurement unit (IMU) sensor, a distance sensor, a pressure sensor, a proximity sensor, or a capacitive sensor, The plurality of components include any one or a combination of an image capture device, a wired communication device, a wireless communication device, or a microphone array.

20. The method of claim 15, wherein the multimodal data comprises a first set of the modal data captured by the plurality of sensors and a second set of the modal data captured by the plurality of components, and Wherein assigning the weights includes assigning a first weight to the modal data of each mode included in the first set of modal data, and assigning a second weight to the modal data of each mode included in the second set of modal data, wherein the first weight is greater than the second weight.

Images