Personalized, context aware input output systems and methods

Abstract

A computer-implemented method and device are disclosed which provide a human-computer interface system which synthesizes control events in the form of natural motion, gestures, and hardware input to a computing device. The system involves receiving user input data from multiple input modalities via various devices, determining the context using user data and preferences, combining sensor data, and calculating control events based on user actions and context using neural networks. The system outputs control inputs to the computing device. The system may utilize network connected processing devices to provide a hybrid computing environment to optimize interface processing.

Description

PERSONALIZED, CONTEXT AWARE INPUT OUTPUT SYSTEMS AND METHODSInventors:Liang Peng Hong Heather Yu Pingfan Wu Xiyun Song Zongfang Lin Yubin Zhou Masood Mortazavi Jiechen Wang Rui Ning Yue Cao Yue Zhang Ning LuCLAIM OF PRIORITY

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 582,186, entitled “Finger Action Enabled Efficient and Real-time Text Entry for Augmented and Virtual Reality”, filed September 12, 2023, which is incorporated by reference herein in its entirety and International Application Number PCT / US2024 / 011494 entitled “High-Capacitance Ionic Liquid Gel / Polymer Matrix Biocompatible Materials for Flexible Pressure Sensor”, filed January 12, 2024, which is incorporated by reference herein in its entirety, and International Application Number PCT / US2024 / 030833 entitled “Input Methods for Smart Eyewear”, filed May 23, 2024, which is incorporated by reference herein in its entirety.FIELD

[0002] The disclosure generally relates to input techniques for computing devices.BACKGROUND

[0003] Human interaction (HCI) with computers (human-computer interaction, HCI) typically occurs through any number and type of different input / output devices. Traditional input devices include keyboards, mice, controllers and touchscreens. These devices are found in conventional computing environments, and while effective, alternative input systems, devices and methodologies can be used especially with theadvent of augmented reality (AR) and virtual reality (VR) devices. Often, users may have a choice of multiple input devices to use simultaneously.

[0004] AR and VR devices include tracking technologies, such as outward facing two-dimensional and three-dimensional cameras, as well as other sensors, to accurately detect and interpret the physical environment of a wearer of such devices. These tracking technologies can be used to detect hand movements and gestures. Traditional computing devices - mobile phones, tablets, computers, etc. - may also include cameras and microphones which may be configured to provide HCI. Typically, a user must determine in advance which method of HCI the user will use during computer interaction, even though multiple input / output systems may be available.SUMMARY

[0005] One general aspect includes a computer implemented method of detecting and providing an indication of natural motion as an input to a computing device. The computer implemented method includes using at least one sensor to detect a human motion to generate input data regarding the detected motion of a user. The input data may include multiple input modalities from multiple input data devices detected from which actions of the user can be determined. The method also includes determining a context of the input data using user context data and user preferences. The context may include information which may be used to interpret the input data from the user movement. The method also includes combining the data from the multiple input data sensors. The method also includes calculating whether one or more control events occurs in the input data based the actions of the user and the context of the input data using one or more neural networks, the one or more neural networks including a trained model associated with each input modality; and outputting one or more control inputs calculated as present in the input data to the computing device. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0006] Implementations may include the computer implemented method where the determining, combining, calculating and outputting are performed partially on the computing device and partially on a second computing device coupled via a network. Implementations may include any of the foregoing methods wherein the determining, combining, calculating and outputting processing tasks are distributed to the computing device or the second computing device based on complexity. Implementations may include any of the foregoing methods wherein the input data devices may include one of a camera, a microphone, a keyboard, a pointing device. Implementations may include any of the foregoing methods wherein the multiple input modalities include one or more of: interaction with virtual keyboards, interaction with ultra-thin and flexible electronic keyboards, biometric inputs, audio inputs, environmental inputs, location, user body motion, in-air gestures, an on-surface gestures, user facial expressions, user voice, user eye motion, and user emotion. Implementations may include any of the foregoing methods wherein the context may include one of application usage, location, temperature. Implementations may include any of the foregoing methods wherein the method further includes: registering each input data device, determining which input devices are providing input data which may include an application input based on the context, resolving conflicts between multiple input devices which may include application input, and generating the control inputs. Implementations may include any of the foregoing methods wherein the method further includes receiving one or more trained models from a service host, updating the one or more trained models received from the host based on user interaction, and forwarding the updated trained models to the service host. Implementations may include any of the foregoing methods wherein the control input may include one of: text, image, hand drawing, an application command, a display command. Implementations may include any of the foregoing methods wherein the context is determined based on context data and user preferences, and the context data and user preferences are stored on the computing device. Implementations may include any of the foregoing methods wherein the context is determined based on context data and user preferences, and the context data and user preferences are stored on the computing device. Implementations may include any of the foregoing methods wherein the context is determined based on context data and user preferences, and the contextdata and user preferences are at least partially stored on one or more network connected processing devices. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer- accessible medium.

[0007] Another general aspect includes a user equipment device. The user equipment device includes a storage medium that may include computer instructions. The device also includes a display device. The device also includes one or more processors coupled to communicate with the storage medium, where the one or more processors retrieve and execute the instructions to cause the device to: receive input data from a user, the input data may include multiple input modalities from multiple input data devices detected from which actions of the user can be determined; determine a context of the input data using user context data and user preferences, the context including information which may be used to interpret the input data from the user; combine the data from the multiple input data sensors; calculate whether one or more control events occurs in the input data based the movement and positioning of the user and the context of the input data using one or more neural networks, the one or more neural networks including a trained model associated with each input modality; and output one or more control inputs calculated as present in the input data to the computing device.

[0008] Implementations may include any of the devices wherein the user equipment device transfers one or more of the instructions to determine, combine, calculate and output to one or more network connected computing devices. Implementations may include any of the devices wherein the transfer to the one or more network computing devices is based on complexity and neural network model size. Implementations may include any of the devices wherein the input data devices may include one of a camera, a microphone, a keyboard, a pointing device. Implementations may include any of the devices wherein the multiple input modalities include one or more of: interaction with virtual keyboards, interaction with ultra-thin and flexible electronic keyboards, biometric inputs, audio inputs, environmental inputs, location, user body motion, in-air gestures, an on-surface gestures, user facial expressions, user voice, user eye motion, and user emotion. Implementations mayinclude any of the devices wherein the context may include one of application usage, location, temperature, Implementations may include any of the devices wherein the one or more processors execute the instructions to cause the device to: register each input data device, determine which input devices are providing input data which may include an application input based on the context, resolve conflicts between multiple input devices which may include application input, and generate the control inputs. Implementations may include any of the devices wherein the one or more processors execute the instructions to cause the device to receive one or more trained models from a service host, update the one or more trained models received from the host based on user interaction, and forward the updated trained models to the service host. Implementations may include any of the devices wherein the control input may include one of: text, image, hand drawing, an application command, a display command. Implementations may include any of the devices wherein the context is determined based on context data and user preferences, and the context data and user preferences are stored on the computing device. Implementations may include any of the devices wherein the context is determined based on context data and user preferences, and the context data and user preferences are at least partially stored on one or more network connected processing devices. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0009] A further general aspect includes a non-transitory computer-readable medium storing computer instructions that configure at least one processor. The non- transitory computer-readable medium storing computer instructions cause the processor to perform receiving input data from a user, the input data may include multiple input modalities from multiple input data devices detected from which input actions of the user can be determined. The instructions also include causing the processor to perform determining a context of the input data using user context data and user preferences, the context including information which may be used to interpret the input data from the user. The instructions also include causing the processor to perform combining the data from the multiple input data sensors. The instructions also include causing the processor to perform calculating whether one or more controlevents occurs in the input data based the movement and positioning of the user and the context of the input data using one or more neural networks, the one or more neural networks including a trained model associated with each input modality; and outputting one or more control inputs calculated as present in the input data to the computing device.

[0010] Implementations may include the non-transitory computer-readable medium where the instructions configure at least one processor, upon execution of the instructions, to distribute one or more of the determining, combining, calculating and outputting to a computing device coupled via a network. Implementations may include the non-transitory computer-readable medium where the instructions configure at least one processor, upon execution of the instructions, to distribute the determining, combining, calculating and outputting processing tasks are distributed based on complexity and neural network model size. Implementations may include the non- transitory computer-readable medium where the input data devices may include one of a camera, a microphone, a keyboard, a pointing device. Implementations may include the non-transitory computer-readable medium where the multiple input modalities include one or more of: interaction with virtual keyboards, interaction with ultra-thin and flexible electronic keyboards, biometric inputs, audio inputs, environmental inputs, location, user body motion, in-air gestures, an on-surface gestures, user facial expressions, user voice, user eye motion, and user emotion. Implementations may include the non-transitory computer-readable medium where the context may include one of application usage, location, temperature, Implementations may include the non-transitory computer-readable medium where the instructions configure at least one processor, upon execution of the instructions, to perform: registering each input data device; determining which input devices are providing input data which may include an application input based on the context; resolving conflicts between multiple input devices which may include application input; and generating the control inputs.

[0011] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter,nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all disadvantages noted in the Background.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying FIG.s for which like references indicate the same or similar elements.

[0013] FIGs. 1 A - 1 C illustrate various types of human computer interactions.

[0014] FIG. 2 is a flowchart illustrating a general method in accordance with the described technology.

[0015] FIG. 3 is a block diagram illustrating a user interacting with a processing device using the described technology.

[0016] FIG. 4 illustrates a use case example which aids in understanding the practical application of the described technology.

[0017] FIG. 5 is a block diagram showing various HCI system components and a basic data flow for the described technology.

[0018] FIG. 6 is a block diagram illustrating a system protocol used by the described technology.

[0019] FIG. 7 is a flowchart illustrating the sensor intelligent fusion process and setup.

[0020] FIG. 8 illustrates a system in accordance with the described technology.

[0021] FIG. 9 is a block diagram illustrating three intelligent fusion processors operating on different types of sensor input.

[0022] FIG. 10 is a flowchart illustrating an HCI system initialization process.

[0023] FIG. 11 is a flowchart illustrating a first embodiment of a method for context aware, seamless input mode transition, where the input mode changes.

[0024] FIG. 12 is a flowchart illustrating another implementation of a method for the HCI system wherein tiered, multimodal inputs are supported.

[0025] FIG. 13 is a flowchart illustrating another implementation of a method for the HCI system wherein a user may allow certain types of sensors to bypass the HCI system and end interaction.

[0026] FIG. 14 illustrates one embodiment of distributed processing enabling the HCI disclosed herein.

[0027] FIG. 15 is a flowchart illustrating task distribution amongst a local processing environment, network nodes and servers.

[0028] FIG. 16 is a block diagram illustrating global model training and privacy control.

[0029] FIG. 17 is a flowchart illustrating training of local and global NN models.

[0030] FIGs. 18 and 19 illustrate processes involved in hybrid computing when a user moves to different geographical regions.

[0031] FIGs. 20A and 20B illustrate use cases for the described technology.

[0032] FIG. 21 is a schematic diagram illustrating exemplary details of a network device, or node.

[0033] FIG. 22 illustrates a network processing device which may comprise any user processing device for implementing the technology discussed herein.

[0034] FIG. 23 illustrates a server network processing device which may comprise a service host for implementing the described technology.WRITTEN DESCRIPTION

[0035] The described embodiments of the disclosure enable an HCI system with context aware and personalized input modalities. This allows a user to seamlessly change between and / or use multiple input devices while interacting with a processing device. The HCI system may be referred to as a Personal Virtual Input Assistant which interprets various input modalities, including physical devices, virtual keyboards, ultrathin and flexible electronic keyboards, virtual inputs, biometric inputs and audio inputs, all of which can be used alone or in combination to interface with processing devices. The HCI system utilizes sensor fusion and intelligent hybrid computing to enable a versatile, ergonomic, adaptable, and scalable solution for and enhanced user experience. A user may thus interact with various traditional computing devices and / or virtual and augmented reality devices, in a more natural way, using native modalities such as hands, voice, eyes, expressions, and the like, as well as traditional input devices. The HCI system synthesizes control events in the form of natural motion, gestures, and hardware input for a computing device. The system receives user input data from multiple input modalities via various devices and sensors, and calculates control events (application controls, application input or display inputs) based on user actions and context, using neural networks. The system may utilize network connected processing devices to provide the hybrid computing environment to optimize interface processing.

[0036] The present disclosure and embodiments present systems and methods for human input motion recognition technology suitable for use with an augmented reality (AR) and / or virtual reality (VR) headset (an “AR / VR device”) as well as processing devices such as personal computers and mobile devices including phones and tablets. In embodiments, the technology interprets multiple possible types of movements as inputs, reconciles the movements based on context and user preferences, and provides an output based on the inputs. Each output may comprise one or more control inputs for an application or a graphics device to display, including application commands, text, and display elements.

[0037] FIGs. 1A - 1 C illustrate various types of human computer interactions. In FIG. 1A, a user 100a is shown interacting with a laptop computer 120. The user may be participating in, for example, an online meeting where the user is speaking 105 with another user or a group of users. User 100 is gesturing with his hand. As is well known, computer 120 may include at least an integrated hardware keyboard and pointing device, as well as integrated sensors such as a camera and microphone. Traditionally, an online meeting on a computer involves the user 100 working with the keyboard, pointing device, camera, and microphone to participate in the meeting by communicating with others and presenting materials. However, in embodiments, the described HCI system allows the user to incorporate free-air hand movements as user gestures and evaluates the context of user actions, detectable by sensors of computer 120, to be used for HCI.

[0038] FIG. 1 B illustrates a second type of human computer interaction where a user 100b in an environment wears an ARA / R device 150. In embodiments, device 150, may include a display integrated with the lenses through which the user views the environment. Where ARA / R device 150 is an augmented reality device, the device can overlay digital information, images, graphics, or holograms onto the user's view of the real world through the lenses. Where AR / VR device 150 is a virtual reality device, the user’s view is entirely focused on a display which blocks the user’s view of the user’s environment. The AR / VR device 150 may include built-in speakers, microphones, sensors (such as gyroscopes, accelerometers, and magnetometers) and imaging sensors such as cameras including red / green / blue (RGB) cameras, depth-sensors, Lidar sensors, ultrasonic sensors, and thermal sensors. In the illustration of FIG. 1 B, the user 100 may be looking at the user’s hands in midair and is gesturing in the air in order to generate input. The gestures may include typing or making application command gestures, detected by the cameras, microphones and other sensors of the AR / VR device, which are interpreted by the HCI system based on the context of the user’s use.

[0039] FIG. 1 C, a user 100c is shown wearing an AR / VR device 130 in, for example, an online meeting where the user is speaking 105 with another user or users. Sensors from both the laptop 120 and the HMD 130 can be used to interpret usermovements and gestures. User gestures may include typing or making application command gestures, detected by the cameras, microphones and other sensors of the ARA / R device, which are interpreted by the HCI system based on the context of the user’s use. The user 100c may type on the blank surface of the table 145 and such typing can be interpreted by the HCI system using for example the methods described in PCT / US2024 / 30833 entitled INPUT METHODS FOR SMART EYEWEAR.

[0040] FIG. 2 illustrates a general method performed by the HCI system in accordance with the described technology. FIG. 3 is a block diagram illustrating a user 100 interacting with a processing device 320 using the described technology. Referring initially to FIG. 3, a user 100 may interact with a processing device 320 using any number of input modalities 300 in addition to traditional input hardware such as keyboards and pointing devices.

[0041] Referring to FIG. 2, at 205, for a user interacting with one or more user processing devices, the HCI system is initialized. Initialization is described below but generally involves detecting the user’s input modalities and available sensors which can be used with the HCI system. At 210, a user profile is accessed to determine user preferences, customized neural network models specifically trained with user preferences and traits, and context data used to determine the context of the user’s interaction with the user’s device. These preferences are utilized to determine the context in which a user may be utilizing different user devices and to make choices specific to the user. User preferences may include a relationship between user applications and input modalities, or preferences for one modality over another when there are two modalities available. For example, when a user is creating text for an application, the user may define a preference to use a table-top typing input over an audio, dictation input. When the two modalities conflict (such as, for example, when the user speaks while typing on a table top,) the preference may define which input to prioritize over the other. At 215, one or more sensor inputs of user actions are received. Input sensors 305 (FIG. 3) collect data from the input modalities and provide it to an input processor 335. Environmental sensors 310 may also retrieve information about the physical environment of the user, such as air temperature, location (whether the user is at home or at work), motion (such as whether the user is in motion in a car),or other environment data. At 220, the input data is interpreted based on the context of the use and the application running on the processing device, and a determination is made as to input actions (if any) that should be generated from among one or more of the actions / inputs. There may be several different input modalities in use, and at 225, the method determines which control inputs the user intends to generate: a command, control input, or display output for the device. Each of these comprise a control event for a display or a computing application. At 230, one or more application commands, audio outputs and / or video outputs are provided to the user device for appropriate action.

[0042] As shown in FIG. 3, a processing device 320 may include an intelligent input module 315 which includes an intelligent input processor 340 performing step 225, for example, a context awareness data processor 330 (which in embodiments includes user profile data) and a fusion processor 345. Each of the illustrated processors 330, 335, 340, 345 may comprise one or more processors (including a single processor) that execute instructions to perform the operations of processors 330-345 which are described herein. Input sensors 305 receive input data based on one or more input modalities 300.

[0043] In the context of human-computer interaction, a modality is the classification of a single independent channel of input / output between a computer and a human. Such channels may differ based on sensory nature (e.g., visual vs. auditory), or other significant differences in processing (e.g., text vs. image). Each input modality is a distinct channel through which a user can provide information or commands to a computer system. It represents a specific method of human-computer interaction, utilizing a particular sensory or cognitive process. Each input modality relies on specific types of sensors, devices, and recognition algorithms to capture and interpret user inputs. Similarly, each output modality is a distinct channel through which a computer system conveys information or feedback to a user and represents an independent channel of presenting information, utilizing a particular sensory or cognitive process. The disclosed HCI system is designed to seamlessly integrate multiple modalities. The one or more sensor inputs may comprise any physical interaction with an input / output device (such as a mouse, keyboard, stylus, joystick,an ultra-thin, flexible keyboard, etc.), an in-air gesture with the hands of the user, an on-surface gesture (such as typing on a plain surface), biometric data such as user temperature, or any of the input modalities 300 shown in FIG. 3. Each of these devices provides an output that may be considered a control event for an application input or command, or a display input. By way of example and without limitation, the input sensors 305 may comprise sensors integrated with augmented reality and virtual reality devices (such as microphones, RGB cameras, and depth sensors), physical keyboards, portable flexible keyboards, video displays, physical pointing devices such a computer mouse, virtual keyboards and keypads (generated and sensed by AR / VR devices or other processing devices), touch keyboards and pointing devices, virtual pointing devices, proximity sensors, inertial measurement units (IMlls), electroencephalogram (EEG) sensors, electrocardiogram (ECG and EKG) sensors, and environmental (temperature and proximity) sensors. Environmental sensors may comprise temperature sensors, proximity sensors, and motion sensors. Sensors can include so-called "e-tattoos" -- ultrathin and ultrasoft epidermal electronics having a multilayer stack of stretchable layers of distinct functionalities, a near field communication (NFC) layer capable of wireless power harvesting and data transmission, a functional circuit layer, and a passive electrode / sensor layer, all of which can be attached to a user's skin to wirelessly measure a variety of biometrics.

[0044] Some input modalities may comprise core modalities, such as expressions on the user’s head 301 , hand and fingers 302, and voice 303. Other input modalities may be considered peripheral modalities, such as eye motion, facial expression, emotion interpretations and body / biometric data. In embodiments, all such modalities are treated the same. In other embodiments, core and peripheral modalities may be processed differently.

[0045] Context awareness processor 330 retrieves context data and user profile information to provide the use context to the intelligent input processor 340. The fusion processor 345 interprets, selects and / or combines multiple inputs when there are multiple inputs from the user. As noted above, at 230, an output appropriate for the application is generated. The output is provided to any of the subsystems 325 of the processing device, including the audio subsystem 350, video subsystem 355 and / orapplication processing 360. This may result in an audio / visual output or any of a number of outputs to output modalities 365 in FIG. 3.

[0046] FIG. 3 illustrates one embodiment of an HCI system on a single processing device 320. In embodiments, hybrid processing involving other network connected processing devices is used to implement the HCI system.

[0047] FIG. 4 illustrates a first use case example to aid in understanding the practical application of the described technology. At 405, a user may be wearing AR glasses and / or sitting in front of a computer, both of which have audio / visual inputs; the computer may also have physical input devices accessible to the user (such as the computers keyboard and pointing device). At 407, the user may initiate the HCI system with an action. The system is configured to detect and interpret the user action. At 410, the user may initiate an online meeting using an online meeting application on the computer or the AR / VR device. Generally, in an online meeting, the input devices, and audio / visual sensors of the computer or A / R device will be used to transmit data to other users. During the meeting the user may stand to get a cup of coffee and while walking, respond to a question by moving fingers in the air (by making a typing motion or a gesture). At 430, the HCI system will detect and interpret the various inputs, select the correct input modality and generate an appropriate command or audio / visual output for the online meeting application, in this example, to send a response that the user made to others in the online meeting. Alternatively, the user may simply stand up in order to draw a graph or figure in the air at 420. At 430, the intelligent input system will interpret the various inputs for drawing the figure, select the correct input modality and generate an appropriate FIG. or graph for the online meeting application to send a response that the user made to others in the online meeting. Although the foregoing illustrates on use case, it will be understood that the described technology is not limited to the number of inputs or types of inputs that may be synthesized and provided to the processing device.

[0048] FIG. 5 illustrates various HCI system components and a basic data flow for the described technology. In embodiments, all the components shown in FIG. 5 may be provided on a processing device. In other embodiments, the components of FIG. 5may be provided on different processing devices and / or split between different connected processing devices to enable hybrid processing. Input sensors 515 integrate various sensory input data types and provide the sensing data to context processing 525 and intelligent processing 540. In embodiments, some processing of the sensor data may occur at the sensor level and while other types of data require input processing on a processing device. Many modem sensors incorporate preprocessing capabilities to enhance data quality and usability. Biometric sensors, such as fingerprint, facial recognition, and iris scanners, may perform image enhancement, segmentation, alignment. Similarly, audio sensors may utilize noise reduction, echo cancellation, voice activity detection, and feature extraction techniques to filter out background noise and identify relevant sound characteristics. The integrated sensing 515 extracts the features provided by the sensors and fuses features from various sensors together. The context processing module 525 accesses context data 535 and user preference / profile data 520. The context data comprises information which allows the NN model 530 to understand the perceived sensory input in relation to the activities of the user. Context data may include location (whether the person is home or at work), environment (whether the person is in a room or cubicle), available input methods (whether a keyboard is available to the user even though the user is waving their hands) and user emotional state. Context information also takes advantage of and user preferences 520 on which inputs are used when performing tasks using the computing device. For example, a user may specify that they don’t like using their voice to command applications but are ok with voice for online meetings. The user preferences 520, context data 535 and sensory input is combined to provide use context intelligent detection and classification processing 540, which uses an adaptive artificial intelligence model 530 to determine a control output 545. The control output may comprise an application command, text, drawing or any needed input for an application running on the computing device.

[0049] FIG. 6 is a block diagram illustrating a system protocol used by the described technology. Devices and sensors that support HCI system processing in accordance with this disclosure are initially discovered and registered with the intelligent input application 315 at 605. Once devices and sensors are discovered, asecure connection 610 between them is negotiated and established. This includes negotiating the type of connection to be used, such as peer-to-peer, and establishing a secure connection protocol using, for example, standard handshaking techniques. Data transmission could be via WiFi, Bluetooth, NearLink, and other types of wired or wireless connections, or a combination thereof. Data may be encrypted per system, application, or user definition or security requirements. At 620, a direct peer-to-peer connection is established between discovered sensors and devices, and the computing device running the intelligent input application 315. At 625, a data transfer confirmation may be provided to the user indicating that the sensor or device is registered and active. Following discovery, at 615, the system determines the input context (such as the environment, whether the input is for an application (application usage), and any user preferences for the input device). At 630, input from the devices or sensors is received and communicated at 635 for sensor data fusion and processing at 640.

[0050] FIG. 7 is a flowchart illustrating the sensor intelligent fusion process and setup. At service initiation at 705, devices may be registered and communication established as discussed above. At 710, for each input modality (each determined sensor or device at 605 above), a neural network model is trained to interpret inputs from the sensor or device relative to the user at 715. The models are trained to understand and process various types of data from devices and sensors. Different types of models can be employed depending on the specific input and the desired interpretation. For example, Natural Language Processing (NLP) Models may include transformer models and recurrent neural network (RNN) models. Convolutional neural networks (CNN) can be used for image and video processing tasks such as object detection, facial recognition, and gesture recognition. Deep Neural Networks (DNNs) and be used for converting spoken language into text. To combine data from the multiple input modalities (e.g., audio, video, text) a combination of CNNs, RNNs, and transformer architectures may be used. Each of the models may be updated and retrained based on user feedback and preferences. Each of the NNs are associated with at least one input modality and can be used to calculate whether user actions which are detected comprise control events for the user’s associated processingdevice. At 720, each of the trained models is stored at one of several locations depending on the size of the model. Smaller models can be stored locally on a user’s processing device. Larger models may be stored on a network connected server or network edge device, as discussed below.

[0051] Once the models are trained and stored, the system is ready to process multi modal inputs. Steps 725 - 765 illustrate one embodiment of step 220 of FIG. 2. At 725, for each type of data received at any given time. At 730, the context of the user’s use and the input will be determined. Context includes things such as the environment the user is operating in (which can include the lighting whether table or surface exists, the location of the user (i.e. whether the user is at home or at the user’s work)), the application that is in use on the processing device of the user, whether other input devices are active, the user’s emotional state (based on biometric sensors), and any current modalities in use. At 735, based on the context, a determination is made as to whether or not this data input is a likely application input. If not, the method ends. If so, at 740, the method waits until the input is completed. At 745, a determination is made as to whether or not there are multiple inputs from different devices or sensors. If not, the method proceeds directly to step 765 where the output is generated. If there are multiple inputs at 745, a determination is made at 750 as to whether or not there are conflicts between the inputs. If there is no conflict, the method proceeds to fuse the multiple inputs based on the use context at 760. If there are conflicts, at 755, one or more inputs are determined to be relevant to the application and use on the processing device, while one or more other inputs may be discarded. For example, if a user is typing and speaking at the same time, but the vocal input is determined to not be relevant to the subject matter of the typed information or the application in use, the vocal input may be discarded and a control event generated for the typed information. At 760, multiple inputs are fused based on the context in the application. For example, if the user stands during an online meeting and makes a gesture while speaking, these two different inputs — the gesture and the spoken word — can be combined to generate control events and outputs for the meeting application at 760. From the user’s perspective, the input actions are taken without a need to indicate to the processing device a switch of input devices or actions. Input modesare interpreted by the HCI system and different modes switched between or merged as appropriate based on the context of use on the processing device, all of which is seamless to the user’s actions and interface. At 765 the control event and / or output is generated for the application.

[0052] Steps 730, 735, 740, 745, 750, 755 and 760 may be performed by one or more of the trained neural network models. As different modes of input are received, the method works to allow the user to seamlessly use different input modalities to interact with a processing device. To fuse different input modalities from a human / computer interface, each trained neural network employs specialized processing for each type of input, such as text, images, and audio. This involves preprocessing steps like tokenizing text, converting audio signals to spectrograms, and normalizing images. Each modality is then processed through a dedicated network to extract relevant features. Different fusion techniques may be employed: early fusion, where raw data from each modality is combined before processing; late fusion, where each modality is processed separately before combining the features; and hybrid fusion, which integrates features at multiple stages. Attention mechanisms, such as self-attention within each modality and cross-attention between modalities, help the network focus on the most relevant aspects of the data.

[0053] Training the network involves optimizing a relevant loss function using backpropagation or other algorithms. The combined features are fed into a final layer for classification or regression, with performance evaluated using metrics like accuracy, precision, and recall. For instance, in a multimodal human / computer interface, text input, audio commands, and visual data are collected and processed through their respective networks. These features are then fused and enhanced using attention mechanisms to improve the model’s ability to understand and respond to complex inputs.

[0054] FIG. 8 illustrates a system in accordance with the described technology. FIG. 8 is similar to FIG. 3 but further details aspects of the system that may be included on a local processing device or on a network connected to the server. In FIG. 8, the processing location for the components of the HCI system are illustrated. In FIG. 8, auser may interact with the system using any of a number of illustrated (and nonlimiting) modalities including voice 802, biometrics 804, touch-based devices 806, gestures 808, vision 810, or inertial sensors (and inertial measurement unit (IMU)) 812. Input sensing 814 occurs to interpret any of the input modalities 802 through 812. As such, at 814 this can result in voice input, gesture typing, gesture writing, peripheral typing, (using for example a hardware keyboard or flexible keyboard), wearable typing, gesture input, or any of a number of different types of inputs. Generally, the input sensing will take place at the user’s local processing device. Some of the sensors that are available to conduct input sensing have some processing capability and can preprocess signals before sending the input data to the human computer interface manager 818. Signal processing at 816 is optional and may not be available from some sensor types. The output of the input sensing is provided to the HCI system 818.

[0055] The HCI system 818 includes a model repository 826, and a context repository 828 which itself includes user preference data 830. A context manager 820 utilizes the context repository and user preferences 830 to determine and process the context of the input sensing signals. In the user preference data 830, for example, users can specify preferences such as preferred types of inputs and user-specific gestures. An intelligent fusion processor 822 utilizes the generated NN models from the model repository 826 the context information from the context repository 828 and user preferences 832 to synthesize the inputs in accordance with the method of FIG. 6. The resource resolver 832 determines which resources (input data sources) are available for use by the system 818, and the action planner 834 determines what actions are available from the available resources. A synthesis and output generator 824 utilizes resource resolution 832 and action planning 834 and output from the fusion processor 822 data to generate the output for both application APIs 838 and output modalities 836 (such as the display screen of a local user device). APIs 838 interact with applications 840, 842 and 844 to provide output commands which may include, for example, initiation of an application, termination of an application, a request to the application a response to the application or an integration of multiple applications.

[0056] Portions of the HCI manager 818 such as model repository and modality and fusion processor may be stored on a network connected server as the processing for certain types of models in the model repository 826 will require more intensive processing power they may be available on a local processing device. Optionally, some of the models may be stored on a local processing device. Optionally, the repositories 826, 828 and 830 may be stored on both a local device and a network connected processing device. Model updates can be provided from the network connected server as feedback or user preferences used to retrain the models.

[0057] It should be understood that there may be multiple instances of the intelligent fusion processor running in parallel on a local device, network connected processing device, or both.

[0058] FIG. 9 is a block diagram illustrating three intelligent fusion processors operating on different types of sensor input. FIG. 9 illustrates various different types of sensors and sensor data processing, and while only three processors are shown, there may be additional processors. Voice sensor data processing 905 may be connected to an audio input, such as a microphone. Vision sensor data processing 908 may be connected to a camera or other optical sensor. Touch sensor data processing 911 may be connected to any type of touch sensor. IMU sensor data processing 914 is connected to an IMU sensor. ECG sensor data processing 920 is connected to an electrocardiogram sensor and EEG sensor data processing 923 is connected to the electroencephalogram sensor. Any number of different types of sensors 917 may be connected to the system. A number of these sensors alone or in combination detect movement and positioning of the user.

[0059] Intelligent fusion processor 822a comprises a first feature level fusion processor. Utilizing the voice and vision sensors, feature level fusion 926 each modality is processed through a NN model to extract relevant features. The NN can estimate the user’s emotions at 941 based on voice, video, ECG and EEG data can utilize these two pieces of data to allow for emotion classification at 941. The NN of each of the feature level fusion 926 and emotion classification can employ different fusion techniques to combine the features for processing by the input decision process929. Attention mechanisms may be used to allow the network to focus on the most relevant aspects of the data. At 929, an input decision based on the fused sensors can be made and output generated at 932. Each of the outputs from modules 932, 950 and 980 discussed herein may be provided to an application, directly to an output modality, or any component of ARA / R device 150, or processing devices 320 and 2200 (described below). Intelligent fusion processor 822b integrates voice data, visual data and touch data are all received by respective voice recognition module 944, visual recognition module 962, and touch classification module 968. Emotion classification module 953 can again determine the emotional state of the user. Emotion classification can be determined from biometric and facial camera data. Biometric sensors allow monitoring physiological signals associated with different emotional states. EEG sensors measure brain wave patterns, where increased beta waves suggest stress or anxiety, and alpha waves correlate with relaxation. ECG sensors track heart rate variability, indicating positive emotions or relaxation with high variability and stress with low variability. Temperature sensors detect user body temperature, which may also indicate stress. By analyzing these signals, the emotion classification module 953 can infer a user's emotional state with significant accuracy.

[0060] Again, each modality is processed through an NN model to extract relevant features. The NN can estimate the user’s emotions based on voice, video, ECG and EEG data and generate an input decision 947 to create an output 950.

[0061] A multi-modal intelligent fusion processor 822c illustrates a general multifeature fusion. This combines feature level fusion 971 with other types of recognition 972, 973 and emotion classification at 974, which are provided to the input decision process 977 to generate an output 980.

[0062] FIG. 10 is a flowchart illustrating an HCI system initialization process. At 1005, the service initiation begins. Service initiation begins by initializing a context engine 1010 and the processing engine 1040. The context engine initiates a context management process 1020 to retrieve information from the context repository 1025 and the user preferences data store 1030. The processing engine 1040 is initialized and retrieves applicable data models from the model repository 1035. Lightweightsensor data 1015 is retrieved by the context engine and used with the context data from the context repository and user preference data to determine the environmental context of the user. Once initialization is complete, input data is checked at 1045 and if input is received, real-time processing occurs at 1055. When user input is completed at 1050, the process ends. Real-time processing can provide output and feedback 1060 to determine if the input has completed at 1050.

[0063] FIG. 11 is a flowchart illustrating a first embodiment of a method for context aware, seamless input mode transition, where the input mode changes. After service is initiated at 1005, context is analyzed at 1105. The context analyzation process will retrieve data from the context repository 1025 and user preferences 1030 to understand, based on the received sensor data 1117 and process lightweight sensor data 1114, the user context for the input. In one example, context analysis can use user head movement data and hand gestures, as well as location information and physiological sensors measuring heart rate and skin conductance. Features such as movement speed, gesture patterns, and physiological changes, may be combined into a contextual model, to determine the user's interaction context. At 1108, input mode selection occurs. The selection can be the use of one or more different input sensor data to provide a control output. The input mode selection can be based on user context and whether different input modes may be combined or conflict. In FIG. 11 , four types of input are shown though it should be understood that numerous types of input modes may be available for selection. Input mode selection 1108 can include one or more of surface typing (typing on a flat surface while wearing an AR or VR device which projects a keyboard on the surface), free typing (typing in free space), a hand gesture in free space, and / or a voice command. Based on the input mode(s) selection, model selection occurs at 1111 . Based on the input mode selection, model selection occurs at 1111. The model selection process 1111 accesses the model repository 1035 as well as the context repository and user preferences to understand which models to use for the particular user.

[0064] If the received sensor data 1117 changes, the context may change as well and a new network model may need to be selected. If, based on a change in the sensor data, the input changes at 1135, then model selection at 1111 can be repeatedas needed. At 1045, a determination is made as to whether or not input has in fact been received. This continues until the input is finished at 1050. Once input is received, real time processing occurs at 1126 to create an output at 1129, which may include a prediction of the input text, drawing, action, control action, or other control feature is made using the intelligent processor. The real time processing at 1126 and output prediction can include determining which of a number of different modalities used simultaneously should provide input to the system (as in step 745- 760 in FIG. 7). Output and feedback 1132 can be provided to determine if the input is finished at 1050. When no additional input is received at 1050, the process terminates.

[0065] FIG. 12 is a flowchart illustrating another implementation wherein tiered, multimodal inputs are supported. As noted above, some inputs may be considered “core” inputs while others are “peripheral”. In the method of FIG. 12, selected core modalities 1302, 1304 are used for model selection while peripheral modalities 1306 are provided directly to the real-time sensor processing. Core modalities the HCI systems comprise primary methods of interaction, including visual, auditory, tactile, kinematic, and textual inputs. Peripheral modalities provide supplementary data, such as physiological, environmental, and contextual information, to enhance and enrich the user interaction experience. Offloading peripheral processing directly to real-time processing can speed the real time processing of user input.

[0066] FIG. 13 is a flowchart illustrating another implementation wherein a user may allow certain types of sensors to bypass the HCI system and end interaction. At 1410, the method may include an operating mode selection 1410 allowing the user to enable or disable a bypass mode using, for example, body sensors 1420 and environment sensors 1440. The body sensors are analyzed by an emotion determination process 1430 to determine whether the user is in an agitated or other state that may indicate distress. Similarly, the environmental sensors are analyzed at 1450 to determine if the user environment has changed and indicates a state where the control system should be neutralized. For example, if the user is in motion in a vehicle, the environmental data may show the user in motion and the system may decide to disable the HCI system while the user is in motion and driving a vehicle. A decision to stop interaction can then be made at 1460, by passing the HCI method steps.

[0067] In embodiments, intelligent hybrid computing is used to enable the HCI system described herein. Intelligent hybrid computing refers to the use of a user’s local processing device(s) and sensors for some tasks, while utilizing different levels of networked elements to perform other processing tasks.

[0068] FIG. 14 illustrates one embodiment of distributed processing enabling the HCI disclosed herein. FIG. 14 shows a user having a local processing device 150a (in this example an AR / VR device) who is present in, for example, the user’s home or office in a local processing environment 1505. Also shown are a plurality of network nodes 1530, 1535, 1540, 1545 which are connected to ARA / R device 150a through a network 1500, and servers 1550 and 1555. The servers 1550 and 1555 may comprise one or more networked processing devices in a cloud computing environment run by a service administrator who provides components of the HCI system as part of a service to users. Nodes 1530, 1535, 1540, 1545 and servers 1550, 1555 may be connected by one or more networks (represented by lines connecting the devices). Nodes 1530 and 1535 may be referred to herein as edge nodes in that they are closer to the local processing environment 1505. An example of a network node is shown in FIG. 21. An example of a network processing device that may comprise a server is illustrated in FIG. 23 As illustrated therein, the local processing environment 1505 and ARA / R device 150a may support on-premises local processing 1520 which may include inferencing, processing text to speech, eye tracking, and local sensor commands. In addition, the AR / VR device supports near sensor / on device processing 1515 comprising sensor processing tasks which are not suitable for in-sensor implementation. In sensor processing 1510 may also be performed in the local environment 1505 and comprises lightweight sensor processing for specific tasks or applications. For example, in an AR / VR device, multiple cameras may be needed to focus on a particular aspect of human interaction, such that many cameras are fighting for the HCI interface. In order to reduce these conflicts, in-sensor processing 1510 can be used to transmit only facial coordinates to the processing systems rather than all the raw data available from the camera(s). The local environment 1505 may also include a context repository, 1525, user preference store 1527 and model repository 1529. Given the potentially more limited processing capacity of the local processingdevice (ARA / R device 150a), the model repository may include tiny NN models which may be used for sentence prediction, grammar correction, emotion detection and userspecific preferences. (In this context, the terms “tiny”, “small”, “medium” and “large” are used only to refer to the relative sizes of the components they modify.)

[0069] Network nodes 1530, 1535, 1540, 1545 may have superior processing capacity to the local processing environment 1505 and may therefore handle tasks such as real time data processing (phrase prediction, text summarizing) and computational intelligence optimization such as updating neural network models. Nodes 1530, 1535, 1540, 1545 may be part of a Multi-access edge computing (MEC) network architecture that enables cloud computing capabilities and a service environment at the edge of a network. Each node may include a context repository 1531 , user preference store 1533 and model repository capable of storing larger models than the local store 1529. Servers 1550, 1555, with even greater processing capacity, may be used for model training and storage, multi-language support, computational intelligence optimization (creating and updating NN models), and processing of data using large neural network models. Each node may include a context repository 1531 for multiple users, user preference store 1533 (for multiple users) and model repository 1541 (with global models and user-specific models) capable of storing larger models than the local store 1529. Processing for the large models may optimally be performed on the servers 1550, 1555.

[0070] FIG. 15 is a flowchart illustrating task distribution amongst the local processing environment 1505, network nodes 1530, 1535, 1540, 1545 and servers 1550, 1555. As user input is received at 1602, the task type will be determined at 1604. The illustrated task types are finger typing action in air 1606, finger drawing action in the air 1608, finger typing action on a surface 1610, a gesture 1612, voice input 1614 or any other type of input (input N) 1615 discussed herein. At 1616, the tasks will be distributed based on complexity. Hand tracking 618 and word correction 620 may be performed locally at a sensor 622. Shape recognition 1624, motion detection 1626, word prediction 1630, grammar correction 1632 and sentence prediction 1634 may be performed at a local processing device 1628. Phrase prediction 1636 and phrase summaries 1638 are assigned to be processed at a network node 1640 and complextasks such as understanding foreign languages 1642 are distributed to the server 1644. A determination is made at 1646 as to whether feedback to the user is required if so, the method returns to the user input at 1602 and if not, the method loops to receive additional input and distribute tasks at 1616. Thus, in general, tasks are distributed to nodes and servers based on processing complexity and model size.

[0071] In embodiments, task distribution at 1616 may be controlled by the user processing device. Tasks pushed to network connected devices and servers may be requested by the user processing device, and acknowledged and tracked by the nodes or servers, in a manner shown in FIG. 19.

[0072] FIG. 16 is a block diagram illustrating global model training and privacy control. FIG. 16 will be understood with respect to FIG. 17 which is a flowchart illustrating training of local and global NN models. It should be understood that the process of FIG. 17 may be repeated for each NN model created for each input modality.

[0073] Referring to FIG. 17, at 1805, the server initializes a global NN model for a given modality. At 1810, the server distributes the global model and a loss function with penalty to all local devices registered with the HCI system. This distribution is illustrated in FIG. 16. At 1815, the server samples end user processing devices which are using the local model. At 1820 and 1825 each local device in parallel performs local training on its own local model. At 1830, local devices and locally trained model updates (not the training data itself) to the server. At 1835, the server aggregates the information from the devices and updates the global model. At 1840, the server updates the model relationships. Model relationships which are dependent on model weights but independent of training data define relationships between input modality NN models. Because the NN models used in the system interact with other modality models, model relationships are used to support multi-modal processing. If the stop criteria is not met at 1845, the method loops to step 1810. If the stop criteria has been met at 1845, the server notifies local devices to discard the training data and it 1855 outputs the latest model. Referring back to FIG. 16, user input 1602 can be used for the local updating of NN models. A privacy filter 1605 ensures that private dataremains on the local device while feature data (training, parameters, gradient, weights and bias) is accessible to the global model. The privacy filter 1605 can employ any of a number of techniques to ensure user privacy, including data anonymization data aggregation, (generalizing data to prevent identification of individuals), federated learning, and data encryption.

[0074] The models trained by the server can be for any of a number of different tasks. For example, the models could recognize the user’s virtual typing habit, the models can be used for word wrecking recognition. The models can be used for word correction, grammar correction, word, phrase, or sentence level predictions and auto completion, hand and finger landmark and movement detection, finger movement or motion gap or space determination, a motion detection, layout functions of a virtual keyboard print varying the location, size, and transparency as a keyboard), automatic pointing device detection and tracking, and multilingual character recognition. The loss function distributed is a regularized loss function with penalty terms which can help migrate challenges from unbalanced data and statistical heterogeneity. Because multitask learning is a complex task, the introduction of model relationships which are dependent on model weights but independent of training data can be done at the central server level during optimization to help with multitask learning.

[0075] FIGs. 18 and 19 illustrate processes involved in hybrid computing when a user moves geographical regions. In FIG. 18, user 100 travels from San Francisco (SF) to New York City (NYC), interacting with a user processing device 150 and other user processing devices (or sensors) 150-1. As shown in FIG. 18, requests for processing jobs too intensive for the user processing device 150-1 are made to the edge node 1530a located in the user’s initial location. In addition, as the users moves, the local edge node 1530a can request to transfer user information to the server 1550. As the user moves and connects to a different edge note 1540a located in NYC, the local edge node 1540a requests the user information to complete processing tasks identified as associated with the user 100. Both user data in the form of user preferences and user-specific models which are stored on edge nodes or on the server may be geographically related to servers and nodes closer to the user’s geographic location.

[0076] FIG. 19 shows the communication between a moving user 100, user device 150-1 , edge nodes 1530 and server 1550. When a user 100 interacts with their device at 2005 and the device 100 requires additional processing, at 2010 the device will attempt to locate the nearest edge node 1530 and after authentication at 2020, the edge node will accept the job, confirm connection and provide job tracking information at 2015. Similarly, when the edge node cannot process a job because it has insufficient processing power, it can request a complex job be performed by the server 1550 at 2025, which, after authentication at 2030, will confirm connection and provide tracking at 2035. Once confirmed, the server will synchronize the data model at 2040, 2045. Once the user travels, at 2050, the method loops to await further interaction.

[0077] FIG. 20A illustrates a use case for the described technology. In the exemplary use case, a user puts on AR / VR device and initiates the HCI system at 2105. At 2110, the user places their hands on an empty surface and starts typing on the empty surface. This typing may take place in accordance with application number serial no. PCT / US2024 / 30833. At 2115, the HCI system will generate output corresponding to the typed text and may display the text for the output of the typing in the ARA / R device. At some point while wearing the AR / VR device, at 2120, the user may start dictating text using their voice. At 2125, the verbal text will be interpreted by the HCI system and at 2125, the verbal dictation text is interpreted and may be displayed in the AR / VR device. In this manner, the user can seamlessly move between different input methods automatically, without affirmatively changing or initiating an audio transcription input.

[0078] FIG. 20B illustrates another use case for the described technology. At 2130, a user sits in front of a laptop and initiates the HCI system. At 2135, the user initiates an online meeting using an online meeting application. During the meeting, at 2140, the user presents material to other members of the meeting using a laptop keyboard and pointing device of their local user processing device. At a later point during the meeting at 2145, the user decides to put on an AR / VR device. At 2150, the user stands and types text (using a virtual keyboard as in PCT / US2024 / 30833) in midair. At 2155, the typed text in mid-air is interpreted by the HCI and may be displayed in the AR / VR device and in the meeting application. At 2160, the user may sit and make commandgesture, such a gesture to end the meeting and close the meeting application. At 2165, the command is interpreted by the HCI, and the meeting application responds to the command gesture.

[0079] FIG. 21 is a schematic diagram illustrating exemplary details of a network device, or node, such as those shown in the network of FIG. 14. A node 1530 may comprise a router, switch, server or other network device, according to an embodiment. The node 1530 can correspond to one of the nodes 1530 and 1540 of FIG. 14. The router or other network node 1530 can be configured to implement or support embodiments of the present technology disclosed herein. The node 1530 may comprise a number of receiving input / output (I / O) ports 2110, a receiver 2112 for receiving packets, a number of transmitting I / O ports 2130 and a transmitter 2132 for forwarding packets. Although shown separated into an input section and an output section in FIG. 21 , often these will be I / O ports 2110 and 2130 that are used for both down-stream and up-stream transfers and the receiver 2112 and transmitter 2132 will be transceivers. Together I / O ports 2110, receiver 2112, I / O ports 2130, and transmitter 2132 can be collectively referred to as a network interface that is configured to receive and transmit packets over a network.

[0080] The node 1530 can also include a processor 2120 that can be formed of one or more processing circuits and a memory or storage section 2122. The storage 2122 can be variously embodied based on available memory technologies and in this embodiment is shown to have a cache 2124, which could be formed from a volatile RAM memory such as SRAM or DRAM, and long-term storage 2126, which can be formed of non-volatile memory such as flash NAND memory or other memory technologies.

[0081] Storage 2122 can be used for storing both data and instructions for implementing the accounting data generating techniques described herein. In particular, instructions causing the processor 2120 to perform the functions of the HCI applications (for real time processing of user data too intensive for a local device), neural network models 2150, context data and user preferences 2132 and an output generator 2160.

[0082] Node 1530 can include a programmable content forwarding plane 2128. Depending on the embodiment, the programmable content forwarding plane 2128 can be part of the more general processing elements of the processor 2120 or a dedicated portion of the processing circuitry. More specifically, the processor(s) 2120, including the programmable content forwarding plane 2128, can be configured to implement embodiments of the present technology described below. In accordance with certain embodiments, the memory 2122 stores computer readable instructions that are executed by the processor(s) 2120 to implement embodiments of the present technology. It would also be possible for embodiments of the present technology described below to be implemented, at least partially, using hardware logic components, such as, but not limited to, Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), special purpose computers, etc.

[0083] FIG. 22 illustrates a network processing device 2200 which may comprise any user processing device 100 for implementing the technology discussed herein. The network processing device 2200 may be a desktop or laptop computer, a mobile device such as a tablet, or be integrated into a headset 150d or may be provided in a housing which is separate from but connected to the headset 150d. In embodiments, network processing device 2200 may contain multiple instances of a component, such as multiple processing units, processors, memories, interfaces, etc.

[0084] Device 2200 may comprise a central processing unit (CPU) 2210, a graphics processing unit (GPU) 2220, a memory 2225, a mass storage device 2230, and an I / O interface 2215 connected to a bus 2270. The I / O interface 2215 may be connected to a headset 150b or other peripherals. The bus 2270 may be one or more of any type of several bus architectures including a memory bus or memory controller, a peripheral bus, or the like. A network interface 2235 enables the network processing device to communicate over a network 1500 with other processing devices such as those described herein.

[0085] The mass storage device 2230 may comprise any type of storage device configured to store data, programs, and other information and to make the data, programs, and other information accessible via the bus 2270. The mass storage device 2230 may comprise, for example, one or more of a solid-state drive, hard disk drive, a magnetic disk drive, an optical disk drive, or the like. The mass storage device 2230 includes instructions which when executed by the CPU (or processor) cause the processor to perform the methods described herein. The mass storage 2230 may include code in the form of application modules and data stored thereon, comprising instructions for causing the CPU to implement the components of HCI system which are illustrated as present in memory 2225 in FIG. 22.

[0086] The CPU 2210 may comprise any type of electronic data processor. Memory 2225 may comprise any type of system memory such as static randomaccess memory (SRAM), dynamic random-access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof, or the like. In an embodiment, memory 2225 may include ROM for use at boot-up, and DRAM for program and data storage for use while executing programs. In embodiments, the memory 2225 is non-transitory. In one embodiment, the memory 2225 includes computer-readable instructions that are executed by the processor(s) 2210 and 2220 to implement embodiments of the disclosed technology.

[0087] In one embodiment, the mass storage 2240 includes user preference data 2234, context data 2236, NN models 2238 and code for implements instances of the intelligent input application 3000. In one embodiment, the memory 2225 includes instances of the input data acquisition and processing module 2242, the intelligent fusion processor 2244, NN models 2244, a context manager 2248, a resource resolver 2250, an action planner 2252 and an output generator 2254, all of which are configured to cause the processor, when executing the instructions thereof, to perform the functions described for such elements herein.

[0088] A NN model updater 2256 configured to cause the processor, when executing the instructions thereof, to update the NN models provided by the server, such as server 1550 described in various embodiments herein.

[0089] FIG. 23 illustrates a server network processing device 1550 which may comprise a service host for implementing the described technology. In FIG 23, like reference numbers refer to like elements described with respect to FIG. 22. A server 1550 in the described technology may include a more powerful central processing unit (CPU) 2210 (or multiple CPUs), a graphics processing unit (GPU) 2220, a memory 2225, a mass storage device 2230, and an I / O interface 2215 connected to a bus 2270.

[0090] In one embodiment, the mass storage 2240 includes user preference data 2334, context data 2336, NN models 2338 and code for implements instances of the intelligent input application 3100. In the server 1550, the user data 2334 comprises data for multiple users of the HCI system. Context data 2336 likewise contains context data for multiple users. The NN models 2338 generally comprise both global models initially sent to user devices as well as updates to those models returned by user devices. In one embodiment, the memory 2225 includes computer-readable instructions that are executed by the processor(s) 2210 and 2220 to implement embodiments of the disclosed technology.

[0091] In one embodiment, the memory 2225 the intelligent fusion processor 2344, NN models 2346, a context manager 2348, an output generator 2354, all of which are configured to cause the processor, when executing the instructions thereof, to perform the functions described for such elements herein. A NN model generator and updater 2356 is configured to cause the processor, when executing the instructions thereof, to update the NN models provided by the server, such as server 1550 described with respect to FIG. 17.

[0092] For the purposes of this document, it should be noted that the dimensions of the various features depicted in the FIGs. may not necessarily be drawn to scale.

[0093] For purposes of this document, reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “another embodiment” may be used to describe different embodiments or the same embodiment.

[0094] For the purposes of this document, a connection may be a direct connection or an indirect connection (e.g., via one or more other parts). In some cases, when an element is referred to as being connected or coupled to another element, the element may be directly connected to the other element or indirectly connected to the other element via intervening elements. When an element is referred to as being directly connected to another element, then there are no intervening elements between the element and the other element. Two devices are “in communication” if they are directly or indirectly connected so that they can communicate electronic signals between them.

[0095] Although the present disclosure has been described with reference to specific features and embodiments thereof, it is evident that various modifications and combinations can be made thereto without departing from the scope of the disclosure. The specification and drawings are, accordingly, to be regarded simply as an illustration of the disclosure as defined by the appended claims, and are contemplated to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the present disclosure.

[0096] The technology described herein can be implemented using hardware, software, or a combination of both hardware and software. The software used is stored on one or more of the processor readable storage devices described above to program one or more of the processors to perform the functions described herein. The processor readable storage devices can include computer readable media such as volatile and non-volatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer readable storage media and communication media. Computer readable storage media may be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information, and which can be accessed by a computer. A computer readable medium or media does (do) not include propagated, modulated, or transitory signals.

[0097] Communication media typically embodies computer readable instructions, data structures, program modules or other data in a propagated, modulated, or transitory data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as RF and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.

[0098] In alternative embodiments, some or all of the software can be replaced by dedicated hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Applicationspecific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), special purpose computers, etc. In one embodiment, software (stored on a storage device) implementing one or more embodiments is used to program one or more processors. The one or more processors can be in communication with one or more computer readable media / storage devices, peripherals and / or communication interfaces.

[0099] It is understood that the present subject matter may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this subject matter will be thorough and complete and will fully convey the disclosure to those skilled in the art. Indeed, the subject matter is intended to cover alternatives, modifications, and equivalents of these embodiments, which are included within the scope and spirit of the subject matter as defined by the appended claims. Furthermore, in the following detailed description of the present subject matter, numerous specific details are set forth in order to provide a thorough understanding of the present subject matter. However, it will be clear to those of ordinary skill in the art that the present subject matter may be practiced without such specific details.

[0100] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0101] The description of the present disclosure has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The aspects of the disclosure herein were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure with various modifications as are suited to the particular use contemplated.

[0102] For purposes of this document, each process associated with the disclosed technology may be performed continuously and by one or more computing devices. Each step in a process may be performed by the same or different computing devices as those used in other steps, and each step need not necessarily be performed by a single computing device.

[0103] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

CLAIMSWhat is claimed is:1 . A computer implemented method of detecting natural motion to generate an input to a computing device, the computer implemented method performed by one or more processors, comprising: using at least one sensor to capture or detect and user movement and generating input data comprising at least one of multiple input modalities from multiple input data devices from which actions of the user can be determined; determining a context of the input data using user context data and user preferences, the context including information which may be used to interpret the input data from the user; calculating whether one or more control events occurs in the input data based the movement and positioning of the user and the context of the input data using one or more neural networks, the one or more neural networks including a trained model associated with each input modality; outputting one or more control inputs calculated as present in the input data to the computing device; detecting at least a different input modality engaged by the user in the input data; calculating whether one or more new control events occurs in the input data based on the different input modality; and outputting the one or more new control inputs calculated as present in the input data to the computing device.

2. The computer implemented method of claim 1 wherein the input data includes multiple input modalities from the multiple data devices, and the method further includes combining the data from the multiple input data sensors.

3. The computer implemented method of claim 1 wherein the determining, combining, calculating and outputting are performed partially on the computing device and partially on a second computing device coupled via a network.

4. The computer implemented method of claim 3 wherein the determining, combining, calculating and outputting processing tasks are distributed to the computing device or the second computing device based on complexity.

5. The computer implemented method of any of claims 1 through 4 wherein the input data devices comprise one of a camera, a microphone, a keyboard, a pointing device, an ECG sensor, an EEG sensor and / or an e-tattoo.

6. The computer implemented method of any of claims 1 through 4 wherein the multiple input modalities include one or more of: interaction with virtual keyboards, interaction with ultra-thin and flexible electronic keyboards, biometric inputs, audio inputs, environmental inputs, location, user body motion, in-air gestures, an on-surface gestures, user facial expressions, user voice, user eye motion, and user emotion.

7. The computer implemented method of claims 1 through 6 wherein the context data comprises one of application usage, user location, user emotional state, user posture, user movement, environment lighting conditions, environment sound, environment weather conditions and environment temperature.

8. The computer implemented method of claims 1 through 7 wherein the method further includes: registering each input data device; determining which input devices are providing input data which may comprise an application input based on the context; resolving conflicts between multiple input devices which comprise application input; and generating the control inputs.

9. The computer implemented method of any of claims 1 and2 wherein the method further includes receiving one or more trained models from a service host, updating the one or more trained models received from the host based on user interaction, and forwarding the updated trained models to the service host.

10. The computer implemented method of any of claims 1 through 8 wherein the control input comprises one of: text, image, hand drawing, an application command, a display command.

11. The computer implemented method of any of claims 1 through 3 wherein the context is determined based on context data and user preferences, and the context data and user preferences are stored on the computing device.

12. The computer implemented method of any of claims 1 through 11 wherein the context is determined based on context data and user preferences, and the context data and user preferences are stored on the computing device.

13. The computer implemented method of any of claims 1 through 11 wherein the context is determined based on context data and user preferences, and the context data and user preferences are at least partially stored on one or more network connected processing devices.

14. The computer implemented method of any of claims 1 through 12 wherein the context include user emotional state and the method includes calculating the emotional state based on the output of an ECG and / or an EEG body sensor.

15. A user equipment device, comprising: a storage medium comprising computer instructions; a display device; one or more processors coupled to communicate with the storage medium, wherein the one or more processors execute the instructions to cause the device to:receive input data from a user, the input data comprising multiple input modalities from multiple input data devices detected from which actions of the user can be determined; determine a context of the input data using user context data and user preferences, the context including information which may be used to interpret the input data from the user; calculate whether one or more control events occurs in the input data based the movement and positioning of the user and the context of the input data using one or more neural networks, the one or more neural networks including a trained model associated with each input modality; and output one or more control inputs calculated as present in the input data to the computing device; detect at least a different input modality engaged by the user in the input data; calculating whether one or more new control events occurs in the input data based on the different input modality; and outputting the one or more new control inputs calculated as present in the input data to the computing device.

16. The user equipment device of claim 15 wherein the input data includes multiple input modalities from the multiple data devices, and the method further includes combining the data from the multiple input data sensors.

17. The user equipment device of claim 15 wherein the user equipment device transfers one or more of the instructions to determine, combine, calculate and output to one or more network connected computing devices.

18. The user equipment device of claims 15 through 17 wherein the transfer to the one or more network computing devices is based on complexity and neural network model size.

19. The user equipment device of any of claims 15 through 17 wherein the input data devices comprise one of a camera, a microphone, a keyboard, a pointing device, an ECG sensor, an EEG sensor and / or an e-tattoo.

20. The user equipment device of any of claims 15 through 18 wherein the multiple input modalities include one or more of: interaction with virtual keyboards, interaction with ultra-thin and flexible electronic keyboards, biometric inputs, audio inputs, environmental inputs, location, user body motion, in-air gestures, an on-surface gestures, user facial expressions, user voice, user eye motion, and user emotion.

21. The user equipment device of claim 15 through 19 wherein the context comprises one of application usage, user location, user location, user emotional state, user posture, user movement, environment lighting conditions, environment sound, environment weather conditions and environment temperature.

22. The user equipment device of claim 15 through 20 wherein the one or more processors execute the instructions to cause the device to: register each input data device; determine which input devices are providing input data which may comprise an application input based on the context; resolve conflicts between multiple input devices which comprise application input; and generate the control inputs.

23. The user equipment device of any of claims 15 through 22 wherein the one or more processors execute the instructions to cause the device to receive one or more trained models from a service host, update the one or more trained models received from the host based on user interaction, and forward the updated trained models to the service host.

24. The user equipment device of any of claims 15 through 22 wherein the control input comprises one of: text, image, hand drawing, an application command, a display command.

25. The user equipment device of any of claims 15 through 20 wherein the context is determined based on context data and user preferences, and the context data and user preferences are stored on the computing device.

26. The user equipment device of any of claims 15 through 23 wherein the context is determined based on context data and user preferences, and the context data and user preferences are at least partially stored on one or more network connected processing devices.

27. The computer implemented method of any of claims 15 through 23 wherein the context include user emotional state and the method includes calculating the emotional state based on the output of an ECG and / or an EEG body sensor.

28. A non-transitory computer-readable medium storing computer instructions that configure at least one processor, upon execution of the instructions, to perform the following steps: receiving input data from a user, the input data comprising multiple input modalities from multiple input data devices detected from which input actions of the user can be determined; determining a context of the input data using user context data and user preferences, the context including information which may be used to interpret the input data from the user; calculating whether one or more control events occurs in the input data based the movement and positioning of the user and the context of the input data using one or more neural networks, the one or more neural networks including a trained model associated with each input modality; and outputting one or more control inputs calculated as present in the input data to the computing device;detecting at least a different input modality engaged by the user in the input data; calculating whether one or more new control events occurs in the input data based on the different input modality; and outputting the one or more new control inputs calculated as present in the input data to the computing device.

29. The non-transitory computer-readable medium of claim 28 wherein the input data includes multiple input modalities from the multiple data devices, and the method further includes combining the data from the multiple input data sensors.

30. The non-transitory computer-readable medium of claim 28 wherein the instructions configure at least one processor, upon execution of the instructions, to distribute one or more of the determining, combining, calculating and outputting to a computing device coupled via a network.

31. The non-transitory computer-readable medium of claim 30 wherein the determining, combining, calculating and outputting processing tasks are distributed based on complexity and neural network model size.

32. The non-transitory computer-readable medium of any of claims 28 through 31 wherein the input data devices comprise one of a camera, a microphone, a keyboard, a pointing device.

33. The non-transitory computer-readable medium of any of claims 28 through 32 wherein the multiple input modalities include one or more of: interaction with virtual keyboards, interaction with ultra-thin and flexible electronic keyboards, biometric inputs, audio inputs, environmental inputs, location, user body motion, in-air gestures, an on-surface gestures, user facial expressions, user voice, user eye motion, and user emotion.

34. The non-transitory computer-readable medium of any of claims 28 through 33 wherein the context comprises one of application usage, user location, and environment temperature.

35. The non-transitory computer-readable medium of claim 28 through 34 wherein the instructions that configure at least one processor, upon execution of the instructions, to perform: registering each input data device; determining which input devices are providing input data which may comprise an application input based on the context; resolving conflicts between multiple input devices which comprise application input; and generating the control inputs.

Images