Haptic feedback based on event classification

Abstract

A computing device may include at least one processor, a haptic device, and a storage device that stores instructions executable by the at least one processor to obtain interaction data associated with input data or output data during execution of a software application. The storage device may further store instructions executable by the at least one processor to provide the interaction data to an event determination system. The storage device may further store instructions executable by the at least one processor to determine, using the event determination system and based on the interaction data, an event from a plurality of events. The storage device may further store instructions executable by the at least one processor to retrieve haptic feedback instructions associated with the event. The storage device may further store instructions executable by the at least one processor to output the haptic feedback instructions.

Description

HAPTIC FEEDBACK BASED ON EVENT CLASSIFICATIONBACKGROUND

[0001] A computing device may use a haptic actuator to produce haptic feedback that can be felt by the user of the computing device. Haptic feedback refers to tactile or kinesthetic sensation created by a haptic actuator that applies forces, vibrations, or motions. A haptic actuator may operate at different vibrational frequencies to produce haptic effects associated with different tactile sensations. For example, a haptic actuator operating at a vibrational frequency at or above 200 Hertz (Hz) may produce haptic feedback associated with a smooth and somewhat penetrating tactile sensation, while a haptic actuator operating at a vibrational frequency at or below 60 Hz may produce haptic feedback associated with a breathing sensation, a flutter sensation, or a roughness sensation.SUMMARY

[0002] In general, aspects of this disclosure are directed to techniques for outputting a haptic feedback pattern based on a real-time or near real-time event classification of interaction data associated with a software application. The techniques may include classifying interaction data (e.g., audio signals, video signals, data generated by one or more inertial measurement units, tactical inputs, etc.), obtained during execution of the software application, as an event of a set of events. The techniques may include classifying the interaction data as an event that is mapped to a particular haptic feedback pattern. The techniques may include outputting a haptic feedback pattern based on a mapping of the haptic feedback pattern to an event associated with the classification of the interaction data.

[0003] In one example, the disclosure is directed to a method that includes obtaining, by one or more processors, interaction data associated with input data or output data during execution of a software application. The method may further include providing, by the one or more processors, the interaction data to an event determination system. The method may further include determining, using the event determination system and based on the interaction data, an event from a plurality of events. The method may further include retrieving, by the one or more processors, haptic feedback instructionsassociated with the event. The method may further include outputting, by the one or more processors, the haptic feedback instructions.

[0004] In another example, the disclosure is directed to a computing device. The computing device includes at least one processor, a haptic device, and a storage device that stores instructions executable by the at least one processor to obtain interaction data associated with input data or output data during execution of a software application. The storage device may further store instructions executable by the at least one processor to provide the interaction data to an event determination system. The storage device may further store instructions executable by the at least one processor to determine, using the event determination system and based on the interaction data, an event from a plurality of events. The storage device may further store instructions executable by the at least one processor to retrieve haptic feedback instructions associated with the event. The storage device may further store instructions executable by the at least one processor to output the haptic feedback instructions.

[0005] In another example, the disclosure is directed to non-transitory computer- readable storage medium storing instructions that, when executed, cause at least one processor of a computing device to obtain interaction data associated with input data or output data during execution of a software application. The instructions may further cause the at least one processor of the computing device to provide the interaction data to an event determination system. The instructions may further cause the at least one processor of the computing device to determine, using the event determination system and based on the interaction data, an event from a plurality of events. The instructions may further cause the at least one processor of the computing device to retrieve haptic feedback instructions associated with the event. The instructions may further cause the at least one processor of the computing device to output the haptic feedback instructions.

[0006] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF DRAWINGS

[0007] FIG. 1 is a conceptual diagram illustrating an example computing device configured to output a haptic feedback pattern based on interaction data, in accordance with one or more aspects of the present disclosure.

[0008] FIG. 2 is a block diagram illustrating an example computing device configured to output haptic feedback patterns based on event classification, in accordance with one or more aspects of the present disclosure.

[0009] FIG. 3 is a conceptual diagram illustrating an example computing device for event classification of interaction data, in accordance with aspects of this disclosure.

[0010] FIG. 4 is a conceptual diagram illustrating an example computing device for classification of interaction data, in accordance with aspects of this disclosure.

[0011] FIG. 5 is a conceptual diagram illustrating processing and synchronization of haptic feedback patterns output based on interaction data, in accordance with aspects of this disclosure.

[0012] FIG. 6 is a conceptual diagram illustrating streaming and segmentation of interaction data for event classification, in accordance with aspects of this disclosure.

[0013] FIG. 7 is a conceptual diagram illustrating featurization of segments of interaction data for event classification, in accordance with aspects of this disclosure.

[0014] FIG. 8 is a flowchart illustrating example operations of an example computing device configured to output a haptic feedback pattern based on interaction data, in accordance with one or more aspects of the present disclosure.DETAILED DESCRIPTION

[0015] FIG. 1 is a conceptual diagram illustrating example computing device 102 configured to output a haptic feedback pattern based on a real-time or near real-time analysis of interaction data, in accordance with one or more aspects of the present disclosure. As shown in FIG. 1, computing device 102 is a mobile computing device (e.g., a mobile phone). However, in other examples, computing device 102 may be a tablet computer, a laptop computer, a desktop computer, a gaming system, a media player, an e-book reader, a television platform, an automobile navigation system, a wearable computing device (e.g., a computerized watch, computerized eyewear such as Al glasses, a computerized glove, a computerized ring, etc.), or any other type of mobile or non-mobile computing device.

[0016] Computing device 102 includes a user interface device (UID) 104. UID 104 of computing device 102 may function as an input device for computing device 102 and as an output device for computing device 102. UID 104 may be implemented using various technologies. For instance, UID 104 may function as an input device using a presence-sensitive input screen, such as a resistive touchscreen, a surface acoustic wave touchscreen, a capacitive touchscreen, a projective capacitive touchscreen, a pressure sensitive screen, an acoustic pulse recognition touchscreen, or another presencesensitive display technology. UID 104 may function as an output (e.g., display) device using any one or more display devices, such as a liquid crystal display (LCD), dot matrix display, light emitting diode (LED) display, microLED, miniLED, organic lightemitting diode (OLED) display, e-ink, or similar monochrome or color display capable of outputting visible information to a user of computing device 102.

[0017] UID 104 of computing device 102 may include a presence-sensitive display that may receive tactile input from a user of computing device 102. UID 104 may receive indications of tactile input by detecting one or more gestures from a user of computing device 102 (e.g., the user touching or pointing to one or more locations of UID 104 with a finger or a stylus pen). UID 104 may present output to a user, for instance at a presence-sensitive display. UID 104 may present the output as a graphical user interface (e.g., user interface 140), which may be associated with functionality provided by computing device 102. For example, UID 104 may present various user interfaces of components of a computing platform, operating system, applications, or services executing at or accessible by computing device 102 (e.g., a gaming application of applications 116, an electronic messaging application of applications 116, a mobile operating system, etc.). A user may interact with a respective user interface to cause computing device 102 to perform operations relating to a function.

[0018] Computing device 102 includes haptic device 114 configured to provide haptic feedback 120 to a user of computing device 102. Haptic device 114 may include one or more haptic actuators, such as linear resonant actuators, eccentric rotating mass vibration motors, piezoelectric transducers, electromechanical devices, and / or other vibrotactile actuators, and drive electronics coupled to the one or more actuators. The drive electronics may cause the one or more haptic actuators to induce a selected vibratory response into at least a portion of the computing device 102, thereby providing haptic feedback in the form of a tactile sensation to a user of computing device 102.

[0019] Haptic device 114 may output patterns of vibrations (also referred to as “vibration patterns” and / or “haptic feedback patterns”) associated with different haptic effects to produce various tactile sensations to a user. For example, haptic device 114 may output a vibration pattern associated with a fluttery haptic effect to produce a fluttery tactile sensation to the user, output a vibration pattern associated with a grainy haptic effect to produce a grainy tactile sensation to the user, or output a vibration pattern associated with a rough haptic effect to produce a rough tactile sensation to the user.

[0020] A vibration pattern may be associated with a vibration frequency, which may be the number of vibrations outputted by haptic device 114 within a specified time period, and may also be associated with a set of vibration intensities that specify the intensities of the vibrations outputted by haptic device 114. Different vibration patterns associated with different haptic effects may have different associated vibration frequencies and different sets of vibration intensities.

[0021] In some examples, computing device 102 may associate haptic effects with various user interface interactions. As the user provides user input to interact with the user interface presented by UID 104, computing device 102 may determine the haptic effect to be produced based on the user interactions with the user interface presented by UID 104, computing device 102, and haptic device 114 may output vibration patterns associated with the haptic effects to produce tactile sensations that can be felt by the user. For example, computing device 102 may, in response to receiving user input to select a button in the user interface, produce a haptic effect associated with the button being selected, or may, in response to receiving user input that corresponds to a gesture to scroll through a list of items in the user interface, produce a haptic effect associated with the scrolling.

[0022] In some examples, computing device 102 may associate haptic effects with the occurrence of events at computing device 102. For example, computing device 102 may select haptic effects to be produced in response to computing device 102 receiving a phone call or a text message, in response to a payment transaction being accepted or declined, in response to the occurrence of an alarm or a reminder, and the like. Computing device 102 may, in response to determining that an event has occurred, determine the haptic effect associated with the event and may output a vibration pattern associated with the haptic effect.

[0023] One or more applications 116 (hereinafter, “applications 116”) of computing device 102 may include functionality to perform any variety of operations on computing device 102. For instance, applications 116 may include a gaming application, a text application, a web browser, a multimedia player, a calendar application, an operating system, a distributed computing application, a graphic design application, a video editing application, a web development application, or any other application. One of applications 116 may be a gaming application (e.g., a software application for a racing game, a fighting game, a first person shooting game, a card game, etc.) developed by software application developers.

[0024] In some instances, applications 116 may include haptic feedback instructions for driving haptic device 114 to generate a haptic feedback pattern (e.g., haptic feedback 120). However, applications 116 with hard-coded haptic feedback may limit experiences that software application developers may provide to users interacting with applications 116. For example, in instances where a gaming application of applications 116 may generate different audio signals based on a virtual environment a character interacts with (e.g., a first audio signal for a car in the gaming application driving on asphalt and a second audio signal for the car driving on off-road terrain), software application developers may not be able to configure the gaming application to output unique haptic feedback patterns for each of the different audio signals. Computing device 102, according to the techniques described herein, may output a particular haptic feedback pattern (e.g., haptic feedback 120) based on a classification of interaction data, such as event classifications of the different audio signals generated by the gaming application.

[0025] In accordance with techniques of this disclosure, computing device 102 may instruct haptic device 114 to output a vibration pattern based on interaction data associated with a software application of applications 116. In some examples, event determination system 110 of computing device 102 may include a machine learning model trained to identify events based on a classification of software application interaction data. Event determination system 110 may apply the machine learning model to determine haptic feedback instructions for a haptic feedback pattern mapped to the identified event. In some instances, event determination system 110 may identify events based on look-up tables specifying mappings or correlations of events to segments of interaction data that may be analyzed in real-time or near-real time. Forinstance, event determination system 110 may query a look-up table based on a segment of interaction data to identify an event corresponding to the segment of interaction data. Event determination system 110 may additionally, or alternatively, determine, using the look-up table, haptic feedback instructions for a haptic feedback pattern mapped to the identified event. Computing device 102 may retrieve the haptic feedback instructions for haptic device 114. Haptic device 114 may output a vibration pattern (e.g., haptic feedback 120) based on the haptic feedback instructions. Although illustrated as stored locally at computing device 102, event determination system 110 may be stored and / or executed on an external computing device or external computing system.

[0026] In operation, computing device 102 may obtain interaction data of a user interacting with a software application of applications 116. Interaction data may include audio signal streams, video signal streams, or other multimodal signals generated by the software application and / or inputs (e.g., data generated by inertial measurement units, tactile input, touch, force applied, etc.) by a user operating computing device 102 during execution of the software application. Computing device 102 may continuously obtain the interaction data, in real-time, during execution of the software application.Computing device 102 may store interaction data in a buffer and provide the interaction data to event determination system 110 of computing device 102. Computing device 102 may collect the interaction data in response to receiving explicit consent from a user operating computing device 102 (e.g., via a prompt output by UID 104).

[0027] In situations in which the systems discussed here collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect user information (e.g. , information about a user’s social network, social actions or activities, profession, a user’s preferences, or a user’s current location), or to control whether and / or how to receive content from the content server that may be more relevant to the user. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information iscollected about the user and used by a content server associated with computing device 102.

[0028] Event determination system 110 may identify events based on interaction data associated with software applications. In some examples, event determination system 110 may apply an algorithm that implements a look-up table to identify one or more events based on interaction data. For instance, event determination system 110 may partition interaction data into segments of interaction data. Event determination system 110 may query a look-up table to identify an event mapped to a segment of interaction data included in the segments of interaction data.

[0029] In some instances, event determination system 110 may apply an algorithm generated and output by a machine learning model to identify events based on interaction data. For instance, event determination system 110 may obtain a first algorithm, generated and output by a machine learning model, for classifying various types of interaction data and a second algorithm, generated and output by a machine learning mode, for determining events based on classifications of interaction data.

[0030] In some examples, event determination system 110 may include one or more machine learning models that are pre-trained to identify, in real-time or near real-time, events based on interaction data associated with software applications. Event determination system 110 may input interaction data to the one or more machine learning models. For example, event determination system 110 may input, to a machine learning model, interaction data including an audio stream (e.g., audio waveforms) generated by a software application of applications 116. In another example, event determination system 110 may input, to a machine learning model, interaction data including a video stream (e.g., a series of images) generated by a software application of applications 116. In another example, event determination system 110 may input, to a machine learning model, interaction data including data generated by one or more inertial measurement units during execution of a software application of applications 116. In another example, event determination system 110, may input, to a machine learning model, interaction data including input data generated by a presence-sensitive display in response to detecting one or more user inputs. Event determination system 110 may apply the one or more machine learning models to segment the interaction data into segments of interaction data that correspond to time frames of signals (e.g., rolling window capturing segments of interaction data signals) included in the interaction data.

[0031] Event determination system 110 may apply the one or more machine learning models to generate feature vectors for each of the segments of interaction data. Feature vectors may include vectors in a high-dimensional space that capture information about interaction data signals, such as spectral envelope, perceptual characteristics, formant structure, timbral characteristics, noise robustness, dynamic changes, temporal resolution, phonetic distinctions, prosodic features, frequency resolution, or the like. Event determination system 110 may apply one or more machine learning models that have been trained to extract and encode features of interaction data as feature vectors based on the types of interaction data being analyzed. For example, event determination system 110 may apply one or more machine learning models that have been trained to extract and encode features of interaction data including audio data, video data, inertial measurement unit data, and / or tactile input data based on training data that includes labeled samples of audio data, video data, inertial measurement unit data, and / or tactile input data.

[0032] Event determination system 110 may apply a classification model that is pretrained to classify feature vectors for segments of interaction data as events. Event determination system 110 may apply the classification model to determine, in real-time or near real-time, probabilities the segments of interaction data correspond to each event of a set of predefined events. Event determination system 110 may store the set of predefined events that have been defined by software developers of the software application associated with the interaction data. Event determination system 110 may determine whether a probability that a segment of interaction data corresponds to an event satisfies an adjustable threshold. Based on whether event determination system 110 determines the probability that the segment of interaction data corresponds to the event satisfies the adjustable threshold, event determination system 110 may output an indication specifying that the segment of interaction data corresponds to the event.

[0033] Computing device 102 may retrieve, based on an indication specifying a segment of interaction data corresponds to an event, haptic feedback instructions.Computing device 102 may retrieve haptic feedback instructions from a plurality of haptic feedback instructions according to a mapping of events to haptic feedback instructions stored as haptic feedback mappings 118. Haptic feedback mappings 118 may include a mapping of an event to haptic feedback instructions for a particular haptic feedback pattern (e.g., haptic feedback 120). Haptic feedback instructions may includea function of an Application Programming Interface (API) (e.g., an API call) to generate haptic waveform signals for haptic feedback patterns. Computing device 102 may execute the retrieved haptic feedback instructions to generate haptic waveform signals for haptic device 114. Computing device 102 may output the haptic waveform signals to haptic device 114. Haptic device 114 may output a vibration pattern according to the haptic waveform signals generated based on the haptic feedback instructions.

[0034] The techniques may provide one or more advantages. For example, computing device 102 may output haptic feedback patterns based on real-time or near real-time determinations of events. By computing device 102 executing event determination system 110 in an application layer of computing device 102 to determine events based on interaction data obtained during execution of a software application of applications 116, the software application may not include hard-coded instructions for outputting haptic feedback patterns; thereby reducing resources a software application developer may use to improve user experiences via haptic feedback. The techniques described herein may enable software application developers to improve user experiences via haptic feedback by allowing developers to define mappings of haptic feedback instructions for various haptic feedback patterns to events recognized in real-time or near real-time (e.g., mappings stored as haptic feedback mappings 118). Computing device 102 may, during execution of a software application developed by the software application developers, output haptic feedback patterns based on an analysis of interaction data and on the mappings of the haptic feedback instructions to the recognized events. In this way, the techniques described herein improve user experiences that a software application developer may develop for a software application, while reducing complexity associated with implementing the improved user experiences in the software application.

[0035] FIG. 2 is a block diagram illustrating an example computing device configured to output haptic feedback patterns based on event classification, in accordance with one or more aspects of the present disclosure. Computing device 202 is only one particular example of computing device 102 of FIG. 1, and many other examples of computing device 102 may be used in other instances. In the example of FIG. 2, computing device 202 may be a wearable computing device, a mobile computing device (e.g., a smartphone), a gaming device, a virtual reality (VR) device, an augmented reality (AR) device, or any other computing device. Computing device 202 of FIG. 2 may include asubset of the components included in example computing device 202 or may include additional components not shown in FIG. 2.

[0036] As shown in the example of FIG. 2, computing device 202 includes user interface device 204 (“UID 204”), one or more processors 240, one or more input devices 242, one or more communication units 244, one or more output devices 246, one or more storage devices 248, and haptic device 214. Storage devices 248 of computing device 202 also include operating system 254, event determination system 210, applications 216, and haptic feedback mappings 218. UID 204, haptic device 214, event determination system 210, applications 216, and haptic feedback mappings 218 of FIG. 2 may be example or alternative implementations of UID 104, haptic device 114, event determination system 110, applications 116, and haptic feedback mappings 118 of FIG. 1, respectively.

[0037] Communication channels 250 may interconnect each of the components 240, 242, 244, 246, 248, 204, and 214 for inter-component communications (physically, communicatively, and / or operatively). In some examples, communication channels 250 may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

[0038] One or more input devices 242 of computing device 202 may be configured to receive input. Examples of input are tactile, audio, and video input. Input devices 242 of computing device 202, in one example, includes a presence-sensitive display, inertial measurement units, touch-sensitive screen, mouse, keyboard, voice responsive system, video camera, microphone or any other type of device for detecting input from a human or machine.

[0039] One or more output devices 246 of computing device 202 may be configured to generate output. Examples of output are tactile, audio, and video output. Output devices 246 of computing device 202, in one example, includes a presence-sensitive display, sound card, video graphics adapter card, speaker, liquid crystal display (LCD), or any other type of device for generating output to a human or machine.

[0040] One or more communication units 244 of computing device 202 may be configured to communicate with external devices via one or more wired and / or wireless networks by transmitting and / or receiving network signals on the one or more networks. Examples of communication unit 244 include a network interface card (e.g. such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GNSS receiver, aGPS receiver, or any other type of device that can send and / or receive information. Other examples of communication units 44 may include short wave radios, cellular data radios, wireless network radios, as well as universal serial bus (USB) controllers.

[0041] In some examples, UID 204 of computing device 202 may include functionality of input devices 242 and / or output devices 246. In the example of FIG. 2, UID 204 may be or may include a presence-sensitive input device. In some examples, a presence sensitive input device may detect an object at and / or near a screen. As one example range, a presence-sensitive input device may detect an object, such as a finger or stylus that is within 2 inches or less of the screen. The presence-sensitive input device may determine a location (e.g., an (x,y) coordinate) of a screen at which the object was detected. In another example range, a presence-sensitive input device may detect an object six inches or less from the screen and other ranges are also possible. The presence-sensitive input device may determine the location of the screen selected by a user’s finger using capacitive, inductive, and / or optical recognition techniques. In some examples, a presence sensitive input device also provides output to a user using tactile, audio, or video stimuli as described with respect to output device 246, e.g., at a display. In the example of FIG. 2, UID 204 may present a user interface.

[0042] While illustrated as an internal component of computing device 202, UID 204 also represents an external component that shares a data path with computing device 202 for transmitting and / or receiving input and output. For instance, in one example, UID 204 represents a built-in component of computing device 202 located within and physically connected to the external packaging of computing device 202 (e.g., a screen on a mobile phone). In another example, UID 204 represents an external component of computing device 202 located outside and physically separated from the packaging of computing device 202 (e.g., a monitor, a projector, etc. that shares a wired and / or wireless data path with a tablet computer).

[0043] Haptic device 214 of computing device 202 is an example of haptic device 114 of FIG. 1 and may be configured to output haptic feedback, such as in the form of vibration patterns, that can be felt by users of computing device 202 that hold computing device 202 and / or that touch an external surface of the enclosure of computing device 202. Haptic device 214 includes one or more haptic actuators 262 and drive electronics 264.

[0044] One or more haptic actuators 262 may include one or more linear resonant actuators, one or more eccentric rotating mass vibration motors, one or more piezoelectric transducers, one or more electromechanical devices, and / or other vibrotactile actuators that may create motion (e.g., vibrate) to impart information to the user of computing device 202 through the user’s sense of touch. For example, a linear resonant actuator may vibrate by moving a mass in a reciprocal manner by means of a magnetic voice coil.

[0045] Drive electronics 264 may be circuitry coupled to one or more haptic actuators 262 to cause one or more haptic actuators 262 to vibrate to induce a selected vibratory response into at least a portion of the computing device 102, thereby providing a tactile sensation to a user of computing device 102. Drive electronics 264 may, in response to haptic device 214 receiving an indication of a vibration pattern, drive one or more haptic actuators 262 to vibrate according to the vibration pattern. That is, drive electronics 264 may drive one or more haptic actuators 262 at the frequency and at the vibration intensities associated with the vibration pattern.

[0046] One or more processors 240 may implement functionality and / or execute instructions within computing device 202. For example, processors 240 on computing device 202 may receive and execute instructions stored by storage devices 248 that execute the functionality of event determination system 210, applications 216, haptic feedback mappings 218, and / or operating system 254. These instructions executed by processors 240 may cause haptic device 214 of computing device 202 to output a vibration pattern to produce a haptic effect. These instructions executed by processors 240 may cause computing device 202 to store and / or modify information within storage device 248 or processors 240 during program execution. Processors 240 may execute instructions of event determination system 210, applications 216, haptic feedback mappings 218, and / or operating system 254 to perform one or more operations. That is event determination system 210, applications 216, haptic feedback mappings 218, and / or operating system 254 may be operable by processors 240 to perform various functions described herein.

[0047] One or more storage devices 248 within computing device 202, in the example of FIG. 2, may include event determination system 210, applications 216, haptic feedback mappings 218, and / or operating system (OS) 254. Storage devices 248 may store event determination system 210 in an application layer. Storage devices 248 maystore information for processing during operation of computing device 202 (e.g., computing device 202 may store data accessed by event determination system 210, applications 216, haptic feedback mappings 218, and / or operating system 254 during execution at computing device 202). In some examples, storage device 248 is a temporary memory, meaning that a primary purpose of storage device 248 is not longterm storage. Storage devices 248 on computing device 202 may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art.

[0048] Storage devices 248, in some examples, also include one or more computer- readable storage media. Storage devices 248 may be configured to store larger amounts of information than volatile memory. Storage devices 248 may further be configured for long-term storage of information as non-volatile memory space and retain information after power on / off cycles. Examples of non-volatile memories include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage devices 248 may store program instructions and / or information (e.g., data) associated with event determination system 210 and operating system 254.

[0049] OS 254 may control the operation of components of computing device 202. For example, OS 254 may facilitate the communication of event determination system 210, applications 216, haptic feedback mappings 218, and / or operating system (OS) 254 with processors 240, storage devices 248, UID 204, input devices 242, output devices 246, and communication units 244. In some examples, OS 254 may manage interactions between software applications and a user operating computing device 202. OS 254 may have a kernel that facilitates interactions with underlying hardware of computing device 202 and provides a fully formed application space capable of executing a wide variety of software applications having secure partitions in which each of the software applications executes to perform various operations.

[0050] In accordance with techniques of this disclosure, computing device 202 may produce haptic effects based on a real-time or near real-time event classification of interaction data associated with applications 216. Event determination system 210 ofcomputing device 202 may be provided interaction data, in real-time or near real-time, from a software application of applications 216 during execution of the software application. For example, computing device 202 may provide event determination system 210 with interaction data such as audio signals, video signals, or the like generated during execution of the software application, as well as input data (e.g., data generated by one or more inertial measurement units of input devices 242, tactile inputs detected by a presence-sensitive display of input devices 242, etc.) received via input devices 242 during execution of the software application. For example, computing device 202 may provide, to event determination system 210, data generated by one or more inertial measurement units of input devices 242 that specifies acceleration, orientation, or other movement associated with computing device 202 and / or tactile inputs detected using input devices 242 that specifies user interactions with a presencesensitive display of input devices 242 during execution of a software application. Event determination system 210 may input raw interaction data to segmentation module 222.

[0051] Segmentation module 222 of event determination system 210 may segment signals of the interaction data. Segmentation module 222 may segment signals of the interaction data to classify segments of the interaction data as potentially corresponding to an event. Segmentation module 222 may segment signals of the interaction data using rolling-window analysis of a time-series model. Segmentation module 222 may generate segments of interaction data by applying a sliding window technique to analyze a time series of signals of interaction data by dividing the signals into overlapping windows. For example, segmentation module 222 may generate segments of interaction data by dividing signals of interaction data into overlapping windows of 25 milliseconds. Segmentation module 222 may input the segments of interaction data to featurization module 224.

[0052] Featurization module 224 of event determination system 210 may generate feature vectors for segments of interaction data. Featurization module 224 may include a machine learning model trained to generate feature vectors as arrays or matrices in a high-dimensional space that capture information about interaction data signals, such as spectral envelope, perceptual characteristics, formant structure, timbral characteristics, noise robustness, dynamic changes, temporal resolution, phonetic distinctions, prosodic features, frequency resolution, or the like. For example, featurization module 224 may include a filter bank trained to generate feature vectors that capture features such asMel-scale Frequency Cepstral Coefficients (MFCC), Gammatone Filter Bank features, Constant-Q Transform (CQT) features, Chroma features, Wavelet Transform features, Linear Predictive Coding (LPC) features, Bark Frequency Cepstral Coefficients (BFCCs), Log-Mel Spectrogram features, or the like. Featurization module 224 may apply the machine learning model to the segments of interaction data to output feature vectors that capture features in a matrix (e.g., MFCC coefficients captured as a 12x10 matrix). Featurization module 224 may include one or more machine learning models that have been specifically trained to extract and embed features for various types of interaction data (e.g., audio data, video data, data generated by inertial measurement units, tactile input data, etc.). For example, featurization module 224 may include one or more machine learning models that have been trained to extract features from audio data based on training data that includes labeled samples of audio data, extract features from video data based on training data that includes labeled samples of video data, extract features from inertial measurement unit data based on training data that includes labeled samples of inertial measurement unit data, and / or extract features from tactile input data based on training data that includes labeled samples of tactile input data. Featurization module 224 may input feature vectors for segments of interaction data to classification module 226.

[0053] Classification module 226 of event determination system 210 may classify segments of interaction data as events. Classification module 226 may classify segments of interaction data as events based on feature vectors corresponding to the segments of interaction data. Classification module 226 may include a machine learning model (e.g., a neural network, hidden Markov models, Gaussian mixture models, support vector machines, random forests, etc.) trained to process feature vectors and output predictions corresponding to probabilities that events of a set of predefined events correspond to the processed feature vectors. The machine learning model of classification module 226 may be trained to correspond feature vectors for segments of interaction data to events defined by software application developers that developed the software application associated with the interaction data. The machine learning model of classification module 226 may be trained based on a training dataset that includes labeled training data of ground truth examples specifying mappings of feature vectors to events stored at haptic feedback mappings 218. For example, the machine learning model of classification module 226 may be trained to classify interaction data as eventsbased at least on training data 228 that may include sample interaction data labeled with known events of a plurality of events. The machine learning model of classification module 226 may be trained using the labeled training data 228 by optimizing parameters to minimize a loss function (e.g., cross-entropy loss).

[0054] Classification module 226 may process feature vectors to predict whether the feature vectors correspond to events of a set of predefined events. For example, classification module 226 may apply the pre-trained machine learning model to predict that a feature vector for a segment of interaction data corresponds to an event based on a probability associated with the event satisfying an adjustable threshold. Classification module 226 may apply the pre-trained machine learning model to output a set of probabilities or likelihoods that include predictions specifying whether a feature vector for a segment of interaction data corresponds to each event of the set of predefined events. Classification module 226 may determine whether a probability associated with an event satisfies an adjustable threshold. Based on the probability satisfying the adjustable threshold (e.g., the probability being greater than a predefined value), classification module 226 may output an indication specifying the segment of interaction data corresponds to the event associated with the probability.

[0055] In some examples, classification module 226 may implement a look-up table to determine an event based on segments of interaction data output by segmentation module 222. For instance, segmentation module 222 may output, to classification module 226, a segment of interaction data. Classification module 226 may determine an event based on the segment of interaction data by querying a look-up table using the segment of interaction data. For example, classification module 226 may input data representing the segment of interaction data into a look-up table that may include a relational database mapping events to segments of interaction data. Classification module 226 may determine, based on an output of querying the look-up table, an event corresponding to the segment of interaction data based on the look-up table specifying a mapping or correlations between the event and the segment of interaction data.

[0056] Computing device 202 may retrieve haptic feedback instructions for haptic device 214 to output a haptic feedback pattern associated with the event. Computing device 202 may retrieve haptic feedback instructions based on a mapping of an event output by classification module 226 to the haptic feedback instructions stored as haptic feedback mappings 218. For example, haptic feedback mappings 218 may include arelational database that may output haptic feedback instructions based on an input of an event determined using classification module 226. Haptic feedback mappings 218 may include mappings of haptic feedback instructions to events that are defined by software application developers that developed the software application associated with the interaction data. Computing device 202 may retrieve, using haptic feedback mappings 218, haptic feedback instructions that include a function or API call for a haptic feedback pattern mapped to the event determined by classification module 226. Computing device 202 may determine a haptic effect based on the haptic feedback instructions. Computing device 202 may drive, via communication channels 250, haptic device 214 to output a vibration pattern associated with the determined haptic effect. In some examples, computing device 202 may generate a haptic effect based on the haptic feedback instructions and the interaction data. For example, computing device 202 may generate, in real-time or near real-time, a unique haptic effect based on haptic feedback instructions and interaction data being associated with multiple events. Computing device 202 may store the haptic effect with the haptic feedback instructions.

[0057] FIG. 3 is a conceptual diagram illustrating example computing device 302 for event classification of interaction data, in accordance with aspects of this disclosure. Computing device 302, featurization module 324, and classification module 326 of FIG. 3 may be example or alternative implementations of computing device 202, featurization module 224, and classification module 226 of FIG. 2, respectively.

[0058] Computing device 302 may generate segment of interaction data 332. Segment of interaction data 332 may include a portion of a signal (e.g., an audio signal, a video signal, signals associated with detected inertial measurement units, user input signals, etc.) included in interaction data collected by computing device 302 during execution of a software application. For example, segment of interaction data 332 may include a 25 millisecond snippet of an audio signal included in interaction data collected by computing device 302 during execution of a gaming application. In another example, segment of interaction data 332 may include a 25 millisecond snippet of a video signal included in the interaction data. In yet another example, segment of interaction data 332 may include snippets of data generated by one or more inertial measurement units (IMUs) of computing device 302 during execution of the software application. In yet another example, segment of interaction data 332 may include snippets of input data generated by a presence-sensitive display (e.g., input devices 242 of FIG. 2) in responseto detecting one or more user inputs. Computing device 302 may provide segment of interaction data 332 as an input to featurization module 324.

[0059] Featurization module 324 may process segment of interaction data 332 to generate feature vector 334. Featurization module 324 may apply a machine learning model (e.g., a filter bank) to encode features of segment of interaction data 332 as feature vector 334. Featurization module 324 may apply a machine learning model that has been specifically trained to extract and encode features of segment of interaction data 332 as feature vector 334 based on the type of interaction data included in segment of interaction data 332. For example, featurization module 324 may apply a machine learning model that has been trained to extract and encode features from audio data, video data, inertial measurement unit data, and / or tactile input data based on training data that includes labeled samples of audio data, video data, inertial measurement unit data, and / or tactile input data.

[0060] Feature vector 334 may include an array or a matrix of values representing features extracted using the machine learning model. The values and dimensions of a matrix included in feature vector 334 are based on types of machine learning models or filter banks applied to extract features (e.g., MFCC values, CQT values, Chroma feature values, BFCC values, etc.). In general, featurization module 324 may generate feature vector 334 to represent features (e.g., spectral envelope, perceptual characteristics, formant structure, timbral characteristics, noise robustness, dynamic changes, temporal resolution, phonetic distinctions, prosodic features, frequency resolution, etc.) extracted from segment of interaction data 332. Featurization module 324 may output feature vector 334 to classification module 326.

[0061] Classification module 326 may process feature vector 334 to determine whether segmentation of interaction data 332 may be classified as an event. Classification module 326 may include a machine learning model (e.g., a neural network) trained to predict whether information captured in feature vector 334 corresponds to an event. Classification module 326 may apply the machine learning model to determine predictions 336 specifying a likelihood that feature vector 334 corresponds to events that may be defined by a software application developer of the software application associated with segment of interaction data 332. For example, in the example of FIG. 3, classification module 326 may determine predictions 336 including a first probability of 0.8 that feature vector 334 corresponds to a first event, a second probability of 0.1 thatfeature vector 334 corresponds to a second event, a third probability of 0.1 that feature vector 334 corresponds to a third event, a fourth probability of 0.2 that feature vector 334 corresponds to a fourth event, a fifth probability of 0.1 that feature vector 334 corresponds to a fifth event, a sixth probability of 0.1 that feature vector 334 corresponds to a sixth event, and a seventh probability of 0.2 that feature vector 334 corresponds to a seventh event. Classification module 326 may output predictions 336.

[0062] Computing device 302 may determine whether segment of interaction data 332 corresponds to an event based on predictions 336. Computing device 302 may apply an adjustable threshold to predictions 336 to determine whether segment of interaction data 332 corresponds to an event. For example, computing device 302 may set a threshold of 0.7 and compare each probability value of predictions 336 to the threshold. In the example of FIG. 3, computing device 302 may determine segment of interaction data 332 corresponds to the first event as a result of the first probability of 0.8 satisfying the threshold of 0.7. Alternatively, or in addition, computing device 302 may determine segment of interaction data 332 corresponds to an event associated with the greatest probability value of predictions 336. Computing device 302 may retrieve haptic feedback instructions based on a mapping of the haptic feedback instructions to the event segment of interaction data has been classified as. Computing device 302 may output a haptic feedback pattern based on the retrieved haptic feedback instructions.

[0063] FIG. 4 is a conceptual diagram illustrating example computing device 402 for classification of interaction data 430, in accordance with aspects of this disclosure. Computing device 402, segmentation module 422, featurization module 424, and classification module 426 of FIG. 4 may be example or alternative implementations of computing device 202, segmentation module 222, featurization module 224, and classification module 226 of FIG. 2, respectively. Segment of interaction data 432A- 432D (collectively referred to herein as “segments of interaction data 432), feature vector 434A-434D (collectively referred to herein as “feature vectors 434”), and prediction 436A-436D (collectively referred to herein as “predictions 436”) of FIG. 4 may be example or alternative implementations of segment of interaction data 332, feature vector 334, and prediction 336 of FIG. 3, respectively.

[0064] Computing device 402 may collect interaction data 430 during execution of a software application (e.g., any one of applications 216 of FIG. 2). Computing device 402 may store interaction data 430 in a buffer. In some instances, interaction data 430may include signals (e.g., audio signals, video signals, data generated by one or more inertial measurement units, input data generated in response to detecting one or more user inputs, etc.) generated and output by computing device 402 during execution of the software application. In some examples, interaction data 430 may include signals (e.g., touch, force, voice, acceleration of computing device 402, orientation of computing device 402, tactical inputs, etc.) input by a user operating computing device 402. In the example of FIG. 4, interaction data 430 may include an audio waveform signal generated and output by computing device 402 during execution of a software application. Interaction data 430, in the example of FIG. 4, may include an audio waveform signal representing 1 millisecond of audio generated and output during execution of the software application. Computing device 402 may input interaction data 430 into segmentation module 422.

[0065] Segmentation module 422 may generate segments of interaction data 432 based on interaction data 430. Segmentation module 422 may apply time-series techniques with a rolling window to generate segments of interaction data 432. For example, in instances where interaction data 430 includes an audio waveform corresponding to 1 millisecond of audio generated and output during execution of a software application, segmentation module 422 may divide the audio waveform into overlapping windows of 0.4 milliseconds with 0.2 millisecond steps. Based on the time-series technique applied by segmentation module 422, segmentation module 422 may generate segment of interaction data 432A to include data corresponding to the 0 millisecond to 0.4 millisecond frame of the audio waveform included in interaction data 430.Segmentation module 422 may generate segment of interaction data 432B to include data corresponding to the 0.2 millisecond to 0.6 millisecond frame of the audio waveform included in interaction data 430. Segmentation module 422 may generate segment of interaction data 432C to include data corresponding to the 0.4 millisecond to 0.8 millisecond frame of the audio waveform included in interaction data 430. Segmentation module 422 may generate segment of interaction data 432D to include data corresponding to the 0.6 millisecond to 1 millisecond frame of the audio waveform included in interaction data 430. Segmentation module 422 may input segments of interaction data 432 to featurization module 424.

[0066] Featurization module 424 may generate feature vectors 434 based on segments of interaction data 432. Featurization module 424 may apply a machine learning modelor filter bank to extract features from segments of interaction data 432 and encode the features as feature vectors 434. Featurization module 424 may apply a machine learning model or filter bank that has been specifically trained to encode features of interaction data 432 as feature vectors 434 based on the type of interaction data (e.g., audio data, video data, inertial measurement unit data, tactile input data, etc.) included in interaction data 432. In the example of FIG. 4, featurization module 424 may generate feature vector 434A that captures features of segment of interaction data 432A, feature vector 434B that captures features of segment of interaction data 432B, feature vector 434C that captures features of segment of interaction data 432C, and feature vector 434D that captures features of segment of interaction data 432D. Featurization module 424 may output feature vectors 434 to classification module 426.

[0067] Classification module 426 may process feature vectors 434 to determine predictions 436 associated with likelihoods that segments of interaction data 432 correspond to an event of a set of predefined events. Classification module 426 may include a machine learning model (e.g., neural network) trained to analyze feature vectors 434 to determine whether features captured by feature vectors 434 correspond to events of the set of predefined events. Classification module 426 may apply the machine learning model to generate predictions 436. Predictions 436 may each include a set of probabilities that a feature vector of feature vectors 434 correspond to each event of the set of predefined events. In the example of FIG. 4, classification module 426 may generate prediction 436A for feature vector 434A, prediction 436B for feature vector 434B, prediction 436C, for feature vector 434C, and prediction 436D for feature vector 434D.

[0068] Computing device 402 may determine whether any one of predictions 436 indicate that any one of segments of interaction data 432 correspond to an event of the set of predefined events. Computing device 402 may implement a probability threshold to determine whether a segment of interaction data corresponds to an event based on predictions 436. Computing device 402 may implement the probability threshold by determining whether a probability associated with an event in predictions 436 satisfies the probability threshold. For example, computing device 402 may determine a first probability associated with a first event in predictions 436A, 436B, and 436C satisfies the probability threshold, and determine a second probability associated with a second event in prediction 436D satisfies the probability threshold.

[0069] Computing device 402 may store the first event and second event as events 438. Events 438 may include the first event with a label indicating the time frames associated with segment of interaction data 432A, 432B, and 432C, and the second event with a label indicating the time frame associated with segment of interaction data 432D. Computing device 402 may retrieve haptic feedback instructions for each event of events 438 based on a mapping of haptic feedback instructions to the predefined set of events. Computing device 402 may output, based on haptic feedback instructions mapped to the first event, a first haptic feedback pattern for a period of time associated with the label indicating the time frames associated with segment of interaction data 432A, 432B, and 432C. Computing device 402 may output, based on haptic feedback instructions mapped to the second event, a second haptic feedback pattern for a period of time associated with the label indicating the time frame associated with segment of interaction data 432D.

[0070] FIG. 5 is a conceptual diagram illustrating processing and synchronization of haptic feedback patterns output based on interaction data, in accordance with aspects of this disclosure. FIG. 5 may be discussed with respect to FIG. 1 for example purposes only.

[0071] In the example of FIG. 5, a first time period “A” may correspond to a period of time between when computing device 102 generates or detects input and / or output of interaction data (e.g., interaction data 430 of FIG. 4). For example, the first time period “A” may be the time between input of digital audio data to an access controller of computing device 102 and arrival of analog audio data at a speaker of computing device 102 (e.g., output device 246 of FIG. 2).

[0072] A second time period “B” may correspond to a period of time of tolerable latency. For example, the second time period “B” may be an amount of time (e.g., 50 milliseconds) that a user operating computing device 102 may not notice a delay between an input or output of interaction data and an output of a haptic feedback pattern (e.g., haptic feedback 120).

[0073] A third time period “C” may correspond to a period of time a haptic feedback pattern (e.g., haptic feedback 120) is output after event classification of interaction data. For example, the third time period “C” may be the time computing device 102 takes to determine an event corresponds to a segment of interaction data, retrieve hapticfeedback instructions mapped to the determined event, and output the haptic feedback pattern based on the retrieved haptic feedback instructions.

[0074] Computing device 102 may, in the example of FIG. 5, limit an amount of time, labeled as “Available Inference Time,” event determination system 110 may have to output an event classification of interaction data. Computing device 102 may define the “Available Inference Time” as the sum of the first time period “A” and the second time period “B,” minus the third time period “C ” Event determination system 110 may include lightweight machine learning models to classify interaction data as events based on the time defined by the “Available Inference Time.” For example, event determination system 110 may include, based on the time defined by the “Available Inference Time, various machine learning models to convert and optimize model files, preprocess raw interaction data, and / or accelerate model inference time. Event determination system 110 may streamline event classification analysis of interaction data based on the time period defined as “Available Inference Time,” to synchronize the output of haptic feedback patterns for corresponding segments of interaction data according to a tolerable latency defined by the second time period “B.” In this way, computing device 102 may output haptic feedback patterns for event classification with low latency between interaction data input or output and haptic feedback output.

[0075] FIG. 6 is a conceptual diagram illustrating streaming and segmentation of interaction data for event classification, in accordance with aspects of this disclosure. FIG. 6 may be discussed with respect to FIG. 2 for example purposes only.

[0076] In the example of FIG. 6, interaction data may include digital audio data with audio waveforms. Event determination system 210 may collect the interaction data from a software application of applications 216 during execution of the software application. Event determination system 210 may stream the digital audio data of the interaction data through segmentation module 222. Segmentation module 222 may continuously segment or divide the digital audio data of the interaction data into segments of audio waveforms with a step size of 25 milliseconds. For example, segmentation module 222 may segment the digital audio data of the interaction data to generate a first interaction data segment “Seg 1 Segmentation” that includes audio waveforms of the digital audio data within the 0 millisecond to 25 millisecond time frame. Segmentation module 222 may input the first interaction data segment “Seg 1 Segmentation” into featurization module 224.

[0077] Featurization module 224 may process the first interaction data segment “Seg 1 Segmentation” to generate a first feature vector “Featurize Seg 1.” In the example of FIG. 6, featurization module 224 may have a featurization latency of 10 milliseconds when generating the first feature vector “Featurize Seg 1.” Featurization module 224 may input the first feature vector “Featurize Seg 1” into classification module 226. Classification module 226 may process the first feature vector “Featurize Seg 1” to generate a first prediction “Predict Feature 1.” In the example of FIG. 6, classification module 226 may have a machine learning model latency of 15 milliseconds. Computing device 202 may retrieve haptic feedback instructions for an event based on the first prediction “Predict Feature 1” (e.g., a probability associated with the event in the first prediction satisfies a threshold). Computing device 202 may output a haptic feedback pattern based on the retrieved haptic feedback instructions.

[0078] While featurization module 224 generates the first feature vector “Featurize Seg 1” and classification module 226 generates the first prediction “Predict Feature 1,” segmentation module 222 may create a second segment of interaction data “Seg 2 Segmentation” as a result of the streaming of the digital audio data of the interaction data. In this way, segmentation module 222 may be configured to segment interaction data according to a time window determined based on latency values associated with featurization by featurization module 224 and classification by classification module 226. Segmentation module 222 may create a second segment of interaction data “Seg 2 Segmentation” that includes audio waveforms within a 25 millisecond time frame of the digital audio data of the interaction data. Segmentation module 222 may input the second segment of interaction data “Seg 2 Segmentation” to featurization module 224. Featurization module 224 may generate, based on the second segment of interaction data “Seg 2 Segmentation,” a second feature vector “Featurize Seg 2” and input the second feature vector into classification module 226. Classification module may generate, based on the second feature vector “Featurize Seg 2,” a second prediction “Predict Feature 2” to potentially classify the second segmentation of interaction data “Seg 2 Segmentation” as an event mapped to a haptic feedback instructions stored at haptic feedback mappings 218. Segmentation module 222, featurization module 224, and classification module 226 may continue to segment interaction data, generate feature vectors, and generate predictions as discussed above until the stream of digitalaudio data has ended or the software application associated with the interaction data has been closed.

[0079] FIG. 7 is a conceptual diagram illustrating featurization of segments of interaction data for event classification, in accordance with aspects of this disclosure. Feature vector 734A, 734B (collectively referred to herein as “feature vectors 734”) of FIG. 7 may be example or alternative implementations of feature vector 334 of FIG. 3. FIG. 7 may be discussed with respect to FIG. 2 for example purposes only.

[0080] Featurization module 224 may include a filter bank configured to extract features of segments of interaction data as Mel Frequency Cepstrum Coefficients (MFCCs). For example, featurization module 224 may input 250 milliseconds of interaction data (e.g., audio data, video data, inertial measurement unit data, tactile input data, etc.) into one or more Mel filter banks. Featurization module 224 may apply the Mel filter banks to extract features from segments of the interaction data and encode the features as feature vectors 734. Featurization module 224 may generate feature vectors 734 as a 12x10 matrix representing MFCC values for each segment of interaction data included in segments of interaction data 732N, as illustrated in FIG. 7. For example, featurization module 224 may generate, for a first segment of interaction data, feature vector 734A as a first portion of a 12x10 matrix representing MFCC values of the interaction data. Featurization module 224 may generate, for a second segment of interaction data, feature vector 734B as a second portion of a 12x10 matrix representing MFCC values of the interaction data Featurization module 224 may provide feature vectors 734 to classification module 226 for event classification of segments of interaction data 732N, as previously discussed.

[0081] FIG. 8 is a flowchart illustrating example operations of an example computing device configured to output a haptic feedback pattern based on interaction data, in accordance with one or more aspects of the present disclosure. FIG. 8 may be discussed with respect to FIG. 1 for example purposes only.

[0082] Computing device 102 may obtain interaction data associated with input data or output data during execution of a software application (802). Computing device 102 may continuously obtain interaction data during execution of the software application (e.g., a gaming application). Computing device 102 may provide the interaction data to event determination system 110 (804). For example, computing device 102 maycontinuously stream interaction data to event determination system 110 during execution of the software application.

[0083] Computing device 102 may determine, using event determination system 110 and based on the interaction data, an event from a plurality of events (806). Computing device 102 may apply event determination system 110 to generate one or more feature vectors based on segmentations of the interaction data. Computing device 102 may apply event determination system 110 to determine probabilities the one or more feature vectors correspond to each event of the plurality of events. Computing device 102 may apply event determination system 110 to output an indication of the event based on the probabilities. Computing device 102 may retrieve haptic feedback instructions associated with the event (808). Computing device 102 may output the haptic feedback instructions (810). For example, computing device 102 may obtain the haptic feedback instructions from a plurality of haptic feedback instructions based on haptic feedback mappings 118 including a mapping of the event to the haptic feedback instructions.

[0084] This disclosure includes the following examples:

[0085] Example 1 : A method includes obtaining, by one or more processors, interaction data associated with input data or output data during execution of a software application; providing, by the one or more processors, the interaction data to system; determining, using the system and based on the interaction data, an event from a plurality of events; retrieving, by the one or more processors, haptic feedback instructions associated with the event; and outputting, by the one or more processors, the haptic feedback instructions.

[0086] Example 2: The method of example 1, wherein providing the interaction data to the event determination system comprises: continuously obtaining the interaction data during execution of the software application; segmenting the interaction data into segmentations of the interaction data based on a time window; and inputting the segmentations of the interaction data to a machine learning model of the event determination system.

[0087] Example 3: The method of any of examples 1-2, wherein determining the event comprises: generating, by the event determination system, one or more feature vectors based on segmentations of the interaction data; determining, by the event determination system, probabilities the one or more feature vectors correspond to each event of theplurality of events; and outputting, by the event determination system, an indication of the event based on the probabilities.

[0088] Example 4: The method of any of examples 1-3, wherein the interaction data includes at least one of: audio data output during execution of the software application, video data output during execution of the software application, data generated by one or more inertial measurement units during execution of the software application, or input data generated by a presence-sensitive display in response to detecting one or more user inputs.

[0089] Example 5: The method of any of examples 1-4, wherein retrieving the haptic feedback instructions associated with the event comprises: obtaining the haptic feedback instructions from a plurality of haptic feedback instructions based on a mapping of the event to the haptic feedback instructions.

[0090] Example 6: The method of any of examples 1-5, wherein outputting the haptic feedback instructions comprises: determining a haptic effect based on the haptic feedback instructions; and driving a haptic device of a user device executing the software application to output a vibration pattern associated with the haptic effect.

[0091] Example 7: The method of any of examples 1-6, wherein the software application is a gaming software application.

[0092] Example 8: The method of any of examples 1-7, further includes generating a haptic effect based on the haptic feedback instructions and the interaction data; and storing the haptic effect with the haptic feedback instructions.

[0093] Example 9: The method of any of examples 1-8, further includes training a machine learning model of the event determination system to determine the event based at least on training data of sample interaction data labeled with known events of the plurality of events.

[0094] Example 10: A computing device includes at least one processor; a haptic device; and a storage device that stores instructions executable by the at least one processor to: obtain interaction data associated with input data or output data during execution of a software application; provide the interaction data to an event determination system; determine, using the event determination system and based on the interaction data, an event from a plurality of events; retrieve haptic feedback instructions associated with the event; and output the haptic feedback instructions.

[0095] Example 11 : The computing device of example 10, wherein to provide the interaction data to the event determination system, the storage device stores instructions executable by the at least one processor to: continuously obtain the interaction data during execution of the software application; segment the interaction data into segmentations of the interaction data based on a time window; and input the segmentations of the interaction data to a machine learning model of the event determination system.

[0096] Example 12: The computing device of any of examples 10 and 11, wherein to determine the event, the storage device stores instructions executable by the at least one processor to: generate, by the event determination system, one or more feature vectors based on segmentations of the interaction data; predict, by the event determination system, probabilities the one or more feature vectors correspond to each event of the plurality of events; and output, by the event determination system, an indication of the event based on the probabilities.

[0097] Example 13: The computing device of any of examples 10 through 12, wherein the interaction data includes at least one of: audio data output during execution of the software application, video data output during execution of the software application, inertial measurement units, or input data generated by a presence-sensitive display in response to detecting one or more user inputs.

[0098] Example 14: The computing device of any of examples 10 through 13, wherein to retrieve the haptic feedback instructions, the storage device stores instructions executable by the at least one processor to: obtain the haptic feedback instructions from a plurality of haptic feedback instructions based on a mapping of the event to the haptic feedback instructions.

[0099] Example 15: The computing device of any of examples 10 through 14, wherein to output the haptic feedback instructions, the storage device stores instructions executable by the at least one processor to: determine a haptic effect based on the haptic feedback instructions; and drive the haptic device to output a vibration pattern associated with the haptic effect.

[0100] Example 16: The computing device of any of examples 10 through 15, wherein the software application is a gaming software application.

[0101] Example 17: The computing device of any of examples 10 through 16, wherein the storage device further stores instructions executable by the at least one processor to:generate a haptic effect based on the haptic feedback instructions and the interaction data; and store the haptic effect with the haptic feedback instructions.

[0102] Example 18: The computing device of any of examples 10 through 17, wherein the storage device further stores instructions executable by the at least one processor to: train a machine learning model of the event determination system to determine the event based at least on training data of sample interaction data labeled with known events of the plurality of events.

[0103] Example 19: Computer-readable storage medium storing instructions that, when executed, cause at least one processor of a computing device to: obtain interaction data associated with input data or output data during execution of a software application; provide the interaction data to an event determination system; determine, using the event determination system and based on the interaction data, an event from a plurality of events; retrieve haptic feedback instructions associated with the event; and output the haptic feedback instructions.

[0104] Example 20: The computer-readable storage medium of example 19, wherein to provide the interaction data to the event determination system, the instructions cause the at least one processor of the computing device to: continuously obtain the interaction data during execution of the software application; segment the interaction data into segmentations of the interaction data based on a time window; and input the segmentations of the interaction data to a machine learning model of the event determination system.

[0105] Example 21 : The computer-readable storage medium of any of examples 19 and 20, wherein to determine the event, the instructions cause the at least one processor of the computing device to: generate, by the event determination system, one or more feature vectors based on segmentations of the interaction data; predict, by the event determination system, probabilities the one or more feature vectors correspond to each event of the plurality of events; and output, by the event determination system, an indication of the event based on the probabilities.

[0106] Example 22: The computer-readable storage medium of any of examples 19 through 21, wherein the interaction data includes at least one of: audio data output during execution of the software application, video data output during execution of the software application, inertial measurement units, or input data generated by a presencesensitive display in response to detecting one or more user inputs.

[0107] Example 23: The computer-readable storage medium of any of examples 19 through 22, wherein to retrieve the haptic feedback instructions, the instructions cause the at least one processor of the computing device to: obtain the haptic feedback instructions from a plurality of haptic feedback instructions based on a mapping of the event to the haptic feedback instructions.

[0108] Example 24: The computer-readable storage medium of any of examples 19 through 23, wherein to output the haptic feedback instructions, the instructions cause the at least one processor of the computing device to: determine a haptic effect based on the haptic feedback instructions; and drive a haptic device of the computing device to output a vibration pattern associated with the haptic effect.

[0109] Example 25: The computer-readable storage medium of any of examples 19 through 24, wherein the software application is a gaming software application.

[0110] Example 26: The computer-readable storage medium of any of examples 19 through 25, wherein the instructions further cause the at least one processor of the computing device to: generate a haptic effect based on the haptic feedback instructions and the interaction data; and store the haptic effect with the haptic feedback instructions.

[0111] Example 27: The computer-readable storage medium of any of examples 19 through 26, wherein the instructions further cause the at least one processor of the computing device to: train a machine learning model of the event determination system to determine the event based at least on training data of sample interaction data labeled with known events of the plurality of events.

[0112] Example 28: A computing system comprising means for performing any of the methods of examples 1-9.

[0113] Example 29: A computer program product, the computer program product comprising at least one computer-readable storage medium encoded with instructions that cause one or more processors of a computing device to perform any of the methods of examples 1-9.

[0114] Example 30: A computer program product, the computer program product comprising at least one computer-readable storage medium encoded with instructions that cause one or more processors of a computing system to perform any of the methods of examples 1-9.

[0115] Example 31 : Computer-readable storage medium encoded with instructions that cause one or more processors of a computing system to perform any of the methods ofexamples 1-9. In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over, as one or more instructions or code, a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer- readable media generally may correspond to (1) tangible computer-readable storage media, which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0116] By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0117] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specificintegrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0118] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0119] Various examples of the disclosure have been described. Any combination of the described systems, operations, or functions is contemplated. These and other examples are within the scope of the following claims.

Claims

CLAIMS:

1. A method comprising: obtaining, by one or more processors, interaction data associated with input data or output data during execution of a software application; providing, by the one or more processors, the interaction data to an event determination system; determining, using the event determination system and based on the interaction data, an event from a plurality of events; retrieving, by the one or more processors, haptic feedback instructions associated with the event; and outputting, by the one or more processors, the haptic feedback instructions.

2. The method of claim 1, wherein providing the interaction data to the event determination system comprises: continuously obtaining the interaction data during execution of the software application; segmenting the interaction data into segmentations of the interaction data based on a time window; and inputting the segmentations of the interaction data to a machine learning model of the event determination system.

3. The method of any of claims 1-2, wherein determining the event comprises: generating, by the event determination system, one or more feature vectors based on segmentations of the interaction data; determining, by the event determination system, probabilities the one or more feature vectors correspond to each event of the plurality of events; and outputting, by the event determination system, an indication of the event based on the probabilities.

4. The method of any of claims 1-3, wherein the interaction data includes at least one of: audio data output during execution of the software application, video data output during execution of the software application, data generated by one or more inertial measurement units during execution of the software application, or input datagenerated by a presence-sensitive display in response to detecting one or more user inputs.

5. The method of any of claims 1-4, wherein retrieving the haptic feedback instructions associated with the event comprises: obtaining the haptic feedback instructions from a plurality of haptic feedback instructions based on a mapping of the event to the haptic feedback instructions.

6. The method of any of claims 1-5, wherein outputting the haptic feedback instructions comprises: determining a haptic effect based on the haptic feedback instructions; and driving a haptic device of a user device executing the software application to output a vibration pattern associated with the haptic effect.

7. The method of any of claims 1-6, wherein the software application is a gaming software application.

8. The method of any of claims 1-7, further comprising: generating a haptic effect based on the haptic feedback instructions and the interaction data; and storing the haptic effect with the haptic feedback instructions.

9. The method of any of claims 1-8, further comprising: training a machine learning model of the event determination system to determine the event based at least on training data of sample interaction data labeled with known events of the plurality of events.

10. A computing device comprising means for performing any of the methods of claims 1-9.

11. A computing system comprising means for performing any of the methods of claims 1-9.

12. A computer program product, the computer program product comprising instructions that cause one or more processors of a computing device to perform any of the methods of claims 1-9.

13. Computer-readable storage medium encoded with instructions that cause one or more processors of a computing system to perform any of the methods of claims 1-9.

Images