Adjusting haptic rendering based on context awareness

Abstract

Systems, apparatus, processes, and computer-readable media for providing haptic feedback in an electronic device based on context awareness are disclosed. According to some aspects, a method of controlling haptic feedback includes obtaining environmental information indicative of an environment of a computing device from at least one sensor, and determining a context associated with the mobile device based on the environmental information. The method may include determining haptic feedback for output by the mobile device based on the context.

Description

Technical Field

[0001] The present disclosure generally relates to tactile rendering or feedback for electronic devices. For example, aspects of the present disclosure relate to systems and techniques for providing adjustments to tactile rendering or feedback for electronic devices based on contextual awareness. Background Art

[0002] Multimedia systems are widely deployed to provide various types of multimedia communication content, such as voice, video, packet data, messaging, broadcasting, etc. These multimedia systems are capable of processing, storing, generating, manipulating, and reproducing multimedia information. Examples of multimedia systems include mobile devices, gaming devices, entertainment systems, information systems, virtual reality systems, models, and simulation systems, etc. These systems can use a combination of hardware and software technologies to support the processing, storage, generation, manipulation, and reproduction of multimedia information, such as client devices, capture devices, storage devices, communication networks, computer systems, and display devices. In some cases, these devices can be configured with tactile actuators or other mechanisms to provide tactile feedback to users when using applications, such as alarms, notifications, or user feedback. The device can also be configured with additional tactile actuators to improve the user experience. The multimedia system can also be configured with additional tactile feedback to improve the user experience. Summary of the invention

[0003] Systems and techniques for controlling (e.g., adjusting) tactile feedback in an electronic device are described. In some aspects, these systems and techniques can control tactile feedback in an electronic device based on contextual awareness. According to at least one example, a method for controlling tactile feedback is provided. The method includes: obtaining environmental information indicating an environment of a computing device from at least one sensor; determining a context associated with the computing device based on the environmental information; and determining tactile feedback for output by the computing device based on the context.

[0004] In another example, an apparatus for controlling tactile feedback is provided, comprising at least one memory and at least one processor coupled to the at least one memory (e.g., configured in a circuit). The at least one processor is configured to: obtain environmental information indicating an environment of a computing device from at least one sensor; determine a context associated with the computing device based on the environmental information; and determine tactile feedback to be output by the computing device based on the context.

[0005] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: obtain environmental information indicating an environment of a computing device from at least one sensor; determine a context associated with the computing device based on the environmental information; and determine tactile feedback to be output by the computing device based on the context.

[0006] In another example, an apparatus for controlling tactile feedback is provided. The apparatus includes: a component for obtaining environmental information indicating an environment of a computing device from at least one sensor; a component for determining a context associated with the computing device based on the environmental information; and a component for determining tactile feedback to be output by the computing device based on the context.

[0007] In some aspects, the apparatus is a wearable device, an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted device (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smartphone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a combination thereof, is a part of, and / or includes a combination thereof. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0008] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0009] The foregoing and other features and aspects will become more fully apparent by reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0011] Figure 1 FIG. 1 illustrates a block diagram of an electronic device configured to determine the context of an environment and adjust haptic feedback according to some aspects of the present disclosure.

[0012] Figure 2 is a conceptual diagram illustrating an example of an environment for an electronic device to determine context and adjust haptic feedback according to some aspects of the present disclosure.

[0013] Figure 3 An overhead view of an example environment is shown in which an electronic device can determine different contexts and adjust haptic feedback according to some aspects of the present disclosure.

[0014] Figure 4 A sequence diagram for training an AI model using a combination of on-device learning techniques and aggregation-based learning in accordance with some aspects of the present disclosure is shown.

[0015] Figure 5 is a three-dimensional (3D) graph illustrating various data points provided by an electronic device that may be grouped to correspond to a context according to some aspects of the present disclosure.

[0016] Figure 6 is a block diagram illustrating a haptic driver for dynamically adapting a drive duty cycle of a haptic device according to some aspects of the present disclosure.

[0017] Figure 7 is a flow chart illustrating an example of a method for controlling haptic feedback based on contextual awareness in accordance with certain aspects of the present disclosure.

[0018] Figure 8 is an illustrative example of a deep learning neural network that can be used to implement machine learning based alignment prediction according to aspects of the present disclosure.

[0019] Fig. 9 is an illustrative example of a convolutional neural network (CNN) in accordance with aspects of the present disclosure.

[0020] Fig.10 is a diagram illustrating an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0021] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of various aspects of the application. However, it is obvious that various aspects can be practiced without these specific details. The drawings and description are not intended to be restrictive.

[0022] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of the example aspects will provide an enabling description for implementing the example aspects for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0023] The following description provides only exemplary aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of exemplary aspects will provide an enabling description for implementing aspects of the present disclosure to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0024] The terms "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term "aspects of the disclosure" does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

[0025] Different types of computing devices may include tactile feedback devices. Tactile refers to the use of technology that stimulates the sense of touch and movement (e.g., the study or use of tactile sensations and the sense of touch) as a method of interacting with computers and electronic devices. For example, wireless gaming systems, mobile devices including cellular communication modules, game controllers, musical instruments, wearable devices, and head-mounted devices (HMDs) may include tactile actuators that are selectively engaged to vibrate the corresponding devices. In another illustrative example, a computing device may activate a tactile actuator to cause the device to provide physical vibrations and provide notifications or other feedback. In an illustrative example, a mobile communication device may receive messages (e.g., emails, text messages, etc.) and be configured to provide auditory notifications and / or physical notifications based on vibrations associated with tactile feedback. Some applications (such as games) may cause a computing device to periodically activate a tactile actuator to improve the user experience.

[0026] In one illustrative example, some electronic devices (e.g., game controllers, etc.) include an eccentric rotating mass (ERM) vibration motor to produce vibrations or rumbles, such as to indicate that a player has run off the road in a virtual racing game, to indicate that a player has been hit in a first-person shooter game, etc. Some computing devices and other consumer electronic devices may use linear resonant actuators (LRAs) for tactile feedback. LRAs are electromechanical devices that vibrate to provide tactile feedback by driving a coil that moves a spring-mounted magnetic block back and forth to produce vibrations. Similar to a speaker, the input to the LRA is a sine wave that "drives" the mass so that it vibrates. The amplitude of the vibration (the displacement of the mass) can be affected by the peak-to-peak input voltage of the sine wave.

[0027] In some cases, haptic actuators are intended to provide silent notifications rather than auditory notifications. However, the haptic feedback function is not silent, and a portion of the energy is transferred to acoustic energy emitted from the device. For example, a person playing a game with a controller may receive continuous haptic feedback events associated with events in the application that engage the haptic actuator. In some cases, the haptic actuator is an audible beep that can be audibly perceived by others nearby, especially in low ambient noise environments such as libraries. The continuous audible beep from the haptic actuator may be distracting and may ruin the enjoyment of the low ambient noise environment.

[0028] Applications can be configured to selectively enable or disable haptic feedback using various mechanisms. For example, a game may have an in-game setting that disables haptic feedback, while some applications such as messaging applications may have settings configured from a central location (such as the settings application of the operating system). Although users can control various settings, such as a do not disturb mode that silences many notifications, notifications can still be output via haptic feedback. When a person transitions between different types of spaces (e.g., a shared transportation space to a library), the person may forget to reconfigure the haptic feedback settings, and based on the context of the current environment, the current settings may be disruptive. It is inconvenient for users to manually adjust settings associated with applications based on the environment because changing settings (e.g., haptic feedback settings) requires time and effort.

[0029] As an illustrated example, a person may be located in a low ambient noise environment, such as a library (e.g., with approximately 40 decibels (dB) of ambient noise), and may begin playing a game with a haptic feedback intensity setting. If the user is also wearing headphones to listen to the audio of the game or audio from another source, the user may not be aware of the amount of noise emitted by the haptic feedback while playing the game. Because the noise emitted by the haptic feedback (which is a beeping sound) is repetitive and emitted on an irregular schedule, other people near the user may be distracted by the haptic feedback. In other scenarios, the intensity of the haptic feedback may be increased based on context to increase immersion.

[0030] In some aspects, systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for context-aware tactile feedback control are described herein. For example, an electronic device including one or more mechanisms for providing tactile feedback can obtain environmental information indicating an environment of a computing device. For example, the environmental information can be obtained from at least one sensor of the electronic device and / or from at least one sensor of another device. The environmental information can include various types of information, such as audio of the environment (e.g., obtained via a microphone), images of the environment (e.g., obtained via one or more image sensors or cameras), brightness from the environment (e.g., obtained via one or more light sensors, image sensors, or cameras, etc.), wireless information associated with one or more objects or devices (e.g., radio wireless communications indicating the presence of one or more nearby people with a wireless device), and the like.

[0031] The electronic device may use environmental information to understand the environment. For example, the electronic device may determine a context associated with the computing device based on the environmental information, and may determine tactile feedback of the computing device based on the context. In some aspects, the electronic device may include an artificial intelligence (AI) model (e.g., one or more machine learning models, such as one or more neural networks) that is configured or trained to understand the context of the computing device based on the environmental information.

[0032] In one illustrative example, the electronic device may be configured to determine an amount of noise associated with tactile feedback that an application will emit during execution of the application. For example, based on the identified context associated with the amount of noise, the electronic device may determine that the tactile feedback of the application may exceed a noise ceiling appropriate for the context of the electronic device, and may reduce the tactile feedback based on the application. In another illustrative example, the electronic device may identify a shared experience with an audience, and may increase tactile feedback to increase the audience's participation in the experience.

[0033] Additional details and aspects of the disclosure are described in more detail below with reference to the accompanying drawings.

[0034] Figure 1A block diagram of an electronic device 100 configured to determine the context of an environment and adjust tactile feedback according to some aspects of the present disclosure is shown. The electronic device 100 includes a processor 105 connected to a memory 110 storing various data and applications. In one illustrative aspect, the memory 110 stores an environment detection engine 112 and profile data 114. The environment detection engine 112 may include one or more components to implement tactile feedback adjustment. In an illustrative example, the environment detection engine 112 may include software instructions that can be executed by the processor 105. The software instructions can be implemented to determine the ambient noise level and control tactile feedback based on the ambient noise. In some aspects, the environment detection engine 112 may include or implement an artificial intelligence (AI) model, such as a machine learning (ML) model that performs predictive analysis and other learning analysis. Examples of learning analysis include contextual speech recognition that identifies the context of speech (e.g., whispers, presentations, meetings, etc.). The processor 105 is configured to control the communication engine 115 to send and receive data from an external source. An example of an external source is a cloud system 120 configured to send and receive data to and from the electronic device 110. In one illustrative example, the cloud system 120 is configured to receive anonymized data from the electronic device 100 to train an AI model (eg, an AI or ML model of the environment detection engine 112 ).

[0035] In another illustrative aspect, the environment detection engine 112 can be configured to detect various spectral qualities that can indicate an environment. Non-limiting examples of spectral qualities include harmonic information and noise type. In one example, the noise type (e.g., gray noise, brown noise, pink noise, white noise, etc.) can indicate properties associated with the environment. For example, gray noise includes random noise with a spectrum that follows loudness contours and can correspond to noise associated with air conditioning, fans, etc. Gray noise can indicate the lack of human speech.

[0036] The processor 105 is also connected to a haptic driver 125, which is configured to actuate and control a haptic actuator 130. Figure 6The example haptic driver 125 is further configured. In one illustrative aspect, the electronic device 100 includes one or more sensors 131 configured to detect an environment 135 of the computing device. Non-limiting examples of the sensor 131 include an image sensor, a photoelectric sensor configured to detect the brightness of the environment, a microphone configured to record auditory information, an infrared (IR) sensor, a light ranging and detection (LIDAR) sensor, an ultrasonic sensor, an atmospheric sensor, a pressure sensor, and a position sensor. Other sensors may be included, and sensors may be omitted based on the primary function of the electronic device 100. As illustrative examples, the electronic device 100 may be a gaming system including a microphone and a photoelectric sensor, or the electronic device 100 may be a mobile wireless communication device for voice or text communication and include each sensor.

[0037] In some aspects, the sensor 131 is configured to obtain different types of environmental data and provide the environmental data to the processor 105. The processor 105 is also configured to be able to detect environmental data based on the communication engine 115. For example, the processor 105 can be configured to control various wireless engines or components to perform functions to obtain wireless neighbor information from nearby wireless devices. An illustrative example of obtaining wireless neighbor information includes transmitting a beacon (such as a wireless probe (e.g., a WiFi 802.11k probe)) to receive neighbor information. The neighbor information can include various qualities of the device, such as available modes, access point information, wireless communication capabilities, and other relevant information.

[0038] The processor 105 is configured to pre-process the environmental data and input the environmental data into the environment detection engine 112. In one aspect, the environment detection engine 112 is configured to analyze the environmental data to output information identifying the context of the electronic device 100. For example, the environment detection engine 112 can identify that the environment 135 is a low ambient noise environment, such as a library with an ambient noise level of approximately 40 decibels (dB). In another example, the environment detection engine 112 identifies that the environment 135 is a high ambient noise environment, such as a restaurant with an ambient noise level of approximately 70 dB.

[0039] In some aspects, the AI ​​model is configured to distinguish combinations of data sources and learn weights to apply to different layers based on various environmental data to output information that identifies the context of the electronic device 100. In some aspects, the environmental data includes suggestion data that may reveal relevant information such as brightness information associated with a photoelectric sensor. In some cases, the brightness information may vary based on the location of the electronic device (e.g., whether the electronic device is stored in a pocket). The environment may also include objective data, such as neighborhood information provided by scanning nearby electronic devices. Figure 2 and Figure 3Various examples of information provided to the environment detection engine 112 and determinations that may be made by the environment detection engine 112 are described.

[0040] The environment detection engine 112 may also be configured to receive multimedia data related to the environment. In some aspects, the environment detection engine 112 is configured to receive environmental audio information to facilitate the determination of context. For example, the environment detection engine 112 may determine various attributes and identify various attributes related to the context of the electronic device 100 on the environmental audio information. For example, in the case where the environment detection engine 112 includes or implements an AI or ML model, the weight of the AI ​​or ML model may be related to various determinations in the hidden layer of the AI ​​or ML model, such as the number of speakers, the distance from the speaker, etc. The environment detection engine 112 may also receive image data related to the environment to facilitate the determination of the context. The weights of different hidden layers of the AI ​​or ML model across the environment detection engine 112 may be trained to understand the diversity of environmental data and determine context information related to the environment. In one illustrative aspect, the context information related to the environment may include a noise floor based on a context corresponding to a high ambient noise environment, or may include a noise ceiling based on a context corresponding to a low ambient noise environment.

[0041] The environment detection engine 112 is configured to combine different types of environmental information and synthesize information to understand the context of the electronic device 100. For example, the weights of the AI ​​or ML model of the environment detection engine 112 can be trained to understand various environmental data and determine contextual information related to the environment. In some cases, the output can be a data object representing various aspects of the environment that can affect auditory perception, and the data object can also represent various aspects related to tactile feedback within the environment.

[0042] According to some aspects, the output of the environment detection engine 112 can vary and include multiple characteristics indicative of the environment. For example, the environment detection engine 112 can determine a noise ceiling (e.g., a maximum noise level) for a low ambient noise environment suitable for different functions associated with the electronic device 100. Illustrative examples of different functions of the electronic device 100 include multimedia content consumption, games, applications, and notifications configured to be output. The environment detection engine 112 can also determine a noise floor (e.g., a minimum noise level) suitable for a high ambient noise environment based on different functions. The environment detection engine 112 can identify the context and configuration of tactile feedback based on the environmental data.

[0043] Based on the environment detection engine 112, the processor 105 is configured to determine to modify the tactile feedback based on the context of the electronic device 100. The modification of the tactile feedback can occur in various ways. In one illustrative example, the processor 105 is configured to modify the settings within the application programming interface (API). For example, Table 1 below shows static constants that can be changed, and Table 2 below shows public static and instance methods of the android.os.VibrationEffect namespace that can be launched based on the static constants from Table 1.

[0044]

[0045]

[0046] Table 1

[0047]

[0048] Table 2

[0049] In other aspects, the processor 105 can be configured to control the sense of touch based on the function being executed. For example, since the hardware controls are outside the browser sandbox, the web browser does not typically output tactile information, so the execution of the web browser will not result in the output of tactile feedback. In another example, the game can be configured to output a large amount of tactile feedback, and the processor 105 may need to adjust the tactile feedback to reduce the noise associated with the tactile feedback. Based on the application or function being executed, the processor 105 can be configured to provide the environment detection engine 112 with environmental data captured simultaneously with the time period that ensures that the environmental data is relevant to the current environment, and then determine whether to adjust the tactile feedback based on the output of the environment detection engine 112.

[0050] In other aspects, the processor 105 may be configured to adjust notifications from various applications. For example, in silent mode, the processor 105 may be configured to output tactile feedback to provide notifications of new messages (e.g., text or email). Messages may be received frequently and quickly, which may cause the electronic device 100 to generate a large amount of unwanted noise. In this case, the electronic device 100 may determine that the noise generated by the notification exceeds a threshold (e.g., power spectral density) for a period of time, and reduce tactile notifications from a portion of the application causing the noise. In some cases, the reduction of tactile notifications can be performed by reducing the intensity of the notification, shortening the tactile feedback period, grouping notifications over a period of time (e.g., every 5 minutes), or disabling notifications from certain applications.

[0051] In some cases, the electronic device 100 can be configured to increase tactile feedback based on context. For example, the environment detection engine 112 can determine that the context involves a shared experience, such as a person consuming multimedia content from an application or playing a game while sharing the experience with at least one audience. The environment detection engine 112 can output information that causes the electronic device 100 to increase tactile feedback to share the experience with the audience. In other aspects, the electronic device 100 can increase tactile feedback based on a shared location with a lot of noise, such as a sporting event.

[0052] The electronic device 100 may be configured to further train the AI ​​or ML model based on the profile data and other information. In some aspects, the various uses of the processor 105 may be recorded in a temporary storage location, and then various aspects of the data are processed. In one illustrative aspect, the processor 105 may construct profile data 114, which may identify various aspects of the use of the device, such as various events and parameters associated with these events. For example, parameters associated with an event may include the time of the event, location information identifying the location of the electronic device 100, duration, and source of the event. Based on the various events and data recorded in the temporary storage location, the processor 105 may execute a machine learning (ML) model to construct a profile identifying various information related to use and application use. In one illustrative aspect, the ML model may be a clustering algorithm that groups data across different dimensions (e.g., location, time, event source, etc.) and constructs profile data 114.

[0053] In some aspects, the electronic device 100 can train the AI ​​or ML model of the environment detection engine 112 based on the events in the profile data 114 or the temporary storage location, which enables the processor 105 to update the weights of the hidden layers within the AI ​​or model. In some cases, joint learning can be used to train the AI ​​or ML model of the environment detection engine 112. In some aspects, the adjustment of the weights can be related to one or more data sources that are primarily responsible for the weight changes. Illustrative examples of data sources responsible for the change include the time of the event and the location of the event. For example, a person can manually enable a silent mode (e.g., do not disturb) when entering a specific room at a specific time. In another illustrative example, the data source primarily responsible for the weight change can be a detection of a person sleeping. In some aspects, the weights of the AI ​​model of the environment detection engine 112 are refined based on data obtained by the electronic device 100, and this can be referred to as joint learning. In some cases, joint learning can also include sharing the refined weights and related anonymized data with the cloud system 120, and then the cloud system 120 can further modify the weights associated with the AI ​​model of the environment detection engine 112, and provide an updated AI model or change for the processor 105 to apply to the AI ​​model.

[0054] Figure 2is a diagram showing an electronic device (eg, Figure 1 100) for determining context. Based on the context, the electronic device can adjust the tactile feedback. In one illustrative aspect, the environment 200 is a low ambient noise environment, where multiple people nearby have various electronic devices that can be used or detected by the electronic device to facilitate understanding of the environment. For example, the environment 200 can be a library or other space designated to minimize ambient noise. In one example, a first person 202 can have multiple devices, such as a computing device 204 that the first person 202 is holding and a wearable device 206 that the person is wearing. A second person 208 can have a computing device 210 placed on a table, which is connected to a true wireless stereo (TWS) earbud 212 worn by the second person 208. A third person 214 can wear an HMD 216 and participate in various functions that can be performed in extended reality (XR), and the third person 214 can have a computing device located in a position that is not perceptible to others in the environment. A fourth person 218 can have an XR device 220, which is also connected to a computing device located in a position that is not perceptible to others in the environment.

[0055] In some aspects, the first person 202, the second person 208, and the third person 214 may each engage in activities that cause various interruptions within a low ambient noise environment. In an illustrative example, the first person 202 may perform a multimedia function (e.g., video, application) that outputs tactile feedback on the computing device 204 to enhance the user experience. The first person 202 may wear earplugs that isolate the audio associated with the multimedia content, and therefore cannot auditorily perceive the tactile feedback generated by the computing device 204. In some aspects, the second person may be disrupted because the computing device 210 is placed on a rigid surface, which will cause the computing device 210 to generate a large amount of noise when outputting tactile feedback. In contrast, if the computing device 210 is placed in the second person's pocket, other people (including the second person) will not perceive the same tactile feedback.

[0056] Computing device 204 may be configured to use an environment detection engine (eg, environment detection engine 112) to determine the context of environment 200. Figure 1As described, the environment detection engine may include or implement an AI or ML model, or may utilize other techniques (e.g., computer vision such as object detection or other techniques) to determine the context of the environment 200. In some aspects, the environment detection engine (e.g., via an AI model, computer vision techniques, etc.) may identify a shift in context based on environmental information such as noise. For example, the environment detection engine may detect that the first person 202 enters a library based on a significant shift in audio noise. In response to detecting a shift in context, the computing device 204 may be configured to determine the context based on multiple sources of environmental information.

[0057] In one illustrative aspect, computing device 204 may record environmental information using a microphone, capture at least one image using an image sensor, and detect the brightness of the environment using a photoelectric sensor. In one illustrative example, computing device 204 may be held in the hand of first person 202, and when computing device 204 detects a transfer of context, it begins to obtain environmental information to identify the context. In illustrative environment 200, environment 200 includes multiple people and indicates daylight.

[0058] The computing device 204 may also use various wireless communication standards (e.g., Bluetooth TM , 802.11 wireless standards, etc.) to detect other nearby devices. For example, the computing device 204 can send a wireless measurement request (e.g., a WiFi 802.11k measurement request) and receive responses from the wearable device 206, the computing device 210 of the second person 208, the TWS earbuds 212 of the second person 208, the HMD worn by the third person 214, the XR device 220 of the fourth person 218, and the laptop 222. The computing device 204 can also receive beacon responses from other devices that are not visible in the environment 200, such as access points for providing wireless Internet services. The wireless measurement request (e.g., an 802.11k measurement request) requests measurements from other devices. In reply to the wireless measurement request, examples of reports from neighboring devices include beacon reports, frame reports, channel load reports, noise histogram reports, hidden node reports, medium sensing time histogram reports, wireless station statistics reports, and location configuration information.

[0059] The computing device 204 can determine various information from the various measurement reports, such as identifying the nearest electronic device based on the received signal strength information. In some cases, the computing device 204 can identify properties of the device that the computing device 204 can use to determine the context. For example, the computing device 204 can identify that the TWS earplug 212 is a wearable device based on the power transmission profile of the TWS earplug 212. In other examples, the channel load report can provide information indicating a large amount of channel usage across many different frequency bands, which can suggest that the environment 200 is a high ambient noise environment.

[0060] Computing device 204 may also record audio using a microphone to determine the context. In some cases, the audio may be combined with profile information to indicate the context. For example, audio information indicating low ambient noise recorded by computing device 204 may be combined with profile information to indicate that the context is a public setting in a shared space. In such an example, computing device 204 may determine to reduce tactile feedback based on the determined context.

[0061] In other examples, the computing device 204 may also use profile information obtained over time (e.g., profile data 114) to identify patterns. In some aspects, the ML model may build profile information by clustering instances of recorded data. For example, the ML model may identify that the first person 202 plays games in short bursts of activity during the day, or that the first person 202 plays games during a particular part of the day (e.g., short bursts of activity while in a library, before sleep). The ML model uses objective data such as time, application information, location, and date to build profile information.

[0062] In another example, computing device 204 can use federated learning techniques to improve an AI model of an environment detection engine used to detect the context of environment 200 and receive updates to the AI ​​model. Figure 4 Let’s discuss examples of federated learning further.

[0063] Figure 3 An overhead view of an example environment 300 according to some aspects of the present disclosure is shown, in which an electronic device can determine different contexts and adjust tactile feedback. In an illustrative example, environment 300 includes a reception area 310, a public area 320 (e.g., a kitchen, a lounge, etc.), a library 330, and a classroom 340. According to some aspects, when a person 312 enters the receiving area, the electronic device can detect a context transfer based on entering a building. Based on the conversation, the ambient noise level of the receiving area 310 can be approximately 60dB. In some aspects, if a person 312 plays a game on an electronic device (e.g., electronic device 100) with context-aware tactile feedback, the electronic device can determine that the context has an ambient noise level that does not require any changes to the tactile feedback.

[0064] In another example, a first person 322 and a second person 324 are in a common area, and the first person 322 executes an application to present multimedia content with the second person 324 who is watching. In this example, the electronic device of the first person 322 can obtain environmental information and identify the second person, the brightness information indicates daylight, and the ambient noise level indicates a conversation of approximately 60 dB. The first person's electronic device can provide environmental information to an environment detection engine (e.g., an environment detection engine 112, which may include or implement an AI model, computer vision technology, and / or other technology for determining the context of the environment) and obtain information identifying the context as shared content in a shared space. Based on the identified context, the electronic device can increase tactile feedback to enable the second person 324 to increase participation in the shared experience.

[0065] The third person 332 can study in the library 330 and can take a break to play multimedia games. When the application is requested to execute or receive focus, the electronic device of the third person 332 can detect the context and output a user interface to play the game. The electronic device of the third person 332 can obtain environmental information and identify brightness information indicating daylight, receive photos indicating that few people are nearby, and obtain information about the ambient noise level corresponding to whispers (e.g., about 40dB). The electronic device of the third person 332 can provide environmental information to an environment detection engine (e.g., environment detection engine 112) and obtain information identifying the context as a shared quiet space. Based on the identified context, the electronic device can reduce tactile feedback associated with multimedia games to reduce the noise generated by the electronic device.

[0066] A fourth person 342 may attend a course in a classroom 340 with a main speaker 344 presenting materials to the classroom. In this case, the electronic device of the fourth person 342 may detect a context transfer when entering the classroom 340 or when the main speaker 344 begins a lecture. In response to detecting a context transfer, the computing device of the fourth person 342 may obtain context-related environmental information and input the environmental information into an environmental detection engine (e.g., environmental detection engine 112). In this case, the electronic device of the fourth person 342 may identify a large number of computing devices (e.g., mobile devices or other devices) in the vicinity based on beacon reports that are part of the environmental information. The electronic device of the fourth person 342 may also detect that the lights in the classroom 340 are dimmed and the brightness is reduced. Based on the environmental information, the environmental detection engine device of the electronic device outputs the context information, and in response, the electronic device device reduces tactile feedback to prevent interrupting others. In one aspect, the electronic device may delay tactile feedback from a notification or may reduce the intensity of the notification.

[0067] Figure 4A sequence diagram 400 is shown for training an AI model of an environment detection engine (e.g., environment detection engine 112) utilizing a combination of on-device learning techniques and aggregate-based learning in accordance with some aspects of the present disclosure. In some aspects, the sequence diagram 400 illustrates training a first electronic device 402 and a second electronic device 404. In an illustrative example, the first electronic device 402 and the second electronic device 404 may be owned by the same person to illustrate how different devices may receive different training weights based on usage, but the first electronic device 402 and the second electronic device 404 may be owned by different persons in the same household or in different households.

[0068] At box 410, the first electronic device 402 and the second electronic device 404 include an initial model, and the model is trained based on initial training, and therefore the weight of the hidden layer is the same. For example, the model can be configured to detect the context of the electronic device and provide information related to the auditory output in the context. In an illustrative example, the model can identify the level of tactile feedback for the context. The model can also be configured to identify the audio level (e.g., the volume of the audio) suitable for the context. When the first electronic device 402 and the second electronic device 404 are used to monitor the environment, each device records data unique to its experience. For example, the first electronic device 402 identifies the ambient noise profile n0 with a volume of 42dB and identifies the location. In some aspects, for clear description, the location can be a data object represented as a string corresponding to the name of the location. The first electronic device 402 can record data in a temporary storage for training the model later. Assume that the second electronic device 404 is located in a different location, but experiences a small noise of 3dB and is located at home.

[0069] Each of the first electronic device 402 and the second electronic device 404 captures data related to its use or monitors its environment and records it over a period of time. Figure 4 For clarity, data recorded by the first electronic device 402 and the second electronic device 404 are depicted at similar times, and each of the first electronic device 402 and the second electronic device 404 may record different data at different times based on their different usage. For example, the second electronic device 404 may incur less usage because it is primarily located at home.

[0070] At block 412, the first electronic device 402 may train a model and Figure 4 In some aspects, some aspects of the training of the model may be irrelevant, and the training and updating of the weights may be based on the use of the model to refine the model, or from Figure 4The second electronic device 404 may also train the model and update the weights at block 414. When the second electronic device 404 uses different data, the second electronic device 404 training will generate different weights.

[0071] In some cases, the first electronic device 402 may include the weight w in the model update information. 402 Sent to server 406, weight w 402 is a trained weight based on the experience of the first electronic device 402. The second electronic device 404 may also set the weight w 404 Sent to server 406, weight w 404 The trained weights are based on the experience of the second electronic device 404. The model update information may also include data related to the weights, such as location and time. In some cases, any personal data is removed in this aspect because personalized information can be consumed within the on-device training (e.g., blocks 412 and 414).

[0072] The server 406 is configured to receive weights for different devices each having different experiences. Based on the updated weights and various information included in the model update information, the server 406 can train the model (to update the weights of the model) based on the unique experiences of other users (and corresponding devices) at block 420 to create a model with refined weights w 420 Then, the server 406 can refine the weight w 420 The first electronic device 402 and the second electronic device 404 may each be based on the refinement weight w 420 To update their unique models. In some cases, the model of the first electronic device 402 or the second electronic device 404 can be replaced.

[0073] As a result of the combination of on-device and aggregated weight training, models can be trained based on a broad range of experiences that cannot be replicated based on a single training dataset.

[0074] Figure 5 4 is a three-dimensional (3D) graph showing various data points that can be grouped to correspond to contexts provided by an electronic device according to some aspects of the present disclosure. Specifically, the 3D graph shows various data points that a server (e.g., server 406) can use to train and refine weights associated with an AI model. For clarity, data points are displayed based on time, location, and environment (e.g., environmental data), but more dimensions can be implemented. Environmental data can represent different data sources, such as audio, images, brightness, and neighboring wireless devices.

[0075] Figure 5 The diagram in illustrates different groups that may be identified that are highly relevant to training an AI model to determine environmental context. For example, a first group 502 may be associated with shared environments with widespread noise (e.g., public transportation), group 504 may be shared environments in quiet environments, group 506 may be associated with private environments (e.g., home) at specific times, group 508 may be associated with public spaces within a specific geographic location at specific times, and group 510 may be shared private spaces with specific noise requirements.

[0076] In some aspects, AI models trained based on different data, such as from federated learning techniques, can provide accurate predictions based on environmental data. By collecting anonymous data from a rich set of experiences, AI models can be accurately trained to accurately predict the context of electronic devices.

[0077] Figure 6 is a block diagram illustrating a haptic driver for dynamically adapting a driving duty cycle of a haptic device 602 according to some aspects of the present disclosure. Figure 6 6, an apparatus 600 that may include a haptic driver circuit includes a digital controller 604 or other processor circuit. In some aspects, the digital controller 604 receives amplitude information from a pattern source 606 and frequency information from a clock generator 608, and outputs a signal to control a driver and power stage 610, which generates a pair of control signals that drive the haptic device 602. The apparatus 600 includes a feedback circuit 612 that generates information 614 about the phase (Φ) and amplitude (A) of a signal detected at an input of the haptic device 602, other information, or a combination thereof. The digital controller 604 uses the information 614 to dynamically adjust the amplitude (or magnitude), period, duration, and / or duty cycle of the control signal. In some aspects, a duty cycle engine 616 controls the vibration generated by the haptic device 602 (e.g., by controlling the duty cycle of the haptic device 602). In some aspects, the apparatus 600 may include a memory 618 communicatively coupled to the digital controller 604 for storing computer instructions, parameters, variables, etc. Figure 6 The specific components and connections shown are illustrative and not limiting.

[0078] Figure 71 is a flowchart illustrating an example of a method 700 for controlling tactile feedback based on contextual awareness in accordance with certain aspects of the present disclosure. The method 700 may be performed by a computing device or apparatus (or a component thereof, such as a chipset, a processor, etc.) that includes or communicates with at least one sensor. For example, the computing device may include a mobile wireless communication device, a camera, an XR device, a wireless-enabled vehicle or a computing system or component of a vehicle, or other computing device. In one illustrative example, the computing system 1000 may be configured to perform all or part of the method 700.

[0079] At block 702, the computing device (or a component thereof) may obtain environmental information indicative of an environment of the computing device from at least one sensor. At block 704, the computing device (or a component thereof) may determine a context associated with the computing device based on the environmental information.

[0080] In some aspects, the environmental information includes ambient audio information of the environment, device information identifying other wireless devices in proximity to the computing device, brightness information, information indicating that at least one person has entered or left the environment, at least one image, any combination thereof, and / or other information. In some cases, to determine the context, the computing device (or its components) may determine whether the ambient audio information is below a threshold for a period of time, determine the number of acoustic devices used to output audio within a specific distance of the computing device, determine the spectral quality of the ambient audio information, determine the number of devices (e.g., mobile devices) within a specific distance of the computing device, determine the number of people within a specific distance of the computing device, any combination thereof. Or perform one or more other operations based on the environmental information.

[0081] In some cases, in order to determine a context associated with a computing device based on the environment, the computing device (or a component thereof) may provide (e.g., output, transmit, or otherwise provide) environmental information to a machine learning model for use in estimating an amount of tactile feedback associated with the environment.

[0082] In some aspects, a computing device (or a component thereof) may obtain (e.g., receive, retrieve, determine, or otherwise obtain) a location of the computing device using a location sensor. In some cases, in order to determine a context associated with the computing device based on the environmental information, the computing device (or a component thereof) may provide (e.g., output, transmit, or otherwise provide) the environmental information and the location of the computing device to a machine learning model for estimating an amount of noise associated with the environment.

[0083] At box 706, the computing device (or its component) may determine tactile feedback for output by the computing device based on the context. In some aspects, to determine the tactile feedback of the computing device based on the context, the computing device (or its component) may determine the amount of noise associated with the tactile feedback based on the surface in contact with the computing device. In some aspects, to determine the tactile feedback of the computing device based on the context, the computing device (or its component) may determine the amount of noise associated with the tactile feedback that the application will emit during execution of the application. In some cases, the amount of noise associated with the tactile feedback that the application will emit is further based on at least one setting associated with the application.

[0084] In some aspects, a computing device (or a component thereof) may identify an application configured to output a notification with tactile feedback. The computing device (or a component thereof) may adjust the tactile feedback associated with notifications for at least a portion of the application (e.g., one or more of the applications).

[0085] In some aspects, a computing device (or a component thereof) may determine a maximum noise level associated with tactile feedback based on a context associated with the computing device. In such aspects, the computing device (or a component thereof) may reduce the tactile feedback of the computing device to reduce the noise associated with the tactile feedback to less than the maximum noise level associated with the tactile feedback. Additionally or alternatively, in some aspects, a computing device (or a component thereof) may determine a minimum noise level associated with tactile feedback based on a context associated with the computing device. In such aspects, the computing device (or a component thereof) may increase the tactile feedback of the computing device to increase the noise associated with the tactile feedback to greater than the minimum noise level associated with the tactile feedback.

[0086] In some examples, a computing device (or components thereof) may adjust the audio volume output of an audio device (eg, increase or decrease the audio volume) based on the context and the haptic feedback.

[0087] In some cases, the computing device (or a component thereof) may obtain additional environmental information indicating a change associated with an environment of the computing device from at least one sensor based on (e.g., after determining) a determination of tactile feedback output by the computing device, the determination being based on the context. The computing device (or a component thereof) may determine a second context associated with the computing device based on the additional environmental information. The computing device (or a component thereof) may also determine a second tactile feedback to be output by the computing device based on the second context.

[0088] In some aspects, the computing device includes multiple feedback events. For example, the multiple feedback events may include a first feedback event that includes activating one or more tactile actuators based on a first parameter of tactile feedback. In an illustrative example, the first parameter of tactile feedback is associated with the amplitude of the tactile feedback, the duty cycle of the tactile feedback, the duration of the tactile feedback, the period of the tactile feedback, any combination thereof, and / or other parameters. The multiple feedback events may additionally or alternatively include a second feedback event that includes activating one or more tactile actuators based on a second parameter of the tactile feedback. In an illustrative example, the second parameter of the tactile feedback is associated with preventing actuation of at least one tactile actuator of the one or more tactile actuators.

[0089] In some cases, a computing device (or a component thereof) may modify a first parameter of tactile feedback associated with a first feedback event based on context and / or modify a second parameter of tactile feedback associated with a second feedback event based on context. For example, different alerts may be associated with different applications. In one example, a first alarm associated with a first action may be a beep with a short duration (e.g., 0.5 seconds or other duration), and a second alarm associated with a second action may be a beep with a longer duration (e.g., 1 second or other duration longer than the short duration) and / or a stronger intensity. A computing device (or a component thereof) may adjust the alarm differently based on context. For example, a computing device (or a component thereof) may maintain a first alarm (with a short duration) based on context, and may not allow a second alarm (e.g., prevent output of a second alarm) based on context.

[0090] In some examples, the processes described herein (e.g., method 700 and / or other processes described herein) can be performed by a computing device or apparatus. For example, method 700 can be performed by a computer having Fig.10 The computing device (eg, Figure 1The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a network-connected watch or smart watch or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device having resource capabilities to perform the methods described herein (including method 700). In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the methods described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive IP-based data or other types of data.

[0091] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0092] The method 700 is illustrated as a logical flow diagram, the operations of which represent a series of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the method.

[0093] The method 700 and / or other methods or processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware, or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0094] As described above, various aspects of the present disclosure may utilize a machine learning model or system. Figure 8 is an illustrative example of a deep learning neural network 800 that can be used to implement the machine learning-based alignment prediction described above. Input layer 820 includes input data. In an illustrative example, input layer 820 may include data representing pixels of an input video frame. Neural network 800 includes multiple hidden layers 822a, 822b to 822n. Hidden layers 822a, 822b to 822n include "n" hidden layers, where "n" is an integer greater than or equal to 1. The number of hidden layers can include as many layers as required for a given application. Neural network 800 also includes an output layer 821, which provides outputs generated by the processing performed by hidden layers 822a, 822b to 822n. In an illustrative example, output layer 821 can provide classification for objects in input video frames. Classification can include categories that identify activity types (e.g., looking up, looking down, closing eyes, yawning, etc.).

[0095] Neural network 800 is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. Information associated with a node is shared between different layers, and each layer retains the information as it is processed. In some cases, neural network 800 can include a feedforward network, in which case there is no feedback connection in which the output of the network is fed back into itself. In some cases, neural network 800 can include a recursive neural network, which can have loops that allow information to be carried across nodes when reading in input.

[0096] Information can be exchanged between nodes by node-to-node interconnection between each layer. The nodes of input layer 820 can activate a group of nodes in the first hidden layer 822a. For example, as shown in the figure, each input node of input layer 820 is connected to each node of the first hidden layer 822a. The nodes of the first hidden layer 822a can transform the information of each input node by applying an activation function to the input node information. Then, the information derived from the transformation can be passed to the nodes of the next hidden layer 822b and can activate the nodes of the next hidden layer 822b, and the nodes of the next hidden layer 822b can perform their own specified functions. Example functions include convolution, upsampling, data transformation and / or any other suitable function. Then, the output of hidden layer 822b can activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 822n can activate one or more nodes of output layer 821, providing output at the one or more nodes. In some cases, although the nodes (e.g., node 826) in neural network 800 are shown as having multiple output lines, the nodes have a single output, and all the lines shown as outputs from the nodes represent the same output value.

[0097] In some cases, each node or interconnection between nodes can have a weight, which is a set of parameters derived from the training of neural network 800. Once neural network 800 is trained, it can be referred to as a trained neural network, which can be used to classify one or more activities. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnections can have tunable digital weights that can be tuned (e.g., based on a training data set), allowing neural network 800 to adapt to the input and be able to learn as more and more data is processed.

[0098] The neural network 800 is pre-trained to process features from the data in the input layer 820 using different hidden layers 822a, 822b to 822n to provide an output through the output layer 821. In examples where the neural network 800 is used to identify features and / or objects in images, the neural network 800 can be trained using training data that includes both images and labels, as described above. For example, training images can be input into the network, where each training frame has a label indicating a feature in the image (for a feature extraction machine learning system) or a label indicating a class of activity in each frame. In one example using object classification for illustrative purposes, the training frame can include an image of the number 2, in which case the label of the image can be [0 0 1 0 0 0 0 0 0 0].

[0099] In some cases, the neural network 800 can use a training process called back propagation to adjust the weights of the nodes. As described above, the back propagation process can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. The process can be repeated for a certain number of iterations for each set of training images until the neural network 800 is trained well enough so that the weights of the layers are accurately tuned.

[0100] For the example of identifying features and / or objects in an image, a forward pass may include passing a training image through the neural network 800. Prior to training the neural network 800, the weights are initially randomized. As an illustrative example, a frame may include an array of numbers representing pixels of an image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that position in the array. In one example, the array may include a 28×28×3 array of numbers having 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or luminance and two chrominance components, etc.).

[0101] As described above, for the first training iteration of the neural network 1400, the output will likely include values ​​that do not favor any particular class due to the weights being randomly selected at initialization. For example, if the output is a vector with probabilities that the object includes different classes, the probability values ​​for each different class may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 1400 cannot determine low-level features and therefore cannot accurately determine what the classification of the object may be. A loss function can be used to analyze the errors in the output. Any suitable loss function definition can be used, such as a cross entropy loss. Another example of a loss function includes a mean squared error (MSE), which is defined as a risk function that corresponds to the expected value of the squared error loss and measures the quality of the estimate provided by the neural network 1400.

[0102] For the first training image, the loss (or error) will be high because the actual value will be very different from the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 1400 can perform a backward pass by determining which inputs (weights) contribute the most to the loss of the network, and the weights can be adjusted so that the loss is reduced and ultimately minimized. The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes the most to the network loss. After calculating the derivative, the weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. The weight update can be expressed as where w represents the weight, wi represents the initial weight, and η represents the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes a larger weight update, and a lower value indicates a smaller weight update.

[0103] The neural network 1400 may include any suitable deep network. An example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layers of the CNN include a series of convolution, nonlinear, pooling (for downsampling), and fully connected layers. The neural network 1400 may include any other deep network other than a CNN, such as an autoencoder, a deep belief network (DBN), a recursive neural network (RNN), etc.

[0104] Fig. 9 is an illustrative example of a CNN 900. The input layer 920 of the CNN 900 includes data representing an image or frame. For example, the data may include an array of numbers representing pixels of the image, each number in the array including a value from 0 to 255 describing the intensity of the pixel at that position in the array. Using the previous example from above, the array may include a 28×28×3 digital array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or luminance and two chrominance components, etc.). The image may pass through a convolutional hidden layer 922a, an optional non-linear activation layer, a pooling hidden layer 922b, and a fully connected hidden layer 922c to obtain an output at an output layer 924. Although Fig. 9 Only one of each hidden layer is shown in , but a person of ordinary skill will understand that multiple convolutional hidden layers, nonlinear layers, pooling hidden layers, and / or fully connected layers may be included in CNN 900. As previously described, the output may indicate a single category of an object, or may include a probability of a category that best describes an object in the image.

[0105] The first layer of CNN 900 is a convolutional hidden layer 922a. The convolutional hidden layer 922a analyzes the image data of the input layer 920. Each node of the convolutional hidden layer 922a is connected to an area of ​​nodes (pixels) of the input image called a receptive field. The convolutional hidden layer 922a can be considered as one or more filters (each filter corresponds to a different activation or feature map), where each convolution iteration of the filter is a node or neuron of the convolutional hidden layer 922a. For example, the area of ​​the input image covered by the filter at each convolution iteration will be the receptive field of the filter. In an illustrative example, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in the convolutional hidden layer 922a. Each connection between a node and the receptive field of the node learns a weight, and in some cases learns an overall bias, so that each node learns to analyze its specific local receptive field in the input image. Each node of the hidden layer 922a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has an array of weights (numbers) and the same depth as the input. For the video frame example, the filter will have a depth of 3 (according to the three color components of the input image). An illustrative example size of the filter array is 5×5×3, corresponding to the size of the node's receptive field.

[0106] The convolutional nature of the convolutional hidden layer 922a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 922a can start at the upper left corner of the input image array and can be convolved around the input image. As described above, each convolution iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 922a. At each convolution iteration, the value of the filter is multiplied by the original pixel value of the corresponding number of the image (for example, a 5×5 filter array multiplied by a 5×5 array of input pixel values ​​at the upper left corner of the input image array). The multiplications from each convolution iteration can be added together to obtain the sum of the iteration or node. Next, the process continues at the next position in the input image according to the receptive field of the next node in the convolutional hidden layer 922a. For example, the filter can move a step amount (called a stride) to the next receptive field. The stride can be set to 1 or other suitable amount. For example, if the stride is set to 1, the filter will move 1 pixel to the right at each convolution iteration. Processing the filter at each unique position in the input volume produces a number representing the filter result at that position, resulting in a summed value being determined for each node of the convolutional hidden layer 922a.

[0107] The mapping from the input layer to the convolutional hidden layer 922a is called an activation map (or feature map). The activation map includes the value of each node that represents the filtering result at each location of the input volume. The activation map may include an array that includes the various summed values ​​produced by each iteration of the filter over the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map will include a 24×24 array. The convolutional hidden layer 922a may include several activation maps in order to identify multiple features in the image. Fig. 9 The example shown in includes three activation maps. Using the three activation maps, the convolutional hidden layer 922a can detect three different kinds of features, each of which is detectable across the entire image.

[0108] In some examples, a nonlinear hidden layer can be applied after the convolutional hidden layer 922a. A nonlinear layer can be used to introduce nonlinearity into a system that already computes linear operations. An illustrative example of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer can apply the function f(x)=max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. Therefore, ReLU can increase the nonlinear properties of the CNN 900 without affecting the receptive field of the convolutional hidden layer 922a.

[0109] The pooling hidden layer 922b can be applied after the convolutional hidden layer 922a (and after the non-linear hidden layer when used). The pooling hidden layer 922b is used to simplify the information in the output from the convolutional hidden layer 922a. For example, the pooling hidden layer 922b can take each activation map output from the convolutional hidden layer 922a and generate a compressed activation map (or feature map) using a pooling function. Maximum pooling is an example of a function performed by a pooling hidden layer. The pooling hidden layer 922a can use other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. The pooling function (e.g., a maximum pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 922a. In Fig. 9 In the example shown, three pooling filters are used for the three activation maps in the convolutional hidden layer 922a.

[0110] In some examples, max pooling can be used by applying a max pooling filter (e.g., having a size of 2×2) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from the convolutional hidden layer 922a. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes in the previous layer (each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by a 2×2 max pooling filter at each iteration of the filter, where the maximum of the four values ​​is output as the "maximum" value. If such a max pooling filter is applied to an activation filter with a dimension of 24×24 nodes from the convolutional hidden layer 922a, the output from the pooling hidden layer 922b will be an array of 18×12 nodes.

[0111] In some examples, an L2 norm pooling filter may also be used. The L2 norm pooling filter involves calculating the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than calculating the maximum value as done in max pooling), and using the calculated value as the output.

[0112] Intuitively, a pooling function (e.g., max pooling, L2 norm pooling, or other pooling functions) determines whether a given feature is found anywhere in an image region and discards the exact location information. This can be done without affecting the results of feature detection, because once a feature is found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (and other pooling methods) provides the following benefits: there are fewer pooled features, thereby reducing the number of parameters required in later layers of CNN 900.

[0113] The last layer of connections in the network is a fully connected layer that connects each node from the pooling hidden layer 922b to each output node in the output layer 924. Using the above example, the input layer includes 28×28 nodes that encode the pixel intensities of the input image, the convolutional hidden layer 922a includes 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for the filter) to the three activation maps, and the pooling hidden layer 922b includes a layer of 3×12×12 hidden feature nodes based on applying a max pooling filter to a 2×2 region on each of the three feature maps. Extending this example, the output layer 924 can include ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 922b is connected to each node of the output layer 924.

[0114] The fully connected layer 922c can obtain the output of the previous pooling hidden layer 922b (which should represent an activation map of high-level features) and determine the features most relevant to a particular category. For example, the fully connected layer 922c layer can determine the high-level features that are most strongly associated with a particular category and can include weights (nodes) for the high-level features. The product between the weights of the fully connected layer 922c and the pooling hidden layer 922b can be calculated to obtain the probabilities of different classes. For example, if the CNN 900 is used to predict that the object in the video frame is a person, a high value will exist in the activation map, which represents the high-level features of the person (e.g., there are two legs, the face is present at the top of the object, two eyes are present at the upper left and upper right of the face, the nose is present in the middle of the face, the mouth is present at the bottom of the face, and / or other features common to people).

[0115] In some examples, the output from the output layer 924 may include an M-dimensional vector (in the previous example, M=10). M indicates the number of classes that the CNN 900 must choose from when classifying an object in an image. Other example outputs may also be provided. Each number in the M-dimensional vector may represent the probability that an object belongs to a certain category. In an illustrative example, if the 10-dimensional output vector represents that objects of ten different categories are [0 0 0.05 0.8 0 0.15 0 0 0 0], the vector indicates that the probability that the image is an object of the third category (e.g., dog) is 5%, the probability that the image is an object of the fourth category (e.g., person) is 80%, and the probability that the image is an object of the sixth category (e.g., kangaroo) is 15%. The probability of a class can be considered as the confidence level that the object is part of that class.

[0116] Fig.10 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, Fig.10 An example of a computing system 1000 is shown, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1005. Connection 1005 may be a physical connection using a bus, or a direct connection into processor 1010, such as in a chipset architecture. Connection 1005 may also be a virtual connection, a networked connection, or a logical connection.

[0117] In some aspects, computing system 1000 is a distributed system, where the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each component performing some or all of the functionality for which the component is described. In some aspects, a component may be a physical or virtual device.

[0118] The example computing system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 that couple various system components including system memory 1015, such as read-only memory (ROM) 1020 and read-only memory (RAM) 1025, to the processor 1010. The computing system 1000 may include a cache 1012 of high-speed memory directly connected to the processor 1010, in close proximity to the processor 1010, or integrated as part of the processor 1010.

[0119] Processor 1010 may include any general purpose processor and hardware services or software services, such as services 1032, 1034, and 1036 stored in storage device 1030, configured to control processor 1010 as well as special purpose processors where the software instructions are incorporated into the actual processor design. Processor 1010 may be essentially a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0120] To enable user interaction, computing system 1000 includes input device 1045, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Computing system 1000 can also include output device 1035, which can be one or more of a plurality of output mechanisms. In some cases, a multimodal system can enable a user to provide multiple types of input / output to communicate with computing system 1000. Computing system 1000 can include communication interface 1040, which can generally govern and manage user input and system output. The communication interface may use a wired and / or wireless transceiver to perform or facilitate receiving and / or transmitting wired or wireless communications, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, a port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, wireless signal transmission, BLE wireless signal transmission, wireless signal transmission, RFID wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 WiFi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), world interoperability for microwave access (WiMAX), IR communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1040 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1000 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based GPS, the Russian-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no limitation to operating on any particular hardware arrangement, and thus the basic features herein may be easily replaced with improved hardware or firmware arrangements when such arrangements are developed.

[0121] The storage device 1030 may be a non-volatile and / or non-temporary and / or computer-readable memory of the device, and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk read-only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a rewritable compact disk (CD) optical disk. A digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, RAM, a static RAM (SRAM), a dynamic RAM (DRAM), a ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASH EPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0122] Storage device 1630 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1610, causes the system to perform a function. In some aspects, a hardware service that performs a particular function may include a software component stored in a computer-readable medium in combination with the necessary hardware components (e.g., processor 1610, connection 1605, output device 1635, etc.) to perform that function. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and which does not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media (e.g., CDs or DVDs), flash memory, memory, or storage devices. Computer-readable media may have stored thereon codes and / or machine-executable instructions, which may represent a program, function, subprogram, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0123] In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, one or more network interfaces configured to transmit and / or receive data, any combination thereof, and / or other components. The one or more network interfaces may be configured to transmit and / or receive wired and / or wireless data, including data according to 3G, 4G, 5G, and / or other cellular standards, data according to Wi-Fi (802.11x) standards, data according to BluetoothTM standards, data according to IP standards, and / or other types of data.

[0124] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0125] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0126] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be appreciated by those skilled in the art that these aspects can be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, which include functional blocks comprising devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. In addition to those components shown in the accompanying drawings and / or described herein, additional components may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form, so as not to obscure these aspects with unnecessary details. In other instances, known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details, so as to avoid obscuring these aspects.

[0127] Various aspects may be described above as a process or method depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart may describe an operation as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0128] The processes and methods according to the above examples can be implemented using computer executable instructions stored in or otherwise available from computer readable media. Such instructions may include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. Parts of the computer resources used may be accessed over a network. Computer executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, etc.

[0129] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may be implemented in any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. (One or more) processors may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or add-on cards. As another example, such functions may also be implemented between different chips or different processes executed in a single device on a circuit board.

[0130] The instructions, the media for carrying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in this disclosure.

[0131] In the foregoing description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative aspects of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations, except for being limited by the prior art. The various features and aspects of the above-mentioned application can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, various aspects can be utilized in any number of environments and applications outside the environment and application described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that, in alternative aspects, the method can be performed in an order different from the described order.

[0132] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by the less than or equal to ("<") and greater than or equal to ("<") symbols, respectively, without departing from the scope of the present specification.

[0133] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform those operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform those operations, or any combination thereof.

[0134] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection, and / or other suitable communication interface).

[0135] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of a set or multiple members of a set (in any combination) satisfy the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0136] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician may implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0137] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses (including applications in wireless communication device handsets and other devices). Any features described as engines, modules, or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions for executing one or more of the methods described above when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a RAM, such as a synchronous dynamic random access memory (SDRAM), a ROM, a non-volatile random access memory (NVRAM), an EEPROM, a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0138] The program code may be executed by a processor, which may include one or more processors, such as one or more DSPs, general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. This processor may be configured to perform any of the techniques described in the present invention. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or implementation suitable for the device in the technology described herein.

[0139] Illustrative aspects of the disclosure include:

[0140] Aspect 1: A method for controlling tactile feedback, comprising: obtaining environmental information indicating an environment of a computing device from at least one sensor; determining a context associated with the computing device based on the environmental information; and determining tactile feedback to be output by the computing device based on the context.

[0141] Aspect 2: The method of aspect 1, wherein determining tactile feedback of the computing device based on the context comprises determining an amount of noise associated with the tactile feedback based on a surface in contact with the computing device.

[0142] Aspect 3: The method of any one of Aspects 1 to 2, wherein determining haptic feedback of the computing device based on the context comprises: determining an amount of noise associated with haptic feedback to be emitted by an application during execution of the application.

[0143] Aspect 4: The method of any one of aspects 2 or 3, wherein the amount of noise associated with the haptic feedback to be emitted by the application is further based on at least one setting associated with the application.

[0144] Aspect 5: The method according to any one of aspects 1 to 4, further comprising: identifying an application configured to output notifications with tactile feedback; and adjusting tactile feedback associated with notifications of at least a portion of the application.

[0145] Aspect 6: A method according to any one of Aspects 1 to 5, wherein the environmental information includes at least one of ambient audio information of the environment, device information identifying other wireless devices close to the computing device, brightness information, information indicating that at least one person has entered or left the environment, or at least one image.

[0146] Aspect 7: A method according to Aspect 6, wherein determining the context includes at least one of: determining whether the ambient audio information is below a threshold for a period of time; determining the number of acoustic devices used to output audio within a specific distance of the computing device; determining the spectral quality of the ambient audio information; determining the number of computing devices within a specific distance of the computing device; or determining the number of people within a specific distance of the computing device.

[0147] Aspect 8: A method according to any one of Aspects 1 to 7, wherein determining the context associated with the computing device based on the environmental information includes: providing the environmental information to a machine learning model for estimating the amount of tactile feedback associated with the environment.

[0148] Aspect 9: The method according to any one of aspects 1 to 8 further comprises: obtaining a position of the computing device using a position sensor.

[0149] Aspect 10: A method according to Aspect 9, wherein determining the context associated with the computing device based on environmental information includes: providing the environmental information and the location of the computing device to a machine learning model for estimating the amount of noise associated with the environment.

[0150] Aspect 11: The method according to any one of aspects 1 to 10, further comprising: determining a maximum noise level associated with the haptic feedback based on a context associated with the computing device.

[0151] Aspect 12: The method of aspect 11, further comprising: reducing tactile feedback of the computing device to reduce noise associated with the tactile feedback to less than a maximum noise level associated with the tactile feedback.

[0152] Aspect 13: The method according to any one of aspects 1 to 12, further comprising: determining a minimum noise level associated with haptic feedback based on a context associated with the computing device.

[0153] Aspect 14: The method of aspect 13, further comprising: increasing tactile feedback of the computing device to increase noise associated with the tactile feedback to greater than a minimum noise level associated with the tactile feedback.

[0154] Aspect 15: The method according to any one of aspects 1 to 14, further comprising: adjusting an audio volume output of an audio device based on the context and the tactile feedback.

[0155] Aspect 16: The method according to any one of Aspects 1 to 15 further includes: after determining tactile feedback to be output by the computing device based on the context, obtaining additional environmental information indicating changes associated with the environment of the computing device from at least one sensor; determining a second context associated with the computing device based on the additional environmental information; and determining second tactile feedback to be output by the computing device based on the second context.

[0156] Aspect 17: A method according to any one of Aspects 1 to 16, wherein the computing device includes multiple feedback events, wherein a first feedback event among the multiple feedback events includes activating one or more tactile actuators based on a first parameter of the tactile feedback, and wherein a second feedback event among the multiple feedback events includes activating the one or more tactile actuators based on a second parameter of the tactile feedback.

[0157] Aspect 18: The method according to Aspect 17 also includes at least one of the following: modifying the first parameter of the tactile feedback associated with the first feedback event based on the context; or modifying the second parameter of the tactile feedback associated with the second feedback event based on the context.

[0158] Aspect 19: The method according to any one of Aspects 17 or 18, wherein the first parameter of the tactile feedback is associated with at least one of an amplitude of the tactile feedback, a duty cycle of the tactile feedback, a duration of the tactile feedback, or a period of the tactile feedback.

[0159] Aspect 20: The method of any one of Aspects 17 to 19, wherein the second parameter of the haptic feedback is associated with preventing actuation of at least one of the one or more haptic actuators.

[0160] Aspect 21: An apparatus for controlling tactile feedback, comprising at least one memory and at least one processor (e.g., implemented in a circuit) coupled to the at least one memory. The at least one processor is configured to: obtain environmental information indicating an environment of the apparatus from at least one sensor; determine a context associated with the apparatus based on the environmental information; and determine tactile feedback to be output by the apparatus based on the context.

[0161] Aspect 22: The apparatus according to aspect 21, wherein, to determine tactile feedback of the apparatus based on the context, at least one processor is configured to: determine an amount of noise associated with the tactile feedback based on a surface in contact with the apparatus.

[0162] Aspect 23: An apparatus according to any one of Aspects 21 or 22, wherein, in order to determine tactile feedback of the apparatus based on the context, at least one processor is configured to: determine an amount of noise associated with tactile feedback to be emitted by an application during execution of the application.

[0163] Aspect 24: The apparatus of any one of Aspects 22 or 23, wherein the amount of noise associated with the haptic feedback to be emitted by the application is further based on at least one setting associated with the application.

[0164] Aspect 25: An apparatus according to any one of Aspects 21 to 24, wherein at least one processor is configured to: identify an application configured to output notifications with tactile feedback; and adjust tactile feedback associated with notifications of at least a portion of the application.

[0165] Aspect 26: An apparatus according to any one of Aspects 21 to 25, wherein the environmental information includes at least one of ambient audio information of the environment, device information identifying other wireless devices approaching the device, brightness information, information indicating that at least one person has entered or left the environment, or at least one image.

[0166] Aspect 27: An apparatus according to Aspect 26, wherein, in order to determine the context, the at least one processor is configured to perform at least one of the following operations: determining whether the ambient audio information is below a threshold for a period of time; determining the number of acoustic devices used to output audio within a specific distance of the apparatus; determining the spectral quality of the ambient audio information; determining the number of computing devices within a specific distance of the apparatus; or determining the number of people within a specific distance of the apparatus.

[0167] Aspect 28: An apparatus according to any one of Aspects 21 to 27, wherein at least one processor is configured to: provide environmental information to a machine learning model for estimating an amount of tactile feedback associated with the environment.

[0168] Aspect 29: The apparatus according to any one of aspects 21 to 28, wherein at least one processor is configured to: obtain the position of the apparatus using a position sensor.

[0169] Aspect 30: An apparatus according to Aspect 29, wherein, in order to determine a context associated with the apparatus based on environmental information, at least one processor is configured to: provide the environmental information and the location of the apparatus to a machine learning model for estimating an amount of noise associated with the environment.

[0170] Aspect 31: An apparatus according to any one of Aspects 21 to 30, wherein the at least one processor is configured to: determine a maximum noise level associated with tactile feedback based on a context associated with the apparatus.

[0171] Aspect 32: The apparatus of aspect 31, wherein the at least one processor is configured to reduce tactile feedback of the apparatus to reduce noise associated with the tactile feedback to less than a maximum noise level associated with the tactile feedback.

[0172] Aspect 33: An apparatus according to any one of Aspects 21 to 32, wherein at least one processor is configured to: determine a minimum noise level associated with tactile feedback based on a context associated with the apparatus.

[0173] Aspect 34: The apparatus of aspect 33, wherein the at least one processor is configured to increase tactile feedback of the apparatus to increase noise associated with the tactile feedback to greater than a minimum noise level associated with the tactile feedback.

[0174] Aspect 35: The apparatus according to any one of Aspects 21 to 34, wherein the at least one processor is configured to: adjust the audio volume output of the audio device based on the context and the tactile feedback.

[0175] Aspect 36: An apparatus according to any one of Aspects 21 to 35, wherein the at least one processor is configured to: obtain additional environmental information indicating changes associated with an environment of the apparatus from the at least one sensor based on a determination of the tactile feedback to be output by the apparatus, the determination being based on the context; determine a second context associated with the apparatus based on the additional environmental information; and determine second tactile feedback to be output by the apparatus based on the second context.

[0176] Aspect 37: An apparatus according to any one of Aspects 21 to 36, wherein the apparatus comprises a plurality of feedback events, wherein a first feedback event among the plurality of feedback events comprises activating one or more tactile actuators based on a first parameter of tactile feedback, and wherein a second feedback event among the plurality of feedback events comprises activating one or more tactile actuators based on a second parameter of tactile feedback.

[0177] Aspect 38: An apparatus according to Aspect 37, wherein at least one processor is configured to: modify a first parameter of tactile feedback associated with a first feedback event based on a context; or modify the second parameter of the tactile feedback associated with the second feedback event based on the context.

[0178] Aspect 39: The apparatus according to any one of Aspects 37 or 38, wherein the first parameter of the tactile feedback is associated with at least one of a magnitude of the tactile feedback, a duty cycle of the tactile feedback, a duration of the tactile feedback, or a period of the tactile feedback.

[0179] Aspect 40: The apparatus of any one of Aspects 37 to 39, wherein the second parameter of the tactile feedback is associated with preventing actuation of at least one tactile actuator of the one or more tactile actuators.

[0180] Aspect 41: An apparatus according to any one of aspects 21 to 38, wherein the apparatus is a computing device.

[0181] Aspect 42: A non-transitory computer-readable medium having instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform the operations according to any one of aspects 1 to 41.

[0182] Aspect 43: An apparatus for controlling tactile feedback, comprising one or more components for performing the operations according to any one of Aspects 1 to 41.

Claims

1. A device for controlling tactile feedback, the device comprising: at least one memory; as well as at least one processor coupled to the at least one memory and configured to: obtaining environmental information indicative of an environment of the device from at least one sensor; determining a context associated with the device based on the environmental information; as well as Haptic feedback for output by the device is determined based on the context.

2. The device according to claim 1, wherein: To determine the tactile feedback of the device based on the context, the at least one processor is configured to: An amount of noise associated with the haptic feedback is determined based on a surface in contact with the device.

3. The device according to claim 1, wherein: To determine the tactile feedback of the device based on the context, the at least one processor is configured to: An amount of noise associated with haptic feedback to be emitted by an application during execution of the application is determined. 4 . The apparatus of claim 3 , wherein the amount of noise associated with the haptic feedback to be emitted by the application is further based on at least one setting associated with the application.

5. The device according to claim 1, wherein: The at least one processor is configured to: identifying an application configured to output a notification with haptic feedback; and Haptic feedback associated with notifications for at least a portion of the application is adjusted.

6. The device of claim 1, wherein the environmental information comprises at least one of ambient audio information of the environment, device information identifying other wireless devices proximate to the device, brightness information, information indicating that at least one person has entered or left the environment, or at least one image.

7. The device according to claim 6, wherein: To determine the context, the at least one processor is configured to perform at least one of the following: Determining whether the ambient audio information is below a threshold for a period of time; determining a number of acoustic devices for outputting audio within a specified distance of the apparatus; determining a spectral quality of the ambient audio information; determining a number of computing devices within a specified distance of the apparatus; or The number of people within a specified distance of the device is determined.

8. The device according to claim 1, wherein: The at least one processor is configured to: The position of the device is obtained using a position sensor.

9. The device according to claim 1, wherein: The at least one processor is configured to: A maximum noise level associated with the haptic feedback is determined based on the context associated with the device.

10. The apparatus of claim 9, wherein the at least one processor is configured to reduce the tactile feedback of the apparatus to reduce noise associated with the tactile feedback to less than the maximum noise level associated with the tactile feedback.

11. The device according to claim 1, wherein: The at least one processor is configured to: A minimum noise level associated with the haptic feedback is determined based on the context associated with the device.

12. The device according to claim 11, wherein The at least one processor is configured to increase the tactile feedback of the apparatus to increase noise associated with the tactile feedback to greater than the minimum noise level associated with the tactile feedback.

13. The device according to claim 1, wherein: The at least one processor is configured to: An audio volume output of an audio device is adjusted based on the context and the haptic feedback.

14. The device according to claim 1, wherein: The at least one processor is configured to: obtaining, from the at least one sensor, additional environmental information indicative of a change associated with the environment of the apparatus based on a determination of the haptic feedback for output by the apparatus, the determination being based on the context; determining a second context associated with the device based on the additional context information; as well as Second haptic feedback for output by the apparatus is determined based on the second context.

15. A device according to claim 1, wherein the device includes multiple feedback events, wherein a first feedback event among the multiple feedback events includes activating one or more tactile actuators based on a first parameter of the tactile feedback, and wherein a second feedback event among the multiple feedback events includes activating the one or more tactile actuators based on a second parameter of the tactile feedback.

16. The device according to claim 15, wherein: The at least one processor is configured to perform at least one of the following: modifying the first parameter of the haptic feedback associated with the first feedback event based on the context; or The second parameter of the haptic feedback associated with the second feedback event is modified based on the context.

17. The device according to claim 15, wherein: The first parameter of the haptic feedback is associated with at least one of a magnitude of the haptic feedback, a duty cycle of the haptic feedback, a duration of the haptic feedback, or a period of the haptic feedback.

18. The device according to claim 15, wherein: The second parameter of the haptic feedback is associated with preventing actuation of at least one of the one or more haptic actuators.

19. The device according to claim 1, wherein: The device is a mobile device.

20. A method for controlling tactile feedback, comprising: obtaining environmental information indicative of an environment of a computing device from at least one sensor; determining a context associated with the computing device based on the environmental information; as well as Haptic feedback for output by the computing device is determined based on the context.

21. The method according to claim 20, wherein: Determining the tactile feedback of the computing device based on the context includes: An amount of noise associated with the haptic feedback is determined based on a surface in contact with the computing device.

22. The method according to claim 20, wherein: Determining the tactile feedback of the computing device based on the context includes: An amount of noise associated with haptic feedback to be emitted by an application during execution of the application is determined.

23. The method of claim 20, further comprising: identifying an application configured to output a notification with haptic feedback; as well as Haptic feedback associated with notifications for at least a portion of the application is adjusted.

24. The method according to claim 20, wherein: The environmental information includes at least one of ambient audio information of the environment, device information identifying other wireless devices proximate to the computing device, brightness information, information indicating that at least one person has entered or left the environment, or at least one image.

25. The method according to claim 24, wherein: Determining the context includes at least one of the following: Determining whether the ambient audio information is below a threshold for a period of time; determining a number of acoustic devices for outputting audio within a specified distance of the computing device; determining a spectral quality of the ambient audio information; determining a number of computing devices within a specified distance of the computing device; or A number of people within a specified distance of the computing device is determined.

26. The method of claim 20, further comprising: determining a maximum noise level associated with the haptic feedback based on the context associated with the computing device; as well as The haptic feedback of the computing device is reduced to reduce noise associated with the haptic feedback to less than the maximum noise level associated with the haptic feedback.

27. The method of claim 20, further comprising: determining a minimum noise level associated with the haptic feedback based on the context associated with the computing device; as well as The haptic feedback of the computing device is increased to increase noise associated with the haptic feedback to greater than the minimum noise level associated with the haptic feedback.

28. The method of claim 20, further comprising: An audio volume output of an audio device is adjusted based on the context and the haptic feedback.

29. The method of claim 20, further comprising: after determining the haptic feedback for output by the computing device based on the context, obtaining additional environmental information from the at least one sensor indicative of a change associated with the environment of the computing device; determining a second context associated with the computing device based on the additional environmental information; as well as Second haptic feedback is determined for output by the computing device based on the second context.

30. The method of claim 20, wherein: The computing device includes multiple feedback events, wherein a first feedback event in the multiple feedback events includes activating one or more tactile actuators based on a first parameter of the tactile feedback, and wherein a second feedback event in the multiple feedback events includes activating the one or more tactile actuators based on a second parameter of the tactile feedback.

Images