Method and system for generating haptic effects

Abstract

A method may include receiving a first media stream from a first terminal and a second media stream from a second terminal; decoding the first media stream and the second media stream to obtain a first content and a second content in a playable format; associating the first media stream and the second media stream with the first identifier and the second identifier; generating a first initial haptic signal based on the first content and the first identifier, and generating a second initial haptic signal based on the second content and the second identifier; processing the first / second content to generate a first / second analysis; modulating the first initial haptic signal according to a first analysis to output a first haptic control signal, and modulating the second initial haptic signal according to a second analysis to output a second haptic control signal; and driving the actuator according to the first haptic control signal and / or the second haptic control signal.

Description

Technical Field

[0001] The present disclosure generally relates to a method and system for generating haptic effects, particularly from a media stream. Background Art

[0002] Electronic devices provide tactile feedback (such as vibration, texture, and heat) to users, commonly referred to as haptic feedback or haptic effects.

[0003] Therefore, a method or system is needed to provide improved tactile effects, thereby enhancing the user experience. Summary of the invention

[0004] According to a first aspect of the present disclosure, a method for generating a tactile effect is provided. The method may include: receiving a first media stream from a first terminal; receiving a second media stream from a second terminal, the second terminal being different from the first terminal; decoding the first media stream to obtain a first content in a playable format, and decoding the second media stream to obtain a second content in a playable format; associating the first media stream and the second media stream with a first identifier and a second identifier, respectively, the first identifier and / or the second identifier being one of a set of predetermined identifiers; generating a first initial tactile signal based on the first content and the associated first identifier, and generating a second initial tactile signal based on the second content and the associated second identifier; processing the first content to generate a first analysis of the first content, and processing the second content to generate a second analysis of the second content; modulating the first initial tactile signal according to the first analysis to output a first tactile control signal, and modulating the second initial tactile signal according to the second analysis to output a second tactile control signal; and driving an actuator according to the first tactile control signal and / or the second tactile control signal.

[0005] According to a second aspect of the present disclosure, a system for generating a haptic effect is provided. The system may include: at least one processor and at least one memory, the at least one memory including computer program code for one or more programs, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the system to at least: receive a first media stream from a first terminal; receive a second media stream from a second terminal, the second terminal being different from the first terminal; decode the first media stream to obtain a first content in a playable format, and decode the second media stream to obtain a second content in a playable format;

[0006] Associating a first media stream and a second media stream with a first identifier and a second identifier, respectively, the first identifier and / or the second identifier being one of a set of predetermined identifiers; generating a first initial tactile signal based on the first content and the associated first identifier, and generating a second initial tactile signal based on the second content and the associated second identifier; processing the first content to generate a first analysis of the first content, and processing the second content to generate a second analysis of the second content; modulating the first initial tactile signal according to the first analysis to output a first tactile control signal, and modulating the second initial tactile signal according to the second analysis to output a second tactile control signal; and driving an actuator according to the first tactile control signal and / or the second tactile control signal.

[0007] According to a third aspect of the present disclosure, a system for generating a haptic effect is provided. The system may include: a receiver configured to receive a first media stream from a first terminal and a second media stream from a second terminal, the second terminal being different from the first terminal; a media processing module configured to decode the first media stream to obtain a first content in a playable format, and to decode the second media stream to obtain a second content in a playable format, and configured to associate the first media stream and the second media stream with a first identifier and a second identifier, respectively, the first identifier and / or the second identifier being one of a set of predetermined identifiers; a tactile module configured to generate a first initial tactile signal based on the first content and the associated first identifier, and to generate a second initial tactile signal based on the second content and the associated second identifier; a machine learning module configured to process the first content through the machine learning module to generate a first analysis of the first content, and to process the second content to generate a second analysis of the second content; a tactile driver configured to modulate the first initial tactile signal according to the first analysis to output a first tactile control signal, and to modulate the second initial tactile signal according to the second analysis to output a second tactile control signal; and an actuator configured to be driven according to the first tactile control signal and / or the second tactile control signal.

[0008] According to a fourth aspect of the present disclosure, a method for generating a tactile effect is provided. The method may include: receiving a plurality of media streams, each of the plurality of media streams coming from a different terminal among a plurality of terminals; decoding the plurality of media streams to obtain a plurality of contents in a playable format; associating each of the plurality of media streams with a corresponding identifier, the corresponding identifier being selected from a set of predetermined identifiers; generating a plurality of initial tactile signals based on the plurality of contents and the associated corresponding identifiers; processing the plurality of contents to generate a plurality of analyses; modulating the plurality of initial tactile signals according to the plurality of analyses to output a plurality of tactile control signals; and actuating the plurality of tactile control signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1Ais a diagram illustrating an example system according to an embodiment of the present disclosure; Figure 1B is a diagram illustrating an example process according to an embodiment of the present disclosure.

[0010] Figure 2 is a flow chart illustrating an example method according to an embodiment of the present disclosure.

[0011] Figure 3 is a block diagram illustrating an example system according to an embodiment of the present disclosure.

[0012] Figure 4 It shows Figure 3 Block diagram of a media processing module of an example system.

[0013] Figure 5A and Figure 5B It shows Figure 3 Block diagram of a haptic module of an example system.

[0014] Fig. 6A and Figure 6B It shows Figure 3 A diagram of the machine learning module of an example system;

[0015] Figure 7 It shows Figure 3 Block diagram of a haptic driver for an example system.

[0016] Figure 8 It shows that Figure 7 Flowchart of the process performed in a haptic driver.

[0017] Fig.9A and Fig. 9B is a diagram illustrating a corresponding example system according to an embodiment of the present disclosure.

[0018] Fig.10 is a diagram illustrating media streaming occurring in an example system according to an embodiment of the present disclosure.

[0019] Fig.11 is a block diagram illustrating an example electronic device 1100 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] The embodiments described below in the context of an apparatus, device or system are similarly valid for the corresponding methods, and vice versa. In addition, it will be understood that the embodiments described below can be combined, for example, a portion of an embodiment can be combined with a portion of another embodiment, and a portion of an embodiment can be combined with a portion of another embodiment.

[0021] It should be understood that the terms "upper", "above", "top", "bottom", "lower", "side", "rear", "left", "right", "front", "rear", "lateral", "sideways", "upward", "downward", "vertical", "horizontal", etc., when used in the following description, are for convenience and to help understand relative positions or directions, and are not intended to limit the orientation of any device or structure or any part of any device or structure. In addition, the singular terms "a, an" and "the" include plural references unless the context clearly indicates otherwise. Similarly, the word "or" is intended to include "and" unless the context clearly indicates otherwise.

[0022] It will be further understood that the terms "comprise" (and any forms thereof, such as "comprises" and "comprising"), "have" (and any forms thereof, such as "has" and "having"), "include" (and any forms thereof, such as "includes" and "including"), and "contain" (and any forms thereof, such as "contains" and "containing") are open linking verbs. Thus, a method or apparatus that "comprises," "has," "includes," or "contains" one or more steps or elements has those one or more steps or elements, but is not limited to having only those one or more steps or elements. Similarly, a step of a method or an element of an apparatus that "comprises," "has," "includes," or "contains" one or more features has those one or more features, but is not limited to having only those one or more features. Furthermore, a device or structure that is configured in a certain manner is configured in at least that manner, but may also be configured in ways that are not listed.

[0023] Approximations as used herein throughout the specification and claims may be applied to modify any quantitative representation that may vary within an allowable range without causing a change in the basic function associated therewith. Therefore, values ​​modified by one or more terms such as "about," "substantially," and the like are not limited to the exact values ​​specified, but are within an acceptable tolerance range for the operation of the embodiment for its intended application. In some cases, approximations may correspond to the precision of the instrument used to measure the value.

[0024] As used herein, a phrase of the form “at least one of A or B” may include A or B, or both A and B. Correspondingly, a phrase of the form “at least one of A or B or C”, or including further listed items, may include any and all combinations of one or more of the associated listed items.

[0025] Various aspects described herein attempt to provide a system including: a receiver configured to receive a first media stream from a first terminal and a second media stream from a second terminal, the second terminal being different from the first terminal; a media processing module configured to decode the first media stream to obtain a first content in a playable format, and to decode the second media stream to obtain a second content in a playable format, and configured to associate the first media stream and the second media stream with a first identifier and a second identifier, respectively, the first identifier and / or the second identifier being one of a set of predetermined identifiers; a tactile module configured to generate a first initial tactile signal based on the first content and the associated first identifier, and to generate a second initial tactile signal based on the second content and the associated second identifier; a machine learning module configured to process the first content through the machine learning module to generate a first analysis of the first content, and to process the second content to generate a second analysis of the second content; a tactile driver configured to modulate the first initial tactile signal according to the first analysis to output a first tactile control signal, and to modulate the second initial tactile signal according to the second analysis to output a second tactile control signal; and an actuator configured to be driven according to the first tactile control signal and / or the second tactile control signal. A corresponding method is also provided.

[0026] According to various aspects, the proposed system can be configured to simultaneously receive a first media stream from a first device and a second media stream from a second device, and process (e.g., decode, manipulate, mix, combine, analyze, etc.) the first media stream and the second media stream through a single system on chip (SoC) having all modules or two SoCs (where each SoC has all modules). The system can be configured to output a playable media signal and a haptic signal synchronized with the playable media signal.

[0027] According to various aspects, the machine learning module can be configured to generate a first analysis to include information about whether the first content includes unexpected content, and to generate a second analysis to include information about whether the second content includes unexpected content. The proposed system can utilize a machine learning module to detect unexpected content (e.g., voice) that is undesirable for actuating a tactile effect. The machine learning module can be trained by multiple known files, including but not limited to game profiles, voice profiles (e.g., speech), and music profiles, so that the machine learning module can detect unexpected content (e.g., voice). The training profile can be customized by the user as a preference reference. Thus, the user can customize tactile events and non-tactile events by loading a customized training profile into the machine learning module.

[0028] In some cases, various aspects of the systems and techniques described herein provide technical improvements and advantages compared to existing approaches. For example, the proposed system and method can provide an improved user experience for at least the following reasons. The proposed system and method can simultaneously (e.g., simultaneously) process two inputs from different terminals and output a combination (a combination of a first media stream and a second media stream) or a selected (e.g., a first media stream or a second media stream) media and a tactile signal. The proposed method may include associating a media stream / content (e.g., an audio stream / audio content) with an identifier so that if the associated identifier is a predefined identifier (e.g., voice), then the undesirable tactile effect is removed / suppressed / weakened for undesirable content (e.g., audio content). The proposed method may further include generating an analysis including another identifier associated with content (e.g., audio content) through a machine learning module so that if the associated another identifier is a predefined identifier (e.g., voice), then the undesirable tactile effect for undesirable content (e.g., audio content) is further removed / suppressed / weakened. Thus, by doubly determining whether the associated (another) identifier is a predefined identifier (e.g., voice), the proposed system can provide a more accurate haptic effect by providing a more accurate haptic control signal, thereby improving the user experience. In particular, by utilizing a machine learning module, an algorithm for processing content (e.g., audio content) to generate an analysis of the content (e.g., audio content) can be customizable, so that a user can determine whether the content (e.g., audio content) can actuate a haptic effect by, for example, adjusting a training accuracy threshold.

[0029] The proposed method can be expanded to include: receiving multiple media streams, each of the multiple media streams coming from a different terminal among multiple terminals; decoding the multiple media streams to obtain multiple contents (e.g., audio contents) in a playable format; associating each of the multiple media streams with a corresponding identifier, the corresponding identifier being selected from a set of predetermined identifiers; generating multiple initial tactile signals based on the multiple contents (e.g., audio contents) and the associated corresponding identifiers; processing the multiple contents (e.g., audio contents) to generate multiple analyses; modulating the multiple initial tactile signals according to the multiple analyses to output multiple tactile control signals; and driving the actuator according to the multiple tactile control signals.

[0030] The following examples relate to various aspects of the disclosure.

[0031] Example 1 is a method, comprising: receiving a first media stream from a first terminal; receiving a second media stream from a second terminal, the second terminal being different from the first terminal; decoding the first media stream to obtain a first content in a playable format, and decoding the second media stream to obtain a second content in a playable format; associating the first media stream and the second media stream with a first identifier and a second identifier, respectively, the first identifier and / or the second identifier being one of a set of predetermined identifiers; generating a first initial tactile signal based on the first content and the associated first identifier, and generating a second initial tactile signal based on the second content and the associated second identifier; processing the first content to generate a first analysis of the first content, and processing the second content to generate a second analysis of the second content; modulating the first initial tactile signal according to the first analysis to output a first tactile control signal, and modulating the second initial tactile signal according to the second analysis to output a second tactile control signal; and driving an actuator according to the first tactile control signal and / or the second tactile control signal.

[0032] In Example 2, the subject matter of Example 1 may optionally include processing the first content and the second content through a machine learning module, the machine learning module being configured to generate a first analysis to include information about whether the first content includes unexpected content, and to generate a second analysis to include information about whether the second content includes unexpected content.

[0033] In Example 3, the subject matter of Example 2 can optionally include that the machine learning module is trained by processing multiple audio profiles, the multiple audio profiles including a game profile, a speech profile, and / or a music profile.

[0034] In Example 4, the subject matter of Example 3 may optionally include that an accuracy threshold for training is preset.

[0035] In Example 5, the subject matter of Example 1 can optionally include that the steps of generating the first initial haptic signal and generating the second initial haptic signal include generating the first initial haptic signal and the second initial haptic signal using an audio-to-haptic algorithm.

[0036] In Example 6, the subject matter of Example 5 can optionally include selecting the first audio-to-haptic algorithm based on a first identifier associated with the first media stream, and selecting the second audio-to-haptic algorithm based on a second identifier associated with the second media stream.

[0037] In Example 7, the subject matter of Example 1 can optionally include that the set of predetermined identifiers includes identifiers sorted by frequency.

[0038] In Example 7, the subject matter of Example 1 may optionally include that if the first content and / or the second content is associated with a predefined identifier in the group of predetermined identifiers, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal associated with the predefined identifier with a tactile reduction indication, and / or modulating the second initial tactile signal associated with the predefined identifier with a tactile reduction indication.

[0039] In Example 9, the subject matter of Example 8 can optionally include that the set of predetermined identifiers includes voice, media stream, etc., wherein the predefined identifier is voice.

[0040] In Example 10, the subject matter of Example 8 can optionally include that the first analysis includes a first further identifier and the second analysis includes a second further identifier, the first further identifier and / or the second further identifier being one of another set of predetermined identifiers.

[0041] In Example 11, the subject matter of Example 10 may optionally include that if the first content and / or the second content is not associated with a predefined identifier, and if the first analysis includes a first another identifier that is a predefined identifier and / or the second analysis includes a second another identifier that is a predefined identifier, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal to output a first tactile control signal having a tactile reduction indication, and / or modulating the second initial tactile signal to output a second tactile control signal having a tactile reduction indication.

[0042] In Example 12, the subject matter of Example 10 may optionally include, if the first analysis does not include a first another identifier that is a predefined identifier and / or the second analysis does not include a second another identifier that is a predefined identifier, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal to output a first tactile control signal having a tactile pattern, and / or modulating the second initial tactile signal to output a second tactile control signal having a tactile pattern.

[0043] In Example 13, the subject matter of Example 1 can optionally include that the first media stream and the second media stream occur simultaneously.

[0044] In Example 14, the subject matter of Example 13 may optionally include receiving a first media stream from a first terminal at a first time; and receiving a second media stream from a second terminal at a second time alternating with the first time.

[0045] In Example 15, the subject matter of Example 1 can optionally include, prior to the step of driving the actuator, combining the first haptic control signal and the second haptic control signal; and driving the actuator according to the combined haptic signal.

[0046] Example 16 is a system comprising at least one processor and at least one memory, the at least one memory comprising computer program code for one or more programs, the at least one memory and the computer program code being configured to, with the at least one processor, cause the system to at least: receive a first media stream from a first terminal; receive a second media stream from a second terminal, the second terminal being different from the first terminal; decode the first media stream to obtain a first content in a playable format, and decode the second media stream to obtain a second content in a playable format; associate the first media stream and the second media stream with a first identifier and a second identifier, respectively, the first identifier and / or the second identifier being one of a set of predetermined identifiers; generate a first initial tactile signal based on the first content and the associated first identifier, and generate a second initial tactile signal based on the second content and the associated second identifier; process the first content to generate a first analysis of the first content, and process the second content to generate a second analysis of the second content; modulate the first initial tactile signal according to the first analysis to output a first tactile control signal, and modulate the second initial tactile signal according to the second analysis to output a second tactile control signal; and drive an actuator according to the first tactile control signal and / or the second tactile control signal.

[0047] Example 17 is a system comprising: a receiver configured to receive a first media stream from a first terminal and a second media stream from a second terminal, the second terminal being different from the first terminal; a media processing module configured to decode the first media stream to obtain a first content in a playable format, and to decode the second media stream to obtain a second content in a playable format, and configured to associate the first media stream and the second media stream with a first identifier and a second identifier, respectively, the first identifier and / or the second identifier being one of a set of predetermined identifiers; a tactile module configured to generate a first initial tactile signal based on the first content and the associated first identifier, and to generate a second initial tactile signal based on the second content and the associated second identifier; a machine learning module configured to process the first content through the machine learning module to generate a first analysis of the first content, and to process the second content to generate a second analysis of the second content; a tactile driver configured to modulate the first initial tactile signal according to the first analysis to output a first tactile control signal, and to modulate the second initial tactile signal according to the second analysis to output a second tactile control signal; and an actuator configured to be driven according to the first tactile control signal and / or the second tactile control signal.

[0048] In Example 18, the subject matter of Example 17 can optionally include that the machine learning module is configured to generate the first analysis to include information about whether the first content includes undesirable content, and to generate the second analysis to include information about whether the second content includes undesirable content.

[0049] In Example 19, the subject matter of Example 18 can optionally include that the machine learning module is trained by processing multiple audio profiles, the multiple audio profiles including a game profile, a speech profile, and / or a music profile.

[0050] In Example 20, the subject matter of Example 19 may optionally include that an accuracy threshold for training is pre-set.

[0051] In Example 21, the subject matter of Example 17 can optionally include that the haptic module is configured such that generating the first initial haptic signal and generating the second initial haptic signal include generating the first initial haptic signal and the second initial haptic signal using an audio-to-haptic algorithm.

[0052] In Example 22, the subject matter of Example 21 can optionally include selecting the first audio-to-haptic algorithm based on a first identifier associated with the first media stream, and selecting the second audio-to-haptic algorithm based on a second identifier associated with the second media stream.

[0053] In Example 23, the subject matter of Example 17 can optionally include that the set of predetermined identifiers includes identifiers sorted by frequency.

[0054] In Example 24, the subject matter of Example 17 may optionally include that if the first content and / or the second content is associated with a predefined identifier in the group of predetermined identifiers, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal associated with the predefined identifier with a tactile reduction indication, and / or modulating the second initial tactile signal associated with the predefined identifier with a tactile reduction indication.

[0055] In Example 25, the subject matter of Example 24 can optionally include that the set of predetermined identifiers includes voice, media stream, etc., wherein the predefined corresponding identifier is voice.

[0056] In Example 26, the subject matter of Example 24 can optionally include that the first analysis includes a first further identifier and the second analysis includes a second further identifier, the first further identifier or the second further identifier being one of another set of predetermined identifiers.

[0057] In Example 27, the subject matter of Example 26 may optionally include that if the first content and / or the second content is not associated with a predefined identifier, and if the first analysis includes a first another identifier that is a predefined identifier and / or the second analysis includes a second another identifier that is a predefined identifier, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal to output a first tactile control signal having a tactile reduction indication, and / or modulating the second initial tactile signal to output a second tactile control signal having a tactile reduction indication.

[0058] In Example 28, the subject matter of Example 26 may optionally include, if the first analysis does not include a first another identifier that is a predefined identifier and / or the second analysis does not include a second another identifier that is a predefined identifier, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal to output a first tactile control signal having a tactile pattern, and / or modulating the second initial tactile signal to output a second tactile control signal having a tactile pattern.

[0059] In Example 29, the subject matter of Example 17 may optionally include that the receiver is configured to: receive a first media stream from a first terminal at a first time; and receive a second media stream from a second terminal at a second time alternating with the first time.

[0060] In Example 30, the subject matter of Example 17 can optionally include that the system is a wireless headset configured to receive the concurrent first media stream and the second media stream.

[0061] In Example 31, the subject matter of Example 17 may optionally include a mixer configured to combine the first haptic control signal and the second haptic control signal.

[0062] Example 32 is a method comprising: receiving multiple media streams, each of the multiple media streams coming from a different terminal among multiple terminals; decoding the multiple media streams to obtain multiple contents in a playable format; associating each of the multiple media streams with a corresponding identifier, the corresponding identifier being selected from a set of predetermined identifiers; generating multiple initial tactile signals based on the multiple contents and the associated corresponding identifiers; processing the multiple contents to generate multiple analyses; modulating the multiple initial tactile signals based on the multiple analyses to output multiple tactile control signals; and actuating the multiple tactile control signals.

[0063] Example 33 is a computer program comprising instructions for causing the system of claim 17 to perform the steps of the method of Example 1.

[0064] Example 34 is a computer readable medium having stored thereon the computer program of Example 33.

[0065] Figure 1A is a diagram illustrating an example system 103 according to various embodiments of the present disclosure; Figure 1B 1 is a diagram illustrating an example process performed in an example system 103 according to various embodiments of the present disclosure. The example system 103 may include a headset, earbuds, or other electronic device capable of delivering tactile effects. The tactile effect may enhance the interaction by applying force, vibration, or motion to the user and convey useful information to the user through the touch experience. The system 103 may receive input from the first device 101 and the second device 102. The system 103 may receive input from the first device 101 and the second device 102 at the same time. The first device 101 and the second device 102 may include a laptop, a mobile phone, a personal computer, or any electronic device capable of sending input to the system 103. The input provided to the system 103 by the first device 101 and / or the second device 102 may include an audio stream, a video stream, a media stream (e.g., including both video and audio data), etc. or any combination thereof. It should be understood that although the media stream is described as input herein, the input may be any suitable input that can be received / streamed to the system 103 by the system 103 (e.g., a headset), for example, a lighting effect (e.g., a Chroma data stream). Thus, various methods (eg, method 300) and systems (eg, system 103) may be adapted to process any suitable input as described above.

[0066] In one embodiment, if Figure 1A As shown, the system 103 can receive a first media stream 101a from the first device 101 and a second media stream 102a from the second device 102. For example, the first media stream 101a from the first device 101 can include game audio, and the second media stream 102a from the second device 102 can include a music media stream and / or a phone call media stream. In another embodiment, the system 103 can receive a media stream from the first device 101 and a video stream from the second device 102. The system 103 may include an interface, such as Bluetooth, 2.4Ghz wireless, USB, etc., for wireless or wired connection with the first device 101 and the second device 102 to receive input from the first device 101 and the second device 102.

[0067] According to various non-limiting embodiments, the system 103 can process (e.g., decode, manipulate, mix, combine, analyze, etc.) a first input from the first device 101 and a second input from the second device 102 to output a first signal (including but not limited to a playable media signal) and a second signal to actuate a haptic effect, as described herein. For example, the system 103 can process (e.g., decode, manipulate, mix, combine, analyze, etc.) a first media stream 101a from the first device 101 and a second media stream 102a from the second device 102 to output a playable media signal and a signal to actuate a haptic effect, as described herein. Figure 1B As shown. In one embodiment, the output playable media signal may include a combination of a processed first media stream 101a from the first device 101 and a processed second media stream 102a from the second device 102. The output signal for actuating a haptic effect may include a combination of a processed first media stream 101a from the first device 101 and a processed second media stream 102a from the second device 102. The system 103 may be configured to play the combination of the output playable media signals. The system 103 may also be configured to perform a combination of output signals for actuating a haptic effect.

[0068] In another embodiment, the outputted playable media signal may include a first individual media signal from the processed first media stream 101a of the first device 101 and a second individual media signal from the processed second media stream 102a of the second device 102. The output signal for actuating the haptic effect may include a first individual haptic signal from the processed first media stream 101a of the first device 101 and a second individual haptic signal from the processed second media stream 102a of the second device 102. The system 103 may be configured to play the first individual media signal and / or the second individual media signal. The system 103 may also be configured to execute the first individual haptic signal and / or the second individual haptic signal.

[0069] Figure 2is a flowchart showing an example method 200 according to an embodiment of the present disclosure. Method 200 may be implemented in system 103 and systems described below (e.g., systems 300, 900a, 900b, 1000). According to various non-limiting embodiments, method 200 may include (step 201, step 202) receiving a first media stream (e.g., first media stream 101a) from a first terminal (e.g., first device 101) and receiving a second media stream (e.g., second media stream 102a) from a second terminal (e.g., second device 102). In one embodiment, receiving the first media stream may occur simultaneously with receiving the second media stream. In another embodiment, receiving the first media stream may occur before receiving the second media stream. In other words, receiving the second media stream may occur subsequently after receiving the first media stream, for example, while the first media stream is being played (e.g., in system 103), i.e., the first media stream may have been processed (e.g., decoded, manipulated, mixed, combined, analyzed, etc.). In various embodiments, receiving a first media stream from a first terminal may occur at a first time, and receiving a second media stream from a second terminal may occur at a second time that alternates with the first time.

[0070] The second terminal may be different from the first terminal. This may include the first terminal and the second terminal being both disposed in an electronic device capable of providing two outputs (e.g., providing two inputs to the system 103). This may also include the first terminal being separable from the second terminal. The first terminal may be physically located at a location different from the location where the second terminal is located. The first terminal and / or the second terminal may be located in the cloud.

[0071] According to various non-limiting embodiments, method 200 may include (step 203) decoding a first media stream to obtain a first content in a playable format, and decoding a second media stream to obtain a second content in a playable format. The first media stream and the second media stream may be transmitted (e.g., from the first device 101 and the second device 102 to the system 103) in encrypted data packets (e.g., in a non-playable format). Thus, decoding the first media stream and the second media stream may include decrypting the first media stream and the second media stream, for example, from a non-playable format to a playable format, optionally using a pre-stored or algorithmically generated encryption key. In one embodiment, decoding the first media stream may occur simultaneously with decoding the second media stream. In another embodiment, decoding the first media stream may occur before decoding the second media stream. In other words, decoding the second media stream may occur subsequently after decoding the first media stream, for example, while the first media stream is being played (e.g., in the system 103), i.e., the first media stream may have been processed (e.g., decoded, manipulated, mixed, combined, analyzed, etc.).

[0072] According to various non-limiting embodiments, method 200 may include (step 204) associating the first media stream and the second media stream with the first identifier and the second identifier, respectively. This may mean that the first media stream and the second media stream may be classified into categories having identifiers, respectively. The first media stream may be classified into the same category as the second media stream, that is, the first identifier may be the same as the second identifier. The first identifier and / or the second identifier may be one of a set of predetermined identifiers. The set of predetermined identifiers may include voice (e.g., a voice call), a media stream (e.g., an audio stream), etc. The identifier may be associated with the media stream based on a wireless connection setting of the media stream.

[0073] According to various non-limiting embodiments, method 200 may divide the media stream processing into multiple time, frequency, or amplitude based segments. In some embodiments, the set of predetermined identifiers may include identifiers sorted by frequency.

[0074] According to various non-limiting embodiments, method 200 may include (step 205) generating a first initial haptic signal based on the first content and the associated first identifier, and generating a second initial haptic signal based on the second content and the associated second identifier. The first initial haptic signal and the second initial haptic signal may include instructions for haptic effects. Generating the first initial haptic signal and generating the second initial haptic signal may include generating the first initial haptic signal and the second initial haptic signal using an audio-to-haptic algorithm. The first audio-to-haptic algorithm may be selected based on the first identifier associated with the first media stream, and the second audio-to-haptic algorithm may be selected based on the second identifier associated with the second media stream. In the case where the first identifier is the same as the second identifier, the first audio-to-haptic algorithm may be the same as the second audio-to-haptic algorithm. In one example, the audio-to-haptic algorithm may be selected based on frequency. The audio-to-haptic algorithm may be a set of mathematical equations (e.g., an algorithm) to convert a media signal into a haptic pattern. Content identified as gaming audio may opt to use an audio-to-haptic algorithm that uses a wider frequency range to provide full haptic effects, while content with both gaming audio and speech (e.g., voice) may opt to use an audio-to-haptic algorithm that uses a narrower frequency range (lower than typical human vocals) to minimize unwanted haptic effects from the human vocals.

[0075] According to various non-limiting embodiments, method 200 may include (step 206) processing the first content by a machine learning module to generate a first analysis of the first content, and processing the second content to generate a second analysis of the second content. The machine learning module may be configured to generate the first analysis to include information about whether the first content includes undesirable content, and to generate the second analysis to include information about whether the second content includes undesirable content. Undesirable content may include human voices or game narration that are typically undesirable for actuating haptic effects. The machine learning module may be trained by processing multiple audio profiles, including game profiles, voice profiles, and / or music profiles. An accuracy threshold for training may be pre-set. This may mean that training is complete once the accuracy is above a threshold.

[0076] According to various non-limiting embodiments, method 200 may include (step 207) modulating the first initial haptic signal to output a first haptic control signal based on the first analysis, and modulating the second initial haptic signal to output a second haptic control signal based on the second analysis. The first haptic control signal and the second haptic control signal may include instructions for actuating a haptic effect.

[0077] According to various non-limiting embodiments, method 200 may include (step 208) driving an actuator according to a first tactile control signal and / or a second tactile control signal. The actuator may be configured to move a touch surface that provides a vibrotactile tactile effect in response to the first tactile control signal and / or the second tactile control signal. Some tactile effects may utilize an actuator coupled to a housing of a system (e.g., system 103), and some tactile effects may use multiple actuators in sequence and / or in coordination. In some embodiments, a touch surface may be simulated by vibrating the surface at different frequencies. In such an embodiment, the actuator may include, for example, one or more of a piezoelectric actuator, a motor, an electromagnetic actuator, a voice coil, a shape memory alloy, an electroactive polymer, a solenoid, an eccentric rotating mass motor (ERM), or a linear resonant actuator (LRA).

[0078] According to various non-limiting embodiments, method 200 may process (e.g., step 204) a media stream (e.g., content) to analyze separately whether certain components (e.g., speech, special effects, background noise, or music) are present. Method 200 may associate content with identifiers, such as classifying each content based on the presence of one or more components. For example, content including sounds associated with gunshots, explosions, and car accelerations may be associated with the identifier "game audio." Further, method 200 may generate (e.g., step 205) an initial haptic signal based on the content associated with the identifier "game audio," the initial haptic signal having instructions for a specific haptic effect or a set of haptic effects (e.g., high intensity vibrations synchronized with the occurrence of components such as gunshots and explosions).

[0079] According to various non-limiting embodiments, method 200 may process a media stream to isolate (e.g., step 206) one or more components (e.g., undesirable content) in the media stream. For example, method 200 may analyze a media stream to detect and isolate various sounds. In an embodiment, a media stream may include a mixed audio signal (e.g., a signal including speech, special effects (e.g., explosions, gunshots, mechanical noises), animal sounds, or musical instruments (e.g., piano, guitar, drums, machines, etc.)). In such an embodiment, method 200 may isolate certain sounds in the media stream, such as isolating speech, and method 200 may associate (e.g., step 206) another identifier to the remaining component(s). In some embodiments, method 200 may isolate multiple sources and associate (e.g., step 206) another identifier to one or more of the multiple sources of the remaining component(s). For example, in an illustrative embodiment, method 200 may separate game narration from gunshots. In such an embodiment, method 200 may associate gunshots with another identifier "game audio". Further, in one embodiment, method 200 can isolate a component (e.g., a game narrative) and determine to modulate (e.g., step 207) an initial haptic signal with a haptic reduction indication. For example, in one embodiment, method 200 can modulate (e.g., step 207) an initial haptic signal with a haptic reduction indication to produce a haptic effect associated with the component. The initial haptic signal with the haptic reduction indication can actuate a reduced or suppressed haptic effect.

[0080] As will be discussed in further detail below, any number of components may be found in a media stream. Embodiments of the present disclosure provide systems and methods for identifying these components and then determining and outputting haptic effects synchronized with these components. Further, in some embodiments, the systems and methods discussed herein may be used to determine haptic effects associated with other types of signals (e.g., pressure, acceleration, velocity, or temperature signals).

[0081] Although the above method is illustrated and described as a series of steps or events, it should be understood that any ordering of such steps or events should not be interpreted as having a restrictive meaning. For example, some steps may occur in different orders and / or occur simultaneously with other steps or events other than the steps or events illustrated and / or described herein. In addition, all illustrated steps will not be required to realize one or more aspects or embodiments described herein. Moreover, one or more steps described herein can be performed in one or more separated actions and / or stages.

[0082] Figure 3 is a block diagram illustrating an example system 300 according to an embodiment of the present disclosure. Figure 4 It shows Figure 3 1 and 10. Block diagram of media processing modules 311a, 311b of an example system 300. Figure 5A and Figure 5B It shows Figure 3 3. A block diagram of a haptic module 320 of an example system 300 is shown. Fig. 6A and Figure 6B It shows Figure 3 FIG. 3 is a diagram of a machine learning module 330 of an example system 300 . Figure 7 It shows Figure 3 300 of the example system 300. Figure 8 It shows that Figure 7 Flow chart of the process performed in the haptic driver 340. Figure 3 to Figure 4 , FIG. 5A to FIG. 5B , FIG. 6A to FIG. 6B as well as Figures 7 and 8 3. The system 300 may be a wireless headset configured to simultaneously receive a first media stream and a second media stream.

[0083] According to various non-limiting embodiments, the system 300 may include receivers 311a, 311b, media processing modules 312a, 312b, a haptic module 320, a machine learning (ML) module 330, a haptic driver 340, and an actuator 350. Some components of the system 300, such as Figure 3 The receivers 311a, 311b and the media processing modules 312a, 312b collectively shown in the figure may be separated as separate components. Some components of the system 300, such as Figure 3 The haptic module 320 and the ML module 330 shown separately in FIG. 3 may be combined into a single component. Also, the system 300 may include Figure 3 (multiple) additional components not shown.

[0084] In various embodiments, receivers 311a, 311b may be configured to receive a first media stream (e.g., first media stream 101a) from a first terminal (first device 101) and a second media stream (e.g., second media stream 102a) from a second terminal (second device 102). This may mean that receiver 311a may be configured to receive a first media stream (e.g., first media stream 101a) from a first terminal (first device 101), and receiver 311b may be configured to receive a second media stream (e.g., second media stream 102a) from a second terminal (second device 102). In one embodiment, receiver 311a may receive the first media stream at the same time as receiver 311b receives the second media stream. By "simultaneously", the first terminal and the second terminal may actively transmit media streams to the system 300 at the same time (e.g., a period of time). "Simultaneously" may not be used to restrict transmissions from the first terminal and the second terminal to occur in the same single time slot of a radio frequency channel. In another embodiment, receiver 311a may receive the first media stream before receiver 311b receives the second media stream. In other words, receiver 311b may receive the second media stream subsequently after receiver 311a receives the first media stream, e.g., while the first media stream is being played (e.g., in system 103), i.e., the first media stream may have been processed (e.g., decoded, manipulated, mixed, combined, analyzed, etc.). In various embodiments, receiver 311a may receive the first media stream from the first terminal at a first time, and receiver 311b may receive the second media stream from the second terminal at a second time alternating with the first time.

[0085] The second terminal may be different from the first terminal. This may mean that the first terminal and the second terminal are both arranged in an electronic device capable of providing two outputs (e.g., providing two inputs to the system 103). This may also mean that the first terminal may be separated from the second terminal. The first terminal may be physically located at a location different from the location where the second terminal is located. The first terminal and / or the second terminal may be located in the cloud. Receivers 311a, 311b may receive input (e.g., a first media stream, a second media stream) wirelessly or wired. Receivers 311a, 311b may include an interface for receiving input. The interface may include one or more adapters, modems, connectors, jacks, terminals, ports, slots, etc.

[0086] refer to Figure 3 and Figure 4In various embodiments, the media processing modules 312a, 312b may include a media stream input module 312c, a media control module 312d, a media decoding module 312e, and a media stream output module 312f. The receivers 311a, 311b may transmit the received first media stream and the received second media stream to the media stream input module 312c of the media processing modules 312a, 312b. The media processing modules 312a, 312b may be configured to decode the first media stream through the media decoding module 312e to obtain the first content in a playable format, and to decode the second media stream through the media decoding module 312e to obtain the second content in a playable format. This may mean that the media processing module 312a may be configured to decode the first media stream to obtain the first content in a playable format, and the media processing module 312b may be configured to decode the second media stream to obtain the second content in a playable format.

[0087] The media processing modules 312a, 312b may also be configured to associate the first media stream and the second media stream with the first identifier and the second identifier, respectively, through the media control module 312d. This may mean that the media processing module 312a may be configured to associate the first media stream with the first identifier, and the media processing module 312b may be configured to associate the second media stream with the second identifier. The first identifier and / or the second identifier may be one of a set of predetermined identifiers. The set of predetermined identifiers may include voice (e.g., voice call), media stream (e.g., audio stream), etc. The identifier may be associated with the media stream based on the wireless connection setting of the media stream. For example, the first identifier (e.g., media stream) may be assigned to the first terminal device based on the first wireless connection setting (e.g., via Wi-Fi) between the first terminal device (e.g., device 101) and the system 300; the second identifier (e.g., voice) may be assigned to the second terminal device based on the second wireless setting (e.g., via Bluetooth) between the second terminal device (e.g., device 102) and the system 300. The set of predetermined identifiers may include identifiers classified by frequency.

[0088] The media processing modules 312a and 312b may be further configured to transmit the first content and the associated first identifier output by the media stream output module 312f of the media processing module 312a and the second content and the associated second identifier output by the media stream output module 312f of the media processing module 312b to the haptic module 320. The media processing modules 312a and 312b may be configured to transmit the first content and the second content to the ML module 330. Further, the media processing modules 312a and 312b may be configured to transmit the associated first identifier and the associated second identifier to the ML module 330.

[0089] Reference now Figure 3 as well as FIG. 5A to FIG. 5B In various embodiments, the haptic module 320 may include a memory 321 on which an audio-to-haptic algorithm is stored, a content type selector 322, and a haptic output module 323. The haptic module 320 may receive a first content and an associated first identifier and a second content and an associated second identifier from the media processing modules 312a, 312b. The haptic module 320 may be configured to generate a first initial haptic signal based on the first content and the associated first identifier, and to generate a second initial haptic signal based on the second content and the associated second identifier. The first initial haptic signal and the second initial haptic signal may include instructions for a haptic effect. Generating the first initial haptic signal and generating the second initial haptic signal may include generating the first initial haptic signal and the second initial haptic signal using the audio-to-haptic algorithm. The first audio-to-haptic algorithm (e.g., 322a or 322b or 322c) can be selected by the content type selector 322 based on the first identifier associated with the first media stream, and the second audio-to-haptic algorithm (e.g., 322a or 322b or 322c) can be selected by the content type selector 322 based on the second identifier associated with the second media stream. When the first identifier is the same as the second identifier, the first audio-to-haptic algorithm can be the same as the second audio-to-haptic algorithm. The first initial haptic signal and the second initial haptic signal can be output by the haptic output module 323 to the haptic driver 340.

[0090] Reference now Figure 3 as well as FIG. 6A to FIG. 6B In various embodiments, the first content and the second content may be transmitted to the ML module 330 by the media processing modules 312a, 312b. The ML module 330 may be trained by processing a plurality of audio profiles (e.g., 301) as training data 331a. The plurality of audio profiles may include a game profile, a voice profile, and / or a music profile. The training data 331a may be processed using a learning / training algorithm 331c and weights may be modified in a neural network (NN) model 331b. The prediction 331d of the processed training data may be compared 331g with the target output 331e. The error signal 331f may be fed back to the learning / training algorithm 331c, and the weights may be modified accordingly. An accuracy threshold for the training data (e.g., the difference between the prediction 331d of the processed training data and the target output 331e) may be pre-set. This may mean that once the accuracy is above a threshold, the training is complete.

[0091] ML module 330 may be configured to process (e.g., using deep neural network (DNN) model 332) first content to generate a first analysis of the first content, and to process (e.g., using deep neural network (DNN) model 332) second content to generate a second analysis of the second content. ML module 330 may be configured to generate the first analysis to include information about whether the first content includes undesirable content, and to generate the second analysis to include information about whether the second content includes undesirable content. Undesirable content may include human voice or game narration that is generally undesirable for actuating haptic effects. The first analysis may include a first further identifier, and the second analysis may include a second further identifier. The first further identifier or the second further identifier may be one of another set of predetermined identifiers.

[0092] In various embodiments, the set of predetermined identifiers may be considered as first-level identifiers, and another set of predetermined identifiers may be considered as second-level identifiers. This may mean that the second-level identifier in the other set of predetermined identifiers is another (e.g., refined, fine-tuned, subdivided, detailed) identifier of the first-level identifier in the set of predetermined identifiers. The other (e.g., refined, fine-tuned, subdivided, detailed) identifier may include a phone call, a voice call, a team call, music, game narration, game audio, etc. The other set of predetermined identifiers may include identifiers in the set of predetermined identifiers that have (e.g., are tagged or attached or marked with) another (e.g., refined, fine-tuned, subdivided, detailed) identifier, such as a voice with another (e.g., refined, fine-tuned, subdivided, detailed) identifier game narration, a media stream with another (e.g., refined, fine-tuned, subdivided, detailed) identifier game audio, etc.

[0093] The ML module 330 may further analyze the content to classify (e.g., refine, tune, subdivide, elaborate) the first level identifier (e.g., speech or media stream) with the second level identifier (e.g., speech with another identifier game narration or a media stream with another identifier game audio). The ML module 330 may be provided to further classify (e.g., refine, tune, subdivide, elaborate) the audio content to identify content that is not desired for haptic effects.

[0094] In various embodiments, the other set of predetermined identifiers may include another (e.g., refined, fine-tuned, segmented, detailed) identifier, such as phone call, voice call, team call, music, game narration, game audio, etc. The ML module 330 may further analyze the content to replace / substitute / replace an identifier (e.g., voice or media stream) in the set of predetermined identifiers with another identifier in the other set of predetermined identifiers (e.g., voice with another identifier voice call, or media stream with another identifier game audio). The ML module 330 may be provided to further classify (e.g., replace, substitute, replace) the audio content in order to identify content that is not desired for the haptic effect.

[0095] The first analysis and the second analysis of the first content and the second content may be output to the haptic driver 340 through the ML output module 333 .

[0096] Reference now Figure 3 and Figure 7 In various embodiments, the haptic driver 340 may include a haptic control module 341 and an actuator driver 342. The haptic driver 340 may be configured to receive a first analysis and a second analysis of the first content and the second content from the ML module 330, and to receive the first content and the second content and the associated first identifier and second identifier from the haptic module 320. The haptic driver 340 may be configured to modulate the first initial haptic signal according to the first analysis to output a first haptic control signal through the haptic control module 341, and to modulate the second initial haptic signal according to the second analysis to output a second haptic control signal through the haptic control module 341. The actuator driver 342 may be configured to output the first control signal and the second control signal to the actuator 350. The first haptic control signal and the second haptic control signal may include instructions for actuating a haptic effect.

[0097] Figure 8A process 800 is depicted that is performed by the haptic driver 340, and in particular by the haptic control module 341. According to various non-limiting embodiments, the haptic driver 340 can determine whether the first content and / or the second content is associated with a predefined identifier in the set of predetermined identifiers 801. The predefined corresponding identifiers may include one or more identifiers, such as speech. If the first content and / or the second content is associated with a predefined identifier in the set of predetermined identifiers, the haptic driver 340 can be configured to modulate the first initial haptic signal associated with the predefined identifier with a haptic reduction indication and / or modulate the second initial haptic signal associated with the predefined identifier with a haptic reduction indication 802. If the first content and / or the second content is not associated with a predefined identifier in the set of predetermined identifiers, the haptic driver 340 can be configured to determine whether the first analysis includes a first another identifier that is a predefined identifier and / or whether the second analysis includes a second another identifier that is a predefined identifier 803. If the first analysis includes the first another identifier that is a predefined identifier and / or the second analysis includes the second another identifier that is a predefined identifier, the haptic driver 340 may be configured to modulate the first initial haptic signal to output a first haptic control signal with a haptic reduction indication and / or modulate the second initial haptic signal to output a second haptic control signal with a haptic reduction indication 804. If the first analysis does not include the first another identifier that is a predefined identifier and / or the second analysis does not include the second another identifier that is a predefined identifier, the haptic driver 340 may be configured to modulate the first initial haptic signal to output a first haptic control signal with a haptic pattern and / or modulate the second initial haptic signal to output a second haptic control signal with a haptic pattern 805.

[0098] In various embodiments, if an identifier in the set of predetermined identifiers (e.g., voice or media stream) is replaced / substituted / superseded by another identifier in another set of predetermined identifiers, then at step 803, the haptic driver 340 can be configured to determine whether the first analysis includes a first another identifier that is another predefined identifier in another set of predetermined identifiers and / or whether the second analysis includes a second another identifier that is another predefined identifier. Another predefined identifier may include one or more identifiers, such as a voice call, a game narration.

[0099] In various embodiments, the actuator 350 can be configured to be driven according to the first tactile control signal and / or the second tactile control signal. The actuator 350 can be configured to move the touch surface that provides the tactile effect of vibrotactile in response to the first tactile control signal and / or the second tactile control signal. Some tactile effects can utilize an actuator coupled to a housing of a system (e.g., system 103), and some tactile effects can use multiple actuators in sequence and / or in coordination. In some embodiments, the touch surface can be simulated by vibrating the surface at different frequencies. In such an embodiment, the actuator 350 can include, for example, one or more of a piezoelectric actuator, a motor, an electromagnetic actuator, a voice coil, a shape memory alloy, an electroactive polymer, a solenoid, an eccentric rotating mass motor (ERM), or a linear resonant actuator (LRA).

[0100] According to various non-limiting embodiments, system 300 can further include a mixer (not shown) configured to combine the first haptic control signal and the second haptic control signal. Thus, actuator 350 can be configured to be driven according to the combined haptic signal.

[0101] Fig.9A and Fig. 9B 900a, 900b according to various embodiments of the present disclosure. The systems 900a, 900b may include at least one processor and at least one memory, the at least one memory including computer program code for one or more programs, the at least one memory and the computer program code being configured to, with the at least one processor, cause the system to at least: receive a first media stream from a first terminal (e.g., a first device 901); receive a second media stream from a second terminal (e.g., a second device 902), the second terminal being different from the first terminal; decode the first media stream to obtain a first content in a playable format, and decode the second media stream to obtain a second content in a playable format; associate the first media stream and the second media stream with a first identifier and a second identifier, respectively. The invention relates to a method for controlling a touch screen device and a touch screen component and a device for controlling a touch screen. The method comprises: a step of: generating a first tactile signal based on the first content and the associated first identifier, wherein the first identifier and / or the second identifier is one of the predetermined group of identifiers; generating a first initial tactile signal based on the first content and the associated first identifier, and generating a second initial tactile signal based on the second content and the associated second identifier; processing the first content through a machine learning module to generate a first analysis of the first content, and processing the second content to generate a second analysis of the second content; modulating the first initial tactile signal according to the first analysis to output a first tactile control signal, and modulating the second initial tactile signal according to the second analysis to output a second tactile control signal; and driving the actuator according to the first tactile control signal and / or the second tactile control signal.

[0102] The systems 900a and 900b may receive input from a first device 901 (e.g., a personal computer) and a second device 902 (e.g., a mobile phone). The systems 900a and 900b may receive input from the first device 901 and the second device 902 at the same time. The system 900a may include a first wireless system on chip (SoC) 911a and a second wireless SoC 912a. The first SoC 911a may be configured to receive input from the first device 901, and the second SoC 912a may be configured to receive input from the second device 902. A receiver (e.g., receiver 311a / 311b), a media processing module (e.g., media processing module 312a / 312b), a haptic module (e.g., haptic module 320), an ML module (ML module 330), a haptic driver (haptic driver 340), and an actuator (e.g., actuator 350) may be arranged in each of the first SoC 911a and the second SoC 912a.

[0103] The system 900b may include a wireless SoC 910b. The SoC 910b may be configured to receive input from both the first device 901 and the second device 902. Two receivers (e.g., receivers 311a, 311b), two media processing modules (e.g., media processing modules 312a, 312b), a haptic module (e.g., haptic module 320), an ML module (ML module 330), a haptic driver (haptic driver 340), and an actuator (e.g., actuator 350) may be arranged in the SoC 910b.

[0104] Systems 900a, 900b can process (e.g., decode, manipulate, mix, combine, analyze, etc.) input from the first device 901 and the second device 902, and output playable content through media processing 920a, 920b and tactile control signals (e.g., tactile effects) through tactile processing 930a, 930b.

[0105] Fig.10 1000 is a diagram illustrating media streaming occurring in an example system according to an embodiment of the present disclosure. The example system may include a multiplexer capable of receiving multiple inputs, selecting among the multiple inputs, and forwarding the selected input to an output. The x-axis 1010 of the diagram 1000 represents time, and the y-axis 1020 of the diagram 1000 represents radio frequency. The example system may receive a first media stream (denoted as 1001) from a first device and a second media stream (denoted as 1002) from a second device.

[0106] According to various non-limiting embodiments, the system may receive a first media stream from a first device at a first time and receive a second media stream from a second device at a second time that is alternating with the first time. In other words, the system may alternately receive the first media stream from the first device and the second media stream from the second device in a periodic manner. This may mean that the system receives one media stream over the RF channel at any one time. This may also mean that the system receives the first media stream over the first RF channel at a first time and receives the second media stream over the first RF channel or the second RF channel at a second time.

[0107] Fig.11 1 is a block diagram illustrating an example electronic device 1100 according to an embodiment of the present disclosure. The electronic device 1100 may be a laptop, a desktop computer, a tablet computer, a car computer, a gaming device, a smart phone, a personal digital assistant, a server, or other electronic device capable of running computer applications. In some embodiments, the electronic device 1100 includes a processor 1102, an input / output (I / O) module 1104, a memory 1106, a power unit 1108, and one or more network interfaces 1110. The electronic device 1100 may include additional components. In some embodiments, the processor 1102, the input / output (I / O) module 1104, the memory 1106, the power unit 1108, and the network interface 1110 are housed together in a common housing or other components.

[0108] The example processor 1102 can execute instructions, for example, to generate output data based on data input. Instructions may include programs, codes, scripts, modules, or other types of data stored in a memory (e.g., memory 1106). Additionally or alternatively, instructions may be encoded as preprogrammed or reprogrammable logic circuits, logic gates, or other types of hardware or firmware components or modules. The processor 1102 may be a multi-core processor with multiple cores, or may include a multi-core processor with multiple cores, and each such core may have an independent power domain and may be configured to enter and exit different operating or performance states based on workload. Additionally or alternatively, the processor 1102 may be a general-purpose microprocessor as a dedicated coprocessor or another type of data processing device, or may include a general-purpose microprocessor as a dedicated coprocessor or another type of data processing device. In some cases, the processor 1102 performs high-level operations of the electronic device 1100. For example, the processor 1102 may be configured to execute or interpret software, scripts, programs, functions, executable files, or other instructions stored in the memory 1106.

[0109] Example I / O module 1104 may include a mouse, keyboard, touch screen, scanner, optical reader and / or stylus (or other input device) through which a user of electronic device 1100 can provide input to electronic device 1100, and may also include one or more speakers for providing audio output and a video display device for providing text, audiovisual and / or graphical output.

[0110] The example memory 1106 may include a computer-readable storage medium, such as a volatile memory device, a non-volatile memory device, or both. The memory 1106 may include one or more read-only memory devices, random access memory devices, buffer memory devices, or a combination of these and other types of memory devices. In some cases, one or more components of the memory may be integrated or otherwise associated with another component of the electronic device 1100. The memory 1106 may store instructions that can be executed by the processor 1102. In some examples, the memory 1106 may store instructions for an operating system 1112 and instructions for an application 1114. The memory 1106 may also store a database 1116.

[0111] The example power unit 1108 provides power to other components of the electronic device 1100. For example, other components may operate based on power provided by the power unit 1108 through a voltage bus or other connection. In some embodiments, the power unit 1108 includes a battery or battery system, such as a rechargeable battery. In some embodiments, the power unit 1108 includes an adapter (e.g., an AC adapter) that receives an external power signal (from an external source) and converts the external power signal into an internal power signal regulated for the components of the electronic device 1100. The power unit 1108 may include other components or operate in other ways.

[0112] The electronic device 1100 may be configured to operate in a wireless, wired, or cloud network environment (or a combination thereof). In some embodiments, the electronic device 1100 may access a network using (multiple) network interfaces 1110. The (multiple) network interfaces 1110 may include one or more adapters, modems, connectors, jacks, terminals, ports, slots, and the like. The wireless network accessed by the electronic device 1100 may operate, for example, according to a wireless network standard or another type of wireless communication protocol. For example, the wireless network may be configured to operate as a wireless local area network (WLAN), a personal area network (PAN), a metropolitan area network (MAN), or another type of wireless network. Examples of WLANs include networks configured to operate according to one or more standards in the 802.11 series of standards developed by the IEEE (e.g., Wi-Fi networks), and the like. Examples of PANs include networks configured to operate according to short-range communication standards (e.g., The wired network accessed by the electronic device 1100 may include, for example, Ethernet, SONET, a circuit switching network (for example, using components such as SS7, cables, etc.), etc.

[0113] Various aspects of what is described herein provide a method or system for providing improved haptic effects, thereby enhancing the user experience.

[0114] Some of the themes and operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Some of the themes described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, which are encoded on a computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. A computer storage medium may be or may be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although a computer storage medium is not a propagation signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagation signal. A computer storage medium may also be one or more separate physical components or media (e.g., multiple CDs, disks or other storage devices) or be included in one or more separate physical components or media.

[0115] Some of the operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0116] The term "data processing apparatus" covers all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or multiple or combinations of the foregoing. The apparatus may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform operating environment, a virtual machine, or a combination of one or more thereof.

[0117] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer, or on multiple computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0118] Some of the processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits (e.g., FPGA (field programmable gate array) or ASIC (application specific integrated circuit)), and the device can also be implemented as a special purpose logic circuit.

[0119] Although this specification contains many details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that are unique to a particular example. Certain features described in this specification or shown in the accompanying drawings in the context of separate embodiments may also be combined. Conversely, various features described or shown in the context of a single embodiment may also be implemented in multiple embodiments, either alone or in any suitable subcombination.

[0120] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single product or packaged into multiple products.

[0121] Various embodiments have been described. However, it should be understood that various modifications can be made. Therefore, other embodiments are within the scope of the following claims.

Claims

1. A method comprising: Receiving a first media stream from a first terminal; receiving a second media stream from a second terminal, the second terminal being different from the first terminal; decoding the first media stream to obtain first content in a playable format, and decoding the second media stream to obtain second content in a playable format; Associating the first media stream and the second media stream with a first identifier and a second identifier, respectively, wherein the first identifier and / or the second identifier is one of a group of predetermined identifiers; generating a first initial haptic signal based on the first content and the associated first identifier, and generating a second initial haptic signal based on the second content and the associated second identifier; processing the first content to generate a first analysis of the first content, and processing the second content to generate a second analysis of the second content; modulating the first initial haptic signal according to the first analysis to output a first haptic control signal, and modulating the second initial haptic signal according to the second analysis to output a second haptic control signal; as well as An actuator is driven according to the first haptic control signal and / or the second haptic control signal.

2. The method of claim 1, wherein: The first content and the second content are processed by a machine learning module, and the machine learning module is configured to generate the first analysis to include information about whether the first content includes undesirable content, and to generate the second analysis to include information about whether the second content includes undesirable content.

3. The method of claim 2, wherein: The machine learning module is trained by processing a plurality of audio profiles, including a gaming profile, a speech profile, and / or a music profile.

4. The method of claim 3, wherein: The accuracy threshold used for the training is preset.

5. The method of claim 1, wherein: The steps of generating a first initial haptic signal and generating a second initial haptic signal include generating the first initial haptic signal and the second initial haptic signal using an audio-to-haptic algorithm.

6. The method of claim 5, wherein: A first audio-to-haptic algorithm is selected based on the first identifier associated with the first media stream, and a second audio-to-haptic algorithm is selected based on the second identifier associated with the second media stream.

7. The method of claim 1, wherein: The set of predetermined identifiers includes identifiers sorted by frequency.

8. The method according to claim 1, in, If the first content and / or the second content are associated with a predefined identifier in the group of predetermined identifiers, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal associated with the predefined identifier with a tactile reduction indication, and / or modulating the second initial tactile signal associated with the predefined identifier with a tactile reduction indication.

9. The method of claim 8, wherein: The set of predetermined identifiers includes voice, media stream, etc., wherein the predefined identifier is voice.

10. The method of claim 8, wherein: The first analysis comprises a first further identifier and the second analysis comprises a second further identifier, the first further identifier and / or the second further identifier being one of a further set of predetermined identifiers.

11. The method of claim 10, wherein: If the first content and / or the second content is not associated with the predefined identifier, and if the first analysis includes the first another identifier that is the predefined identifier and / or the second analysis includes the second another identifier that is the predefined identifier, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal to output the first tactile control signal having the tactile reduction indication, and / or modulating the second initial tactile signal to output the second tactile control signal having the tactile reduction indication.

12. The method of claim 10, wherein: If the first analysis does not include the first another identifier that is the predefined identifier and / or the second analysis does not include the second another identifier that is the predefined identifier, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal to output the first tactile control signal having a tactile pattern, and / or modulating the second initial tactile signal to output the second tactile control signal having a tactile pattern.

13. The method of claim 1, wherein: The first media stream and the second media stream occur simultaneously.

14. The method of claim 13, further comprising: receiving the first media stream from the first terminal at a first time; The second media stream is received from the second terminal at a second time alternating with the first time.

15. The method of claim 1, before the step of driving the actuator, further comprising: combining the first haptic control signal and the second haptic control signal; as well as The actuator is driven according to the combined haptic signal.

16. A system comprising at least one processor and at least one memory, the at least one memory comprising computer program code for one or more programs, the at least one memory and the computer program code being configured to, with the at least one processor, cause the system to at least: Receiving a first media stream from a first terminal; receiving a second media stream from a second terminal, the second terminal being different from the first terminal; decoding the first media stream to obtain first content in a playable format, and decoding the second media stream to obtain second content in a playable format; Associating the first media stream and the second media stream with a first identifier and a second identifier, respectively, wherein the first identifier and / or the second identifier is one of a group of predetermined identifiers; generating a first initial haptic signal based on the first content and the associated first identifier, and generating a second initial haptic signal based on the second content and the associated second identifier; processing the first content to generate a first analysis of the first content, and processing the second content to generate a second analysis of the second content; modulating the first initial haptic signal according to the first analysis to output a first haptic control signal, and modulating the second initial haptic signal according to the second analysis to output a second haptic control signal; as well as An actuator is driven according to the first haptic control signal and / or the second haptic control signal.

17. A system comprising: a receiver configured to receive a first media stream from a first terminal and a second media stream from a second terminal, the second terminal being different from the first terminal; a media processing module, the media processing module being configured to decode the first media stream to obtain first content in a playable format, and to decode the second media stream to obtain second content in a playable format, and being configured to associate the first media stream and the second media stream with a first identifier and a second identifier, respectively, the first identifier and / or the second identifier being one of a group of predetermined identifiers; a haptic module configured to generate a first initial haptic signal based on the first content and the associated first identifier, and to generate a second initial haptic signal based on the second content and the associated second identifier; a machine learning module configured to process the first content via the machine learning module to generate a first analysis of the first content, and to process the second content to generate a second analysis of the second content; a haptic driver configured to modulate the first initial haptic signal to output a first haptic control signal according to the first analysis, and to modulate the second initial haptic signal to output a second haptic control signal according to the second analysis; as well as An actuator is configured to be driven according to the first haptic control signal and / or the second haptic control signal.

18. The system of claim 17, wherein: The machine learning module is configured to generate the first analysis to include information regarding whether the first content includes undesirable content, and to generate the second analysis to include information regarding whether the second content includes undesirable content.

19. The system of claim 18, wherein: The machine learning module is trained by processing a plurality of audio profiles, including a gaming profile, a speech profile, and / or a music profile.

20. The system of claim 19, wherein: The accuracy threshold used for the training is preset.

21. The system of claim 17, wherein: The haptic module is configured such that generating a first initial haptic signal and generating a second initial haptic signal include generating the first initial haptic signal and the second initial haptic signal using an audio-to-haptic algorithm.

22. The system of claim 21, wherein: A first audio-to-haptic algorithm is selected based on the first identifier associated with the first media stream, and a second audio-to-haptic algorithm is selected based on the second identifier associated with the second media stream.

23. The system of claim 17, wherein: The set of predetermined identifiers includes identifiers sorted by frequency.

24. The system of claim 17, in, If the first content and / or the second content are associated with a predefined identifier in the group of predetermined identifiers, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal associated with the predefined identifier with a tactile reduction indication, and / or modulating the second initial tactile signal associated with the predefined identifier with a tactile reduction indication.

25. The system of claim 24, wherein: The set of predetermined identifiers includes voice, media stream, etc., wherein the predefined corresponding identifier is voice.

26. The system of claim 24, wherein: The first analysis comprises a first further identifier and the second analysis comprises a second further identifier, the first further identifier or the second further identifier being one of a further set of predetermined identifiers.

27. The system of claim 26, wherein: If the first content and / or the second content is not associated with the predefined identifier, and if the first analysis includes the first another identifier that is the predefined identifier and / or the second analysis includes the second another identifier that is the predefined identifier, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal to output the first tactile control signal having the tactile reduction indication, and / or modulating the second initial tactile signal to output the second tactile control signal having the tactile reduction indication.

28. The system of claim 26, wherein: If the first analysis does not include the first another identifier that is the predefined identifier and / or the second analysis does not include the second another identifier that is the predefined identifier, the steps of modulating the first initial tactile signal and modulating the second initial tactile signal include modulating the first initial tactile signal to output the first tactile control signal having a tactile pattern, and / or modulating the second initial tactile signal to output the second tactile control signal having a tactile pattern.

29. The system of claim 17, wherein: The receiver is configured to: receiving the first media stream from the first terminal at a first time; The second media stream is received from the second terminal at a second time alternating with the first time.

30. The system of claim 17, wherein: The system is a wireless headset configured to receive the first media stream and the second media stream simultaneously.

31. The system of claim 17, further comprising: A mixer is configured to combine the first haptic control signal and the second haptic control signal.

32. A method comprising: Receiving a plurality of media streams, each of the plurality of media streams coming from a different terminal among a plurality of terminals; Decoding the plurality of media streams to obtain a plurality of contents in a playable format; associating each media stream of the plurality of media streams with a corresponding identifier, the corresponding identifier being selected from a set of predetermined identifiers; generating a plurality of initial haptic signals based on the plurality of contents and the associated corresponding identifiers; processing the plurality of contents to generate a plurality of analyses; modulating the plurality of initial haptic signals according to the plurality of analyses to output a plurality of haptic control signals; as well as The plurality of haptic control signals are actuated.

33. A computer program comprising instructions for causing the system of claim 17 to perform the steps of the method of claim 1.

34. A computer readable medium having stored thereon a computer program as claimed in claim 33.

Images