Voice cancellation system and method

Abstract

To provide an audio cancellation system which generates a response so as to prevent the quality of audio content from being degraded, a method, and a recording medium.SOLUTION: A system comprises: a feedback device 700 comprising one or more elements operable to generate an output in response to an input from a processing device; a microphone 710 configured to capture one or more audio inputs; an input detection unit 720 configured to detect one or more inputs to the feedback device from the processing device; and a response generation unit 730 configured to generate a response to the detected input in dependence upon the input and a distance between a component associated with a corresponding output and the microphone.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to voice cancellation systems and methods. [Background technology]

[0002] A "Background" is provided to provide a general context for the present disclosure. In this Background section, no work by the inventors is admitted, expressly or impliedly, to be prior art to the present invention unless specifically stated to be prior art at the time of filing.

[0003] The use of online voice communication is becoming more widespread with the increase in high-speed internet connections. Over the past few years, the use of online voice communication has accelerated due to both social factors (e.g., the Covid-19 pandemic has reduced opportunities for people to meet in person) and technological factors. In recent years, an increasing number of devices have been equipped with high-quality microphones to help capture audio. Examples of such devices include laptops, mobile phones, and gaming consoles (and / or their associated controllers).

[0004] Online voice communication is used for a variety of purposes (e.g., socializing, business meetings, etc.), but is particularly relevant to gaming communities. This may include both multiplayer gaming contexts (communicating with competitors and / or teammates) and video game streaming contexts (where a single player broadcasts their gameplay to a larger audience of non-players, optionally including audio commentary). Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the increasing use of microphones for voice communication, various challenges have been identified. One challenge is the ambient noise picked up by the microphone. This ambient noise can be annoying and / or irritating to the listener, as well as making it difficult to hear the speaker. Existing technology addresses this issue through push-to-talk functionality, which reduces the amount of time the microphone is active. Additionally, software-based noise reduction technology is sometimes used, which separates the user's voice from the picked-up sounds.

[0006] Such techniques are inconvenient for users who need to control audio quality, and may impose excessive processing load on the system by performing noise suppression, which may degrade the quality of the audio content in an effort to maintain the quality of the audio itself (which may, for example, annoy the user or introduce significant time delays).

[0007] The present disclosure aims to solve these problems. [Means for solving the problem]

[0008] The present disclosure is defined in claim 1.

[0009] Further particular aspects of the present disclosure are defined in the appended claims.

[0010] Both the foregoing general description of the invention and the following detailed description are explanatory only and are not intended to be limiting. [Brief explanation of the drawings]

[0011] A more complete understanding of the present disclosure and its many advantages will be obtained by reading the following detailed description in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of a handheld controller. [Figure 2] FIG. 1 is a schematic diagram of a processing system. [Figure 3] FIG. 1 is a schematic diagram of a sound reduction method. [Figure 4] FIG. 1 is a schematic diagram of a method for generating a response to a detected input by a user. [Figure 5] FIG. 1 is a schematic diagram of how a processing device generates a response to a sensed input. [Figure 6] FIG. 1 is a schematic diagram of a system for implementing one or more embodiments of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram of a system for implementing one or more embodiments of the present disclosure. [Figure 8] FIG. 1 is a schematic diagram of a system for implementing one or more embodiments of the present disclosure. [Figure 9] FIG. 1 is a schematic diagram of a method for generating a transfer function. [Figure 10] FIG. 1 is a schematic diagram of a system for generating a transfer function. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings, in which like reference numerals indicate the same or corresponding parts throughout the drawings.

[0013] FIG. 1 illustrates an example game controller 100 suitable for implementing several embodiments of the present disclosure.

[0014] The game controller 100 is configured to transmit one or more inputs provided by a user to a processing device (e.g., a game console). These inputs may include signals indicative of the operation of one or more of the buttons 110, trigger 120, or joystick 130. The game controller 100 may also include one or more hardware motion detectors (e.g., accelerometers or gyroscopes) capable of detecting movement of the game controller 100. The game controller 100 may generate one or more inputs to a processing device to control the operation of the processing device (and / or one or more applications running on the device, e.g., a computer game). The game controller 100 may further include one or both of a microphone 140 and a speaker 150, which allow for audio input from and audio output to the user, respectively.

[0015] Game controller 100 may also be configured to receive input from an associated processing device as well as input from a user, examples of such input include audio input (related to output from a speaker associated with the game controller) or haptic input (used to control one or more haptic feedback elements, such as rumble characteristics).

[0016] In some embodiments, the game controller 100 may include one or more processing units operable to implement the functionality of the game controller 100. This has the advantage that processing (e.g., input detection) can be performed more quickly because there is no processing load on the associated processing device and no delay due to transmitting a signal to the processing device.

[0017] Of course, the controller may take any configuration suitable for use with embodiments of the present disclosure, as long as the device is capable of receiving input from a user and generating sound in response to manipulation of the controller.

[0018] 2 is a schematic diagram of an example of a system in which such a controller is used. The system includes one or more controllers 100, a processing device 200, and a display 210. An example of a system according to FIG. 2 is a game console that includes multiple controllers for user input and a display that displays gameplay to the user. Similarly, a personal computer may be provided as the processing device 200 instead of a game console. In this case, the one or more controllers 100 may include a keyboard and / or a mouse or other suitable input device.

[0019] While FIG. 2 shows each element of the system as separate, in some embodiments, the elements may be implemented in fewer devices. A first example of such a device is a laptop, where the keyboard, processing unit, and display are integrated into a single physical unit. Thus, in this case, each element of FIG. 2 is integrated into a single device. Similarly, handheld game consoles or forms are also examples of such devices. Mobile phones or tablets are often used with separate game controllers (via wired or wireless connections). Such configurations are examples of integrated processing device 200 and display 210 used with a separate controller 100. In addition to such examples, the integrated controller 100 and processing device 200 may also be provided with a separate display. Examples of such devices include devices that provide a shared or assignable screen (allowing for flexible display device selection). For example, in the past, there have been handheld game controllers (with processing elements) that could be directly connected to televisions to provide gaming functionality to users.

[0020] As is clear from the above description, the game controller shown in Figure 1 is used alongside a game console, but these concepts can be applied to a wider range of devices.

[0021] 3 illustrates a schematic diagram of an example method according to one or more embodiments of the present disclosure. Such a method may be used to mitigate the effect of input-related sounds (such as button presses or haptic feedback) on audio captured within the controller's environment (e.g., using a microphone on the controller or on the processing device, or a separate microphone (e.g., a headset worn by the controller user)). That is, such a method may be used to mitigate the effect of button presses or the like on captured audio (e.g., game player commentary, voice input for controlling the processing device, conversations between friends, etc.).

[0022] At step 300, one or more inputs provided to the controller are detected. This may include inputs provided by a user (e.g., actuation of one or more buttons (which may be simultaneous or sequential), or inputs provided by an associated processing device, such as audio input or haptic feedback generating input). This detection is based on detecting actuation of a button(s) on the controller, while the generation of a haptic feedback signal is detected in the processing device or controller. Further examples of such detection are discussed below.

[0023] At step 310, an audio effect associated with the detected signal is determined. This may be done in any suitable manner relative to the input. In some embodiments, this may be done by detecting a button press and using a lookup table or the like to identify a corresponding sound. Alternatively or additionally, the applied input sound may be detected using a microphone. A further alternative or additional example is to determine a sound associated with the signal applied to the controller. This may, for example, determine a sound resulting from the haptic feedback associated with the haptic feedback generating signal.

[0024] Determining the audio impact may involve using information about the propagation of sound between a sound source and a microphone associated with the controller and / or user. For example, the distance between the sound source and the microphone may represent a difference in audio level between the transmitted and received sound. Similarly, different propagation paths may change the pitch or loudness of the sound. This may also be due to the structure of the controller itself. For example, for sound traveling through plastic parts, the difference between the transmitted and received sound may be significant. Therefore, it may be useful to use information about the controller (e.g., the structure, material, and location of input / output elements) and information from prior testing to determine the audio impact associated with the input.

[0025] At step 320, a response is generated that reduces the impact of the audio on sounds captured by the microphone in the controller's environment. In some embodiments, this may include applying processing to the captured audio signal in accordance with the determination of the audio impact to perform a noise cancellation function. Alternatively or additionally, a speaker associated with the controller may be controlled to emit a signal received by the microphone, where the signal may be determined to perform a noise cancellation function for the audio impact associated with the input.

[0026] The method of FIG. 3 may also be configured to implement a noise cancellation-style process on the audio generated by the controller. This is useful because it can leverage existing knowledge (e.g., sounds generated by the input) to more effectively reduce the impact of undesired sounds in the captured audio stream. This is also useful because it can be implemented in conjunction with the audio capture process (rather than a traditional noise cancellation-like audio playback process). This results in more efficient audio playback and higher quality captured audio, since the amount of undesired sounds can be reduced during the audio capture process.

[0027] As mentioned above, identifying characteristics of the controller or input itself in advance can aid in effectively implementing the methods of the present disclosure. Such characteristics may be derived from the design of the device (e.g., the shape of the buttons, microphone, materials, and / or controller) or from testing in which inputs are given and audio output associated with those inputs is recorded. Such characteristics are advantageous in that they allow for a priori determination of what expected (unwanted) sounds will be in the captured audio.

[0028] In many cases, the sounds emitted for each element can be predetermined, enabling a modeling process. For example, different modes of operation for haptic feedback may be known, and materials and forces on different buttons may be determined. Using a modeling process is useful when the sound impact depends on the user using the controller or peripheral factors. For example, a strong grip or large hands may reduce the transmission of sound through the controller, or a keyboard may produce different sounds depending on the surface it is resting on (e.g., a hard surface like a desk may generate more noise than a soft surface like the user's leg).

[0029] Such identification may also be given on an element-by-element basis. For example, information may be generated about the operation of a particular button or trigger, or about a particular haptic feedback element. This may be appropriate when the location of a particular element is fixed within the controller (i.e., when the spatial component for identification is constant). Such an approach may be particularly applicable when there are many distinct inputs, allowing for many possible actions, since this allows for easier identification.

[0030] Alternatively or additionally, the identification may be given for each action (e.g., a particular input operation). Examples of this may be the specific operation of a haptic feedback element (e.g., a particular rumble pattern defined with a particular frequency and amplitude) or a button (e.g., a trigger with multiple input options may generate different sounds for half-press and full-press actions). Such an approach may be particularly suitable when considering embodiments in which the input rules depend on the signals generated for these inputs, since in this case the action is easily determined.

[0031] Further alternative (or additional) methods for generating and storing the identified characteristics may also be suitable. Therefore, the above examples should not be construed as limiting in this disclosure. For example, identification of the spatial dependency of the sound's influence with respect to the microphone position may be performed, which is used to modify the predicted effect on the captured sound.

[0032] 4 and 5 illustrate two exemplary methods for implementing embodiments of the present disclosure. The former involves providing user input (such as a button press), while the latter involves input to a controller (such as haptic feedback or voice input). While there are differences between each method, each provides useful background for the other, as there are significant similarities between the methods.

[0033] 4 illustrates a schematic of a method for generating a response to a detected user input, which is used to modify the captured audio. The detected input may include the manipulation of a physical element (e.g., a button, a key, a trigger, a touchscreen, and / or a trackpad, etc.).

[0034] At step 400, one or more audio inputs are obtained using a microphone associated with the controller. These audio inputs may include, for example, speech from a user of the controller, but are not limited to such, and any suitable audio may be considered.

[0035] At step 410, one or more inputs from a user to the controller are detected. This detection may be performed in any suitable manner. In a first example, this may be performed by audio detection, where the sound of an element manipulated by the user to provide the input is detected. In a second example, signals output from the controller in response to the user input are analyzed, allowing a determination to be made as to which input was provided at a particular time.

[0036] Step 420 involves generating a response to the detected input that depends on the input and at least the distance between the microphone and the element associated with that input. For example, based on the detection of a particular input, an expected sound at the microphone location due to that input may be identified. As noted above, the audio impact of applying the input also depends at least on the distance between the microphone and the element associated with the input, since attenuation or penetration through the controller is a relevant factor. The response generated depends on the desired method for modifying the audio data.

[0037] Step 430 includes outputting the response generated in step 420. In some embodiments, this may include providing an audio signal to an output speaker, thereby reducing the impact of sounds associated with the detected user input. In such cases, the response may include an audio signal that is the inverse of the sound associated with the detected input. This may involve one or more further modifications to the relative positions of elements relative to the input, microphone, and / or speaker that outputs the response.

[0038] Alternatively or additionally, outputting a response may include providing information for directly modifying the captured audio, thereby modifying the captured audio to reduce the impact of audio associated with the detected user input. For example, an inverse signal of a sound associated with the detected input may be generated. Alternatively, time, intensity, and / or frequency information may be generated that allows for more effective removal of undesired audio (which corresponds to the user input) from the captured audio.

[0039] Figure 5 shows a schematic diagram of how a response to a detected input by a processing device can be generated, which is used to modify the captured audio.

[0040] At step 500, one or more audio inputs are detected by a microphone associated with the controller. These audio inputs may include, but are not limited to, the speech of a user of the controller, and any suitable audio may be considered.

[0041] At step 410, one or more inputs from the processing device to the controller are detected. These inputs may include sounds (e.g., audio effects corresponding to events in the game) or signals (e.g., haptic feedback signals) output from a speaker associated with the controller. These may be detected using a microphone or analysis of data received from the processing device, similar to the detection discussed at step 410.

[0042] Step 520 involves generating a response to the detected input that depends on the distance between the microphone and an element (e.g., a speaker or haptic feedback element) associated with the input and at least a corresponding output (i.e., an output generated based on the detected input). For example, based on the detection of a particular input, an expected sound at the microphone location due to that input may be identified. As noted above, the audio impact of providing an input also depends at least on the distance between the microphone and an element associated with the input, since attenuation or transmission through the controller is a relevant factor. The response generated depends on the desired method for modifying the audio data.

[0043] Step 530 includes outputting the response generated in step 520. In some embodiments, this may include providing an audio signal to an output speaker, thereby reducing the impact of sounds associated with the detected user input. In such cases, the response may include an inverse signal of the sound associated with the detected input. This may involve one or more further modifications to the relative positions of elements relative to the input, microphone, and / or speaker that outputs the response.

[0044] Alternatively or additionally, outputting a response may include providing information for directly modifying the captured audio, thereby modifying the captured audio to reduce the impact of sounds associated with the detected user input. For example, an inverse signal of a sound associated with the detected input may be generated. Alternatively, time, intensity, and / or frequency information may be generated that allows for more effective removal of undesired sounds (which correspond to inputs by the processing device) from the captured audio.

[0045] In some embodiments, these processing device inputs may be determined in advance by the processing device (e.g., during a scripted scene in the game). In this case, a response may be generated to be output simultaneously with the input itself. Similarly, the processing device may be configured to automatically generate a response when generating an input (e.g., depending on the specific controller being used). For example, the processing device may be configured to generate the haptic feedback signal and the response simultaneously (or as part of the same process) to reduce the audio impact of the haptic feedback signal. This may be useful to reduce latency in the response generation process, especially if the processing device implements the response.

[0046] The above discussion has primarily focused on controllers and the like that are integrated with microphones for recording user commentary. However, the microphone may be located in a separate device, such as a headset. In such cases, the distance between the controller (i.e., the hardware that generates sound from input) and the microphone may be fixed. In such cases, the position of each device may be tracked, and changes in audio influence may then be determined. For example, the intensity of audio influence may be reduced with increasing distance between the devices. Any suitable tracking method may be implemented. For example, both the controller and the headset (or the user's head, which defines the position of the head-mounted microphone) may be camera-based tracked.

[0047] 6 illustrates a schematic diagram of a system for implementing one or more embodiments of the present disclosure. The system includes an input device 600, a microphone 610, an input detection unit 620, and a response generation unit 630. The input detection unit 620 and the response generation unit 630 may be implemented using a central processing unit or the like located in one or more devices (e.g., the input device 600, an associated processing device, and / or a server). Further functional units may be provided but are not shown for the sake of brevity. For example, storage means (such as a hard drive) may be provided for saving parameters of the input device 600, and a processing unit may be provided for performing audio processing of sounds captured by the microphone 610.

[0048] The input device 600 comprises one or more elements that a user can manipulate to generate input to a processing device. The processing device may be associated with a game console or the like or a cloud-based server. The elements may comprise one or more buttons, keys, analog sticks, touch inputs, and / or triggers. In some embodiments, the input device 600 may be configured as a game controller, keyboard, or touchscreen device. Thus, discussions of controllers elsewhere herein may apply to the input device 600. Alternatively or additionally, the input device 600 may be integrated with one or more other devices. For example, a device may be provided that includes the input device 600 and a display device for displaying the output of the processing device. Examples of such devices include a laptop, a portable game console, a mobile phone, and the like.

[0049] The microphone 610 is operable to obtain one or more audio inputs. In some embodiments, the microphone is provided as part of the input device 600. Alternatively or additionally, the microphone 610 may be provided as separate hardware (e.g., a headset integrated with a processing device worn by a user, or a standalone unit).

[0050] The input detection unit 620 may be operable to detect one or more inputs by a user to the input device 600. In some embodiments, the input detection unit 620 may be operable to detect the input using captured audio, as discussed with reference to step 410 of Figure 4. The captured audio may be captured using the microphone 610 or another alternative microphone. Alternatively or additionally, the input detection unit 620 may be operable to detect the input in response to a control signal generated by the input device 600 in response to the user's input.

[0051] The response generation unit 630 is operable to generate a response to the detected input depending on the input and the distance of the microphone from an element associated with the input. The response generation unit 630 may be implemented by a processing unit located in either (or both) of the input device 600 and an associated processing device, etc. The response may be an inverse of the sound associated with the detected input, modified to accommodate additional audio effects.

[0052] In some embodiments, the response generation unit 630 may be operable to generate an audio response for output on a speaker associated with the input device 600. Alternatively or additionally, the response generation unit 630 may be operable to generate a response for output to the audio processing unit that modifies the audio input captured by the microphone 610. Examples of such response generation were discussed with reference to steps 420 and 430 of FIG.

[0053] The response generation unit 630 may be operable to generate a response depending on one or more physical characteristics of the input device 600, as well as the distance between the element associated with the input and the microphone 610 (and / or other factors). Examples of such physical characteristics include the size, shape, structure, material, and / or sound transmissivity of the input device 600. These characteristics may be used to estimate or calculate the difference between the sound emitted from the element during the input process and the sound detected by the microphone 610, as these characteristics affect sound propagation through the input device 600. Such characteristics may be important even if the microphone 610 is not integrated with the input device 600, as the sound generated by the user input may still be affected by these characteristics. This is particularly true for the rear trigger of a controller, for example, when a user wears a head-mounted microphone. In this case, sound from the trigger still propagates through the controller and reaches the microphone.

[0054] Alternatively or additionally, the response generation unit 630 may be operable to generate a response depending on one or more operational parameters of the input device 600. Such parameters include how the user holds the input device 600. For example, a pressure sensor in the input device 600 may be used to estimate the grip strength of the input device 600 or the size of the user's hand. Another operational parameter may include information about the surface on which the input device 600 is placed during use. The operational parameters may be determined using images of the placement or audio cues (e.g., that determine the difference between the expected sound and the sound detected upon operation of the input device 600 due to such parameters). Furthermore, the operational parameters may be used to modify the expected audio influence on the sound captured by the microphone 610.

[0055] In some embodiments, the response generation unit 630 may be configured to generate responses using one or more transfer functions defined for the input device and / or one or more elements of the input device (e.g., each button). These may be generated, for example, based on information about the structure of the elements and their placement within the controller, or based on an analysis of sound propagation through the controller. The latter may be performed, for example, by providing standardized sounds and comparing the responses measured by a microphone with information about the sounds. The standardized sounds may be, for example, pre-recorded sounds or may be estimated based on information about the operation of the buttons (e.g., it may be possible to detect how difficult some buttons are to press, thereby predicting the sound that a button press will produce). Based on this comparison, information about time delays, attenuation, and / or changes in frequency or frequency profile may be obtained.

[0056] In some cases, these transfer functions may be defined depending on information about the user's grip on the controller or information about the position of the input device (e.g., whether the controller is located on a hard or soft surface), which may affect the propagation of sound through the input device. This can be determined based on measurements of grip force (e.g., by using or estimating pressure sensors in the input device), and used to, for example, predict or evaluate changes in sound propagation. Similarly, spatial tracking of the input device may be used to determine whether the input device is placed on a particular surface in the environment (together with surface identification providing information about stiffness or other properties).

[0057] The above discussion has focused on what sounds are picked up. However, instead (or in addition), it may be useful to consider determining a transfer function that represents the sound a user hears when operating an input device. Such a transfer function may be predicted depending on the relative position of the input device and the user's head (or more specifically, ears). Furthermore, consideration may be given to attenuation caused by a user wearing headphones or the like. These transfer functions may be determined for each input element (or at least selected input elements) or a group of input elements (e.g., shoulder buttons on a controller), or sufficient consideration may be given to determining a transfer function for the input device itself (which in some embodiments may be used in conjunction with the transfer function for the input device).

[0058] In such an embodiment, an optional transfer function determination unit may be provided that is capable of determining a transfer function in response to at least one characteristic of the input device and a distance between an element associated with the input and the microphone. Alternatively, an existing transfer function may be selected in response to the determined input provided to the processing device from the input device. For example, a transfer function associated with a particular button or input key operated by a user may be selected. In this case, the response generation unit may be operable to generate a response to the detected input in response to the input and the determined transfer function, without explicitly utilizing the distance between the element associated with the input and the microphone.

[0059] The response output unit 640 is operable to output the response generated by the response generation unit 630. The response output unit 640 may be configured as appropriate depending on the particular embodiment. For example, the response output unit 640 may be implemented as a processing unit and a speaker that generates sound in response to the response. Alternatively, the response output unit 640 may be implemented as a processor that provides a signal to an audio processing unit that modifies the sound captured by the microphone 610. In some embodiments, the response output unit 640 may comprise each of these features so that an appropriate type of response can be output. For example, in some embodiments, audio output may be optimal for one input, while audio processing may be optimal for another input. This may be determined on an input-by-input basis, e.g., to be optimal for the particular input device 600.

[0060] The configuration of Figure 6 is an example of a processor (e.g., a GPU and / or CPU located in a game console or any other computing device) for use with a system that includes an input device that includes one or more elements that can be manipulated by a user to generate input to a processing device and a microphone that can obtain one or more audio inputs. The processor may operate as follows: · Detecting input by one or more users to an input device. Generate a response to the detected input depending on the distance of the microphone to the input and the elements associated with the input.

[0061] 7 illustrates a schematic diagram of a system for implementing one or more embodiments of the present disclosure. The system includes a feedback device 700, a microphone 710, an input detection unit 720, and a response generation unit 730. The input detection unit 720 and the response generation unit 730 may be implemented using a central processing unit or the like located in one or more devices (e.g., the feedback device 700, associated processing devices and / or servers, etc.). Further functional units may be provided but are not shown for the sake of brevity. For example, storage means (e.g., a hard drive) may be provided for saving parameters of the feedback device 700, and a processing unit may be provided for performing audio processing of sounds captured by the microphone 710.

[0062] Feedback device 700 comprises one or more elements operable to generate an output in response to input from a processing device. Examples of these include audio output (via a speaker associated with feedback device 700) and haptic feedback via a haptic feedback unit (e.g., a rumble generator). In accordance with the above discussion, feedback device 700 may be implemented as part of a game controller or the like. However, it is not necessary for feedback device 700 to allow user input. In other words, feedback device 700 is only required to be operable to generate an output in response to input from a processing device. Examples of suitable feedback devices include those that provide haptic feedback via rumble motors or the like. Thus, feedback device 700 may be implemented as a head-mounted display that provides haptic feedback. Alternatively, feedback device 700 may be another wearable device that provides output with limited user input capabilities (e.g., a vest that can simulate impacts, a wristband that can simulate weight using torque generation, etc.).

[0063] The microphone 710 is operable to obtain one or more audio inputs. In some embodiments, the microphone is provided as part of the feedback device 700, but may also be provided as separate hardware (e.g., a headset integrated with a processing device worn by a user, or a standalone unit).

[0064] The input detection unit 720 may operate to detect one or more inputs by a user to the feedback device 700. In some embodiments, the input detection unit 720 may operate to detect an input using audio representing a corresponding captured output. That is, the input may be detected from the captured audio (which includes the sound of the output generated in response to the input). For example, if the audio includes a rumble, it is inferred that a haptic feedback signal is to be provided to the feedback device 700.

[0065] Alternatively or additionally, the input detection unit 720 may be operable to detect an input in response to a control signal in an input provided by the processing device to the feedback device 700. Such detection may include analysis of the input signal to the feedback device 700 to determine which output is commanded. Alternatively or additionally, this may include analysis of the control signal as part of the generation process.

[0066] The response generation unit 730 is operable to generate a response to the detected input depending on the input and the distance of the microphone from an element associated with the input. The response generation unit 730 may be implemented by a processing unit located in either (or both) of the feedback device 700 and an associated processing device, etc. The response may be an inverse of the expected output associated with the detected input, modified to accommodate additional audio effects.

[0067] In some embodiments, the response generation unit 730 may be operable to generate an audio response for output on a speaker associated with the feedback device 700. Alternatively or additionally, the response generation unit 730 may be operable to generate a response for output to the audio processing unit that modifies audio input captured by the microphone 710. Examples of such response generation were discussed with reference to steps 520 and 530 of FIG.

[0068] The response generation unit 730 may be operable to generate a response depending on one or more physical characteristics of the feedback device 700, as well as the distance between the element associated with the input and the microphone 710 (and / or other factors). Examples of such physical characteristics include the size, shape, structure, material, and / or sound transmissivity of the feedback device 700. These characteristics may be used to estimate or calculate the difference between the sound emitted from the element during the input process and the sound detected by the microphone 710, as these characteristics affect the sound propagation through the feedback device 700. Such characteristics may be important even if the microphone 710 is not integrated with the feedback device 700, as the sound generated by the user input may still be affected by these characteristics. This is particularly true for the rear trigger of a controller, for example, when the user wears a head-mounted microphone. In this case, sound from the trigger still propagates through the controller and reaches the microphone.

[0069] Alternatively or additionally, the response generation unit 730 may be operable to generate a response depending on one or more operational parameters of the feedback device 700. Such parameters include how the feedback device 700 is held. For example, a pressure sensor in the feedback device 700 may be used to estimate the grip strength of the feedback device 700 or the size of the user's hand. Another operational parameter may include information about the surface on which the feedback device 700 is placed during use. The operational parameters may be determined using images of the placement or audio cues (e.g., determining the difference between the expected sound and the sound detected upon operation of the feedback device 700 due to such parameters). Furthermore, the operational parameters may be used to modify the expected audio influence on the sound captured by the microphone 710.

[0070] In some embodiments, the response generation unit 730 may be configured to generate responses using one or more transfer functions defined for the feedback device 700 and / or one or more elements of the feedback device 700 (e.g., a particular haptic feedback element). These may be generated based on, for example, information about the structure of the elements and their placement within the controller, or based on an analysis of sound propagation through the controller. The latter may be performed, for example, by providing a standardized sound and comparing the response measured by the microphone with information about the sound. The standardized sound may be, for example, a haptic feedback having a predetermined duration, frequency, intensity, and / or timing. Based on this comparison, information about time delays, attenuation, and / or changes in frequency or frequency profile may be obtained.

[0071] In some cases, these transfer functions may be defined depending on information about the user's grip on the controller or information about the position of the feedback device (e.g., whether the controller is located on a hard or soft surface), which may affect the propagation of sound through the feedback device. This can be determined based on measurements of grip force (e.g., by using or estimating pressure sensors in the feedback device), and used to, for example, predict or evaluate changes in sound propagation. Similarly, spatial tracking of the feedback device can be used to determine whether the feedback device is placed on a particular surface in the environment (with identification of the surface providing information about stiffness or other properties).

[0072] The above discussion has focused on what sounds are picked up. However, instead (or in addition), it may be useful to consider determining a transfer function that represents the sound a user hears when operating a feedback device. Such a transfer function may be predicted based on the relative position of the feedback device and the user's head (or, more specifically, ears). Furthermore, consideration may be given to factors such as attenuation caused by a user wearing headphones or the like. These transfer functions may be determined for each output element (or at least selected output elements) or a group of output elements (e.g., vibration elements of a controller), or sufficient consideration may be given to determining a transfer function for the feedback device itself (which, in some embodiments, may be used in conjunction with the transfer function for the feedback device). Information about the transfer functions and / or other outputs may be used to determine the overall sound effect on the microphone or user. For example, consideration may be given to the mutual interference of outputs generated by each output element of a feedback device (e.g., each of multiple vibrations generated by a motor of a controller).

[0073] This interference may be constructive (increasing the overall sound effect), destructive (reducing or completely eliminating the overall sound effect), or a combination thereof. This may increase some parts of the output (e.g., certain vibration frequencies) while decreasing others. Thus, even if the overall sound effect of an output element evaluated for a particular output falls below a threshold, no response may be generated. In such embodiments, an optional transfer function determination unit may be provided that is capable of determining a transfer function depending on at least one characteristic of the feedback device and the distance between the element associated with the output and the microphone. Alternatively, an existing transfer function may be selected depending on the determined input provided to the feedback device from the processing device. For example, a transfer function associated with a particular vibration element may be selected. In this case, the response generation unit may be operable to generate a response to the detected input depending on the input and the determined transfer function, without explicitly utilizing the distance between the element associated with the output and the microphone.

[0074] In some embodiments, the input detection unit 720 may be provided in a processing device and may be operable to detect an input before the input is provided to the feedback device 700. In such a case, a response to the detected input (generated by the response generation unit 730) may be provided to the feedback device 700 along with the detected input.

[0075] The response output unit 740 is operable to output the response generated by the response generation unit 730. The response output unit 740 may be configured as appropriate depending on the particular embodiment. For example, the response output unit 740 may be implemented as a processing unit and a speaker that generates sound in response to the response. Alternatively, the response output unit 740 may be implemented as a processor that provides a signal to an audio processing unit that modifies the sound captured by the microphone 710. In some embodiments, the response output unit 740 may comprise each of these features so that an appropriate type of response can be output. For example, in some embodiments, audio output may be optimal for one input, while audio processing may be optimal for another input. This may be determined on an input-by-input basis, e.g., to be optimal for the particular feedback device 700.

[0076] The configuration of Figure 7 is an example of a processor (e.g., a GPU and / or CPU located in a game console or any other computing device) for use with a system that includes one or more elements operable to generate output in response to input from a processing device and a feedback device that includes a microphone capable of obtaining one or more audio inputs. The processor may be operable as follows: · Detecting input by one or more users to an input device. Generate a response to the detected input depending on the distance between the microphone and the elements associated with the input and output.

[0077] FIG. 8 schematically illustrates a system for implementing one or more embodiments of the present disclosure. The input device of FIG. 8 functions, for example, as the controller 100 of FIG. 2. The input device 600 and feedback device 700 of FIGS. 6 and 7 may be implemented according to the example of FIG. 8. In this case, one or more elements may be optional, where appropriate. For example, the output element 810 may be omitted if haptic feedback or audio input is desired. And / or the processing unit 840 may be omitted if the processing function is performed by an associated processing device (e.g., a game console). Similarly, in embodiments where there is no user input (i.e., a feedback-only device), the input element 800 may be omitted.

[0078] The input element 800 comprises a feature operable by a user to provide input to a process, such as a computer program or game. Examples of suitable input elements include buttons, keys, switches, analog sticks, touch input, and / or triggers. Alternative (or additional) input elements may be provided to provide additional functionality. For example, a trackable indicator (such as a colored light), an inside-out tracking camera, and / or hardware motion detectors such as an accelerometer or gyroscope may be provided to provide motion input. Similarly, a microphone may be provided as an input element to allow voice command by the user.

[0079] Output element 810 comprises any suitable element for providing feedback to a user in response to signals generated by the processing device.

[0080] Examples of suitable elements include tactile feedback elements such as vibration or rumble units and speakers that provide audio feedback to the user.

[0081] The microphone 820 may include one or more microphones associated with an input device. A single microphone may be provided to capture audio for output (e.g., voice communication by a user). As discussed with reference to FIGS. 4 and 5, this same microphone may be used for input / output detection. In some embodiments, multiple microphones may be provided. For example, multiple microphones may be provided to capture output audio. This may increase the accuracy of sound source localization. Alternatively or additionally, additional microphones may be provided solely for the purpose of input / output detection. These microphones may be of lower quality than those capturing output audio (e.g., operating in a narrower frequency band or producing a noisier signal).

[0082] A transmitter 830 (or alternatively, a separate transmitter and receiver) may be provided to facilitate communication between the input device and an external processing device. Where the input device is integrated with the processing device (e.g., a laptop or handheld game console), such features may be omitted. This unit 830 may operate to send and receive communications via wired or wireless communication protocols to convey input and output between the input device and the associated processing device.

[0083] A processing unit 840 is provided to perform any suitable processing on the input device. In some embodiments, this processing unit 840 may be used to perform input detection and / or response generation processes (e.g., applying responses to generated speech). Alternatively, in some embodiments, such processing unit 840 (or at least selected functionality thereof) may be implemented by an associated processing device.

[0084] Speaker 850 may comprise one or more speakers capable of providing audio output. In some embodiments, a single speaker may be provided for providing audio (e.g., game audio or voice communication sounds) to the user. This speaker may be repurposed to generate audio signals to reduce the effect of audio input by the user or processing device. Alternatively, multiple speakers may be provided to provide ambient sound and / or dedicated speakers for generating reduced audio output. Similar to the discussion of microphone 820 above, lower quality speakers (e.g., those with a narrower frequency range or outputting a lower quality sound to reduce audio output) may be provided where appropriate.

[0085] Although the above only describes the use of a single input device in the system, multiple devices suitable for such processing may be provided. Each of the input devices may perform additional processing, thereby reducing the audio effect of sounds generated by other input devices in the system. This effectively extends the benefits of the present disclosure to multi-player configurations, or to single-player configurations where multiple input devices are provided.

[0086] In such a configuration, operation information (e.g., information about user input and / or output from the processing device to each input device) for each input device can be generated separately. Furthermore, if sound effect reduction processing is performed on each input device, this operation information can be shared between the input devices. Alternatively or additionally, processing can be performed using a common device (e.g., a game console to which each input device is associated), thereby reducing the amount of information that needs to be transmitted. Further useful information is the distance between each input device. This can be obtained using camera-based tracking means or proximity detection, for example. Such processing can be performed on a threshold basis. For example, input devices that are located at a distance greater than a threshold can be determined to have negligible (or no) effect on the other input devices. A threshold distance can be determined for each input / output, since each threshold distance is related to different volume levels and sound effects.

[0087] It should be noted that the teachings of the present disclosure do not require a microphone to capture sound in order to provide a benefit to the user. Such noise cancellation (or noise reduction) may be desirable for the user's own benefit. In other words, sound may be output by an audio device. In this case, the impact of the input / output sound may be reduced, if appropriate. For example, sound associated with haptic feedback may be reduced using the techniques described above. This allows the user to benefit from the benefits of the feedback without experiencing significant disruption from the associated sound.

[0088] As mentioned above, the sound cancellation process may use a transfer function to more accurately identify the audio impact of a particular input or output manipulation on the audio recorded by a microphone. The transfer function may be used to represent the difference between an initial sound (e.g., a button press or rumble) and the sound captured by a particular microphone. For example, vibrations of a particular frequency may cause secondary vibrations of a different frequency within the input device, which may be due to resonances in the input device's shell. Alternatively, the distance of an input / output element (e.g., a button or haptic feedback element) may cause a change in volume.

[0089] Thus, utilizing a transfer function that represents such considerations may allow for improved processing of the captured audio, thereby eliminating or reducing the audio's influence. For example, the audio's influence on the haptic feedback element may be reduced by more effectively generating one or more inverse haptic signals. Thus, while consideration of such transfer functions is not required, it may be desirable in some embodiments of the present disclosure.

[0090] FIG. 9 illustrates a schematic diagram of a method for generating such a transfer function. This will be discussed in the context of input devices in general. As previously mentioned, such an input device may be a controller for use in a game console, a keyboard, an integrated input device (e.g., a portable game console, a laptop input device), or any other device that provides input for controlling the operation of a processing device. The input device may correspond to any of the input and feedback devices described above, or may include both input and output elements (in which case both functions are realized). The following method is exemplary and not limiting. Accordingly, any suitable method may be implemented to generate the transfer function according to embodiments of the present disclosure.

[0091] For example, one alternative method for generating transfer functions is to present a standardized stimulus (e.g., a given tactile feedback, a button press of known pressure / duration, etc.) (as discussed above) and determine the difference between the sound impact at the stimulus microphone and the expected (known) sound impact at the stimulus source. This effectively provides a comparison between the output (generated) sound and the captured sound. While effective, this may be insufficient because it may be necessary to perform a large number of measurements to provide a sufficient set of transfer functions (or even a single transfer function) that can represent a wide enough range of behavioral and structural parameters.

[0092] 9, step 900 includes determining parameters of one or more input devices, including the location of a microphone relative to the input device and the location of one or more input / output elements relative to the input device. These parameters can be considered characteristics of the input device and its components, and can also be considered information about how the input device is being used, since they are factors that can modify the transmission of sound and vibrations through the input device.

[0093] Other example parameters include the position of the input device in the environment, the movement of the input device, the identity of the surface on which the input device is placed, whether the user holds the input device (and if so, which hand is used and with what strength the input device is gripped), the shape and / or construction of the input device, the mass of the device, input / output operational parameters (such as whether a particular button works) and the type of switch (i.e., how the button is implemented). Any modifications to the controller (functional or cosmetic, e.g., replacing buttons with alternative or additional elements to change the appearance of the device), and the attachment of peripherals (plug-in keyboards or headsets) (which can change the degree to which sound / vibration is transmitted through the input device) may also be taken into account.

[0094] The parameters may be determined in any suitable manner. For example, some parameters may be entered by a user, while others may be taken from a database of information about particular input devices. Alternatively or additionally, some parameters may be identified based on acquired images of the input device (e.g., the location of each button, the location of the input device, etc.). Identification of materials used to construct the input device may also be taken into account, based on any suitable technique (e.g., images, audio characteristics, determining the density of the input device, etc.). Data from any suitable sensor (e.g., camera, microphone, inertial sensor, etc.) may be used to determine the parameters.

[0095] Rather than direct detection, it may be preferable to estimate parameters of the input device and / or its use. For example, rather than explicitly determining the surface on which the input device is placed, it may be sufficient to determine that the input device is motionless and oriented in a particular resting direction (e.g., horizontally or standing on one edge). This determination may take into account whether the input device is located on a hard surface, since the input device will not move if it is fixed, and will be oriented differently (e.g., tilted) and sink into the surface in an irregular manner if it is on a softer surface. Similarly, based on the detected pattern of input device movement, it may be possible to determine which hand (or both hands) the user uses to operate the input device. Identifying the user's hand on the input device does not require analyzing an image or asking the user to indicate how they hold the input device.

[0096] Step 910 includes determining one or more characteristics of the audio transmission between each input / output element and the microphone. This characteristic may include any of the attenuation, response, and / or change in the frequency profile of the audio associated with each input / output element. Audio transmission here may be considered to refer to sound or vibration from any input and / or output element (e.g., button clicks, haptic feedback through a vibrating element, etc.). This determination may be made by modeling the propagation of audio / vibration through the input device. For example, a model may be generated based on the specific parameters determined in step 900, or an existing model may be applied according to the parameters determined in step 900.

[0097] In some embodiments, this step may involve determining the effect of one or more parameters on sound transmission instead of fully calculating the sound transfer characteristic. For example, the damping factor for sound associated with a particular input or output element may be calculated, or the resonant frequencies of an output may be determined. Such information may be used to adapt an existing transfer function or may be used as input for the generation of a new transfer function.

[0098] Step 920 involves generating a transfer function (or multiple transfer functions) in response to the determined audio characteristics. As previously mentioned, this may involve the overall generation of a new transfer function, or it may involve the modification of an existing transfer function. For example, a "basic" transfer function for an input device defined by the input device manufacturer may be modified in response to information about how the input device is held (example parameters) and its effect on acoustic transmission (example characteristics). Once generated, the transfer function may be saved for future use. Alternatively, new transfer functions may be generated with regularity (each play session, after a predetermined time period, or after a threshold change in one or more parameters).

[0099] Transfer functions may be defined in any suitable combination for a given input device. In some embodiments, a single transfer function may be defined for an input device, which characterizes the overall sound transfer. This is particularly suitable for simple input devices and configurations where the impact of the generated sound remains largely unchanged during use. Alternatively, transfer functions may be defined for each microphone associated with the input device and / or each input / output element associated with the input device. In this case, a separate transfer function may be specified for each microphone / input or output pair identified for the input device.

[0100] Optional step 930 includes performing a noise cancellation process on audio corresponding to one or more inputs / outputs of the input device in response to the generated transfer functions. Such a step is optional because the generation of the transfer functions in step 920 may be performed before the transfer functions are needed, such as as part of a calibration process to create a set of transfer functions to use. This noise cancellation process may be performed according to any of the methods described with reference to FIGS. 4 and 5. For example, a response may be generated in response to the transfer functions generated in step 920. In other words, the noise cancellation process includes generating output audio and / or modifying captured audio (which may include audio corresponding to one or more inputs and / or outputs of the input device).

[0101] If multiple transfer functions are associated with an input device, consideration may be given to selecting an appropriate transfer function depending on the input or output identified as corresponding to a particular sound, for example, operation of a particular button on the input device may be identified and the transfer function corresponding to this button may be selected.

[0102] 10 is a schematic diagram illustrating a system for generating a transfer function representing an audio transfer characteristic of an input device associated with a processing device. The input device may include multiple input and / or output elements. The input elements may include one or more buttons, keys, triggers, joysticks, and / or touchpads that are operable by a user of the input device. The output elements may include one or more haptic feedback units and / or speakers that provide output to a user of the input device.

[0103] The system includes a parameter determination unit 1000, an audio characteristic determination unit 1010, a transfer function generation unit 1020, and a noise cancellation unit 1030. This configuration may be implemented using any suitable processing unit (e.g., a CPU and / or GPU) associated with any suitable device (e.g., a processor located in a game console to which an input device is associated), and may be used to implement the following functions: The system of Figure 10 is configured to perform the method discussed with reference to Figure 9.

[0104] The parameter determination unit 1000 is configured to determine one or more parameters of the input device. These parameters include the location of a microphone associated with the input device and the locations of one or more input / output elements associated with the input device. The parameter determination unit 1000 may be configured to determine sound parameters associated with each of the input / output elements whose locations have been determined. For example, this may include determining characteristics of a sound or range of sounds associated with a particular input action (e.g., a button press sound) or a particular output frequency or frequency range / profile (e.g., frequencies associated with a haptic feedback element).

[0105] In some embodiments, the parameter determination unit 1000 is configured to determine the parameters based on a model of the input device, acquired images of the input device, and / or information acquired while a user is manipulating the input device. The model of the input device may include any suitable information about the device, such as shape, size, arrangement of elements, material information, weight, and any other information that may be related to the sound generated by the device or its operation, or how well the sound propagates through the input device. Images may be used to derive such information (e.g., using images to determine the size and materials to be used for the input device).

[0106] Information acquired while the user is manipulating the input device may include sound acquired by a microphone (or other microphones) associated with the input device, which can be compared to the predicted sound, allowing one or more further parameters related to sound propagation to be derived. The further information may include information about grip strength, how the user holds the input device (number of hands and how the hands are placed on the input device), movement of the input device, any other factors that may affect the propagation of sound through the input device, etc.

[0107] The audio characteristic determination unit 1010 is configured to determine one or more additional characteristics of the audio transmission between each input / output element and the microphone. The characteristics include attenuation, resonance, and / or changes in the frequency profile of the audio associated with each input / output element. The audio characteristic determination unit 1010 may be configured to determine parameters based on a model of the input device and / or information obtained while the user is manipulating the input device. The model may be the same model and / or information obtained as described for the parameter determination unit 1000.

[0108] In embodiments where the parameter determination unit 1000 is configured to determine the position and / or movement of the input device as a parameter, the audio characteristic determination unit 1010 may be configured to determine one or more audio transfer characteristics depending on the position and movement of the input device. Similarly, in embodiments where the parameter determination unit 1000 is configured to determine the parameter how the user holds the input device, the audio characteristic determination unit 1010 may be configured to determine one or more audio transfer characteristics depending on how the user holds the input device.

[0109] The transfer function generation unit 1020 is configured to generate a transfer function in response to the determined audio characteristics. In some embodiments, the transfer function generation unit 1020 is configured to generate a respective transfer function for a microphone associated with the input device. Alternatively or additionally, the transfer function generation unit 1020 may be configured to generate respective transfer functions for multiple input and / or output elements associated with the input device.

[0110] The noise cancellation unit 1030 is configured to perform a noise cancellation process on audio corresponding to one or more inputs and / or outputs of the input device according to the generated transfer function. This noise cancellation may be performed using any of the techniques described above (e.g., the techniques described with reference to FIGS. 4 and 5). The noise cancellation process may include generating an output audio (or vibration) and / or modifying the captured audio (which includes audio corresponding to one or more inputs and / or outputs of the input device), which respectively refer to outputting an inverse signal to reduce the effect of the audio and processing the captured audio.

[0111] In some embodiments, a transfer function generation process may be used to generate a transfer function for the input device with additional transfer functions representing variations in the sound propagation caused by factors external to the input device (e.g., the surface on which the input device is placed, the way the input device is held, etc.) In some embodiments, the noise cancellation unit 1030 may be configured to select and combine (or apply sequentially) two or more transfer functions to more accurately represent the sound effects associated with a particular sound.

[0112] The noise cancellation process does not necessarily cancel sounds generated by the input device itself. For example, a transfer function for the input device may be generated to represent the sound propagation of sounds outside the input device. This can be used to evaluate the impact of sounds on microphones outside the input device (e.g., a second input device, such as another player's controller or a second device used by the same player). This information can be used for noise cancellation between devices, thereby achieving more accurate noise cancellation of external sound sources (e.g., other input devices).

[0113] The configuration of Figure 10 is an example of a processor (e.g., a GPU and / or CPU located in a game console or other computing device) operable to generate a transfer function representing the sound propagation characteristics of an input device associated with the processing device and to perform a noise cancellation process. The processor may, among other things, be operable as follows: Determining one or more parameters of the input device, the parameters including a position of a microphone relative to the input device and a respective position of one or more input / output elements relative to the input device. Determining one or more characteristics of the audio transmission between each input / output element and the microphone, including changes in the audio attenuation, resonance, and / or frequency profile associated with each input and / or output element. Generate a transfer function according to the determined voice characteristics. · Performing a noise cancellation process on the audio corresponding to one or more inputs and / or outputs of the input device according to the generated transfer function.

[0114] The above techniques may be implemented by hardware, software, or a combination thereof. A software-controlled data processing device may implement one or more aspects of the present embodiment. Such software and non-transitory recording media having such software stored thereon are also included in the embodiments of the present disclosure.

[0115] Accordingly, the disclosure and description discussed above are exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other embodiments without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, and not to limit the scope of the invention or the claims. This disclosure includes readily discernible variations, in part to define the language of the claims, and is not intended to dedicate the inventive subject matter to the public.

[0116] The embodiments of the present disclosure are realized according to the following provisions. Clause 1 a feedback device comprising one or more elements operative to generate an output in response to an input from the processing device; a microphone for obtaining one or more audio inputs; an input detection unit for detecting one or more inputs from the processing device to the feedback device; a response generation unit for generating a response to the detected input in response to a distance between the microphone and a component associated with the input and the corresponding output; A system comprising: Clause 2 2. The system described in clause 1, wherein the microphone is part of the feedback device. Clause 3 3. The system according to clause 1 or 2, characterized in that the element comprises one or more speakers and / or haptic feedback units. Clause 4 4. The system of any of clauses 1 to 3, wherein the input detection unit detects an input using a captured sound corresponding to a corresponding output. Clause 5 5. The system of any one of clauses 1 to 4, characterized in that the input detection unit detects an input in response to a control signal among the inputs provided to the feedback device by the processing device.

[0117] Clause 6 6. The system of any of clauses 1 to 5, wherein the response generation unit generates an audio response related to the output using a speaker associated with the feedback device. Clause 7 7. The system of any of clauses 1 to 6, wherein the response generation unit generates an audio response for output to an audio processing unit that modifies the audio input captured by the microphone. Article 8 8. The system of any of clauses 1 to 7, wherein the response generation unit generates an audio response in response to one or more physical characteristics of the feedback device. Article 9 9. The system of claim 8, wherein the physical characteristics include any of size, shape, structure, material, and acoustic conductivity. Article 10 the response generating unit generates a response in response to one or more operating parameters of the feedback device; 10. A system according to any one of clauses 1 to 9, characterized in that the operational parameters represent how a user holds the feedback device.

[0118] Article 11 11. The system of any of clauses 1 to 10, wherein the input detection unit is provided to the processing device and detects the input before the input is provided to the feedback device. Article 12 12. The system of clause 11, wherein a response to a detected input is provided to a feedback device along with said detected input. Article 13 1. A method of using a system comprising: a feedback device comprising one or more elements operative to generate an output in response to an input from a processing device; and a microphone for obtaining one or more audio inputs, the method comprising: detecting one or more inputs from the processing device to the feedback device; generating a response to the detected input in response to a distance between the microphone and a component associated with the input and corresponding output; A method comprising: Article 14 Computer software which, when executed by a computer, causes the computer to carry out the method of clause 13. Article 15 A non-transitory recording medium on which the computer software described in clause 14 is recorded.

Claims

1. a feedback device comprising one or more elements operative to generate an output in response to an input from the processing device; a microphone for obtaining one or more audio inputs; an input detection unit that detects one or more inputs from the processing device to the feedback device by analyzing data from the processing device; a response generation unit for generating a response to one or more detected inputs in response to a distance between the microphone and components associated with the inputs and corresponding outputs; Equipped with the input detection unit detects the one or more inputs in response to a control signal in the one or more inputs provided to the feedback device by the processing device; The system, wherein the response generation unit generates a response for output to an audio processing unit that modifies at least one audio input of the one or more audio inputs captured by the microphone.

2. The system of claim 1 , wherein the microphone is part of the feedback device.

3. The system of claim 1 , wherein the elements comprise one or more speakers and / or haptic feedback units.

4. 2. The system of claim 1, wherein the input detection unit detects an input using a captured sound corresponding to a corresponding output.

5. 2. The system of claim 1, wherein the input detection unit detects an input in response to a control signal among the inputs provided by the processing device to the feedback device.

6. 10. The system of claim 1, wherein the response generation unit generates an audio response for the output using a speaker associated with the feedback device.

7. 10. The system of claim 1, wherein the response generation unit generates an audio response for output to an audio processing unit that modifies audio input captured by the microphone.

8. The system of claim 1 , wherein the response generation unit generates an audio response in response to one or more physical characteristics of the feedback device.

9. The system of claim 8 , wherein the physical characteristics include any of size, shape, structure, material, and acoustic conductivity.

10. 2. The system of claim 1, wherein the input detection unit is provided to the processing device and detects an input before the input is provided to the feedback device.

11. 11. The system of claim 10, wherein a response to a detected input is provided to a feedback device along with the detected input.

12. 1. A method of using a system comprising: a feedback device comprising one or more elements operative to generate an output in response to an input from a processing device; and a microphone for obtaining one or more audio inputs, the method comprising: detecting one or more inputs from the processing device to the feedback device by analyzing data from the processing device; generating a response to one or more detected inputs in response to a distance between the microphone and components associated with the inputs and corresponding outputs; detecting the one or more inputs in response to a control signal in the one or more inputs provided by the processing device to the feedback device; generating a response for output to an audio processing unit that modifies at least one audio input of the one or more audio inputs captured by the microphone; A method comprising:

13. A non-transitory recording medium on which computer software is recorded, The computer software, when executed by a computer, causes the computer to perform a method of using a system comprising: a feedback device comprising one or more elements operative to generate output in response to input from a processing device; and a microphone for obtaining one or more audio inputs; The method comprises:

1. A method of using a system comprising: a feedback device comprising one or more elements operative to generate an output in response to an input from a processing device; and a microphone for obtaining one or more audio inputs, the method comprising: detecting one or more inputs from the processing device to the feedback device by analyzing data from the processing device; generating a response to one or more detected inputs in response to a distance between the microphone and components associated with the inputs and corresponding outputs; detecting the one or more inputs in response to a control signal in the one or more inputs provided by the processing device to the feedback device; generating a response for output to an audio processing unit that modifies at least one audio input of the one or more audio inputs captured by the microphone; A non-transitory recording medium comprising:

Images