Gesture-Controlled Hardware for Immersive Audio and Other Applications

Abstract

The present disclosure is directed to an audio spatialization system that includes a gesture controller for controlling sound source position and multiple audio parameters. The audio spatialization system can be used in live music settings, such as performances, and in studio settings. The gesture controller can control sound source position using inertial sensing and can include a user input device to allow a user to control parameters of the immersive audio generated by the system.

Description

[Technical Field]

[0001] Related Cross-References

[0001] This application claims the benefit under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 483,874, entitled "GESTURAL CONTROL HARDWARE FOR IMMERSIVE AUDIO AND OTHER APPLICATIONS," filed February 8, 2023, the entire contents of which are incorporated herein by reference.

[0002] Technical Field

[0002] The present disclosure is directed to an audio spatialization system and method utilizing a gesture controller. [Background technology]

[0003] background

[0003] Spatial audio is a broad technical field that can create virtual auditory spaces (via binaural headsets) and real auditory spaces (in performances with surround / multidimensional speaker arrays) in two-dimensional (2D) and three-dimensional (3D) playback formats. Summary of the Invention [Means for solving the problem]

[0004] overview

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0005] According to one aspect of the present technology, an audio spatialization system is provided, the audio spatialization system including a handheld controller including an inertial measurement unit (IMU), the handheld controller configured to wirelessly output IMU data indicative of a position and / or orientation of the handheld controller and parameter control sensor data specifying at least two parameters of audio generated by the audio spatialization system. The audio spatialization system further includes a receiver configured to wirelessly receive the IMU data and the parameter control sensor data from the handheld controller, the receiver further configured to convert the IMU data and the parameter control sensor data into spatialization control data and audio parameter data. The audio spatialization system further includes a digital audio workstation (DAW) coupled to a plurality of audio reproduction devices disposed at respective positions in an environment, the digital audio workstation (DAW) configured to receive the spatialization control data and the audio parameter data from the receiver and to control generation of audio from one or more of the plurality of audio reproduction devices based on the spatialization control data and the audio parameter data.

[0006] According to one aspect of the present technology, there is provided a method for performing audio spatialization, the method including: using a handheld controller including an inertial measurement unit (IMU) to wirelessly output, using the handheld controller, IMU data indicative of a position and / or orientation of the handheld controller and parameter control sensor data specifying at least two parameters of audio to be generated. The method further includes: wirelessly receiving, using a receiver, the IMU data and the parameter control sensor data from the handheld controller; converting, using the receiver, the IMU data and the parameter control sensor data into spatialization control data and audio parameter data; receiving, at a digital audio workstation (DAW) coupled to a plurality of audio reproduction devices disposed at respective positions in an environment, the spatialization control data and the audio parameter data from the receiver; and controlling, using the DAW, generation of audio from one or more of the plurality of audio reproduction devices based on the spatialization control data and the audio parameter data.

[0007]

[0007] According to one aspect of the present technology, an audio spatialization controller is provided, the audio spatialization controller including a mobile handheld housing; an inertial measurement unit (IMU) disposed within the mobile handheld housing, the IMU configured to generate IMU data indicative of a position and / or orientation of the mobile handheld housing; a plurality of input devices on the mobile handheld housing configured to generate parameter control sensor data specifying respective parameters of the audio to be generated; a processor disposed within the mobile handheld housing, the processor configured to control operation of the mobile handheld housing; and a wireless transceiver disposed within the mobile handheld housing, the wireless transceiver configured to wirelessly transmit the IMU data indicative of the position and / or orientation of the mobile handheld housing and the parameter control sensor data generated by the plurality of input devices to a receiving device.

[0008] According to one aspect of the present technology, there is provided a method of operating an audio spatialization controller having a mobile handheld housing and an inertial measurement unit (IMU) disposed within the mobile handheld housing, the method including: using the IMU disposed within the mobile handheld housing to generate IMU data indicative of a position and / or orientation of the mobile handheld housing; using a plurality of input devices on the mobile handheld housing to generate parameter control sensor data specifying respective parameters of audio to be generated; and wirelessly transmitting the IMU data indicative of the position and / or orientation of the mobile handheld housing and the parameter control sensor data generated by the plurality of input devices from the mobile handheld housing to a receiving device.

[0009]

[0009] According to one aspect of the present technology, an apparatus is provided, the apparatus including a processing module configured to receive wireless data from a handheld audio spatialization controller, the wireless data including inertial measurement unit (IMU) data indicating a position and / or orientation of the handheld audio spatialization controller and parameter control sensor data specifying at least two parameters of the audio to be generated, the processing module being further configured to convert the IMU data and the parameter control sensor data into spatialization control data and audio parameter data.

[0010] According to one aspect of the present technology, there is provided a method for processing data for input to a spatial audio workstation, the method including receiving, at a processing module, wireless data from a handheld audio spatialization controller, the wireless data including inertial measurement unit (IMU) data indicating a position and / or orientation of the handheld audio spatialization controller and parameter control sensor data specifying at least two parameters of audio to be generated, the method further including converting, with the processing module, the IMU data and the parameter control sensor data into spatialization control data and audio parameter data.

[0011]

[0011] These and other features, aspects, and advantages of various embodiments of the present disclosure will be understood with reference to the following description and accompanying drawings.

[0012] BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Various aspects and embodiments of the present application are described with reference to the following figures. It should be understood that the figures are not necessarily drawn to scale. Items that appear in more than one figure are designated by the same reference numeral in all figures in which they appear. [Brief explanation of the drawings]

[0013] [Figure 1A] 1 illustrates an audio spatialization system employed in a live music setting, in accordance with aspects of the present technology. [Figure 1B] 1 illustrates an audio spatialization system employed in a live music setting, in accordance with aspects of the present technology. [Figure 2A]

[0014] 1A-1B illustrate user control over sound source position in the audio spatialization system of FIGS. [Figure 2B]

[0014] Figures 1A-1B illustrate user control over sound source position in the audio spatialization system. [Figure 2C]

[0014] Figures 1A-1B illustrate user control over sound source position in the audio spatialization system. [Figure 3A]

[0015] 10A-10C illustrate exemplary gestures that a user may employ to control sound source position and / or audio parameters in connection with the operation of an audio spatialization system. [Figure 3B]

[0015] Figure 3 illustrates exemplary gestures that a user may employ to control sound source position and / or audio parameters in connection with the operation of an audio spatialization system. [Figure 3C]

[0015] Figure 3 illustrates exemplary gestures that a user may employ to control sound source position and / or audio parameters in connection with the operation of an audio spatialization system. [Figure 4]

[0016] FIG. 1 is a block diagram of an audio spatialization system according to some embodiments. [Figure 5]

[0017] 1 is a flowchart of a method of operation of an audio spatialization system according to some embodiments. [Figure 6]

[0018] FIG. 1 is a block diagram of a mobile gesture controller according to some embodiments. [Figure 7]

[0019] 1 illustrates a program mode selection menu for a mobile gesture controller according to some embodiments. [Figure 8A]

[0020] FIG. 1 illustrates a top view of a mobile gesture controller according to some embodiments. [Figure 8B]

[0020] FIG. 1 is a side perspective view of a mobile gesture controller according to some embodiments. [Figure 8C] 1 illustrates a close-up view of the tip of a mobile gesture controller according to some embodiments. [Figure 8D] 1 illustrates a bottom view of a mobile gesture controller according to some embodiments. [Figure 9]

[0021] 1 illustrates an example of a mobile gesture controller according to another embodiment. [Figure 10]

[0022] 1 is a flowchart of a method of operation of a mobile gesture controller according to some embodiments. [Figure 11A]

[0023] 4 shows the configuration of a receiver operating in a live setting according to different embodiments; [Figure 11B]

[0023] Figure 3 shows a receiver configuration operating in a studio setting according to different embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0014] Detailed Description

[0024] Aspects of the technology described herein provide an audio spatialization system including a mobile gesture controller configured to control sound source position and audio parameters for both live performances and studio settings. The mobile gesture controller may be a handheld controller that responds to natural human musical gestures and generates position and / or orientation data that can be used to indicate a desired location of a sound source generated by an audio playback device of the audio spatialization system. Thus, a user can manipulate the location of sound through gestures with the mobile gesture controller, such as panning, flicking, waving, pointing, twisting, throwing, hitting, or other gestures. For example, physical interactions such as throwing, swiping, or hitting can be mapped to different user gestures, thereby enabling physical interaction with the spatialized audio. The mobile gesture controller also includes sensors that provide output signals associated with audio and / or room parameters. Thus, the mobile gesture controller can be used like an instrument in controlling multiple aspects of the generated audio.

[0015]

[0025] The audio spatialization system may also include a receiver configured to receive position and orientation data from the mobile gesture controller and convert such data into data that audio spatialization software, such as that executed by a digital audio controller (DAC), can manipulate. The receiver may be located at different points within the audio spatialization system depending on whether the performance is a live performance or a studio setting.

[0016]

[0026] The DAC can receive data from the receiver as input and provide control signals to an audio reproduction device, which outputs desired audio indicative of desired sound source positions and audio parameters.

[0017]

[0027] The audio spatialization system described herein provides fine control over sound source location and audio parameters with low latency, facilitating its use as a music controller in live performances. The audio spatialization system can respond in real time to user gestures and sensor controls, as if the user were playing a musical instrument. Thus, a user may be a conductor, musician, singer, dancer, or other type of performer or sound engineer controlling a musical performance in a live setting such as a concert hall, stadium, or other setting. The user can control the audio spatialization and audio or room parameters to create desired effects as part of the performance and to create an immersive experience for the audience.

[0018]

[0028] Alternatively, the gesture mobile controller can be used in a studio setting to control sound from, for example, a live sound booth or to control a pre-recorded track. An audio engineer in a studio setting can use the mobile gesture controller in the same way as a live performer. Controlling audio in this way provides an intuitive means for achieving panning and / or effects compared to using a soundboard, mouse, joystick, or trackball.

[0019]

[0029] According to one aspect of the present technology, an audio spatialization system is provided that includes a mobile gesture controller, a receiver, and an audio workstation. The mobile gesture controller is configured to output both position and / or orientation data and sensor data for controlling audio parameters of generated sounds. The mobile gesture controller is configured to wirelessly communicate with a receiver that receives the position and / or orientation data and the audio parameter control sensor data and converts the received data into a format suitable for input to the audio workstation. The audio workstation processes the received data input and controls sound generation via multiple audio playback devices.

[0020]

[0030] In some embodiments, the mobile gesture controller is a dedicated handheld instrument such as a wand. Thus, the gesture controller may be a handheld controller in at least some embodiments. The mobile gesture controller may include an inertial measurement unit (IMU) configured to output fused data from a multi-axis accelerometer and a multi-axis gyroscope, and optionally a multi-axis geomagnetic sensor. The mobile gesture controller also includes multiple sensors for sensing user inputs used to control parameters of the audio or room the user is interacting with. For example, the mobile gesture controller may include pressure sensors, squeeze sensors, twist sensors, touch sensor plates, sliders, dials, push buttons, or other input devices (including any combination of such input devices). A user can control two or more parameters of the generated audio, such as volume, brightness, room liveliness, or other parameters, by controlling the input devices.

[0021]

[0031] The receiver may execute software in the form of a software plug-in configured to convert data received from the mobile gesture controller into data suitable for input to the audio workstation. In some embodiments, the receiver is a stand-alone hardware component, such as a box, that can be placed in a suitable location within the audio environment. In other embodiments, the receiver may be integrated into a mixing board, CPU, digital audio workstation, or other audio playback processor.

[0022]

[0032] The audio workstation may be one of a variety of suitable audio workstations configured to control audio playback from a multi-channel array of audio playback devices. The audio playback devices may be speakers, speaker stacks, or other audio playback technologies positioned at various points within a room, studio, auditorium, or other environment to enable sound to be generated from specific desired locations within their projection range. The audio workstation, in some embodiments, is a digital audio workstation (DAW).

[0023]

[0033] According to another aspect of the present technology, a mobile gesture controller for audio spatialization is provided. The mobile gesture controller is configured to wirelessly output both IMU data and parameter control sensor data. The mobile gesture controller includes a high-resolution IMU configured to output fused multi-axis gyroscope and multi-axis accelerometer data, and optionally multi-axis geomagnetic sensor data as part of the fused data. Thus, movements of the mobile gesture controller can be tracked with very high resolution. The mobile gesture controller further includes two or more sensors configured to output data used to control audio parameters of an audio spatialization system in which the mobile gesture controller is integrated.

[0024]

[0034] According to another aspect of the present technology, a receiver for an audio spatialization system is provided. The receiver communicates between a mobile gesture controller and an audio workstation. In some embodiments, the receiver is a stand-alone component, and in other embodiments, the receiver is integrated into the audio workstation. The receiver is configured to convert position and orientation data (e.g., fused IMU data) and parameter control sensor data from the mobile gesture controller into spatialization control data and audio parameter data suitable for input to the audio workstation.

[0025]

[0035] In one embodiment, an audio spatialization system is provided that includes a processor for audio spatialization parameter data. The system includes a gesture controller with various sensors, and data generated by the sensors is wirelessly transmitted via Bluetooth Low Energy (BLE) to a receiver that includes a central processing unit (CPU). The receiving CPU is configured to interact with a standalone software processor, a Virtual Studio Technology (VST) / VST3 software plug-in applied to an audio channel strip, or a similar software application compatible with a third-party digital audio workstation (DAW) or other third-party software package.

[0026]

[0036] The CPU can receive data streamed from the gesture controller and perform conversion and processing to enable use of the data by third-party software or general audio spatialization equipment. In some embodiments, position data from the gesture controller can be converted to MIDI data, facilitating integration with spatialization automation systems in third-party DAWs. The CPU also facilitates mapping data from other sensors on the gesture controller to automation parameters within third-party DAWs or other spatialization equipment. This mapping function is fully user-configurable, allowing customization to suit the specific preferences and needs of individual users.

[0027]

[0037] For example, the gesture controller may include a squeeze sensor, and the signal from the squeeze sensor may be mapped by the CPU to automation data in the DAW, thereby producing real-time changes to the audio signal based on changes in pressure applied to the gesture controller. This functionality allows for dynamic changes such as applying a variable filter or granulator in response to changes in pressure, adjusting the filter state (open or closed) or adjusting the granulation strength accordingly.

[0028]

[0038] Additionally, the disclosed CPU manipulates gesture controller data to enhance physical interaction with spatialized audio objects. Activating sensors on the wand can automatically activate pre-determined or user-programmed spatialized trajectories that can be deployed with third-party software. Additionally, virtually assigning physical properties such as weight, inertia, and viscosity to the behavior of spatialized audio objects enables responsive and nuanced interpretation of complex physical gestures. For example, gestures such as thrusting, accelerating, releasing, twisting, turning, or flicking the wand can correspondingly affect spatial audio, firing sound along a predetermined path within a spatial matrix and emulating the properties of a weighted ball, including its velocity and inertial characteristics.

[0029]

[0039] The above aspects and embodiments, as well as additional aspects and embodiments, are further described below. These aspects and / or embodiments may be used individually, all together, or in any combination of two or more, as their application is not limited in this respect.

[0030]

[0040] 1A-1B illustrate an audio spatialization system employed in a live music setting in accordance with aspects of the present technology. A user 102 stands on a stage 104 and holds a gesture controller 106. Multiple audio reproduction devices 108a-108g are positioned on the stage and around the auditorium. In this example, audio reproduction device 108a is located on the left side of the stage, audio reproduction device 108b is located in the center of the stage, audio reproduction device 108c is located on the right side of the stage, audio reproduction device 108d is located on the left side of the audience seats, audio reproduction device 108e is located in the center of the audience seats, audio reproduction device 108f is located on the right side of the audience seats, and audio reproduction device 108g is located at ground level offstage below audio reproduction device 108e. In some embodiments involving standard immersive audio and surround speaker configurations, additional and / or alternative locations for the audio reproduction devices may also be included. A receiver 110 and a digital audio controller (DAC) 112 are also provided.

[0031]

[0041] The user 102 can be any of a variety of types of users. For example, the user 102 can be a conductor, musician, singer, dancer, or sound engineer. As mentioned above and further described below, the gesture controller 106 can be used to control various aspects of the generated audio, the skilled use of which can be considered a performance in itself, just as the user 102 can be considered a performer.

[0032]

[0042] The users 102 may be positioned at various points within the auditorium and may move freely without limitations regarding the performance space. In the illustrated example, the users 102 are standing on the stage 104, such as a conductor, musician, singer, or dancer. Alternatively, the users 102 may be located at the back of the stage, in the orchestra pit, or to the side of the stage, such as an audio engineer. Furthermore, the users 102 may move around as part of operating the mobile gesture controller to control the sound source position. For example, the users 102 may move around on the stage or even enter the audience area.

[0033]

[0043] The mobile gesture controller 106 may be of any of the types described herein. For example, in some embodiments, the mobile gesture controller 106 may be shaped like a wand and may be comfortably held and operated in one hand. The mobile gesture controller 106 may have an ergonomic design to facilitate holding and operation by the user 102. The mobile gesture controller 106 may also include an IMU and two or more additional sensors. The IMU may generate data indicative of the movement of the mobile gesture controller 106, such as data indicative of the position and / or orientation of the mobile gesture controller 106. The IMU data may be fused data that identifies the pose of the mobile gesture controller. In some embodiments, the mobile gesture controller 106 is a dedicated audio spatialization controller.

[0034]

[0044] The receiver 110 in this embodiment is a stand-alone hardware component that may be located in close proximity to the DAC 112. The receiver 110 is configured to run software for mapping data from the mobile gesture controller 106 into audio input data that the DAC 112 can manipulate.

[0035]

[0045] In operation, the user 102 can move the mobile gesture controller 106 to place a sound source at a desired location within the auditorium. For example, the user 102 can point at a given audio reproduction device 108a-108g, and the sound source can move to emanate from that audio reproduction device 108a-108g. The user 102 can point at a space between the audio reproduction devices 108a-108g, and the audio spatialization system can control the audio reproduction devices 108a-108g to manipulate the resulting sound to appear to emanate from the space the user 102 is pointing at. Several examples are shown.

[0036]

[0046] 2A, the user 102 may point the gesture controller 106 at the audio reproduction device 108a, so that sound, whether from an instrument being played live by a person on stage or from a pre-recorded track, can be controlled by the DAC 112 to emanate from the audio reproduction device 108a.

[0037]

[0047] 2B, the user 102 may point the gesture controller 106 at the audio playback device 108c, so that sound, whether from a live instrument or a pre-recorded track, can be controlled by the DAC 112 to emanate from the audio playback device 108c.

[0038]

[0048] 2C, the user need not point to a particular audio reproduction device, but may instead point to any location within the range of the audio reproduction device where the user desires to place the sound source. In this example, the user 102 points the gesture controller 106 toward a point 202 located between audio reproduction devices 108a and 108b. The audio workstation implemented by the DAC 112 appropriately controls the generation of sound from audio reproduction devices 108a-108g to create the impression that the sound is emanating from point 202, thereby acting as the sound source.

[0039]

[0049] Additionally, the user 102 can gesture with the mobile gesture controller 106 in a variety of ways to control either the sound source location or some parameter of the generated sound. The user 102 can pan the mobile gesture controller 102 to create the effect of moving the sound throughout the auditorium, or wave it from front to back or left to right to create a wave-like sensation of sound enveloping the audience, for example. For example, as shown in FIG. 3A , the user 102 can wave their arm from one side to the other, and the sound source can follow in real time, thus creating the impression of sound flowing laterally across the auditorium.

[0040]

[0050] As shown in FIG. 3B , the user 102 can perform a twisting motion with the gesture controller 106. The twisting motion can have a variety of effects. For example, the twisting motion can be mapped to controlling a sound source, creating the impression that the sound is rotating around a vertical axis. Alternatively, the twisting motion can be mapped to a parameter of the sound, such as the playback speed of the audio (e.g., slowing down or speeding up the audio), its volume, or brightness. In some embodiments, the apparent distance of a moving audio object is controlled by twisting the gesture controller along its horizontal axis.

[0041]

[0051] 3C, the user 102 can perform a flicking motion using the gesture controller 106. The flicking motion can control the sound source position and / or parameters of the generated sound, such as volume or brightness, or can trigger a spatialization system to project sound in a predetermined trajectory through the performance space.

[0042]

[0052] The illustrated gestures are non-limiting. The gesture controller 106 can be configured to associate various gestures with changing the sound source location and / or changing the sound parameters, giving the user 102 intuitive control over such location and parameters.

[0043]

[0053] Additionally, the user 102 may provide input to sensors on the mobile gesture controller 102 to control audio parameters. For example, the mobile gesture controller 102 may include one or more mechanical sensors, such as switches, buttons, knobs, dials, pressure sensors, touch-sensitive plates, or sliders, as examples. The user 102 may control audio parameters, such as sound source and orientation parameters (SSPOPs) and / or room modal parameters (RMPs). Examples of SSPOPs include position (X, Y, Z), azimuth, elevation, sound source size, directionality, presence of reproduced sound, distance from the sound source to the audience, brightness, and warmth. Examples of RMPs include room size, presence, early reflections, reverberation, reverberation cutoff frequency, Doppler effect, equalization, filtering, air absorption, distance, attenuation, microphone, sink (speaker-like device or virtual speaker) or speaker position, microphone or sink directionality, and sense of pressure / liveness. The sensors may also be configured to control acoustic tonal characteristics such as timbre, texture density, or additional music processors such as rhythm generators, synthesized audio, pitch generators, digital signal processors, dynamic processors, predefined trajectories, sudden position changes, or non-linear trajectories. Different sensors of the mobile gesture controller 102 may be associated with controlling different parameters. In some embodiments, the mobile gesture controller can operate in different modes, with different modes associating sensors with different parameters. For example, in one mode, a pressure sensor may be configured to control audio volume, and in a different mode, a pressure sensor may be configured to control brightness. The modes may be selectable by the user using a mode control circuit, for example, in a manner further described below in connection with FIG. 7.

[0044]

[0054] 4 is a block diagram of an audio spatialization system 400 in accordance with one aspect of the present technology. The audio spatialization system includes a mobile gesture controller 402, a receiver 404, a DAC 406, and a speaker array 408. The gesture controller 106 is an example implementation of the gesture controller 402. The receiver 110 is an example of the receiver 404. The DAC 112 is an example of the DAC 406. The audio playback devices 108a-108g represent example implementations of the speaker array 408.

[0045]

[0055] In operation, the gesture controller 402 can be used in the manner described above with respect to the gesture controller 106. A user can move the gesture controller to indicate the location or movement of a desired audio source. The gesture controller 402 includes an IMU configured to output inertial motion data, such as fused IMU data, indicating the position and / or orientation (e.g., pose) of the gesture controller 402. A user may additionally or alternatively provide input to the gesture controller 402 via an input device, such as a button, slider, or any of the types described above. The gesture controller 402 may generate parameter control sensor data for controlling any of the above-listed types of parameters. Thus, in some embodiments, the gesture controller outputs absolute position in 3D as parameter control data, such as, for example, rotation vector, linear acceleration, gravity, orientation, and multi-axis geomagnetic data fused with Euler angles and / or quaternion data, and serial MIDI data. In some embodiments, the gesture controller outputs a position in x, y, and z coordinates, and may output a roll, pitch, and yaw.

[0046]

[0056] The gesture controller 402 may wirelessly output IMU data and parameter control sensor data to the receiver 404. Communication may be via Bluetooth, Bluetooth low energy (BLE), WiFi, or other suitable communication protocols. The data is output in a manner that allows it to be received by the receiver 404 regardless of the direction the gesture controller 402 is pointing, thereby allowing a user to point the gesture controller throughout an audio environment such as that shown in FIGS. 1A and 1B, and the wirelessly transmitted data will be received by the receiver 404. In at least some embodiments, the gesture controller 402 includes an omnidirectional transceiver configured to transmit IMU data and parameter control sensor data omnidirectionally.

[0047]

[0057] The gesture controller 402 can transmit IMU data and parameter control sensor data in a suitable low-latency format. Low-latency operation allows the gesture controller to be used in live music performances or other scenarios requiring high responsiveness. In some embodiments, the IMU data and parameter control sensor data are transmitted as a multi-threaded serial data stream to maintain the lowest possible latency. In other embodiments, other techniques that provide low-latency transmission may be used.

[0048]

[0058] The gesture controller 402 can also receive data from the receiver 404. For example, the gesture controller 402 can receive serial data for visualizing parameter data on the gesture controller's display screen, for speaker position calibration or mode selection, among other parameters. The receiver 404 can also send data to the gesture controller 402 for generating a haptic response.

[0049]

[0059] The receiver 404 is configured to convert the IMU data and parameter control sensor data into data that the DAC 406 can manipulate. The DAC 406 may include a digital audio workstation that facilitates mapping sound source locations to positions within the range of an audio reproduction device. The receiver 404 converts the IMU data and parameter control data into a format suitable for input to the DAC 406. For example, the receiver 404 can convert IMU data into spatialization control data (e.g., azimuth and elevation angles) and parameter control sensor data into audio parameter data (e.g., distance from the audience and room lighting).

[0050]

[0060] As an example, the receiver may require position data (degrees) for the sound source position (x,y,z), distance (meters), aperture (degrees), and pitch / yaw / roll (degrees). Data such as room modal parameters may be processed in units of cubic meters, scaled floating-point numbers, or pre-set parameters (integers). Filters may be controlled in frequency bands. Using appropriate processing, such as via a software plug-in, the receiver can recognize which parameters are being manipulated by the gesture controller and convert the data from the gesture controller into the appropriate output format listed above. In some cases, MIDI data may be required for conversion to fit the input requirements of the DAW.

[0051]

[0061] As a further example, a user can preselect a parameter to be controlled by a given sensor on the gesture controller. The user's selection informs the receiver's data processor that all incoming data from that sensor will be mapped to the appropriate format required by the desired parameter. For example, a rotary encoder on the gesture controller can be set to filter mode for a given sensor, such as a squeeze sensor. The squeeze sensor then transmits data to the processor, which converts the value (e.g., an analog value between 0 and 4095 with 12-bit resolution) into a frequency slope previously set by the user. Thus, when the gesture controller is squeezed, the filter frequency increases by a predefined amount depending on the value of the transmitted pressure value (a portion of the 0-4095 range), and the tone changes accordingly. When the user releases the squeeze pressure, the value decreases back to the initial setting. For example, the analog value is converted into a frequency slope that can be read by the filter. This process is performed for each parameter in the spatialization system. The receiver's processor converts and formats the wand's output signal into a data format that can be read by the parameter processor in the DAW or spatialization engine. The output of the gesture controller can also be in other formats, such as quaternions. These can be converted into a format that a DAW or spatialization engine can read, such as x, y, and z coordinates. However, such processing is user-customizable. For example, the z-axis, in some embodiments, can be mapped to a representation parameter such as room brightness, which then involves converting the data from degrees to a brightness level between 0 and 255 (at 8-bit resolution). In practice, the z-axis may have only a limited usable operating range, so the receiver can select an appropriate mapping algorithm to facilitate this conversion to maximize expressiveness. The distance value, expressed as a decimal fraction of meters as the distance from the audience, is converted from the output of a slider sensor, which is typically sampled at 12 bits (as described above) and has an output value between 0 and 4095.

[0052]

[0062] After the appropriate data conversion is performed, the receiver 404 transmits such data to the DAC 406. The receiver 404 may transmit the data per channel such that the DAC 406 receives the data from the receiver 404 on the correct channel.

[0053]

[0063] The DAC 406 can then process the input data from the receiver 404 and generate control signals for the audio reproduction devices of the speaker array 408 to produce sound from the indicated sound source locations and indicate desired parameters. For example, the DAC 406 may receive information regarding the overall audio level, audio level changes in individual output channels, the number of output channels, phase relationships between output channels, and other control and status data. The DAC 406 may include software such as Dolby Atmos.

[0054]

[0064] As described above, the DAC 406 may facilitate mapping sound source positions anywhere within the range of deployed audio reproduction devices. The coordinates of audio reproduction devices deployed at a particular fixed set of locations may be maintained in a table, a selectable menu, or other suitable configurable format. When spatialization control data is received from the receiver, the data may be processed by the DAW on the DAC to determine appropriate control signals to apply to different (fixed) audio reproduction devices of the speaker array to create the effect of sound moving through physical space.

[0055]

[0065] A calibration process can also be performed by the system 400 to calibrate the gesture controller position and the speaker positions within the array. For example, a user can stand in a default position within an audio environment, such as the auditorium of FIGS. 1A-1B, point the gesture controller 402 at a desired audio playback device, and activate the calibration system. The calibration system can include an optical calibration system. For example, the gesture controller 402 can include a high-power LED, laser, or other light emitter. A signal can be received by the receiver 404 or a receiver on the gesture controller itself, thereby establishing IMU data indicating the position and / or orientation as a reference position. The position and orientation of the gesture controller can then be measured by the system relative to that point.

[0056]

[0066] In another example, the exact positions of the speakers deployed in a performance space / studio can be programmed into the table or menu described above by the gesture controller. This is achieved by simply repeating the same method described above for each speaker position. This reduces system setup time and allows the spatialization algorithm on the DAW / DAC to read this data and calculate the correct signals to feed each speaker. Furthermore, the optical calibration system can be used for real-time environment mapping beyond the initial calibration. Using the optical calibration system for real-time environment mapping allows the gesture controller to dynamically adapt to changes in the user's surroundings, ensuring continued accuracy in audio spatialization even in evolving environments.

[0057]

[0067] Thus, in some embodiments, the optical calibration system provides a simple pointing mechanism to enable accurate readings for setting the gesture controller (and spatialization system) to the correct speaker location relative to the performer's position. The user points to the speaker cone and sets its location by clicking one of the rotary encoders (or other input devices) on the gesture controller. Such a configuration of gesture controllers may be used in dance, live concert performances, theatrical situations, or other situations where real-time audio spatialization is required by performers interacting with an audio performance space. The optical calibration system may also provide the user with distance information regarding the speaker the gesture controller is pointing at. Haptic feedback may confirm data lock on the speaker location.

[0058]

[0068] If the gesture controller includes an optical calibration system, the optical signal may be a visible signal that the user can also use to determine where the gesture controller is pointing. For example, the optical calibration system may include a laser that emits a laser light in the direction the gesture controller is pointing, so that the user can activate the laser to identify where the gesture controller is pointing and where the sound source will be located. In some embodiments, an optical aiming system, such as a laser aiming system, for indicating to the user where the gesture controller is pointing may be separate from the optical calibration system.

[0059]

[0069] In some embodiments, the system 400 may provide haptic feedback to the user. The haptic feedback may be emitted by the gesture controller 402 or other components of the system, such as the receiver 404 or the DAC 406. For example, the haptic feedback may be generated in response to a particular type of gesture or a particular type of input from the user to the gesture controller 402. Such haptic feedback may serve as, for example, confirmation of a particular input or an indication that a particular gesture has been performed. Such haptic feedback may be emitted on the gesture controller 402. In some embodiments, haptic feedback may be generated to indicate parameters of the audio generated by the audio spatialization system 400. For example, some parameters of the audio may be difficult for a user to hear in a performance situation, either because of their frequency, amplitude, or other reasons; therefore, the gesture controller may provide a haptic signal to the user to inform the user that the sound being generated by the audio spatialization system is exhibiting that particular characteristic. In some embodiments, the haptic feedback may be provided to alert the user that the position and / or orientation of the gesture controller has exceeded some boundary. For example, if the user points the gesture controller 402 in a direction that is out of range of the virtual sound source, the DAC 406 and / or receiver 404 may generate a signal that is sent to the gesture controller 402 to trigger a haptic response.

[0060]

[0070] The haptic feedback may be generated by suitable on-board haptic actuators in the gesture controller and may therefore take a variety of forms, for example, it may take the form of a vibration, a click, a shake or a buzzer.

[0061]

[0071] 5 is a flowchart illustrating the operation of an audio spatialization system 400 according to one embodiment. The method 500 begins at stage 502 with transmitting IMU data and parameter control sensor data using a gesture controller 402 to a receiver 404. In at least some embodiments as described herein above, the transmission is wireless.

[0062]

[0072] At stage 504 , the receiver 404 receives the IMU data and parameter control sensor data from the gesture controller 402 .

[0063]

[0073] At stage 506, the receiver converts the IMU data and parameter control sensor data into spatialized control data and audio parameter data, which may be in a format suitable for input to the DAC 406. In some embodiments, the spatialized control data and audio parameter data is MIDI data.

[0064]

[0074] At stage 508, the receiver 404 provides the spatialization control data and audio parameter data to the DAC 406. If the receiver 404 is integrated into the DAC 406, stage 508 may be implemented by providing the spatialization control data and audio parameter data to appropriate processing modules of the DAC 406.

[0065]

[0075] At stage 510, the DAC 406 uses internal processing of the gesture controller's position and / or orientation to map the received spatialization control data to control signals for a speaker array to generate sound in a manner that provides the effect of sound emanating from a user-specified sound source position. The DAC 406 also generates control signals for controlling sound parameters based on the audio parameter data generated by the gesture controller 402.

[0066]

[0076] At stage 512, the DAC 406 outputs the audio signal to a speaker array in a wired or wireless manner.

[0067]

[0077] At stage 514 , the audio playback device generates output audio based on the audio signal received from the DAC 406 .

[0068]

[0078] Aspects of the present technology provide a gesture controller for use as part of an audio spatialization system. The gesture controller, in some embodiments, may be mobile, such as a handheld device or a device that may be worn by a user, for example, on the wrist or arm. In some embodiments, the gesture controller includes an ergonomic housing shaped like, for example, a wand or a conductor's baton. When in the form of a handheld device, the gesture controller may have sufficient weight to give the user a sense of weight. The gesture controller may be constructed of multiple materials and configured to withstand impacts, such as being dropped or thrown. In at least some embodiments, the gesture controller is a dedicated audio spatialization controller.

[0069]

[0079] FIG. 6 shows a block diagram of a gesture controller 600 according to some embodiments. The gesture controller 600 is an example of the gesture controller 106 of FIG. 1A. The gesture controller 600 includes at least one high-resolution inertial measurement unit (IMU) 602, a power supply 604, multiple sensors 606, an on-board microprocessor or CPU 608, at least one micro-antenna 610, at least one transceiver 612 (herein encompassing a separate transmitter and receiver), a haptic feedback system 614, a display 616, and an optical calibration system 618. The gesture controller 600 may also include a Secure Digital (SD) card reader for reading and processing data and / or instructions. For example, the CPU 608 can communicate with an SD card to read or store data or processing instructions.

[0070]

[0080] In some embodiments, the IMU 602 may include a commercially available unit, such as the BNO055 9 DOF (degrees of freedom) available from Bosch Sensortec, for measuring (among other data) absolute position, angular velocity, acceleration vector, linear acceleration vector, magnetic field strength vector, and gravity vector. The IMU 602, in at least some embodiments, may provide a fused data output rather than individual inertial sensor data outputs. Providing fused data transmitted by the transceiver 612 (via the antenna 610) may facilitate rapid processing by the receiver of an audio spatialization system, instead of requiring the receiver to process data from individual accelerometers and gyroscopes. Thus, providing fused IMU data rather than individual inertial sensor data facilitates the use of gesture controllers in real-time operation and situations where precise timing and accurate time control are important.

[0071]

[0081] Additionally, the IMU may be a high precision IMU that outputs data that can be used to detect even small gestures, such as a flick of the wrist, a slight upward or downward movement, a twist or rotation of the controller along its long axis, etc. Such gestures are typical human gestures in the context of musical performance, and therefore providing the gesture controller with sufficient precision in detecting such gestures facilitates the use of the gesture controller in musical performance and the creation of an immersive musical experience.

[0072]

[0082] Furthermore, providing an optical targeting system, such as a laser targeting system or a high intensity LED targeting system, facilitates precise speaker position calibration and ensures a high level of accuracy in determining the exact location of speakers in a spatial audio setup, thereby providing an accurate mapping of physical space to audio parameters, which can be important in creating an immersive experience.

[0073]

[0083] In some embodiments, the plurality of sensors 606 may include any of the sensors described previously herein. For example, the sensors 606 may include various sensors including, but not limited to, touch sensors, touch strips, squeeze sensors, potentiometers, rotary encoders, sliders, and button switches. In some embodiments, the gesture controller includes an IMU and two or more input sensors including at least a slider (e.g., to control the distance of a sound source). The more input sensors provided, the more audio parameters can be controlled and the more operating modes can be used with the gesture controller, thus making the gesture controller more widely applicable.

[0074]

[0084] According to embodiments, data received by the IMU 600 and / or the plurality of sensors 606 may be transmitted to at least one CPU 608 for processing. For example, the CPU 608 may operate to read, process, and format incoming data received at the CPU 608 into a custom output structure depending on the device being controlled by the embodiment (e.g., an immersive audio device, a gameplay console, an interactive performance element, and a user-defined output format). The output data format is determined by the software mode setting of the embodiment and may be set by at least one of the plurality of sensors 606 (e.g., the rotary encoder 700 of FIG. 7 described below). In some embodiments, as described below, all control data may be transmitted to the transceiver 612 for transmission to and processing by a receiver and ultimately a DAC as previously described herein.

[0075]

[0085] In some embodiments, the CPU 608 can receive and process data output from the sensor 606. For example, touch sensors can be used to activate the optical calibration system 618 and lock the coordinates of the speaker position. A touch strip can be used to manipulate additional user-defined control parameters (e.g., those not mapped to gesture outputs) as enumerated herein. A squeeze sensor embedded in the handle of the device can be used to control parameters such as those enumerated herein. Potentiometers can be used to map output data to user-selected spatialization parameters as enumerated herein. In some embodiments, such potentiometer sensors can also be custom configured by the user. Rotary encoders can be used to control mode and menu navigation. They can also be used to lock spatial parameter data from the laser calibration mode, such as distance. For example, as will be explained in view of Figure 7, a push button or switch sensor may be used to "quick select" a pre-programmed spatialization mode, such as a performance spatialization mode that directs sound objects to a precise physical location when the wand is pointed at a precise location in 2D / 3D space, and / or a floorplan spatialization mode that directs sound objects to a location where the wand is pointed on a projected floor plan of the physical space. The user may also program other functions for the button, as determined by the desired output.

[0076]

[0086] According to an embodiment, the CPU 608 provides configurable operating software and can provide access to a Secure Digital (SD) card reader database of custom software device settings. The operating software can be designed to read, format, and process all incoming data, simultaneously mapping the data to multiple spatialization parameters while configuring and controlling haptic feedback configurations. For example, there are two larger groups of parameters that the disclosed embodiments can control: sound source position and orientation parameters (SSPOPs) of the type previously described herein and room model parameters (RMPs). The device easily enables real-time manipulation of multiple parameters from each of the above categories by mapping appropriate sensor outputs to parameters. The operating software distinguishes modes, menus, and on-board operational data from control data for use with external devices.

[0077]

[0087] Returning to FIG. 6 , in some embodiments, the device may include a power supply 604 for powering the entire device. The power supply 604 may include a 3.7-5.0 VDC rechargeable battery, which may be charged via a USB-C connection in the base or a wireless charging mat via the base. The battery may be selected to provide 800-1000 mAh; an example is a LiPo battery that meets this standard. The battery may provide several hours of battery life, e.g., 5-8 hours, which may be sufficient for music performance. Additionally, the gesture controller may include circuitry for reducing power consumption when the gesture controller is not in use. For example, if IMU data indicates that the controller has not moved for a user-defined time frame, the gesture controller may enter a sleep or low activity mode.

[0078]

[0088] In some embodiments, the transceiver 612 may be operable to receive power control data from the CPU 608 and transmit such received data via the micro-antenna 610 using Bluetooth Classic (BT), Bluetooth Low Energy (BLE), or Wi-Fi data transfer protocols. The transceiver 612 may also be operable for data pairing with the device being controlled. The transceiver 612 may operate in conjunction with the CPU 608 to ensure smooth and secure transfer of data to the controlled device (typically a computer).

[0079]

[0089] The transceiver may provide sufficient signal strength to enable use of the gesture controller in large spaces, such as concert halls, auditoriums, or stadiums. The transceiver may also provide suitable operation in smaller spaces, such as in music studios. As mentioned above, the transceiver and antenna may operate to transmit in such a way that a receiver of the audio spatialization system receives the transmitted data even when the gesture controller is not directly pointing at the receiver. For example, the transceiver and antenna may, in at least some embodiments, transmit data omnidirectionally.

[0080]

[0090] In some embodiments, the antenna 610 may also be used to locate the gesture controller. That is, the antenna may serve a dual purpose of transmitting IMU data and parameter control sensor data and also providing location services. Signals transmitted to and from the antenna may be used to locate the antenna (e.g., by triangulation or other techniques), thus enhancing the determination of the gesture controller location provided by the IMU data. In some embodiments, two different antennas may be provided, one for transmitting IMU data and parameter control sensor data, and the other for transmitting and receiving location signals for locating the gesture controller.

[0081]

[0091] In some embodiments, at least one feedback system 614 provides advanced haptic feedback as described herein by using at least two (and up to four or more) haptic feedback actuators (pager micromotors, smartphone haptic motors, and / or other advanced inertial manipulation mechanisms) positioned at precise axial locations on the device. These devices are used to create user-configurable haptic feedback. For example, feedback system 614 may be used to depict spatialized boundaries (e.g., an actuator beeps when a specific physical boundary limit is reached or exceeded) or other desired haptic feedback. Feedback system 614 may be configured by the user for any mode and parameters, but is preconfigured in the operating system for operation in the mode described in FIG. 7.

[0082]

[0092] In some embodiments, at least one display 616 is provided. The display 616, in some embodiments, includes a data visualization microscreen. The display 616 may provide visualization of battery life, mode settings, menu options, programming options, memory slots, spatial parameters, audio parameters, and device capabilities, as well as real-time visual readouts of various data being transmitted. The display may also display information identifying the location of a sound source. Such visual cues may complement the gesture control aspects of the gesture controller. The visualization format on the display 616 may be configured using multiple sensors 606, such as rotary encoder sensors.

[0083]

[0093] The inclusion of a data visualization screen on the gesture controller that displays position coordinates and modal movement data helps users more effectively understand and interact with the gesture controller itself and complex spatial audio data. The display provides the user with meaningful feedback and insight into the ongoing operation of the device, enhancing their control and understanding of the spatial audio environment.

[0084]

[0094] In some embodiments, an optical calibration system 618 is provided, which in some embodiments may be a laser aiming system or may include one or more high intensity LEDs. As mentioned above, the optical calibration system 618 may facilitate the speaker position calibration process for use in recording studios and live performance situations.

[0085]

[0095] According to an embodiment, the device may be wirelessly linked (via BT, BLE, or Wi-Fi) to the receiver 110 or other processing console. The precise data output from the sensors 606 in the device is used by software (and hardware attached to the receiver 110 or processing console) to manipulate the position of audio objects in actual physical space and control the actual implementation of SSPOP, RMP, floor plan, and other creative data mappings. In one embodiment, the gesture controller is accompanied by a software plug-in package for the DAW of the receiver 404 running on a CPU, allowing the gesture controller to be connected to an appropriate hardware device connected to the CPU running the DAC 112. In yet another embodiment, the receiver 404 may be connected to an audio spatialization engine running on a CPU connected to the DAC 112. The device also outputs data specifically formatted for mapping to objects on a computer for activating characters and objects within a commercial video gaming console or during gameplay.

[0086]

[0096] According to some embodiments, the gesture controller described herein can be operated in one of multiple selectable modes. For example, different modes can correspond to different types of performance, and different input sensors of the gesture controller are associated with different audio parameters. In some embodiments, different modes are associated with different users or different user preferences. For example, one user may prefer to use a given sensor (e.g., a slider) of the gesture controller to control audio volume, while another user may prefer to use that sensor to control a different audio parameter, such as distance from the user. Thus, different users may set the gesture controller to their preferred mode.

[0087]

[0097] FIG. 7 is a block diagram illustrating how the gesture controller can be operated in different modes, according to some embodiments. In this embodiment, the gesture controller can include at least one rotary encoder sensor 700. A top-level operating system menu is also shown. In some embodiments, the device can be activated by clicking one of multiple sensors 606 (e.g., shown as rotary encoder 700), and such activation provides access to various operational modes 701, 703, 705, and 707. All operational mode data is displayed on 607 (e.g., via display 607). Rotary encoder 700 can function to scroll through operational modes 701, 703, 705, and 707. Once an operational mode is selected, the user clicks rotary encoder 200 to access that mode and scroll through various submenus 702, 704, 706, and 708. Rotary encoder 700 can be clicked again to select a submenu mode. It should be noted that the various modes of operation described herein are contemplated for the device, and that as new technology becomes available, any number of additional or different modes of operation may be incorporated herein.

[0088]

[0098] 7, as described above, a Performance Spatialization Mode ("PSPAT") 701 is provided. In some embodiments, accessing a submenu 702 of the Performance Spatialization Mode 701 provides the performer with multiple memory locations (currently up to 10 user-defined speaker and data configurations) that can be recalled to meet the artistic or technical needs of the performance. Such a Performance Spatialization Mode 701 can be designed to facilitate the incorporation of the device into an actual performance act and is calibrated from the performer's perspective on stage or other performance location. All spatialization effects can be optimized for the performer's location. All data in the Performance Spatialization Mode 701 can be accessed directly, for example, by scrolling through the submenu 702 and clicking the rotary encoder 700 to activate configurations within the CPU 608. In some embodiments, the Operational Mode 702 data can be linked to output from a Laser Program Mode ("LPRO") 707, as described herein.

[0089]

[0099] 7, as described above, a Floor Plan Spatialization Mode ("FSPAT") 703 is provided. In some embodiments, by accessing a submenu 704 of the Floor Plan Spatialization Mode 703, the user is provided with multiple memory locations (currently up to 10 user-defined speaker / floor plan and data configurations) for recalling custom-programmed floor plan projection and gesture spatialization data mappings to meet the artistic or technical needs of the performance. Typically, a floor plan of a physical space (e.g., a recording studio or performance space) is projected onto a screen, and visual avatar sound objects are controlled by pointing the device at precise map locations on the screen. Once a floor plan configuration is programmed and calibrated, the data can be accessed directly by, for example, scrolling through the submenu 704 and clicking the rotary encoder 700 to activate the configuration within the CPU 608. All floor plan data, spatialization software, and hardware connections to the audio diffusion or gaming system reside in the computer or game console. In some embodiments, the Floor Plan Spatialization Mode 703 is calibrated from the perspective of the audience (who may coincidentally be the user), and that data can be linked to the output from the Laser Program Mode 707 as described herein.

[0090]

[0100] 7, as described above, a game controller mode (“CONT”) 705 is provided. In some embodiments, accessing a submenu 706 of the game controller mode 705 provides the user with multiple memory locations (currently up to about 10 video game controller configurations and other possible data configurations for use in game play) for recalling custom-programmed video game controller data mappings to suit specific games being played on a commercially available game console or computer. The game data configurations may be accessed directly, for example, by scrolling through the submenu 706 to the desired configuration and clicking the rotary encoder 700 to activate the configuration within the CPU 608. In some embodiments, the game controller mode 705 data may be linked to output from a laser program 707, as described herein.

[0091]

[0101] With further consideration of FIG. 7 , as described above, a laser program mode 707 is provided. In some embodiments, the laser program mode 707 is designed to provide calibrated data for the three other operating modes 701, 703, and 705. When the laser program mode 707 is enabled, the user can scroll to an available memory slot in a submenu 708. When the user clicks on a slot, a series of programmable parameters are displayed on the display 616. The user can input data from either the gesture controller or other onboard sensors 606 and select how to scale the data from submenu options. Additionally, the laser can be unlocked for operation and enabled, for example, via a touch sensor. Once a desired position is defined, data from that position (typically x, y, and z coordinates) can be locked into memory by clicking the rotary encoder 700. In some embodiments, the laser program mode 707 is designed to provide calibrated data for a wide range of uses in audio spatialization and video game software and hardware. In some embodiments, the laser calibration mode is fully automated, including laser distance measurement.

[0092]

[0102] According to an embodiment, and considering FIGS. 8A-8D , the device may include a gesture controller in a wand-like form factor. In some embodiments, an outlet 800 for the optical calibration system 618 may be provided, with such outlet 800 having an appropriate aperture and / or lens at the tip of the wand. The optical calibration system 618 may include a 5 mW, green (520 nm and 532 nm), Class 3R laser. In another embodiment, the optical calibration system 618 may include a high-intensity LED. The optical calibration system 618 is operated in laser program mode 707 and can be energized and operated, for example, by touching either switch 803 or 804 on either side of the wand. Other mounting configurations are possible, including, but not limited to, a wristband mounting, a glove mounting, a wearable device, mounting the controller to a musical instrument, or embedding the controller in clothing.

[0093]

[0103] In some embodiments, the device may form a cone 801 along the inside of which to house the optical calibration system 618 and IMU 602, as well as several haptic feedback actuators.

[0094]

[0104] In some embodiments, a touch strip sensor 802 may be provided on the top surface of the cone 801 for use in any of the above-described operational modes 701, 703, 705, and 707. The touch strip sensor 802 may be used to manipulate data variables mapped to coordinates other than position data, but may include position data.

[0095]

[0105] In some embodiments, switches 803 and 804 may comprise touch switches mounted on the surface of either side of cone 801. Switches 803 / 804 may be activated in any of operational modes 701, 703, 705, and 707 and may be mapped to any user-defined on / off parameters. In some embodiments, at least one touch switch (e.g., switch 303) may be dedicated to laser activation. In some embodiments, at least one other touch switch (e.g., switch 804) may include an illuminated power switch for the entire device, configured to flash while the wand is charging (via USB-C or a charging mat) and emit a steady light when fully charged. Alternative embodiments may indicate charging status by placing the illuminated indicator in an alternative location and / or displaying charging status data on screen 616.

[0096]

[0106] In some embodiments, at least one radially or axially mounted encoder 805 (e.g., a rotary encoder) may be provided, with such encoder 805 serving as the primary operational mode navigation device for the device. The encoder 805 shown in FIGS. 8A and 8B is a radially mounted rotary encoder. For example, clicking the protruding disk of the encoder 805 activates a switch within the rotary encoder, while scrolling up and down the protruding disk and clicking the disk is the primary means of navigating the operational modes (e.g., 701, 703, 705, and 707) of the device. In other embodiments of the present disclosure, an axially mounted rotary encoder may also be used with a top-push switch-click method.

[0097]

[0107] In some embodiments, a display screen 806 is provided, which is an exemplary implementation of display 616. This small screen displays a visual readout of the wand's operating mode, submenu selections, sensor mapping data, real-time location data, battery status and charge capacity indicators, memory location, connection status to a CPU or game console, and other data useful to the user. The screen is attached to a collar-like extension of the gesture controller's base 807. The collar section of the wand directly behind the display screen 806 may house electronics and power connectors important to the operation of the device, including a rotary encoder. In one embodiment of the present disclosure, a small Secure Digital (SD) card slot is located above the rotary encoder for swapping and storing operational data.

[0098]

[0108] In some embodiments, a base 807 is provided that includes a removable top half that provides access to the internal components. The base 807 may serve to house the main CPU 608, the BT, BLE, and Wi-Fi transceiver 610, the haptic feedback control system 614, the power and charging electronics 604, and a USC-C port 809 accessible from the base at a flat end of the base 807. The base 807 may also have haptic feedback actuators located along its interior.

[0099]

[0109] In some embodiments, a hidden squeeze sensor array 808 may be provided beneath an overlying base grip coating. In one embodiment, this coating is a silicone sheet that is adhered in two halves around the entire perimeter of the base 807, securing the squeeze sensor array between the grip coating and the body of the base 807. The sensors are directly connected to the CPU 608.

[0100]

[0110] In some embodiments (consider FIG. 8D), a USB-C programming and charging port is provided for uploading changes to the device's firmware and charging the power supply 604 within the body of the wand.

[0101]

[0111] In some embodiments, the base plate 810 of the device may serve to house a wireless, mat-based charging system that may eliminate the need for a USB-C port. Note that the on-board CPU 608 in the device may be programmed wirelessly in addition to the USB-C port.

[0102]

[0112] 9 shows an example of a mobile gesture controller according to another embodiment. The mobile gesture controller 900 includes a power button 902, two touchpads 904, a display screen 906, a rotary encoder 908, and a slider 910. Wireless signals are transmitted and received via a transceiver embedded in the tip 912 of the gesture controller. A user can hold the gesture controller near its base 914 and use their thumbs to control the slider 910 and / or the rotary encoder 908.

[0103]

[0113] 8A-8D and 9, it should be appreciated that embodiments of the present application provide a gesture controller that presents an ergonomic, intuitive, and easy-to-use user interface for controlling spatial audio. Incorporating multiple tactile and other input sensors within easy reach of a user's thumb and fingers facilitates intuitive and efficient control of spatialization parameters using only one hand. Users can easily select and manipulate audio parameters in a spatial environment without feeling overwhelmed by or limited by a lack of control.

[0104]

[0114] 10 illustrates a method of operation of a gesture controller according to one embodiment of the present application. The method 1000 begins at stage 1002 by the gesture controller transmitting an optical calibration signal based on user actuation of an optical calibration system of the gesture controller.

[0105]

[0115] Once calibrated, the user may select an operating mode at stage 1004. For example, the user may operate a rotary dial to select a preferred mode associated with the user. The mode may be any of the types previously described herein.

[0106]

[0116] At stage 1006, the method includes generating IMU data in response to the user moving the gesture controller. The IMU data may be fusion data as described above in this specification. The IMU data may indicate a position and / or an orientation of the gesture controller.

[0107]

[0117] At stage 1008, parameter control sensor data for two or more audio parameters is generated based on user actuation of two or more sensors of the gesture controller.

[0108]

[0118] At stage 1010, the IMU data and parameter control sensor data are wirelessly transmitted from the gesture controller to a receiver in the audio spatialization system.

[0109]

[0119] Aspects of the present technology provide a receiver configured to convert data between a gesture controller and a DAC. Examples are receiver 110 and receiver 404, both of which are described earlier in this specification. The receiver can execute software such that the functionality of the receiver is performed. In some embodiments, a software plug-in may be utilized.

[0110]

[0120] 11A-11B show different configurations based on whether the audio spatialization system is being used in a live performance (FIG. 11A) or a studio setting (FIG. 11B).

[0111]

[0121] 11A shows a receiver configuration in a live audio setting. The configuration includes a gesture controller 1102, which can be any of the types described herein. The system also includes a receiver 1108, which can be any of the types of receivers described previously herein. The configuration further includes a live performer 1104 on stage. The live performer 1104 is connected to a microphone or has audio sent via an analog-to-digital converter 1106 to an audio spatialization control engine 1114 within the receiver 1108. For example, the audio spatialization control engine 1114 can be Max or PureData and can run third-party spatialization software.

[0112]

[0122] The gesture controller 1102 transmits data (e.g., IMU data and parameter control sensor data) to the data processor software interface 1110. In this embodiment where the gesture controller 1102 is used on a stage, the gesture controller 1102 transmits multiple data streams to the receiver 1108 via BT / BLE or some other multi-threaded, low-latency data format. The data is sent directly to the data processor software interface 1110, which runs on the spatialization control engine. The information transmitted is 9 degrees of freedom (DOF) fused position data, audio parameter data, gesture pattern data, output data from on-board sensors (squeeze, rotary encoder, push button, touch slider, touchpad) (which can trigger audio parameter data), operating mode data, battery life, room / speaker calibration data (size, dimensions, etc.).

[0113]

[0123] Recorded audio 1112 can also be combined via output from a DAW on the same receiver 1108 (e.g., CPU) and fed to a spatialization control engine 1114, which then feeds to a DAC 1116. The gesture controller 1102 can move any single sound or combination of sounds using gestures in real time. Wherever the performer points, the sound plays in real time. The performer can also control a combination of SSPOP and RMP using onboard sensors in the manner previously described herein. Typically, a user can control two or three audio parameters simultaneously with one hand. If the user uses both hands, more parameters can be controlled simultaneously.

[0114]

[0124] The gesture controller 1102 controls the spatialization control engine 1114 and third-party software through a data processor software interface 1110 that runs on the spatialization control engine. This software converts the output of the gesture controller 1102 into a readable format (such as degrees, a range of numbers, integers, floating-point values, or other formats) required by the individual control parameters and / or passes the data values ​​to the correct features, allowing the gesture controller 1102 to control all parameter features of the third-party spatialization software.

[0115]

[0125] The gesture controller 1102 also receives data from the data processor software interface regarding mode status, enabled parameter features, engaged audio channels, etc. This is displayed on the gesture controller's visualization screen. It also provides haptic feedback to the user regarding the type of audio being spatialized (heavy, airy) and the physical attributes utilized in the performance (weight, acceleration, drag, resistance, viscosity, frequency shift, etc.).

[0116]

[0126] The output of the DAC 1116 is provided to a multi-channel speaker array 1118 to produce the desired sound.

[0117]

[0127] FIG. 11B shows a receiver configuration in a studio setting, such as when recording or mixing spatial audio.

[0118]

[0128] In this mode, an audio engineer in the studio sends audio 1120 directly to the DAW via the input of the ADC 1122, or the audio is pre-recorded 1126 in the DAW 1128. The DAW resides on the receiver 1108, e.g., a CPU, and is running third-party spatialization software or other DAW-based spatialization software (often via a third-party plug-in on the DAW 1128). The output of the DAW is sent to the DAC 1116 and then to the multi-channel speaker array 1118. The gesture controller 1102 controls the third-party spatialization software or other DAW-based spatialization software via a data processor software interface plug-in 1124 running on the DAW 1128. This software plug-in converts the output of the gesture controller 1102 into a readable format (e.g., degrees, numeric range, integer, floating-point value, MIDI data, or other format) required by the individual control parameters or spatialization features of the DAW, allowing the gesture controller 1102 to control all parameter features of third-party spatialization software on the DAW. The gesture controller 1102 can move any single note or combination of notes using gestures in real time or when automation is enabled on the DAW. Wherever the performer points, the notes will play. They can also control combinations of SSPOPs and RMPs using onboard sensors, allowing for two or three simultaneous controls with one hand, or more when using both hands.

[0119]

[0129] When used in a studio, the gesture controller 1102 transmits multiple data streams to the receiver 1108 (CPU) via BT / BLE or some other multi-threaded, low-latency data format. The data is sent directly to a data processor software plug-in running on the DAW. The transmitted information includes 9DOF fused position data, audio parameter data, gesture pattern data, output data from onboard sensors (squeeze, rotary encoder, push button, touch slider, touchpad) (which can trigger audio parameter data), operating mode data, battery life, and room / speaker calibration data (size, dimensions, etc.). All data from the gesture controller is appropriately converted by the software plug-in to access the spatialization features of the DAW.

[0120]

[0130] A gesture controller of the type described herein may also or alternatively be used as a game controller. For example, the gesture controller may be used as a video game controller to control the generation of sounds from the game and / or to control actions within the video game. For example, particular gestures and / or sensor inputs may control the actions or movements of a character (e.g., an avatar) within the game, such as running, jumping, or manipulating an object (e.g., a weapon). In some embodiments, the gesture controller may be used to control a sound object (e.g., a singing voice represented by an image of a ball or puck in an x,y or x,y,z visualization / spatialization system) of a visual avatar in two or three dimensions around the head of a second visual avatar superimposed over the floor plan of a physical space.

[0121]

[0131] For example, when configured for binaural (headset-enabled) spatialization, these visual images reflect the gesture controller's control of the sound's trajectory around the audience member's head in three dimensions. When the gesture controller is pointed away from the avatar's head, the gesture controller moves the ball / puck avatar in conjunction with the physical gesture, thereby moving the sound in space around the individual (e.g., the gesture controller can be moved to trace a circle around the individual's head or traverse any pattern deemed artistic or creatively valuable). Twisting the gesture controller or manipulating a potentiometer can change the audience's height channel, causing the sound to appear to be emanating from above their head. As the gesture controller is moved in real time, the system manipulates the audio objects the individual hears through headphones to accurately reflect that movement (e.g., the individual hears a sound circling around their head, traveling from their left ear above their head through the back of their head to their right ear). It can also operate with 2D or 3D speaker arrays in larger live performance spaces. The system thus operates to physically represent sound trajectory control. Such an embodiment may also provide all the data necessary to shape the trajectories of audio objects in complex audio spatialization systems.

[0122]

[0132] In the context of video games, the sensors of the gesture controller may generate output signals associated with character actions or game selections instead of audio parameters. In some embodiments, the gesture controller may be used to control both audio and actions within a video game and may thus be a multimodal game controller.

[0123]

[0133] Aspects of the present technology provide various advantages. Some of these advantages have been described above. Some advantages will now be listed. It should be understood that not all embodiments provide all of the listed advantages, and that advantages other than those listed herein may be provided in one or more embodiments.

[0124]

[0134] Some embodiments enable an audio spatialization system that allows intuitive, real-time user control of sound source location and audio parameters. Users of the audio spatialization system's gesture controllers can create an immersive audio experience for themselves and their audiences, enhancing performances and recordings. The gesture controllers may represent a new kind of instrument that can be controlled to create multiple different audio effects with precision. The gesture controllers may also allow personalized use by multiple different users via simple mode selection means. Several controllers may be used simultaneously by transmitting position and parameter data on separate, unique, user-defined channels.

[0125]

[0135] The audio spatialization systems described herein, in at least some embodiments, promote immersive listening, allowing users to focus on the acoustic material rather than the technical process of manipulating software with cumbersome items such as mice, joysticks, and trackballs. The controllers described herein are immersive audio native, allowing users to use intuitive gestures in physical space (the realm of musical expression), such as those naturally used in musical performance, to drive the creative process. Users can respond to perceived audio in real time and easily construct creative, interactive, and immersive experiences. Furthermore, the gesture controllers incorporate various types of haptic feedback that provide users with information about parameters such as the gesture controller's orientation, acoustic physics, space boundaries, mode switching, and power status, among others.

[0126]

[0136] As described, at least some embodiments of the audio spatialization system and gesture controller of the present technology provide low-latency operation, including real-time control over the spatialization of live and / or pre-recorded audio. The low-latency operation allows a performer or sound artist to dynamically change the spatial characteristics of the audio during a live performance, significantly enhancing the audience experience and creating a more dynamic, engaging, and immersive acoustic environment for both the audience and the performing artist.

[0127]

[0137] The audio spatialization system and gesture controller of at least some embodiments provide precise single-handed control over a range of specific spatial parameters, for example, precise positioning of sound elements in three-dimensional space, precise control of room or reverberation characteristics, precise manipulation of the perceived distance or direction of a sound source, or sound source projection characteristics, all of which, in at least some embodiments, can be controlled by a user with one hand.

[0128]

[0138] The audio spatialization systems and gesture controllers described herein, in at least some embodiments, offer users a new level of creative expression. By enabling sound artists to physically gesture and shape sound in space, the gesture controllers facilitate new forms of artistic exploration and offer unique ways to express musical or audio ideas. Such control facilitates entirely new forms of audiovisual performance or interactive recordings and installations that were previously not possible.

[0129]

[0139] User feedback can be provided in multiple forms, such as through haptic feedback from a display and gesture controller. Multiple forms of user feedback, including data visualization and sensory experiences, provide an intuitive user experience beyond that provided by the individual components.

[0130]

[0140] The audio spatialization system and gesture controller according to the embodiments described herein assist studio-based audio engineers, live audio engineers, professional musicians, home studio users, live performers (e.g., EDM artists), stage production personnel, concert venue personnel, film and video game audio designers, and others who wish to manipulate immersive audio for creative purposes by reducing the complexity of interacting with the software and enabling a more transparent, intuitive, music / listening-based experience for the user. They can be used in augmented reality (AR) and virtual reality (VR) sound design.

[0131]

[0141] The gesture controller may represent a dedicated hardware solution designed specifically for the complexities of immersive audio mixing. Unlike traditional mixing consoles or mouse-based interfaces, the gesture controller described herein allows sound engineers and audio professionals to interact with sound elements in three-dimensional space using intuitive physical gestures. This level of physical interaction is not only ergonomic, but also more naturally aligned with human perception of sound in real-world environments, allowing the gesture controller to function as an extension of the engineer's musical expression. The gesture controller's precise tactile and motion sensors and ergonomic design provide a level of control and expressiveness that is difficult and time-consuming to achieve with standard studio equipment, making it a valuable tool for addressing the nuances of immersive audio.

[0132]

[0142] Gesture controllers of the type described herein can be user-friendly: a user does not need to have knowledge of how spatialization software works to use the gesture controller to position sound sources and control audio parameters.

[0133]

[0143] One or more aspects and embodiments of the present disclosure involving the execution of a process or method may utilize program instructions executable by a device (e.g., a computer, processor, or other device) to perform or control the execution of the process or method. In this regard, various inventive concepts may be embodied as a computer-readable storage medium (or multiple computer-readable storage media) (e.g., computer memory, one or more floppy disks, compact disks, optical disks, magnetic tapes, flash memories, circuitry in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) that are encoded with one or more programs and that, when executed on one or more computers or other processors, perform methods that implement one or more of the various embodiments described above. One or more computer-readable media may be portable such that one or more programs stored thereon can be loaded into one or more different computers or other processors to implement the various aspects described above. In some embodiments, the computer-readable medium may be non-transitory.

[0134]

[0144] In some embodiments, a non-transitory computer-readable storage medium is provided that stores processor-executable instructions, which, when executed by a processor, perform a method described herein. For example, the computer-readable storage medium may store instructions that, when executed by a processor of a receiver described herein, may perform a method performed by the receiver of an audio spatialization system described herein. Similarly, the computer-readable storage medium may store instructions that, when executed by a CPU of a gesture controller, cause the gesture controller to perform a method attributed to the gesture controller and described herein. Examples of computer-readable storage media include solid-state memory, disks, tapes, and flash drives, or the examples thereof described above.

[0135]

[0145] While several embodiments have been shown and described, it will be understood by those skilled in the art that various changes and modifications can be made to these embodiments without changing or departing from the scope, spirit, or function thereof. The terms and expressions used in the preceding specification are used herein as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude the features shown and equivalents of the described portions thereof.

[0136]

[0146] For example, various embodiments have been described in which the gesture controller outputs IMU data and parameter control sensor data to a receiver, which then processes such data appropriately and provides it to an audio workstation. In an alternative embodiment, the gesture controller itself includes a software processor, and the gesture controller autonomously performs all necessary data conversion and manipulation algorithms. In this case, third-party software interfaces with the BLE data line directly or via a plug-in or standalone program, simplifying integration into existing audio generation workflows.

[0137]

[0147] As used in this specification and in the claims, the term "and / or" should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with "and / or" should be construed in the same manner, i.e., "one or more" of the elements so conjoined. Other elements other than the elements specifically identified by the "and / or" clause may optionally be present, whether related or unrelated to those specifically identified elements.

[0138]

[0148] As described, some aspects may also be embodied as one or more methods. The actions performed as part of a method may be ordered in any suitable manner. Thus, embodiments may be constructed in which actions are performed in an order different from that illustrated, and these embodiments may include performing some actions simultaneously although shown as sequential actions in the illustrated embodiments.

Claims

1. 1. An audio spatialization system comprising: A handheld controller including an inertial measurement unit (IMU), IMU data indicative of the position and / or orientation of the handheld controller; parameter control sensor data specifying at least two parameters of audio produced by the audio spatialization system; a handheld controller configured to wirelessly output the a receiver configured to wirelessly receive the IMU data and the parameter control sensor data from the handheld controller, the receiver further configured to convert the IMU data and the parameter control sensor data into spatialized control data and audio parameter data; a digital audio workstation (DAW) coupled to a plurality of audio reproduction devices disposed at respective locations within the environment, the digital audio workstation (DAW) configured to receive the spatialization control data and audio parameter data from the receiver and to control generation of the audio from one or more of the audio reproduction devices based on the spatialization control data and the audio parameter data; An audio spatialization system comprising:

2. The audio spatialization system of claim 1 , wherein the IMU data indicative of the position and / or orientation of the handheld controller is fused data that identifies a pose of the handheld controller.

3. 10. The audio spatialization system of claim 1, wherein the handheld controller further comprises a wireless transceiver configured to wirelessly output the IMU data indicative of a position and / or orientation of the handheld controller in an omnidirectional manner.

4. 10. The audio spatialization system of claim 1, wherein the handheld controller includes an optical calibration system configured to emit an optical signal and set a position of the handheld controller as a reference position when a user activates the optical calibration system.

5. 10. The audio spatialization system of claim 1, wherein the handheld controller further includes a mode control circuit configured to receive user input to select a mode, and the handheld controller is configured to adjust data transmitted to the receiver based on which mode is selected.

6. The audio spatialization system of claim 5 , wherein the mode control circuitry is configured to select among a plurality of modes calibrated with respective user preferences.

7. The audio spatialization system of claim 1 , wherein the handheld controller further comprises one or more input devices operable to generate the parameter control sensor data.

8. The audio spatialization system of claim 7 , wherein the one or more input devices include a pressure sensor.

9. The audio spatialization system of claim 7 , wherein the one or more input devices include a rotary dial.

10. The audio spatialization system of claim 7 , wherein the one or more input devices include a slider.

11. The audio spatialization system of claim 7 , wherein the one or more input devices include one or more push buttons.

12. The audio spatialization system of claim 7 , wherein the one or more input devices include a touch slider.

13. The audio spatialization system of claim 7 , wherein the one or more input devices include one or more touch sensor plates.

14. The audio spatialization system of claim 1 , wherein the receiver is integrated with the digital audio workstation.

15. The audio spatialization system of claim 1 , wherein the receiver is in a housing separate from the digital audio workstation.

16. 1. A method for performing audio spatialization, comprising: Using a handheld controller that includes an inertial measurement unit (IMU), IMU data indicating the position and / or orientation of the handheld controller; parameter control sensor data specifying at least two parameters of the audio to be generated; and wirelessly outputting the wirelessly receiving the IMU data and the parameter control sensor data from the handheld controller using a receiver; converting the IMU data and the parameter control sensor data into spatialized control data and audio parameter data using the receiver; receiving the spatialization control data and audio parameter data from the receiver at a digital audio workstation (DAW) coupled to a plurality of audio reproduction devices located at respective positions within the environment; using the DAW to control generation of the audio from one or more audio reproduction devices of the plurality of audio reproduction devices based on the spatialization control data and the audio parameter data; A method comprising:

17. 17. The method of claim 16, wherein wirelessly outputting the IMU data indicative of the position and / or orientation of the handheld controller comprises wirelessly outputting fusion data identifying a pose of the handheld controller.

18. 17. The method of claim 16, wherein wirelessly outputting the IMU data and the parameter control sensor data indicative of the position and / or orientation of the handheld controller comprises outputting the IMU data and the parameter control sensor data indicative of the position and / or orientation of the handheld controller in all directions.

19. The method of claim 16 , further comprising calibrating a reference position of the handheld controller using an optical calibration system of the handheld controller.

20. 17. The method of claim 16, further comprising receiving a user input to select a mode of the handheld controller, selecting the mode of the handheld controller in response to receiving the user input, and adjusting data transmitted to the receiver based on which mode is selected.

21. The method of claim 20 , wherein the user input for selecting a mode of the handheld controller identifies a mode from among a plurality of modes calibrated with respective user preferences.

22. The method of claim 16 , further comprising receiving, by the handheld controller, one or more input signals that are provided to generate the parameter control sensor data.

23. The method of claim 22 , wherein the one or more input signals include a signal from a pressure sensor in the handheld controller.

24. 23. The method of claim 22, wherein the one or more input signals include a signal from a rotary dial on the handheld controller.

25. 23. The method of claim 22, wherein the one or more input signals include a signal from a slider on the handheld controller.

26. 23. The method of claim 22, wherein the one or more input signals include signals from one or more push buttons on the handheld controller.

27. The method of claim 22 , wherein the one or more input signals include a signal from a touch slider on the handheld controller.

28. 23. The method of claim 22, wherein the one or more input signals include signals from one or more touch sensor plates of the handheld controller.

29. The method of claim 16 , wherein the receiver is integrated with the DAW.

30. The method of claim 16 , wherein the receiver is in a separate housing from the DAW.

31. 1. An audio spatialization controller, comprising: a mobile handheld housing; an inertial measurement unit (IMU) disposed within the mobile handheld housing, the IMU configured to generate IMU data indicative of a position and / or orientation of the mobile handheld housing; a plurality of input devices on the mobile handheld housing configured to generate parameter control sensor data specifying respective parameters of the audio to be generated; a processor disposed within the portable handheld housing, the processor configured to control operation of the portable handheld housing; a wireless transceiver disposed within the mobile handheld housing configured to wirelessly transmit the IMU data indicative of a position and / or orientation of the mobile handheld housing and the parameter control sensor data generated by the plurality of input devices to a receiving device; Audio spatialization controller including:

32. 32. The audio spatialization controller of claim 31 , further comprising an optical calibration system configured to emit an optical signal and set a position of the mobile handheld housing as a reference position when a user activates the optical calibration system.

33. 32. The audio spatialization controller of claim 31, wherein the plurality of input devices includes a mode selection input on the mobile handheld housing configured to receive a mode selection from a user.

34. 34. The audio spatialization controller of claim 33, operable in a plurality of modes representing respective user preferences, the mode selection input configured to receive a mode selection of a first mode from among the plurality of modes.

35. 32. The audio spatialization controller of claim 31, wherein the plurality of input devices includes pressure sensors.

36. 32. The audio spatialization controller of claim 31, wherein the plurality of input devices include a rotary dial.

37. 32. The audio spatialization controller of claim 31, wherein the plurality of input devices include sliders.

38. 32. The audio spatialization controller of claim 31, wherein the plurality of input devices includes one or more push buttons.

39. 32. The audio spatialization controller of claim 31, wherein the plurality of input devices include touch sliders.

40. 32. The audio spatialization controller of claim 31, wherein the plurality of input devices includes one or more touch sensor plates.

41. 32. The audio spatialization controller of claim 31, wherein the mobile handheld housing further comprises a haptic feedback device configured to provide haptic feedback to a user of the audio spatialization controller.

42. 1. A method of operating an audio spatialization controller having a mobile handheld housing and an inertial measurement unit (IMU) disposed within the mobile handheld housing, comprising: generating IMU data indicative of a position and / or orientation of the mobile handheld housing using the IMU located within the mobile handheld housing; generating parameter control sensor data specifying respective parameters of the generated audio using a plurality of input devices on the mobile handheld housing; wirelessly transmitting the IMU data indicative of a position and / or orientation of the mobile handheld housing and the parameter control sensor data generated by the plurality of input devices from the mobile handheld housing to a receiving device; A method comprising:

43. 43. The method of claim 42, further comprising using an optical calibration system of the audio spatialization controller to emit an optical signal when a user activates the optical calibration system and to set a position of the mobile handheld housing as a reference position.

44. 43. The method of claim 42, further comprising selecting an operational mode of the audio spatialization controller using an input device of the plurality of input devices on the mobile handheld housing.

45. 45. The method of claim 44, wherein selecting the operational mode of the audio spatialization controller comprises selecting a mode associated with a user preference of a respective user.

46. 43. The method of claim 42, wherein the plurality of input devices include pressure sensors, and generating the parameter control sensor data is based at least in part on receiving signals representative of user actuation of the pressure sensors.

47. 43. The method of claim 42, wherein the plurality of input devices include a rotary dial, and generating the parameter control sensor data is based at least in part on receiving a signal representative of a user actuation of the rotary dial.

48. 43. The method of claim 42, wherein the plurality of input devices include sliders, and generating the parameter control sensor data is based at least in part on receiving signals representative of user actuation of the sliders.

49. 43. The method of claim 42, wherein the plurality of input devices includes one or more pushbuttons, and generating the parameter control sensor data is based at least in part on receiving signals representative of user actuation of the one or more pushbuttons.

50. 43. The method of claim 42, wherein the plurality of input devices include touch sliders, and generating the parameter control sensor data is based at least in part on receiving signals representative of user actuation of the touch sliders.

51. 43. The method of claim 42, wherein the plurality of input devices includes one or more touch sensor plates, and generating the parameter control sensor data is based at least in part on receiving signals representative of user actuation of the one or more touch sensor plates.

52. 43. The method of claim 42, wherein the mobile handheld housing further includes a haptic feedback device configured to provide haptic feedback to a user of the audio spatialization controller, the method further including providing a haptic signal to the user in response to the user moving the mobile handheld housing outside a boundary.

53. 43. The method of claim 42, wherein the mobile handheld housing further includes a haptic feedback device configured to provide haptic feedback to a user of the audio spatialization controller, the method further including providing a haptic signal to the user in real time response to changes in a user-defined audio parameter.

54. 1. An apparatus comprising: a processing module configured to receive wireless data from a handheld audio spatialization controller, the wireless data comprising: Inertial Measurement Unit (IMU) data indicating the position and / or orientation of the handheld audio spatialization controller; and parameter control sensor data specifying at least two parameters of the audio to be generated; wherein the processing module is further configured to convert the IMU data and parameter control sensor data into spatialized control data and audio parameter data.

55. 55. The apparatus of claim 54, coupled to a digital audio workstation (DAW) and configured to provide the spatialization control data and audio parameter data to the DAW.

56. 55. The apparatus of claim 54, wherein the processing module comprises a hardware processor configured to implement a software plug-in.

57. 55. The apparatus of claim 54, wherein the processing module is configured to convert the IMU data and parameter control sensor data into MIDI data.

58. 1. A method of processing data for input to a spatial audio workstation, comprising: receiving, at a processing module, wireless data from a handheld audio spatialization controller, the wireless data comprising: Inertial Measurement Unit (IMU) data indicating the position and / or orientation of the handheld audio spatialization controller; and parameter control sensor data specifying at least two parameters of the audio to be generated; receiving, converting the IMU data and parameter control sensor data into spatialized control data and audio parameter data using the processing module; A method comprising:

59. 60. The method of claim 58, further comprising providing the spatialization control data and audio parameter data from the processing module to a digital audio workstation.

60. 60. The method of claim 58, wherein the processing module comprises a software plug-in executable by a hardware processor.

61. 59. The method of claim 58, wherein converting the wireless data into spatialized control data and audio parameter data comprises converting the wireless data into MIDI data.

Images