Gestural control hardware for immersive audio and other applications

Abstract

The present disclosure is directed to audio spatialization systems including a gestural controller for controlling sound source location and multiple audio parameters. The audio spatialization systems are usable in live music settings, such as performances, and in studio settings. The gestural controller may control sound source location using inertial sensing, and may include user input devices to permit a user to control parameters of immersive audio generated by the system.

Description

GESTURAL CONTROL HARDWARE FOR IMMERSIVE AUDIO AND OTHER APPLICATIONSCROSS-REFERENCE TO RELATED

[0001] This application claims the benefit under 35 U.S.C. § 1 19(e) to U.S. Provisional Application No. 63 / 483,874, filed February 8, 2023 and entitled “GESTURAL CONTROL HARDWARE FOR IMMERSIVE AUDIO AND OTHER APPLICATIONS”, the entire contents of which are incorporated by reference herein.TECHNICAL FIELD

[0002] The present disclosure is directed to audio spatialization systems and methods utilizing a gestural controller.BACKGROUND

[0003] Spatial audio is a wide technical field that can create virtual auditory spaces (via binaural headsets) and actual auditory spaces (in performance with surround / multi-dimensional speaker arrays) in two-dimensional (2D) and three- dimensional (3D) reproduction formats.SUMMARY

[0004] Aspects and advantages of the embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0005] According to an aspect of the technology, an audio spatialization system is provided, comprising: a handheld controller comprising an inertial measurement unit (IMU) and configured to wirelessly output: IMU data indicating position and / or orientation of the handheld controller; and parametric control sensor data specifying at least two parameters of audio to be produced by the audio spatialization system. The audio spatialization system further comprises a receiver configured to wirelessly receive from the handheld controller the IMU data and the parametric control sensor data, and further configured to translate the IMU data and parametric control sensor data into spatialization control data and audio parametric data. The audio spatialization systems further comprises a digital audio workstation (DAW) coupledto a plurality of sound reproduction devices positioned at respective locations within an environment, the DAW configured to receive the spatialization control data and audio parametric data from the receiver and control generation of the audio from one or more sound reproduction devices of the plurality of sound reproduction devices based on the spatialization control data and audio parametric data.

[0006] According to an aspect of the technology, a method of performing audio spatialization is provided, comprising: wirelessly outputting, with a handheld controller comprising an inertial measurement unit (IMU): IMU data indicating position and / or orientation of the handheld controller of the handheld controller; and parametric control sensor data specifying at least two parameters of audio to be produced. The method further comprises wirelessly receiving from the handheld controller, with a receiver, the IMU data and the parametric control sensor data; translating, with the receiver, the IMU data and the parametric control sensor data into spatialization control data and audio parametric data; receiving, at a digital audio workstation (DAW) coupled to a plurality of sound reproduction devices positioned at respective locations within an environment, the spatialization control data and audio parametric data from the receiver; and controlling, with the DAW, generation of the audio from one or more sound reproduction devices of the plurality of sound reproduction devices based on the spatialization control data and the audio parametric data.

[0007] According to an aspect of the technology, an audio spatialization controller is provided, comprising: a mobile, handheld housing; an inertial measurement unit (IMU) disposed within the mobile, handheld housing and configured to generate IMU data indicating position and / or orientation of the mobile, handheld housing; a plurality of input devices on the mobile handheld housing configured to generate parametric control sensor data specifying respective parameters of audio to be produced; a processor disposed within the mobile, handheld housing and configured to control operation of the mobile, handheld housing; and a wireless transceiver disposed within the mobile, handheld housing, and configured to wirelessly transmit to a receiving device the IMU data indicating position and / or orientation of the mobile, handheld housing and the parametric control sensor data generated by the plurality of input devices.

[0008] According to an aspect of the present technology, a method of operating an audio spatialization controller having a mobile, handheld housing, and an inertialmeasurement unit (IMU) disposed within the mobile, handheld housing, is provided. The method comprises: generating, using the IMU disposed within the mobile, handheld housing, IMU data indicating position and / or orientation of the mobile, handheld housing; generating, using a plurality of input devices on the mobile handheld housing, parametric control sensor data specifying respective parameters of audio to be produced; and wirelessly transmitting from the mobile, handheld housing to a receiving device the IMU data indicating position and / or orientation of the mobile, handheld housing and the parametric control sensor data generated by the plurality of input devices.

[0009] According to an aspect of the technology, an apparatus is provided, comprising: a processing module configured to receive wireless data from a handheld audio spatialization controller, the wireless data including: inertial measurement unit (IMU) data indicating position and / or orientation of the handheld audio spatialization controller; and parametric control sensor data specifying at least two parameters of audio to be produced, wherein the processing module is further configured to translate the IMU data and parametric control sensor data into spatialization control data and audio parametric data.

[0010] According to an aspect of the technology, a method of processing data for input to a spatial audio workstation is provided, comprising: receiving, at a processing module, wireless data from a handheld audio spatialization controller, the wireless data including: inertial measurement unit (IMU) data indicating position and / or orientation of the handheld audio spatialization controller; and parametric control sensor data specifying at least two parameters of audio to be produced. The method further comprises translating, with the processing module, the IMU data and parametric control sensor data into spatialization control data and audio parametric data.

[0011] These and other features, aspects, and advantages of various embodiments of the present disclosure will be understood with reference to the following description, and appended figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Various aspects and embodiments of the application will be described with reference to the following figures. It should be appreciated that the figures are not necessarily drawn to scale. Items appearing in multiple figures are indicated by the same reference number in all the figures in which they appear.

[0013] FIGs. 1A-1 B illustrate an audio spatialization system employed in a live music setting according to aspects of the present technology.

[0014] FIGs. 2A-2C illustrates control over sound source location by a user of the audio spatialization system of FIGs. 1 A-1 B.

[0015] FIGs. 3A-3C illustrate example gestures a user may employ to control sound source location and / or audio parameters in connection with operation of an audio spatialization system.

[0016] FIG. 4 is a block diagram of an audio spatialization system according to some embodiments.

[0017] FIG. 5 is a flowchart of a method of operation of an audio spatialization system according to some embodiments.

[0018] FIG. 6 is a block diagram of a mobile gestural controller according to some embodiments.

[0019] FIG. 7 is a diagram illustrating a program mode selection menu of a mobile gestural controller according to some embodiments.

[0020] FIGs. 8A-8D illustrate an example of a mobile gestural controller according to some embodiments. FIG. 8A is a top view. FIG. 8B is a side perspective view. FIG. 8C represents a zoomed in view of a tip of the gestural controller. FIG. 8D represents a base view of the gestural controller.

[0021] FIG. 9 illustrates an example of a mobile gestural controller according to another embodiment.

[0022] FIG. 10 is a flowchart of a manner of operation of a mobile gestural controller according to some embodiments.

[0023] FIGs. 11 A and 11 B illustrate configurations of a receiver operating in live and studio settings, respectively, according to different embodiments.DETAILED DESCRIPTION

[0024] Aspects of the technology described herein provide an audio spatialization system including a mobile gestural controller configured to control sound source position and audio parameters both for live performances and studio settings. The mobile gestural controller may be a handheld controller responsive to natural human musical gestures and producing position and / or orientation data which may be used to indicate a desired position of a sound source generated by sound reproduction devices of the audio spatialization system. Thus, the user may manipulate sound position by gesturing with the mobile gestural controller, such as by panning, flicking, sweeping, pointing, twisting, throwing, batting, or otherwise gesturing. For example, physical interactions such as throwing sound, swiping sound, or batting sound may be mapped to different user gestures, allowing for physical interaction with the spatialized audio. The mobile gestural controller also includes sensors providing output signals associated with parameters of the audio and / or room. Thus, the mobile gestural controller may be used like an instrument in controlling multiple aspects of the produced audio.

[0025] The audio spatialization system may also include a receiver configured to receive the position and orientation data from the mobile gestural controller and translate such data into data on which audio spatialization software, such as executed by a digital audio controller (DAC), may operate. The receiver may be located at different points within the audio spatialization system depending on whether the performance is a live performance or a studio setting.

[0026] The DAC may receive as input the data from the receiver and provide control signals to sound reproduction devices, which output the desired audio exhibiting desired sound source location and audio parameters.

[0027] The audio spatialization system(s) described herein may provide fine control over sound source location and audio parameters with low latency, facilitating its use as a music controller in live performances. The audio spatialization system may respond in real-time to user gestures and sensor controls, as if the user were playing an instrument. Thus, the user may be a conductor, musician, singer, dancer, or other type of performer or sound engineer controlling a musical performance in a live setting, such as in a concert hall, stadium, or other setting. They may control thespatialization of the audio and the parameters of the audio or room to create a desired effect as part of the performance, creating an immersive experience for listeners.

[0028] Alternatively, the gestural mobile controller may be used in a studio setting, for example to control sound from the live sound booth or to control pre-recorded tracks. The audio engineer in a studio setting may use the mobile gestural controller in the same manner as would a live performer. The control of the audio in this manner provides an intuitive means for achieving panning and or effects compared to use of a sound board, mice, joysticks, or trackballs.

[0029] According to an aspect of the present technology, an audio spatialization system is provided, including a mobile gestural controller, a receiver, and an audio workstation. The mobile gestural controller is configured to output both location and / or orientation data and sensor data for controlling audio parameters of the sound to be produced. The mobile gestural controller is configured to communicate wirelessly with the receiver, which receives the location and / or orientation data and the audio parameter control sensor data and translates the received data into a format suitable for input to the audio workstation. The audio workstation processes the received data input and controls the production of sound via a plurality of sound reproduction devices.

[0030] In some embodiments, the mobile gestural controller is a dedicated, handheld instrument, such as a wand. Thus, the gestural controller may be a handheld controller in at least some embodiments. The mobile gestural controller may include an inertial measurement unit (IMU) configured to output fused data from a multi-axial accelerometer and multi-axial gyroscope, and optionally from a multi- axial geo-magnetic sensor. The mobile gestural controller also includes multiple sensors for sensing user inputs used to control parameters of the audio or room in which the user is operating. For example, the mobile gestural controller may include a pressure sensor, squeeze sensor, twist sensor, touch sensitive plates, slider, dial, pushbutton, or other input device, including any combination of such input devices. The user may control the input devices to control two or more parameters of the audio to be produced, such as volume, brightness, room liveness, or other parameters.

[0031] The receiver may execute software in the form of a software plugin configured to translate the data received from the mobile gestural controller into data suitable for input to the audio workstation. In some embodiments, the receiver is astandalone hardware component, such as a box which may be positioned at a suitable location within the audio environment. In other embodiments, the receiver may be integrated into a mixing board, CPU, digital audio workstation, or other sound reproduction processor.

[0032] The audio workstation may be one of a variety of suitable audio workstations configured to control sound reproduction from a multi-channel array of sound reproduction devices. The sound reproduction devices may be speakers, speaker stacks, or other sound reproduction technologies positioned at various points within the room, studio, auditorium, or other environment, to permit production of sound from a specific, desired location within their field of projection. The audio workstation in some embodiments is a digital audio workstation (DAW).

[0033] According to another aspect of the present technology, a mobile gestural controller is provided for audio spatialization. The mobile gestural controller is configured to wirelessly output both IMU data and parametric control sensor data. The mobile gestural controller comprises a high-resolution IMU configured to output fused multi-axial gyroscope and multi-axial accelerometer data, and optionally multi- axial geo-magnetic sensor data as part of the fused data. Thus, movements of the mobile gestural controller may be tracked with very high resolution. The mobile gestural controller further comprises two or more sensors configured to output data used to control audio parameters of the audio spatialization system with which the mobile gestural controller integrates.

[0034] According to another aspect of the present technology, a receiver is provided for an audio spatialization system. The receiver communicates between a mobile gestural controller and an audio workstation. The receiver is a standalone component in some embodiments, and in other embodiments is integrated into an audio workstation. The receiver is configured to translate location and orientation data (e.g., fused IMU data), as well as parametric control sensor data from the mobile gestural controller into spatialization control data and audio parametric data suitable for input to the audio workstation.

[0035] In an embodiment, an audio spatialization system is provided, including a processor for audio spatialization parametric data. The system includes a gestural controller equipped with various sensors, wherein data generated by the sensors is wirelessly transmitted via Bluetooth Low Energy (BLE) to a receiver comprising aCentral Processing Unit (CPU). The receiving CPU is configured to interact with a stand-alone software processor or a Virtual Studio Technology (VST) / VST3 software plugin, applied to an audio channel strip, or a similar software application compatible with third-party Digital Audio Workstations (DAWs) or other third-party software packages.

[0036] The CPU may receive the streamed data from the gestural controller and perform translations and processing to render the data usable by third-party software or generic audio spatialization equipment. In some embodiments, positional data from the gestural controller may be translated into MIDI data, facilitating integration with the spatialization automation system of a third-party DAW. The CPU also facilitates the mapping of data from other sensors on the gestural controller to automation parameters within the third-party DAW or other spatialization equipment. This mapping functionality is fully user-configurable, allowing customization to accommodate the specific preferences and needs of individual users.

[0037] For instance, the gestural controller may include a squeeze sensor, and the signals from the squeeze sensor may be mapped by the CPU to automation data in the DAW, thereby effecting real-time changes to audio signals based on variations in pressure applied to the gestural controller. This functionality enables dynamic alterations, such as the application of a variable filter or granulator in response to changes in pressure, with the filter state adjusting (opening or closing) or the granulation intensity modulating accordingly.

[0038] Additionally, the disclosed CPU manipulates gestural controller data to enhance physical interactivity with spatialized audio objects. Activation of sensors on the wand can automatically engage pre-determined or user-programmed spatialization trajectories which can be deployed in 3rdparty software. Additionally, virtual assignment of physical characteristics, such as weight, inertia, viscosity, etc., to the behaviour of spatialized audio objects allows for responsive and nuanced interpretation of complex physical gestures. For example, gestures like thrusting, accelerated releasing, twisting, whirling, or flicking the wand may correspondingly affect the spatialized audio, launching sound in a predetermined pathway within the spatial matrix, emulating the characteristics of a weighted ball, including velocity and inertial properties.

[0039] The aspects and embodiments described above, as well as additional aspects and embodiments, are described further below. These aspects and / or embodiments may be used individually, all together, or in any combination of two or more, as the application is not limited in this respect.

[0040] FIGs. 1 A-1 B illustrate an audio spatialization system employed in a live music setting according to aspects of the present technology. A user 102 is on a stage 104 and holding a gestural controller 106. A plurality of sound reproduction devices 108a-108g are positioned on the stage and around the auditorium. In this example, sound reproduction device 108a is located stage left, sound reproduction device 108b is stage center, sound reproduction device 108c is stage right, sound reproduction device 108d is left audience, sound reproduction device 108e is center audience, sound reproduction device 108f is right audience, and sound reproduction device 108g is at the ground level off the stage below sound reproduction device 108e. Additional and / or alternative locations of the sound reproduction devices may also be included in some embodiments, including standard immersive audio and surround speaker configurations. A receiver 1 10 and digital audio controller (DAC) 1 12 are also provided.

[0041] The user 102 may be any of various types of users. For example, the user 102 may be a conductor, a musician, a singer, a dancer, or a sound engineer. As described above and further below, the gestural controller 106 may be used to control various aspects of the produced audio, and its skilled use may itself be considered a performance such that the user 102 may be considered a performer.

[0042] The user 102 may be positioned at various points within the auditorium, and may move freely, without restriction about the performance space. In the illustrated example the user 102 is standing on the stage 104, as might a conductor, musician, singer, or dancer. Alternatively, the user 102 may be in a back part of the stage, in an orchestra pit, or to the side of the stage, as might an audio engineer, for example. Moreover, the user 102 may move about as part of manipulating the mobile gestural controller to control sound source location. For example, the user 102 may move about on the stage or even into the audience.

[0043] The mobile gestural controller 106 may be any of the types described herein. For example, the mobile gestural controller 106 in some embodiments may be shaped like a wand, being able to be held comfortably in and manipulated by onehand. The mobile gestural controller 106 may have an ergonomic design to facilitate holding and movement by the user 102. The mobile gestural controller 106 may also include an IMU and two or more additional sensors. The IMU may produce data indicative of movement of the mobile gestural controller 106, such as data indicative of the position and / or orientation of the mobile gestural controller 106. The IMU data may be fused data identifying a pose of the mobile gestural controller. In some embodiments, the mobile gestural controller 106 is a dedicated audio spatialization controller.

[0044] The receiver 1 10 in this embodiment is a standalone hardware component which may be positioned proximate the DAC 1 12. The receiver 1 10 is configured to execute software for performing mapping of the data from the mobile gestural controller 106 to audio input data on which the DAC 112 may operate.

[0045] In operation, the user 102 may move the mobile gestural controller 106 to position a sound source at a desired location within the auditorium. For example, the user 102 may point at a given sound reproduction device 108a-108g and the sound source may be moved to emanate from that sound reproduction device 108a-108g. The user 102 may point to a space between sound reproduction devices 108a-108g and the audio spatialization system may control the sound reproduction devices 108a-108g to operate such that the resulting sound seems to emanate from the space at which the user 102 is pointing. Some examples are illustrated.

[0046] As shown in FIG. 2A, the user 102 may point the gestural controller 106 at sound reproduction device 108a. As a result, the sound, whether from a live instrument being played by someone on stage or from a pre-recorded track, may be controlled by the DAC 1 12 to emanate from the sound reproduction device 108a.

[0047] As shown in FIG. 2B, the user 102 may point the gestural controller 106 at the sound reproduction device 108c. As a result, the sound, whether from a live instrument or a pre-recorded track, may be controlled by the DAC 1 12 to emanate from the sound reproduction device 108c.

[0048] Moreover, as shown in FIG. 2C, the user need not point at a specific sound reproduction device, but instead may point anywhere where it is desired to place the sound source within the field of the sound reproduction devices. In this example, the user 102 points the gestural controller 106 at a point 202 located between sound reproduction devices 108a and 108b. The audio workstation implemented by theDAC 1 12 controls production of sound from the sound reproduction devices 108a- 108g suitably to create the impression that the sound is emanating from point 202, which therefore serves as the sound source.

[0049] Further still, the user 102 may gesture with the mobile gestural controller 106 in various ways, either to control sound source location or some parameter of the generated sound. The user 102 may pan the mobile gestural controller 102 to create the effect of pushing the sound around the auditorium, or sweeping it from front to back or left to right, for example to create a wave-like sensation of sound washing over or around the audience. For example, as shown in FIG. 3A, the user 102 may sweep his arm from one side to another, and the sound source may follow in realtime, thus creating the impression of the sound traveling across the auditorium.

[0050] As shown in FIG. 3B, the user 102 may make a twisting motion with the gestural controller 106. The twisting motion may create various affects. For example, the twisting motion may be mapped to control of the sound source to give the impression it is rotating on a vertical axis. Alternatively, the twisting motion may be mapped to a parameter of the sound, for example, the reproduction speed of the audio (e.g., slowing down or speeding up the audio), its volume, or brightness. In some embodiments, the apparent distance of the moving audio object is controlled by twisting the gestural controller along its horizontal axis.

[0051] As shown in FIG. 3C, the user 102 may make a flicking motion with the gestural controller 106. The flicking motion may control sound source location and / or may control a parameter of the generated sound, such as volume or brightness, or may trigger the spatialization system to hurl the sound through the performance space in some pre-determined trajectory.

[0052] The illustrated gestures are non-limiting. The gestural controller 106 may be configured to associate various gestures with changes to sound source location and / or changes to parameters of the sound, giving the user 102 control over such location and parameters in an intuitive manner.

[0053] Moreover, the user 102 may provide input to the sensors of the mobile gestural controller 102 to control parameters of the audio. For example, the mobile gestural controller 102 may include one or more mechanical sensors, such as switches, buttons, knobs, dials, pressure sensors, touch sensitive plates, or sliders, as an example. The user 102 may control parameters of the audio such as soundsource and orientation parameters (SSPOPs) and / or room modal parameters (RMPs). Examples of SSPOPs include: position (X,Y,Z); azimuth; elevation; size of the sound source; directivity; presence of the reproduced sound; sound source distance from listener; brilliance; and warmth. Examples of RMPs include: room size; presence; early reflections; reverberation; reverberation cutoff frequency; doppler effect; equalization; filtering; air absorption; distance; decay; microphone, sink (a speaker-like device or virtual speaker) or speaker position; microphone or sink directivity; heaviness / liveness. The sensors may also be configured to control sonic timbral characteristics, such as sound color, textural density, or additional musical processors such as rhythm generators, synthetic audio, pitch generators, digital signal processors, dynamic processors, pre-defined trajectory, sudden position changes, or non-linear trajectories. Different sensors of the mobile gestural controller 102 may be associated with controlling different parameters. In some embodiments, the mobile gestural controller may be operated in different modes with the different modes associating the sensors with different parameters. For example, in one mode a pressure sensor may be configured to control audio volume while in a different mode the pressure sensor may be configured to control brilliance. The mode may be selectable by the user using mode-control circuitry, for example in the manner described further below in connection with FIG. 7.

[0054] FIG. 4 is a block diagram of an audio spatialization system 400 according to an aspect of the present technology. The audio spatialization system includes a mobile gestural controller 402, a receiver 404, a DAC 406, and a speaker array 408. The gestural controller 106 is an example implementation of the gestural controller 402. The receiver 110 is an example of the receiver 404. The DAC 1 12 is an example of the DAC 406. The sound reproduction devices 108a-108g represent an example implementation of the speaker array 408.

[0055] In operation, the gestural controller 402 may be used in the manner described previously herein with respect to gestural controller 106. A user may move the gestural controller to indicate desired sound source location or motion. The gestural controller 402 includes an IMU configured to output inertial motion data, such as fused IMU data indicating position and / or orientation (e.g., the pose) of the gestural controller 402. The user may additionally or alternatively provide input to the gestural controller 402 via input devices such as buttons, sliders, or any of those typesdescribed previously herein. The gestural controller 402 may generate parametric control sensor data for controlling any of the types of parameters listed above. Thus, in some embodiments, the gestural controller outputs absolute position in 3D, for example as Euler angles and / or Quaternion data, fused with rotation vector, linear acceleration, gravity, heading, and multi-axial geo-magnetic data, and parametric control data, such as MIDI data in a serial format. In some embodiments, the gestural controller outputs position in x, y, and z coordinates, and may output roll, pitch, and yaw.

[0056] The gestural controller 402 may output the IMU data and parametric control sensor data wirelessly to the receiver 404. The communication may be via Bluetooth, Bluetooth low energy (BLE), WiFi, or other suitable communication protocol. The data is output in a manner such that it will be received by the receiver 404 irrespective of which way the gestural controller 402 is pointed, which allows the user to point the gestural controller throughout the audio environment, such as that of FIGs. 1 A-1 B, and the wirelessly transmitted data will be received by the receiver 404. In at least some embodiments, the gestural controller 402 includes an omnidirectional transceiver configured to transmit the IMU data and parametric control sensor data omnidirectionally.

[0057] The gestural controller 402 may transmit the IMU data and parametric control sensor data in a suitable low latency format. Low latency operation allows the gestural controller to be used in live music performances, or other scenarios requiring high responsiveness. In some embodiments, the IMU data and parametric control sensor data are transmitted as a multi-threaded serial data stream to maintain the lowest latency possible. Other technologies providing for low latency transmission may be used in other embodiments.

[0058] The gestural controller 402 may also receive data from the receiver 404. For example, the gestural controller 402 may receive serial data for visualization of parametric data on a display screen of the gestural controller, for speaker location calibration, or mode selection, among other parameters. The receiver 404 may also send data to the gestural controller 402 for generating haptic responses.

[0059] The receiver 404 is configured to translate the IMU data and parametric control sensor data into data on which the DAC 406 can operate. The DAC 406 may include a digital audio workstation which facilitates mapping of sound source locationsto the positions within the field of the sound reproduction devices. The receiver 404 performs translation of the IMU data and parametric control data into a format appropriate for input to the DAC 406. For example, the receiver 404 may translate the IMU data into spatialization control data (for example azimuth and elevation) and may translate the parametric control sensor data into audio parametric data (such as distance from listener and room brightness, for example).

[0060] As an example, the receiver may need positional data of the sound source location x, y, z, in degrees, distance in meters, aperture in degrees, and pitch / yaw / roll in degrees. Data such as room modal parameters may be processed in cubic meters, scaled floating point numbers, or preset parameters (whole numbers). Filters may be controlled in frequency bands. The receiver may, using appropriate processing such as via a software plugin, recognize which parameter is being addressed by the gestural controller and convert the data from the gestural controller into the appropriate output format listed above. In some cases, MIDI data will be required in the conversion to match the input requirement of a DAW.

[0061] As a further example, the user may pre-select which parameter is being controlled by a given sensor of the gestural controller. The user selection would notify the data processor in the receiver that all incoming data from that sensor is to be mapped to the appropriate format required by the desired parameter. For instance, a rotary encoder of the gestural controller could be set to filter mode for a given sensor, such as a squeeze sensor. The squeeze sensor will then send data to the processor which will translate the values (for example, analog values of 0-4095 at 12-bit resolution) to frequency gradations that are set by the user ahead of time. Thus, when the gestural controller is squeezed, the filter frequency will increase by a preset amount according to the value of the pressure values sent (some fraction of 0-4095) and alter the timbre of the sound accordingly. When the pressure of the user’s squeeze is released, the value drops to the initial setting. The analog values, for example, are translated to frequency gradients that the filter can read. This process takes place for each parameter in the spatialization system. The processor of the receiver translates and formats the output signals of the wand to a readable data format for the parametric processor in the DAW or spatialization engine. The output of the gestural controller may also be other forms of data, for example quaternions. These may be translated into a form the DAW or spatialization engine can read, suchas x, y, z coordinates. However, such processing is user-customizable. For example, the z axis can be mapped onto expressive parameters like room brightness in some embodiments, which would then involve a data translation from degrees to brightness levels of say 0-255 (at 8 bit resolution). The receiver may choose an appropriate mapping algorithm to facilitate this translation for maximum expressive potential - since the z axis might only have a limited useful degree of motion in practicality. Distance values are expressed in fractions of meters from the listener and may be translated from the output of the slider sensor, which typically is sampled at 12 bits (as above) with output values between 0 and 4095.

[0062] After the appropriate data translation is performed, the receiver 404 sends such data to the DAC 406. The receiver 404 may send data on a channelized-basis, so that the DAC 406 receives the data from the receiver 404 on the correct channel.

[0063] The DAC 406 may then process the input data from the receiver 404 and generate control signals for the sound reproduction devices of the speaker array 408 to generate sounds from an indicated sound source location and exhibiting the desired parameters. For example, the DAC 406 may receive information about overall audio levels, changing audio levels in the individual output channels, the number of output channels, the phase relationships between the output channels, and other control and status data. The DAC 406 may include software such as Dolby Atmos.

[0064] As described above, the DAC 406 may facilitate a mapping of sound source locations to positions anywhere within the field of the sound reproduction devices being deployed. The coordinates of the sound reproduction devices, being deployed in a particular, fixed set of locations, may be maintained in a table, selectable menu, or other suitable configurable format. When spatialization control data is received from the receiver, that data is processed by the DAW on the DAC and may determine the appropriate control signals to apply to the different (fixed) sound reproduction devices of the speaker array to produce the effect of sound moving in the physical space.

[0065] A calibration process may also be performed by the system 400 to calibrate the gestural controller positions and speaker positions in the array. For example, the user may stand at a default position within the audio environment, such as the auditorium of FIGs. 1 A-1 B, and point the gestural controller 402 at a desired soundreproduction device and activate a calibration system. The calibration system may include an optical calibration system. For example, the gestural controller 402 may include a high-power LED, a laser, or other optical emitter. The signal may be received by the receiver 404, or a receiver on the gestural controller itself, which may set the IMU data indicating position and / or orientation as the reference position. Location and orientation of the gestural controller may subsequently be measured relative to that point by the system.

[0066] In another example, the exact locations of the speakers deployed in the performance space / studio can be programmed into the above table or menu by the gestural controller. This is accomplished in the same manner as the method above, only repeated for the locations of each speaker. This speeds system set-up times and enables the spatialization algorithms on the DAW / DAC to read this data and calculate the correct signals to feed each speaker. Moreover, the optical calibration system may be used beyond initial calibration, for real-time environmental mapping. Using the optical calibration system for real-time mapping of the environment allows the gestural controller to dynamically adapt to changes in the user's surroundings, ensuring continued accuracy in audio spatialization even in evolving environments.

[0067] Thus, in some embodiments, the optical calibration system provides a simple pointing mechanism to permit accurate readings for setting the gestural controller (and the spatialization system) to the correct speaker locations in relation to a performer’s position. The user points to the speaker cone and sets that location with a click on one of the rotary encoders (or other input device) of the gestural controller. Such a configuration of the gestural controller can be used in dance, live concert performances, dramatic situations, or other situations where real-time audio spatialization is required by a performer interacting with the audio performance space. The optical calibration system may also provide the user with distance information in relation to the speaker at which the gestural controller is pointed. Haptic feedback may confirm the data lock of the speaker position.

[0068] When the gestural controller includes an optical calibration system, the optical signal may be a visible signal which the user may also use to determine where the gestural controller is pointed. For example, the optical calibration system may include a laser emitting laser light in the direction in which the gestural controller is pointed, so that the user may activate the laser to determine the point at which thegestural controller is pointed and at which the sound source will be located. In some embodiments, an optical sighting system, such as a laser sighting system for indicating to the user where the gestural controller is pointed may be separate from the optical calibration system.

[0069] In some embodiments, the system 400 may provide haptic feedback to the user. The haptic feedback may originate with the gestural controller 402 or with the other components of the system, such as the receiver 404 or DAC 406. For example, the haptic feedback may be generated in response to a particular type of gesture or a particular type of input from the user to the gestural controller 402. Such haptic feedback may serve as confirmation of a particular input, or an indication that a particular gesture was performed, as examples. Such haptic feedback may originate on the gestural controller 402. In some embodiments, haptic feedback may be generated to indicate a parameter of the audio generated by the audio spatialization system 400. For example, some parameters of audio may be difficult for the user to hear in the context of a performance, either because of its frequency, amplitude, or otherwise, and thus the gestural controller may provide a haptic signal to the user to let the user know that the sound being generated by the audio spatialization system is exhibiting that particular characteristic. In some embodiments, haptic feedback may be provided to alert the user that the gestural controller position and / or orientation crosses some boundary. For example, if the user points the gestural controller 402 in a direction which is outside the field of the virtual sound sources, the DAC 406 and / or receiver 404 may generate a signal which is sent to the gestural controller 402 to trigger a haptic response.

[0070] The haptic feedback may be generated by suitable on-board haptic actuators within the gestural controller, and thus take various forms. For example, the haptic feedback may take the form of vibrations, clicks, shakes, or buzzing.

[0071] FIG. 5 is a flowchart illustrating operation of the audio spatialization system 400 according to an embodiment. The method 500 begins at stage 502 with transmitting IMU data and parametric control sensor data with the gestural controller 402 to the receiver 404. The transmission is wireless in at least some embodiments as described previously herein.

[0072] At stage 504, the receiver 404 receives the IMU data and parametric control sensor data from the gestural controller 402.

[0073] At stage 506, the receiver converts the IMU data and parametric control sensor data into spatialization control data and audio parametric data. The spatialization control data and audio parametric data may be in a format suitable for input to the DAC 406. In some embodiments, the spatialization control data and audio parametric data are MIDI data.

[0074] At stage 508, the receiver 404 provides the spatialization control data and audio parametric data to the DAC 406. When the receiver 404 is integrated into the DAC 406, stage 508 may be implemented by providing the spatialization control data and audio parametric data to appropriate processing module(s) of the DAC 406.

[0075] At stage 510, the DAC 406 uses its internal processing of gestural controller position and / or orientation to map the received spatialization control data to control signals for the speaker array to generate sound in a manner which provides the effect of the sound emanating from the user-indicated sound source location. The DAC 406 also generates control signals for controlling parameters of the sound based on the audio parametric data generated by the gestural controller 402.

[0076] At stage 512, the DAC 406 outputs audio signals to the speaker array in a wired or wireless fashion.

[0077] At stage 514, the sound reproduction devices produce output audio based on the audio signals received from the DAC 406.

[0078] Aspects of the present technology provide a gestural controller for use as part of an audio spatialization system. The gestural controller may be mobile in some embodiments, for example being a handheld device or a device that may be worn by the user, for example on the wrist or arm. In some embodiments, the gestural controller includes an ergonomic housing, for example being shaped like a wand or a conductor’s baton. When taking the form of a handheld device, the gestural controller may have sufficient weight to provide a sense of heft to the user. The gestural controller may be constructed of materials and configured to withstand shocks, such as being dropped or thrown. In at least some embodiments, the gestural controller is a dedicated audio spatialization controller.

[0079] FIG. 6 illustrates a block diagram of a gestural controller 600 according to some embodiments. The gestural controller 600 is an example of the gestural controller 106 of FIG. 1 A. The gestural controller 600 comprises at least one high- resolution inertial measurement unit (IMU) 602, a power source 604, a plurality ofsensors 606, an onboard microprocessor or CPU 608, at least one micro antenna 610, at least one transceiver 612 (which herein encompasses a separate transmitter and receiver), a haptic feedback system 614, a display 616, and an optical calibration system 618. The gestural controller 600 may also include a secure digital (SD) card reader for reading data and / or instructions and processing the same. For example, the CPU 608 may communicate with an SD card to read or store data or process instructions.

[0080] In some embodiments, IMU 602 may comprise a commercially available unit, such as BNO055 9 DOF (degrees of freedom) available from Bosch Sensortec, for measuring absolute position, angular velocity, acceleration vector, linear acceleration vector, magnetic field strength vector, and gravity vector (among other data). The IMU 602 may provide fused data output in at least some embodiments, rather than individual inertial sensor data output. Providing fused data to be sent by the transceiver 612 (via antenna 610) may facilitate rapid processing by the receiver of the audio spatialization system instead of requiring the receiver to process data from individual accelerometer and gyroscopes. Therefore, providing fused IMU data rather than individual inertial sensor data facilitate real-time operation, and use of the gestural controller in situations in which precise timing and exact temporal control is important.

[0081] Moreover, the IMU may be a high precision IMU outputting data which may be used to detect even small gestures, such as the flick of a wrist, a slight upward or downward motion, a twisting motion of the controller along its long axis, or a twirling motion. Such gestures are typical human gestures in the context of musical performances, and thus providing the gestural controller with sufficient precision in detection of such gestures facilitates the use of the gestural controller in musical performances and creation of immersive musical experiences.

[0082] Moreover, providing an optical sighting system, such as a laser sighing system or high-intensity LED sighting system, facilitates accurate speaker location calibration and ensures a high level of precision in determining the exact positions of speakers in a spatial audio setup. This provides for accurate mapping of the physical space to the audio parameters, which may be important for creating an immersive experience.

[0083] In some embodiments, the plurality of sensors 606 may comprise any of those sensors described previously herein. For example, the sensors 606 may include a variety of sensors including, without limitation, touch sensors, a touch strip, a squeeze sensor, potentiometers, rotary encoders, sliders, and button switches. In some embodiments, the gestural controller includes a IMU and two or more input sensors, including at least a slider (e.g., to control distance of the sound source). The more input sensors provided, the more audio parameters may be controlled and the more modes of operation that may be used with the gestural controller, thus making the gestural controller applicable more broadly.

[0084] According to embodiments, data received at the IMU 600 and / or at the plurality of sensors 606 may be sent to the at least one CPU 608 for processing. For example, CPU 608 may be operative to read, process, and format incoming data received at the CPU 608 into a custom output structure depending on the device being controlled by this embodiment (immersive audio devices, game play consoles, interactive performance elements, and user defined output formats). Output data formats may be determined by the software mode setting of the embodiment and set by at least one of the plurality of sensors 606 (e.g., rotary encoder 700, FIG. 7, described below). In some embodiments, as will be described, all control data may be sent to transceiver 612 for transmission to, and processing by, a receiver and ultimately a DAC as described previously herein.

[0085] In some embodiments, CPU 608 may receive and process data outputs from sensors 606. For example, touch sensors may be used to activate the optical calibration system 618 and to lock in coordinates for speaker location. A touch strip may be used to manipulate additional, user defined, control parameters (e.g., those not mapped onto gestural output) such as those listed herein. A squeeze sensor embedded in the handle of the present apparatus may be used to control parameters such as those listed herein. Potentiometers may be used to map their output data onto user selected spatialization parameters as listed herein. In some embodiments, such potentiometer sensors may also be custom configured by the user. Rotary encoders may be used to control mode and menu navigation. They may also be used to lock in spatial parameter data from the laser calibration mode, such as distance. For example, as will be described having regard to FIG. 7, pushbuttons or switch sensors may be used to “quick select” pre-programmed spatialization modes, suchas in performance spatialization mode, where the wand directs a sound object to a precise physical location when it is pointed at that exact location in 2-D / 3D space, and / or in floorplan spatialization mode, where the wand directs a sound object to a location to which it is pointed on a projected floorplan of a physical space. The user may also program other functions onto the buttons as determined by the desired output.

[0086] According to embodiments, CPU 608 may provide a configurable operating software and access to an SD (Secure Digital) card reader database of custom software device settings. The operating software may be designed to read, format, and process all incoming data, and map data onto multiple spatialization parameters simultaneously while setting and controlling haptic feedback configurations. For example, there are two larger groups of parameters which the disclosed embodiment may control: sound source position and orientation parameters (SSPOPs) and room model parameters (RMPs) of the types described previously herein. The present apparatus easily permits the real-time manipulation of multiple parameters from each category above by mapping the appropriate sensor output onto the parameter. The operating software differentiates mode, menu, and onboard operating data from control data for use by external devices.

[0087] Returning to FIG. 6, in some embodiments, the present apparatus may comprise a power source 604 for providing power to the entire apparatus. Power source 604 may comprise a 3.7-5.0 VDC rechargeable battery, where recharging the battery may be accomplished via a USB-C connection in the base or a wireless charging mat through the base. The battery may be selected to provide between SOO- WOO mAh, an example being a LiPo battery meeting that criterion. The battery may provide several hours of battery life, for example between 5-8 hours of battery life, which may be sufficient for a musical performance. Moreover, the gestural controller may include circuitry for reducing power draw when the gestural controller is not in use. For example, a sleep mode or low activity mode may be entered by the gestural controller when the IMU data indicates the controller has not been moved for a user- defined timeframe.

[0088] In some embodiments, transceiver 612 may be operative to receive output control data from CPU 608 and to transmit such received data, via micro antenna 610, using Bluetooth classic (BT), Bluetooth Low Energy (BLE), or Wi-Fi data transferprotocols. Transceiver 612 may also be operative for data pairing with the device being controlled. Transceiver 612 may work in concert with CPU 608, ensuring data is transferred smoothy and securely to the controlled device (usually a computer).

[0089] The transceiver may provide sufficient signal strength to permit use of the gestural controller in large spaces, such as concert halls, auditoriums, or stadiums. The transceiver may also provide suitable operation within smaller spaces, such as a in a music studio. As described previously, the transceiver and antenna may operate to transmit in a manner that the receiver of the audio spatialization system will receive the transmitted data even when the gestural controller is not pointed directly at the receiver. For example, the transceiver and antenna may transmit data omnidirectionally in at least some embodiments.

[0090] The antenna 610 may also be used to locate the gestural controller in some embodiments. That is, the antenna may serve dual purposes of transmitting IMU data and parametric control sensor data and also provide location services. Signals sent to / from the antenna may be used to locate the antenna (e.g., by triangulation or other techniques), thus enhancing the determination of gestural controller position provided by the IMU data. In some embodiments, two different antennae may be provided, one for transmitting IMU data and parametric control sensor data and another for transmitting and receiving location signals to locate the gestural controller.

[0091] In some embodiments, at least one feedback system 614 provides advanced haptic feedback as described herein by using at least two (up to four or more) haptic feedback actuators (pager micro-motors, smart-phone haptic motors, and / or other advanced inertial manipulation mechanisms) positioned at precise axial locations on the present apparatus. These devices are used for the purpose of creating user configurable tactile feedback. For example, feedback system 614 could be used to delineate spatialization boundaries (e.g., actuators buzz when certain physical boundary limits are reached or exceeded) or other desired tactile feedback. Feedback system 614 may be set for any mode and parameter by the user but is preset in the operating system for operation in the modes set out in FIG. 7.

[0092] In some embodiments, at least one display 616 is provided. The display 616 comprises a data visualization micro-screen in some embodiments. Display 616 may provide visualizations of battery life, mode setting, menu options, programming options, memory slots, spatial parameters, audio parameters, and device functions,and the like, and may also provide a real-time visual readout of the various data being transmitted. The display may also display information identifying sound source location. Such visual cues may complement the gestural control aspect of the gestural controller. Visualization formats on the display 616 can be set by using the plurality of sensors 606, such as the rotary encoder sensors.

[0093] Including on the gestural controller a data visualization screen displaying location coordinates and modal operation data helps users comprehend and interact with the gestural controller itself and the complex spatial audio data more effectively. The display provides users with meaningful feedback and insights into the ongoing operations of the device, and enhances their control and understanding of the spatial audio environment.

[0094] In some embodiments, an optical calibration system 618 is provided, which in some embodiments may be a laser sighting system or may include one or more high-intensity LEDs. As described above, optical calibration system 618 may facilitate the speaker location calibration process for use in recording studios and in live performance situations.

[0095] According to embodiments, the present apparatus may wirelessly link (via BT, BLE, or Wi-Fi) to a receiver 1 10 or other processing console. Precise data output from sensors 606 in the present apparatus are used by software (and hardware attached to the receiver 1 10 or processing console) to manipulate the location of the audio objects in actual physical space and control the actual implementation of the SSPOPs, RMPs, floorplans, and other creative data mappings. In one embodiment, a receiver 404 software plug-in package for a DAW running on a CPU may accompany the gestural controller to connect it with the appropriate hardware device connected to a CPU running a DAC 1 12. In yet another embodiment, the receiver 404 may be connected to an audio spatialization engine running on a CPU connected to a DAC 112. The present apparatus also outputs data specifically formatted for mapping onto objects in commercially available video gaming consoles or on computers for activation of characters and objects in game play.

[0096] According to some embodiments, the gestural controllers described herein may be operated in one of multiple selectable modes. For example, the different modes may correspond to different types of performance, in which the different input sensors of the gestural controller are associated with different audio parameters. Insome embodiments, the different modes are associated with different users or different user preferences. For example, one user may prefer the a given sensor of the gestural controller - such as a slider - to be used to control audio volume, while a different user may prefer that sensor be used to control a different audio parameter, such as distance-from-user. Thus, different users may set the gestural controller to their preferred mode.

[0097] FIG. 7 is a block diagram illustrating the manner in which a gestural controller may be operated in different modes according to some embodiments. In this embodiment, a gestural controller may include at least one rotary encoder sensor 700. Also illustrated is the highest level of the operating system menu. In some embodiments, the present apparatus may be activated by clicking one of the plurality of sensors 606 (e.g., shown as rotary encoder 700), where such activation accesses various operating modes 701 , 703, 705, and 707. All operating mode data is displayed on 607 (e.g., via display 607). Rotating encoder 700 may serve to scroll through the operating modes 701 , 703, 705, and 707. Once an operating mode is chosen, the user clicks rotary encoder 200 to access that mode and to scroll through various sub-menus 702, 704, 706, and 708. To select a sub-menu mode, rotary encoder 700 may be clicked again. It should be noted that the various operating modes described herein are contemplated for the present apparatus, and that any number of additional or different operating modes may be incorporated herein as new technology becomes available.

[0098] Having further regard to FIG. 7, a performance spatialization mode (“PSPAT”) 701 is provided, as described above. In some embodiments, accessing a sub-menu 702 of performance spatialization mode 701 provides the performer with multiple memory locations (currently up to ~10 user-defined speaker and data configurations) which are recalled to suit the artistic or technical needs of the performance. Such performance spatialization mode 701 may be designed to facilitate incorporating the present apparatus in the actual performative act and is calibrated from the point-of-view of the performer on stage or in other performance locations. All spatialization effects may be optimized to the performer’s location. All data in performance spatialization mode 701 can be accessed directly by, for example, scrolling through the sub-menu 702 and clicking rotary encoder 700 to enable the configuration in the CPU 608. In some embodiments, operating mode 702may have its data linked to output from a laser program mode (“LPRO”) 707, as described herein.

[0099] Having further regard to FIG., 7, floorplan spatialization mode (“FSPAT”)703 is provided, as described above. In some embodiments, accessing a sub-menu704 of floorplan spatialization mode 703 provides the user with multiple memory locations (currently up to ~10 user-defined speaker / floorplan and data configurations) to recall custom programmed floorplan projections and gestural spatialization data mappings to suit the artistic or technical needs of the performance. Typically, a floorplan of a physical space (e.g., a recording studio or performance space) is projected on a screen with visual avatars of the sound objects which are controlled by pointing the present apparatus to precise map locations on the screen. Once a floorplan configuration has been programmed and calibrated, its data can be accessed directly by, for example, scrolling through sub-menu 704 and clicking rotary encoder 700 to enable the configuration in the CPU 608. All floorplan data, spatialization software, and hardware connections with the audio diffusion or gaming system reside on a computer or gaming console. In some embodiments, floorplan spatialization mode 703 is calibrated from the point-of-view of the listener (who also happens to be the user) and may have its data linked to output from laser program mode 707, as described herein.

[0100] Having further regard to FIG. 7, a game controller mode (“CONT”) 705 is provided, as described above. In some embodiments, accessing a sub-menu 706 of the game controller mode 705 provides the user with multiple memory locations (currently up to ~10 video came controller configurations and other possible data configurations for use in game play) to recall custom programmed video game controller data mappings to match specific games played on commercially available gaming consoles or computers. Game data configurations can be accessed directly by, for example, scrolling through sub-menu 706 to the desired configuration and clicking rotary encoder 700 to enable the configuration in the CPU 608. In some embodiments, game controller mode 705 may have its data linked to output from laser program 707, as described herein.

[0101] Having further regard to FIG. 7, a laser program mode 707 is provided, as described above. In some embodiments, laser program mode 707 is designed to provide calibrated data to the three other operating modes 701 , 703, 705. When laserprogram mode 707 is enabled, the user may scroll to a free memory slot in sub-menu 708. When the user clicks on the slot, a series of programmable parameters appear on the display 616. The user can input data from the gestural controller, or any of the other on-board sensors 606, and can select how to scale the data from choices in the sub-menu. Additionally, the laser is unlocked for activation and may be enabled via a touch sensor, for example. When it is sighted to the desired location, the data from that location (typically x, y, and z coordinates) may be locked in the memory by clicking rotary encoder 700. In some embodiments, laser program mode 707 is designed to provide calibrated data for use by a wide array of audio spatialization and video game software and hardware. In some embodiments, the laser calibration mode is fully automated to include distance measurements by the laser.

[0102] According to embodiments, having regard to FIGS. 8A - 8D, the present apparatus may comprise a gestural controller in a wand-like form-factor. In some embodiments, an outlet 800 of optical calibration system 618 may be provided, such outlet 800 having an appropriate aperture and / or lens at the tip of the wand. Optical calibration system 618 may comprise a 5 mW, green (520 nm and 532 nm), class 3R laser. In another embodiment, optical calibration system 618 may comprise a high- intensity LED. Optical calibration system 618 may be activated in laser program mode 707 and energized to operation by, for example, touching either switches 803 or 804 in either side of the wand. Other mounting configurations are also possible, such as, without limitation, wristband mounted, glove mounted, wearable device, attaching the controller to a musical instrument, or embedding the controller in clothing.

[0103] In some embodiments, the present apparatus may form a cone 801 for housing optical calibration system 618 and IMU 602, as well as several haptic feedback actuators along its inner sides.

[0104] In some embodiments, a touch-strip sensor 802 may be provided on the top surface of cone 801 for use with any of the above-described operating modes 701 , 703 705, and 707. Touch-strip sensor 802 may be used to manipulate data variables mapped on to coordinates other than positional data, though it may include it.

[0105] In some embodiments, switches 803 and 804 may comprise touch switches attached to the surface of either side of cone 801 . Switches 803 / 804 may be activated in any operating mode 701 , 703 705, and 707 and mapped onto any user-defined on / off parameter. In some embodiments, at least one touch switch (e.g., switch 303)may be dedicated to laser activation. In some embodiments, at least one other touch switch (e.g., switch 804) may comprise a lighted power switch for the entire apparatus, and may be configured to flash while the wand is charging (via USB-C or charging mat) and glow steadily when fully charged. Alternative embodiments may indicate charge status with a lighted indicator in an alternate location and / or charge state data may be displayed on screen 616.

[0106] In some embodiments, at least one radially or axially mounted encoder 805 (e.g., a rotary encoder) may be provided, such encoders 805 serving as the primary operating mode navigation devices in the present apparatus. Encoders 805 as depicted in FIGS. 8A and 8B are radially mounted rotary encoders. For example, clicking the protruding disk of encoder 805 enables the switch inside the rotary encoder, whereas scrolling the protruding disk up or down and clicking the disk are the primary means of navigating through the operating modes (e.g., 701 , 703 705, and 707) of the present apparatus. Axially mounted rotary encoders may also be used in other embodiments of the disclosure with a top-push, switch clicking method.

[0107] In some embodiments, display screen 806 is provided, which is an example implementation of display 616. This small screen presents a visual read-out of the operating mode of the wand, the sub-menu selections, sensor mapping data, realtime positional data, battery status and charge capacity indicator, memory location, connection status to a CPU or gaming console, and other data useful to the user. The screen is mounted on a collar-like extension of the base 807 of the gestural controller. Directly behind display screen 806, in the collar section of the wand, electronics and power connectors may be found which are important to the operation of the present apparatus including the rotary encoders. A small secure digital (SD) card slot above the rotary encoders is present in one embodiment of the disclosure to swap and store operational data.

[0108] In some embodiments, a base 807 is provided, the base comprising a detachable top half that provides access to the components inside. Base 807 may serve to house the main CPU 608, the BT, BLE, and Wi-Fi transceivers 610, the haptic feedback control system 614, the power source and charging electronics 604, as well as a base-accessible USC-C port 809 on the flat end of base 807. Base 807 may also have haptic feedback actuators installed along its inner sides.

[0109] In some embodiments, a concealed squeeze sensor array 808 may be provided underneath an overlaid base-grip coating. In one embodiment, this coating is a silicone sheet which bonds in two halves to base 807 on all sides, securing the squeeze sensor array between the grip coating and the body of base 807. The sensors are connected directly to the CPU 608.

[0110] In some embodiments, having regard to FIG. 8D), a USB-C programming and charging port is provided for uploading changes to the firmware of the present apparatus and for charging the power sources 604 within the body of the wand.

[0111] In some embodiments, base plate 810 of the present apparatus may serve to house a wireless, mat-based, charging system which may abrogate the need for the USB-C port. It should be noted that onboard CPU 608 in the present apparatus may be programmed wirelessly in addition to the USB-C port.

[0112] FIG. 9 illustrates an example of a mobile gestural controller according to another embodiment. The mobile gestural controller 900 includes a power button 902, two touchpads 904, a display screen 906, a rotary encoder 908, and a slider 910. Wireless signals are transmitted and received via a transceiver embedded in the tip 912 of the gestural controller. A user holds the gestural controller near base 914, and may use their thumb to control the slider 910 and / or rotary encoder 908.

[0113] It should be appreciated from reference to FIGs. 8A-8D and 9 that embodiments of the present application provide gestural controllers representing an ergonomic, intuitive, and easy-to-use user interface for controlling spatial audio. Incorporating multiple tactile and other input sensors in positions that allow for simple manipulation with a user’s thumb and fingers facilitates intuitive and efficient control of spatialization parameters using just one hand. Users may easily select and manipulate audio parameters in a spatial environment without feeling overwhelmed by controls or restricted by the lack of them.

[0114] FIG. 10 illustrates a method of operation of a gestural controller according to an embodiment of the present application. The method 1000 begins at stage 1002 with the gestural controller sending an optical calibration signal based on user activation of the optical calibration system of the gestural controller.

[0115] Once calibrated, then at stage 1004 the user may select the mode of operation. For example, the user may operate a rotary dial to select a preferred modeassociated with the user. The mode may be any of the types described previously herein.

[0116] At stage 1006, the method comprises generating IMU data in response to the user moving the gestural controller. The IMU data may be fused data as described previously herein. The IMU data may indicate position and / or orientation of the gestural controller.

[0117] At stage 1008, parametric control sensor data is generated for two or more audio parameters based on user activation of two or more sensors of the gestural controller.

[0118] At stage 1010, the IMU data and parametric control sensor data is transmitted wirelessly from the gestural controller to a receiver of the audio spatialization system.

[0119] Aspects of the present technology provide a receiver configured to translate data between a gestural controller and a DAC. An example is the receiver 110 and the receiver 404, both of which have been described previously herein. The receiver may execute software resulting in performance of the receiver’s functions. In some embodiments, a software plugin may be utilized.

[0120] FIGs. 11 A-11 B illustrate different configurations based on whether the audio spatialization system is being used in a live performance (FIG. 11 A) or in a studio setting (FIG. 11 B).

[0121] FIG. 11 A illustrates the configuration of a receiver in a live audio setting. The configuration includes a gestural controller 1102, which may be any of the types described herein. The system also includes receiver 1108 which may be any of the types of receivers described previously herein. This configuration further includes live performers 1104 on stage. The live performers 1104 are connected to microphones or have their audio sent via an analog to digital converter 1106 to an audio spatialization control engine 1114 within the receiver 1108. For example, the audio spatialization control engine 1114 may be Max or PureData, and may run 3rdparty spatialization software.

[0122] The gestural controller 1102 transmits data (e.g., IMU data and parametric control sensor data) to the data processor software interface 1110. In this embodiment, in which the gestural controller 1102 is used on stage, it transmits a plurality of data to the receiver 1108 via BT / BLE or some other multi-threaded, lowlatency data format. The data is sent directly to a data-processor software interface 1 1 10 running on a spatialization control engine. The transmitted information is 9 degree-of-freedom (DOF) fused positional data, audio parametric data, gestural pattern data, output data (which can trigger audio parametric data) from on-board sensors (squeeze, rotary encoder, pushbuttons, touch slider, touch pad), operational mode data, battery life, and room / speaker calibration data (size, dimensions, etc.).

[0123] Recorded audio 1 1 12 can also be combined via output from a DAW on the same receiver 1 108 (e.g., CPU) and fed into the spatialization control engine 1 1 14 and then on to the DAC 1 1 16. The gestural controller 1 102 can move any single sound or combination of sounds by using gestures in real-time. Wherever the performer points, that is where the sound goes in real-time. The performer can also control combinations of SSPOPs and RMPs using the onboard sensors in the manner described previously herein. Usually, the user may control two or three audio parameters simultaneously with one hand. More parameters may be controlled simultaneously when the user uses two hands.

[0124] The gestural controller 1 102 controls the spatialization control engine 1 114 and the 3rdparty software via a data-processor software interface 1 1 10 running on the spatialization control engine. This software permits the gestural controller 1102 to control all the parametric features of the 3rdparty spatialization software by translating the output of the gestural controller 1 102 into readable formats (such as degrees, numerical ranges, whole numbers, floating point values, or other formats) required by the individual control parameters and / or passing data values to the correct features.

[0125] The gestural controller 1 102 also receives data from the data-processor software interface about mode status, parametric features enabled, audio channels engaged, etc. This is displayed on the visualization screen of the gestural controller. It also provides haptic feedback to the user about the type of audio being spatialized (heavy sound, light sound) and the physical attributes leveraged in the performance (weight, acceleration, drag, resistance, viscosity, frequency shift, etc.).

[0126] The output of the DAC 1 1 16 is provided to a multichannel speaker array 1 1 18 to produce the desired sounds.

[0127] FIG. 1 1 B illustrates the configuration of a receiver in a studio setting, such as when recording or mixing spatial audio.

[0128] In this mode, audio engineers in a studio have their audio 1120 sent directly to a DAW via the inputs of an ADC 1 122 or audio is pre-recorded 1126 on the DAW 1 128. The DAW is on a receiver 1 108, for instance a CPU, and is running either 3rdparty spatialization software, or other DAW-based spatialization software (often via a 3rdparty plugin on the DAW 1128). The output of the DAW goes to the DAC 1 1 16 and then to a multichannel speaker array 11 18. The gestural controller 1102 controls the 3rdparty spatialization software or other DAW-based spatialization software via a data-processor software interface plugin 1124 running on the DAW 1128. This software plugin permits the gestural controller 1 102 to control all the parametric features of the 3rdparty spatialization software on the DAW by translating the output of the gestural controller 1 102 into readable formats (such as degrees, numerical ranges, whole numbers, floating point values, MIDI data, or other formats) required by the individual control parameters or spatialization features on the DAW. The gestural controller 1 102 can move any single sound or combination of sounds by using gestures in real-time or when automation is enabled on the DAW. Wherever the performer points, that is where the sound goes. They can also control combinations of SSPOPs and RMPs using the onboard sensors, for example two to three simultaneously with one hand or more if both hands are used.

[0129] Used in the studio, the gestural controller 1 102 transmits a plurality of data to the receiver 1 108 (CPU) via BT / BLE or some other multi-threaded, low latency data format. The data is sent directly to a data-processor software plugin running on the DAW. The transmitted information is 9 DOF fused positional data, audio parametric data, gestural pattern data, output data (which can trigger audio parametric data) from on-board sensors (squeeze, rotary encoder, pushbuttons, touch slider, touch pad), operational mode data, battery life, and room / speaker calibration data (size, dimensions, etc.). All data from the gestural controller is translated appropriately by the software plugin to access the spatialization features of the DAW.

[0130] Gestural controllers of the types described herein may also or alternatively be used as game controllers. For example, the gestural controller may be used as a video game controller to control the generation of sound from the game and / or to control actions within the video game. For example, certain gestures and / or sensor inputs may control actions or movements of characters (e.g., avatars) within a game,such as running, jumping, or controlling an object (e.g., a weapon). In some embodiments, the gestural controllers can be used to control a visual avatar of a sound-object in two or three dimensions (such as a singing voice represented by an image of a ball or puck in x, y or x, y, z visualization / spatialization systems) around a second visual avatar of a head all overlaid on a floorplan of a physical space.

[0131] For example, when configured for binaural (headset-enabled) spatialization, these visual images mirror control by the gestural controller of the sound trajectory around the listener’s head in three dimensions. If the gestural controller is pointed away from the avatar of the head, the gestural controller moves the ball / puck avatar in concert with the physical gesture and consequently moves the sound within the space around the individual (e.g., the gestural controller may be moved to make circles around the individual’s head, or traverse any pattern deemed of artistic or production value). Twisting the gestural controller or manipulating the potentiometers can change the height channels for the listener, so that it appears that the sound is emanating from above their head. As the gestural controller moves, in real time, the system manipulates the audio-objects heard by the individual through the headphones to reflect the movement exactly (e.g., the individual will hear the sound circling around their head - where the sound might appear to travel from their left ear, behind their head, to their right ear, above their head, and so on). This may also be operationalized in 2D or 3D speaker arrays in larger live performance spaces. The present system thus operates to physicalize sound trajectory control. Such embodiments may also provide all the data needed to shape audio-object trajectories in complex audio spatialization systems.

[0132] In the context of a video game, the sensors of the gestural controller may produce output signals associated with character actions or game selections instead of audio parameters. In some embodiments, the gestural controller may be used to control both audio and actions within a videogame, and therefore may be a multimodal game controller.

[0133] Aspects of the present technology provide various benefits. Some such benefits have been described already. Some benefits are now listed. It should be appreciated that not all embodiments provide all listed benefits, and that benefits besides those now listed may be provided in one or more embodiments.

[0134] In some embodiments, an audio spatialization system permitting intuitive, real-time user control of sound source location and audio parameters is permitted. The user of a gestural controller of the audio spatialization system may create immersive audio experiences for themselves and an audience, enhancing performances and recordings. The gestural controller may represent a new kind of instrument, controllable to create multiple different audio effects with high precision. The gestural controller may also permit personalized use by multiple different users through simple mode selection means. Several controllers may be used simultaneously by transmitting positional and parametric data on separate, unique user defined channels.

[0135] The audio spatialization systems described herein, in at least some embodiments, facilitate immersive listening, allowing users to focus on the sonic materials and not the technical process of operating software with cumbersome items such as mice, joy sticks and trackballs. The controllers described herein are immersive audio native, and users employ intuitive gestures in physical space (the realm of musical expression), such as those that might naturally be employed in musical performances, to drive the creation process. Users respond to perceived audio in real-time and thereby easily construct creative, interactive, immersive experiences. Furthermore, the gestural controller incorporates various types of haptic feedback that provide information to the user about parameters such as gestural controller orientation, sonic physicality, spatial boundaries, mode switching, and power status, among others.

[0136] As has been described, at least some embodiments of the audio spatialization systems and gestural controllers of the present technology provide low latency operation, including real-time control over the spatialization of live and / or prerecorded audio. The low latency operation allows performers or sound artists to dynamically alter the audio's spatial characteristics during live performances, significantly enhancing an audience's experience, and creating a more dynamic, engaging, and immersive sonic environment for both the listener and performing artist.

[0137] The audio spatialization systems and gestural controllers of at least some embodiments offer precise control by one hand over a range of specific spatial parameters. For example, precise positioning of sound elements in three-dimensionalspace, precise control of room or reverberation characteristics, precise manipulation of perceived distance or direction of sound sources, or sound source projection characteristics are all controllable by a user with a single hand in at least some embodiments.

[0138] The audio spatialization systems and gestural controllers described herein, in at least some embodiments, provide a new level of creative expression to the user. By allowing sound artists to physically gesture and shape sound in space, the gestural controller facilitates a new form of artistic exploration and offers unique ways of expressing musical or audio ideas. Such control facilitates entirely new forms of audiovisual performances or interactive recordings and installations previously unachievable.

[0139] User feedback may be provided in multiple forms, including via a display and via haptic feedback from the gestural controller. The multiple forms of feedback to the user, including visualization of data and sensory experiences provides an intuitive user experience that goes beyond that provided by the individual components.

[0140] Audio spatialization systems and gestural controllers according to embodiments described herein may help studio-based audio engineers, live audio engineers, professional musicians, home studio users, live performers (e.g., EDM artists), those involved in stage productions, concert venues, and audio designers for movies and video games who want to manipulate immersive audio for creative purposes by reducing the complexity of interacting with the software and enabling a more transparent, intuitive, and musical / listening-based experience for the user. They may be used in augmented reality (AR) and virtual reality (VR) sound design.

[0141] The gestural controller may represent a dedicated hardware solution specifically designed for the complexities of immersive audio mixing. Unlike traditional mixing consoles or mouse-based interfaces, the gestural controllers described herein allow sound engineers and audio professionals to interact with sound elements in a three-dimensional space using intuitive physical gestures. This level of physical interaction is not just ergonomic but also aligns more naturally with the human perception of sound in a real environment, making the gestural controller perform as an extension of the engineer’s musical expression. The gestural controllers precise tactile and motion sensors, and ergonomic design provide a level of control andexpressiveness that is difficult and time consuming to achieve with standard studio equipment, making it beneficial as a tool for the nuanced demands of immersive audio.

[0142] Gestural controllers of the types described herein may be user friendly. A user need not have knowledge of how spatialization software works to use the gestural controller to place sound sources and control audio parameters.

[0143] One or more aspects and embodiments of the present disclosure involving the performance of processes or methods may utilize program instructions executable by a device (e.g., a computer, a processor, or other device) to perform, or control performance of, the processes or methods. In this respect, various inventive concepts may be embodied as a computer readable storage medium (or multiple computer readable storage media) (e.g., a computer memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement one or more of the various embodiments described above. The computer readable medium or media can be transportable, such that the program or programs stored thereon can be loaded onto one or more different computers or other processors to implement various ones of the aspects described above. In some embodiments, computer readable media may be non-transitory media.

[0144] In some embodiments, a non-transitory computer readable storage medium is provided, storing processor-executable instructions which, when executed by a processor, perform the methods described herein. For example, a computer readable storage medium may store instructions which, when executed by a processor of the receivers described herein, may perform the methods performed by the receivers of the audio spatialization systems described herein. Similarly, a computer readable storage medium may store instructions which, when executed by the CPU of a gestural controller, cause the gestural controller to perform the methods described herein attributed to the gestural controllers. Examples of computer readable storage media include solid state memory, disks, tapes, and flash drives or those examples described above.

[0145] Although a few embodiments have been shown and described, it will be appreciated by those skilled in the art that various changes and modifications can be made to these embodiments without changing or departing from their scope, intent, or functionality. The terms and expressions used in the preceding specification have been used herein as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding equivalents of the features shown and the described portions thereof.

[0146] For example, various embodiments have been described in which the gestural controller outputs IMU data and parametric control sensor data to a receiver, which processes such data suitably to supply it to an audio workstation. In an alternative embodiment, the gestural controller itself contains the software processor, wherein the gestural controller autonomously performs all required data translations and manipulation algorithms. In this instance, any third-party software would interface directly with the BLE data line, or through a plugin or a standalone program, simplifying integration into existing audio production workflows.

[0147] The phrase “and / or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified.

[0148] Also, as described, some aspects may be embodied as one or more methods. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

Claims

What is claimed is:1 . An audio spatialization system, comprising: a handheld controller comprising an inertial measurement unit (IMU) and configured to wirelessly output:IMU data indicating position and / or orientation of the handheld controller; and parametric control sensor data specifying at least two parameters of audio to be produced by the audio spatialization system; a receiver configured to wirelessly receive from the handheld controller the IMU data and the parametric control sensor data, and further configured to translate the IMU data and parametric control sensor data into spatialization control data and audio parametric data; and a digital audio workstation (DAW) coupled to a plurality of sound reproduction devices positioned at respective locations within an environment, the DAW configured to receive the spatialization control data and audio parametric data from the receiver and control generation of the audio from one or more sound reproduction devices of the plurality of sound reproduction devices based on the spatialization control data and audio parametric data.

2. The audio spatialization system of claim 1 , wherein the IMU data indicating position and / or orientation of the handheld controller is fused data identifying a pose of the handheld controller.

3. The audio spatialization system of claim 1 , wherein the handheld controller further comprises a wireless transceiver configured to wirelessly output omnidirectionally the IMU data indicating position and / or orientation of the handheld controller.

4. The audio spatialization system of claim 1 , wherein the handheld controller comprises an optical calibration system configured to emit an optical signal and set as a reference position a position of the handheld controller when a user activates the optical calibration system.

5. The audio spatialization system of claim 1 , wherein the handheld controller further comprises mode-control circuitry configured to receive user input to select a mode, and wherein the handheld controller is configured to adjust data sent to the receiver based on which mode is selected.

6. The audio spatialization system of claim 5, wherein the mode-control circuitry is configured to select among a plurality of modes calibrated with respective user preferences.

7. The audio spatialization system of claim 1 , wherein the handheld controller further comprises one or more input devices operable to produce the parametric control sensor data.

8. The audio spatialization system of claim 7, wherein the one or more input devices comprises a pressure sensor.

9. The audio spatialization system of claim 7, wherein the one or more input devices comprises a rotary dial.

10. The audio spatialization system of claim 7, wherein the one or more input devices comprises a slider.11 . The audio spatialization system of claim 7, wherein the one or more input devices comprises one or more pushbuttons.

12. The audio spatialization system of claim 7, wherein the one or more input devices comprises a touch slider.

13. The audio spatialization system of claim 7, wherein the one or more input devices comprises one or more touch sensitive plates.

14. The audio spatialization system of claim 1 , wherein the receiver is integrated with the digital audio workstation.

15. The audio spatialization system of claim 1 , wherein the receiver is in a housing distinct from the digital audio workstation.

16. A method of performing audio spatialization, comprising: wirelessly outputting, with a handheld controller comprising an inertial measurement unit (IMU):IMU data indicating position and / or orientation of the handheld controller of the handheld controller; and parametric control sensor data specifying at least two parameters of audio to be produced; wirelessly receiving from the handheld controller, with a receiver, the IMU data and the parametric control sensor data; translating, with the receiver, the IMU data and the parametric control sensor data into spatialization control data and audio parametric data; receiving, at a digital audio workstation (DAW) coupled to a plurality of sound reproduction devices positioned at respective locations within an environment, the spatialization control data and audio parametric data from the receiver; and controlling, with the DAW, generation of the audio from one or more sound reproduction devices of the plurality of sound reproduction devices based on the spatialization control data and the audio parametric data.

17. The method of claim 16, wherein wirelessly outputting the IMU data indicating position and / or orientation of the handheld controller comprises wirelessly outputting fused data identifying a pose of the handheld controller.

18. The method of claim 16, wherein wirelessly outputting the IMU data indicating position and / or orientation of the handheld controller and the parametric control sensor data comprises outputting omnidirectionally the IMU data indicating position and / or orientation of the handheld controller and the parametric control sensor data.

19. The method of claim 16, further comprising calibrating a reference position of the handheld controller using an optical calibration system of the handheld controller.

20. The method of claim 16, further comprising receiving user input to select a mode of the handheld controller, selecting the mode of the hand-held controller in response to receiving the user input, and adjusting data sent to the receiver based on which mode is selected.21 . The method of claim 20, wherein the user input to select a mode of the handheld controller identifies a mode from among a plurality of modes calibrated with respective user preferences.

22. The method of claim 16, further comprising receiving, by the handheld controller, one or more input signals provided to produce the parametric control sensor data.

23. The method of claim 22, wherein the one or more input signals comprises a signal from a pressure sensor of the handheld controller.

24. The method of claim 22, wherein the one or more input signals comprises a signal from a rotary dial of the handheld controller.

25. The method of claim 22, wherein the one or more input signals comprises a signal from a slider of the handheld controller.

26. The method of claim 22, wherein the one or more input signals comprises a signal from one or more pushbuttons of the handheld controller.

27. The method of claim 22, wherein the one or more input signals comprises a signal from a touch slider of the handheld controller.

28. The method of claim 22, wherein the one or more input signals comprises a signal from one or more touch sensitive plates of the handheld controller.

29. The method of claim 16, wherein the receiver is integrated with the DAW.

30. The method of claim 16, wherein the receiver is in a housing distinct from the DAW.31 . An audio spatialization controller, comprising: a mobile, handheld housing; an inertial measurement unit (IMU) disposed within the mobile, handheld housing and configured to generate IMU data indicating position and / or orientation of the mobile, handheld housing; a plurality of input devices on the mobile handheld housing configured to generate parametric control sensor data specifying respective parameters of audio to be produced; a processor disposed within the mobile, handheld housing and configured to control operation of the mobile, handheld housing; and a wireless transceiver disposed within the mobile, handheld housing, and configured to wirelessly transmit to a receiving device the IMU data indicating position and / or orientation of the mobile, handheld housing and the parametric control sensor data generated by the plurality of input devices.

32. The audio spatialization controller of claim 31 , further comprising an optical calibration system configured to emit an optical signal and set as a reference position a position of the mobile, handheld housing when a user activates the optical calibration system.

33. The audio spatialization controller of claim 31 , wherein the plurality of input devices comprises a mode selection input on the mobile, handheld housing configured to receive a mode selection from a user.

34. The audio spatialization controller of claim 33, wherein the audio spatialization controller is operable in a plurality of modes representing respective user preferences, and wherein the mode selection input is configured to receive a mode selection of a first mode from among the plurality of modes.

35. The audio spatialization controller of claim 31 , wherein the plurality of input devices comprises a pressure sensor.

36. The audio spatialization controller of claim 31 , wherein the plurality of input devices comprises a rotary dial.

37. The audio spatialization controller of claim 31 , wherein the plurality of input devices comprises a slider.

38. The audio spatialization controller of claim 31 , wherein the plurality of input devices comprises one or more pushbuttons.

39. The audio spatialization controller of claim 31 , wherein the plurality of input devices comprises a touch slider.

40. The audio spatialization controller of claim 31 , wherein the plurality of input devices comprises one or more touch sensitive plates.41 . The audio spatialization controller of claim 31 , wherein the mobile, handheld housing further comprises a haptic feedback device configured to provide haptic feedback to a user of the audio spatialization controller.

42. A method of operating an audio spatialization controller having a mobile, handheld housing, and an inertial measurement unit (IMU) disposed within the mobile, handheld housing, the method comprising: generating, using the IMU disposed within the mobile, handheld housing, IMU data indicating position and / or orientation of the mobile, handheld housing;generating, using a plurality of input devices on the mobile handheld housing, parametric control sensor data specifying respective parameters of audio to be produced; and wirelessly transmitting from the mobile, handheld housing to a receiving device the IMU data indicating position and / or orientation of the mobile, handheld housing and the parametric control sensor data generated by the plurality of input devices.

43. The method of claim 42, further comprising using an optical calibration system of the audio spatialization controller to emit an optical signal and set as a reference position a position of the mobile, handheld housing when a user activates the optical calibration system.

44. The method of claim 42, further comprising selecting a mode of operation of the audio spatialization controller using an input device of the plurality of input devices on the mobile, handheld housing.

45. The method of claim 44, wherein selecting the mode of operation of the audio spatialization controller comprises selecting a mode associated with user preferences of a respective user.

46. The method of claim 42, wherein the plurality of input devices comprises a pressure sensor, and wherein generating the parametric control sensor data is based at least in part on receiving a signal representing user activation of the pressure sensor.

47. The method of claim 42, wherein the plurality of input devices comprises a rotary dial, and wherein generating the parametric control sensor data is based at least in part on receiving a signal representing user activation of the rotary dial.

48. The method of claim 42, wherein the plurality of input devices comprises a slider, and wherein generating the parametric control sensor data is based at least in part on receiving a signal representing user activation of the slider.

49. The method of claim 42, wherein the plurality of input devices comprises one or more pushbuttons, and wherein generating the parametric control sensor data is based at least in part on receiving a signal representing user activation of the one or more pushbuttons.

50. The method of claim 42, wherein the plurality of input devices comprises a touch slider, and wherein generating the parametric control sensor data is based at least in part on receiving a signal representing user activation of the touch slider.51 . The method of claim 42, wherein the plurality of input devices comprises one or more touch sensitive plates, and wherein generating the parametric control sensor data is based at least in part on receiving a signal representing user activation of the one or more touch sensitive plates.

52. The method of claim 42, wherein the mobile, handheld housing further comprises a haptic feedback device configured to provide haptic feedback to a user of the audio spatialization controller, and wherein the method further comprises providing a haptic signal to the user in response to the user moving the mobile, handheld housing outside a boundary.

53. The method of claim 42, wherein the mobile, handheld housing further comprises a haptic feedback device configured to provide haptic feedback to a user of the audio spatialization controller, and wherein the method further comprises providing a haptic signal to the user in response to changes in user defined audio parameters in real-time.

54. An apparatus, comprising: a processing module configured to receive wireless data from a handheld audio spatialization controller, the wireless data including: inertial measurement unit (IMU) data indicating position and / or orientation of the handheld audio spatialization controller; andparametric control sensor data specifying at least two parameters of audio to be produced, wherein the processing module is further configured to translate the IMU data and parametric control sensor data into spatialization control data and audio parametric data.

55. The apparatus of claim 54, wherein the apparatus is coupled to a digital audio workstation (DAW), and is configured to provide the spatialization control data and audio parametric data to the DAW.

56. The apparatus of claim 54, wherein the processing module comprises a hardware processor configured to implement a software plugin.

57. The apparatus of claim 54, wherein the processing module is configured to translate the IMU data and parametric control sensor data into MIDI data.

58. A method of processing data for input to a spatial audio workstation, comprising: receiving, at a processing module, wireless data from a handheld audio spatialization controller, the wireless data including: inertial measurement unit (IMU) data indicating position and / or orientation of the handheld audio spatialization controller; and parametric control sensor data specifying at least two parameters of audio to be produced; and translating, with the processing module, the IMU data and parametric control sensor data into spatialization control data and audio parametric data.

59. The method of claim 58, further comprising providing from the processing module to a digital audio workstation the spatialization control data and audio parametric data.

60. The method of claim 58, wherein the processing module comprises a software plugin executable by a hardware processor.61 . The method of claim 58, wherein translating the wireless data into spatialization control data and audio parametric data comprises translating the wireless data into MIDI data.

Images