Utilizing inaudible ultrasonic frequencies to embed additional audio asset channels within existing audio channels

Abstract

The computer game transmitter can shift (504) the frequency of game assets, such as audio (306) or haptic assets (304), into the ultrasonic range, then mix (506) the assets with game audio (302) and transmit the mix. The receiver (400) then separates the audio from the audible band, reproduces the audio, and downmixes the assets for that reproduction. The frequencies of the assets may be companded (904) before transmission.

Description

[Technical Field]

[0001] This application relates generally to using inaudible ultrasonic frequencies to embed additional audio asset channels within existing audio channels. [Background technology]

[0002] As understood herein, the audio bandwidth available from computer simulated audio, such as computer game audio, can be relatively large, for example, 24 kHz from an audio sampling rate of 48 kHz. Summary of the Invention

[0003] As further understood herein, human hearing is generally limited to a maximum of approximately 12 kHz, or in some circumstances, a maximum of approximately 14 kHz in younger individuals. The present principles recognize that this leaves a 10-12 kHz bandwidth available at ultrasonic frequencies that is otherwise unused and inaudible to most individuals. The present principles describe leveraging the otherwise unused bandwidth for additional assets (such as additional game audio channels or haptic devices) that can be used to drive additional speaker channels or haptic devices.

[0004] Accordingly, the apparatus includes at least one processor configured to identify at least one computer simulation asset (CSA) associated with at least one computer simulation and shift at least one frequency of the CSA to at least one ultrasonic frequency bandwidth.

[0005] The processor is configured to mix the CSA with audio associated with the computer simulation and transmit the CSA along with the audio associated with the computer simulation for presentation by the at least one receiver assembly.

[0006] The CSA may include audio or haptic assets, or even smell, taste, or video assets from the simulation.

[0007] In an embodiment, the processor may be configured to frequency compand the at least one CSA before transmitting the at least one CSA.

[0008] In some embodiments, the CSA is a first CSA, and the processor is configured to identify at least a second CSA, shift at least one frequency of the first CSA to at least a first ultrasonic frequency bandwidth, shift at least one frequency of the second CSA to at least a second ultrasonic frequency bandwidth, and mix the first CSA and the second CSA with audio associated with the computer simulation before transmitting the CSA and audio associated with the computer simulation.

[0009] The at least one ultrasonic frequency bandwidth may include at least one bandwidth in the range of 12 to 24 kHz.

[0010] In another aspect, the device includes at least one computer storage device, which is not a transitory signal, that includes instructions executable by at least one processor for receiving at least one computer simulation asset (CSA) associated with at least one computer simulation. The CSA is mixed with audio associated with the computer simulation. The instructions are executable to extract the audio associated with the computer simulation and play the audio associated with the computer simulation on at least one speaker. Further, the instructions are executable to downshift at least one frequency of the CSA and present the CSA on at least one output device after downshifting the at least one frequency of the CSA.

[0011] The CSA can be an audio asset and the output device can be a speaker. The CSA can be a haptic asset and the output device can be a haptic generator.

[0012] The instructions may be executable to expand a frequency of the at least one CSA before presenting the at least one CSA.

[0013] In some examples, the at least one CSA is a first CSA, and the instructions may be executable to receive at least a second CSA mixed with audio associated with the computer simulation, downshift at least one frequency of the first CSA from at least one first ultrasonic frequency bandwidth, downshift at least one frequency of the second CSA from at least one second ultrasonic frequency bandwidth, and present the first CSA and the second CSA.

[0014] In another aspect, a method includes mixing audio associated with at least one computer simulation with an ultrasonic frequency representing at least one first computer simulation asset (CSA) associated with the at least one computer simulation. The method includes transmitting the audio and the first CSA, receiving the audio and the first CSA, and playing the audio on at least one speaker. The method further includes downshifting the ultrasonic frequency and presenting the first CSA on at least one output device after downshifting.

[0015] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram of an exemplary system in accordance with present principles; [Figure 2]1 illustrates an exemplary specific system consistent with the present principles. [Figure 3] 1 illustrates an exemplary transmitter assembly consistent with present principles. [Figure 4] 1 illustrates an exemplary transmitter assembly consistent with present principles. [Figure 5] 1 illustrates a first exemplary transmitter logic in exemplary flow chart form. [Figure 6] Signal processing consistent with Figure 5 is shown. [Figure 7] 1 illustrates a first exemplary receiver logic in exemplary flow chart form. [Figure 8] Signal processing consistent with Figure 6 is shown. [Figure 9] 10 illustrates a second exemplary transmitter logic in exemplary flow chart form. [Figure 10] Signal processing consistent with FIG. 9 is shown. [Figure 11] 10 illustrates a second exemplary receiver logic in exemplary flow chart form. [Figure 12] The signal processing is shown in accordance with FIG. DETAILED DESCRIPTION OF THE INVENTION

[0017] The present disclosure generally relates to computer ecosystems, including, but not limited to, aspects of consumer electronics (CE) device networks, such as computer gaming networks. The systems herein may include server and client components that may be connected via a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices, including game consoles such as Sony PlayStation®, or game consoles made by Microsoft® or Nintendo®, or other manufacturers; extended reality (XR) headsets, such as virtual reality (VR) headsets and augmented reality (AR) headsets; portable televisions (e.g., smart TVs, Internet-enabled televisions); portable computers, such as laptop computers and tablet computers; and smartphones and other mobile devices, including additional examples described below. These client devices may operate in a variety of operating environments. For example, some of the client computers may use, as examples, the Linux® operating system, an operating system manufactured by Microsoft®, or a Unix® operating system, or an operating system manufactured by Apple, Inc.®, or Google®, or the Berkeley Software Distribution or Berkeley Standard Distribution (BSD) OS (including derivatives of BSD). These operating environments may be used to run one or more browsing programs, such as browsers made by Microsoft®, Google®, or Mozilla®, or other browser programs capable of accessing websites hosted by the Internet servers described below. An operating environment according to present principles may also be used to run one or more computer game programs.

[0018] Servers and / or gateways may be used, which may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, clients and servers may be connected via a local intranet or virtual private network. The server or controller may be instantiated by a game console such as a Sony PlayStation®, a personal computer, or the like.

[0019] Information may be exchanged between the client and the server over a network. To this end, and for security, the server and / or client may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method for providing a secure community, such as an online social website or gamer network, to network members.

[0020] The processor may be a single-chip processor or a multi-chip processor capable of performing logic through various lines, such as address lines, data lines, and control lines, as well as registers and shift registers. A processor, including a digital signal processor (DSP), may be an embodiment of a circuit.

[0021] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or depicted in the figures may be combined, substituted, or excluded from other embodiments.

[0022] "A system having at least one of A, B, and C" (and similarly, "a system having at least one of A, B, or C" and "a system having at least one of A, B, and C") includes a system having only A, a system having only B, a system having only C, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having both A, B, and C.

[0023] Referring now to FIG. 1 , an exemplary system 10 is shown. System 10 can include one or more of the exemplary devices in accordance with the present principles, as referenced above and further described below. A first of the exemplary devices included in system 10 is a consumer electronics (CE) device, such as an audio-video device (AVD) 12, including, but not limited to, a theater display system, which may be projector-based, or an Internet-enabled television with a TV tuner (or, equivalently, a set-top box that controls a TV). Alternatively, AVD 12 can also be a computerized Internet-enabled (“smart”) phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and / or a headset (such as smart glasses or a VR headset), another wearable computerized device, a computerized Internet-enabled music player, computerized Internet-enabled headphones, a computerized Internet-enabled implantable device (such as an implantable skin device, etc.). In any event, it will be understood that AVD12 is configured to implement the present principles (e.g., to communicate with other CE devices to implement the present principles, to execute the logic described herein, and to perform any other functions and / or operations described herein).

[0024] Thus, to implement such principles, AVD 12 may be established by some or all of the components shown. For example, AVD 12 may include one or more touch-enabled displays 14, which may be implemented by high-resolution or ultra-high-resolution "4K" or higher flat screens. Touch-enabled display(s) 14 may include, for example, a capacitive or resistive touch-sensing layer with a grid of electrodes for touch sensing consistent with the present principles.

[0025] The AVD 12 may also include one or more speakers 16 for outputting audio in accordance with the present principles and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to the AVD 12 to control it. The example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, or a LAN, under the control of one or more processors 24. Thus, the interface 20 may be a Wi-Fi® transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It will be understood that the processor 24 controls the AVD 12 to implement the present principles, including controlling the display 14 to present images thereon, receiving input from the display 14, and other elements of the AVD 12 described herein. Furthermore, it should be noted that the network interface 20 may be a wired or wireless modem or router, or a wireless telephony transceiver, or other suitable interface, such as the Wi-Fi® transceiver described above.

[0026] In addition to the foregoing, AVD 12 may also include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI) port or a Universal Serial Bus (USB) port for physically connecting to another CE device, and / or a headphone port for connecting headphones to AVD 12 to present audio from AVD 12 to a user via headphones. For example, input port 26 may be wired or wirelessly connected to a cable or satellite source 26a of audio-video content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. If implemented as a game console, source 26a may include some or all of the components described below in connection with CE device 48.

[0027] AVD 12 may further include one or more computer memory / computer-readable storage media 28, such as disk-based or solid-state storage, which may be embodied within the AVD's chassis as a standalone device, or as a personal video recording device (PVR) or video disc player either internal or external to the AVD's chassis for playing AV programs, or as a removable storage medium or a server as described below. In some embodiments, AVD 12 may also include a location or position receiver, such as, but not limited to, a cellular telephone receiver, a GPS receiver, and / or an altimeter 30 configured to receive geographic location information from a satellite or cellular tower and provide that information to processor 24 and / or configured in cooperation with processor 24 to determine the altitude at which AVD 12 is located.

[0028] Continuing with the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be a thermal imaging camera, a digital camera such as a webcam, an IR sensor, an event-based sensor, and / or a camera integrated into AVD 12 and controllable by processor 24 to collect pictures / images and / or video in accordance with the present principles. AVD 12 may also include a Bluetooth transceiver 34 and other NFC elements 36 for communicating with other devices using Bluetooth and / or near field communication (NFC) technology, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0029] Furthermore, AVD 12 may include one or more auxiliary sensors 38 that provide input to processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors forming a layer of touch-enabled display 14 itself, and may be, without limitation, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc. Examples of other sensors include pressure sensors, motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, and gesture sensors (e.g., sensors for sensing gesture commands). Thus, sensors 38 may be implemented by an inertial measurement unit (IMU), which typically includes one or more motion sensors such as individual accelerometers, gyroscopes, and magnetometers, and / or a combination of accelerometers, gyroscopes, and magnetometers, or by an event-based sensor such as an event detection sensor (EDS), to determine the position and orientation of AVD 12 in three dimensions. An EDS consistent with the present disclosure provides an output indicative of a change in light intensity sensed by at least one pixel of the light-sensing array. For example, if the light sensed by the pixel is decreasing, the output of the EDS may be −1. If it is increasing, the output of the EDS may be +1. No change in light intensity below a certain threshold may be indicated by an output binary signal of 0.

[0030] The AVD 12 may also include a wireless TV broadcast port 40 for receiving over-the-air TV broadcasts, which provides input to the processor 24. Note that, in addition to the foregoing, the AVD 12 may also include an infrared (IR) transmitter and / or receiver and / or transceiver 42, such as an Infrared Data Association (IRDA) device. The AVD 12 may be provided with a battery (not shown) for powering itself, which may be a kinetic energy harvester capable of converting kinetic energy into electrical power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field-programmable gate array 46 may also be included. One or more haptic / vibration generators 47 may be provided for generating haptic signals that can be sensed by a person holding or interacting with the device. Thus, the haptic generator 47 may vibrate all or part of the AVD 12 using an electric motor connected to an off-center and / or unbalanced weight via the motor's rotating shaft, the shaft rotating under the control of the motor (which may be controlled by a processor such as processor 24) to create simulations of vibrations of various frequencies and / or amplitudes, and forces in various directions.

[0031] A light source such as an infrared (IR) projector may also be included.

[0032] In addition to the AVD 12, the system 10 may include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to transmit computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or via a server, as described below, while the second CE device 50 may include similar components to the first CE device 48. In the illustrated example, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. The HMD may include a head-up transparent display or a head-up opaque display that presents AR / MR content or VR content (more generally, extended reality (XR) content), respectively. The HMD may be configured as a glasses-type display or as a large VR-type display sold by a computer game console manufacturer.

[0033] In the illustrated example, only two CE devices are shown, and it will be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown for AVD 12. Any of the components shown in the figures below may incorporate some or all of the components shown in the AVD 12 example.

[0034] Referring now to the aforementioned at least one server 52, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as disk-based or solid-state storage, and at least one network interface 58 that, under the control of the server processor 54, enables communication with the other illustrated devices over the network 22 and, indeed, may facilitate communication between the server and client devices in accordance with the present principles. It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi® transceiver, or other suitable interface, such as, for example, a wireless telephony transceiver.

[0035] Thus, in some embodiments, server 52 may be an entire Internet server or server "farm" and may include or perform "cloud" functionality such that, in an exemplary embodiment, for example, for a network gaming application, devices of system 10 may access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown.

[0036] The components shown in the following figures may include some or all of the components shown herein. Any user interfaces (UIs) described herein may be integrated and / or extended, and UI elements may be mixed and matched between UIs.

[0037] The present principles may use a variety of machine learning models, including deep learning models. Machine learning models consistent with the present principles may use a variety of algorithms trained using methods including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms may be implemented by computer circuitry, including one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN known as a long-short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks may also be considered examples of machine learning models. In addition to the types of networks listed above, the models herein may also be implemented by classifiers.

[0038] As understood herein, performing machine learning may include accessing and training a model on training data to enable the model to process additional data and make inferences. As a result, an artificial neural network / artificial intelligence model trained through machine learning may include a weighted input layer, an output layer, and multiple hidden layers in between, configured to make inferences about a suitable output.

[0039] Please refer now to Figure 2. A computer simulation, such as a computer game, may be transmitted from a computer game console 200 or a computer game server 202 to a display device 204, such as a television, for presenting the computer simulation under the control of one or more computer simulation controllers 206, such as, but not limited to, a PlayStation® controller or other controller.

[0040] One or more haptic generators 208 may be provided on the controller 206, which may be manipulated by the player to control the presentation of the computer simulation. Audio sourced from the game console 200 or the server 202 is played on one or more speakers 210 of a speaker system. Elements of the system shown in Figure 2 may incorporate some or all of the suitable devices and components described above with reference to Figure 1. One or more of the components of Figure 2 may include one or more microphones 212.

[0041] An embodiment of an exemplary Transmitter assembly is shown in Figure 3. The Transmitter assembly may be part of, for example, a server, a game console, or any other suitable component described herein.

[0042] A computer simulation 300, such as a computer game, provides simulated audio 302 in the audible frequency range (typically below 12 Khz) and one or more computer game assets (abbreviated CGA), such as one or more haptic assets 304 and one or more audio assets 306. Audio assets may be computer objects that represent sound effects, such as special effects, associated with the game, while haptic assets are typically objects that represent haptic effects associated with the game.

[0043] 3, assets 304, 306 are provided to a processor, such as a DSP 308, for manipulation as described further herein, and the manipulated assets output by the DSP 308 are mixed or aggregated with game audio 302 in a mixer 310. The output of the mixer 310 is sent to a transmitter 312 for wired and / or wireless transmission to a receiver assembly, which is typically part of a player component. Although shown as a separate component from the DSP 308, the mixer 310 may, in some embodiments, be implemented by the DSP 308.

[0044] 4 shows an example of such a receiver assembly, which may be implemented by any suitable device herein. The receiver 400 receives a signal from the transmitter 312. The receiver 400 passes the signal to a receiver DSP 402, which may access or implement one or more low-pass filters 404, and outputs the audio 302 portion of the signal (shown in FIG. 4 as audio 406) for audio playback on one or more speakers 410.

[0045] The DSP 402 also has access to or includes one or more bandpass filters 412 to process portions of the signal from the transmitter 312 representing the CGAs 304, 306, labeled 414, 416, respectively, in Figure 4. The haptic assets 414 are presented to one or more haptic generators, such as any of the haptic generators described herein, while the audio assets 416 are played on one or more speakers 420.

[0046] Figure 5 illustrates a first embodiment of logic that may be executed by the transmitter shown in Figure 3, and Figure 6 illustrates accompanying signal waveforms associated with the logic of Figure 5. Beginning at block 500, game audio 302 is received from the simulation 300. Additional CGA 304, 306 is received at block 502.

[0047] Proceeding to block 504, the frequency of the signal representing the CGA is shifted into the ultrasonic range. Typically, this requires an upward shift in frequency. In one embodiment, the ultrasonic range is approximately 12 Khz to 24 Khz, plus or minus 5 percent, or plus or minus 10 percent, or plus or minus 20 percent.

[0048] Proceeding to block 506, the upshifted (in frequency) CGA is mixed with the game audio received in block 500. Proceeding to block 508, the resulting waveform superposition is transmitted to an output device using the receiver assembly of FIG.

[0049] When multiple CGAs are mixed with the game audio 302, the ultrasonic frequency can be subdivided into smaller bands, each representing an asset channel. For example, if two CGAs are transmitted, the first CGA can have a frequency shifted to the 12KHz to 18Khz band, and the second CGA can have a frequency shifted to the 18KHz to 24Khz band.

[0050] Figure 6 shows in the top panel game audio 302 in the audible frequency range below the ultrasonic frequency range, while the middle panel shows multiple audio assets 306 originally having frequencies in the audible range (e.g., 0 KHz to 12 Khz) being shifted to respective bands within the ultrasonic range in the bottom panel of Figure 6 and mixed with the game audio 302 in the audible range.

[0051] FIG. 7 illustrates a first embodiment of exemplary logic that may be executed by the receiver assembly of FIG. 4. A signal from the transmitter assembly shown in FIG. 3 is received via a wireless and / or wired path in block 700. Moving to block 702, the LPF 404 is used by the DSP 402 to extract gameplay audio 302 in the audible portion of the signal received from the transmitter. Moving to block 704, the DSP 402 uses the bandpass filter 412 to extract CGA from the ultrasonic portion of the signal received from the transmitter, and the frequency of the extracted CGA is downshifted to the original audible range in block 706 as a source from the simulation 300. In block 708, the CGA at its original frequency is played by one or more haptic generators in the case of haptic assets 304 or one or more speakers in the case of audio assets 306.

[0052] Figure 8 shows, in the top panel, gameplay audio 302 being received in the audible range and CGA 306 being received in the ultrasonic range of the signal received from the transmitter assembly. The second panel shows the use of LPF 404 to separate the gameplay audio from the audible portion of the signal, and the third panel of Figure 8 shows the use of bandpass filter 412 to separate an ultrasonic version of a first CGA from a first portion of the ultrasonic band. The bottom panel of Figure 8 shows the use of bandpass filter 412 to separate an ultrasonic version of a second CGA from a second portion of the ultrasonic band.

[0053] Figure 9 illustrates a second embodiment of logic that may be executed by the transmitter shown in Figure 3, and Figure 10 illustrates accompanying signal waveforms associated with the logic of Figure 9. Starting at block 900, game audio 302 is received from the simulation 300. Additional CGA 304, 306 is received at block 902.

[0054] Proceeding to block 904, each ultrasound asset channel may be companded, which is a process in which its full bandwidth is compressed to the bandwidth of the subdivided ultrasound segment range.

[0055] Proceeding to block 906, the frequency of the signal representing the CGA is shifted into the ultrasonic region. Typically, this requires an upward frequency shift. If multiple CGAs are being mixed with the game audio 302, the ultrasonic frequencies are shifted to their respective compound channels. For example, one compound channel may range from 12 Khz to 14 Khz, a second compound channel may range from 14 Khz to 16 Khz, a third compound channel may range from 16 Khz to 18 Khz, and so on. Note that the compound channels need not have the same bandwidth as each other. The compound channels may have identical bandwidths other than 2 Khz; for example, each compound channel may have a bandwidth of 3 Khz. Generally, the ultrasonic frequency band is divided into a number of compound channels equal to or approximately equal to the number of CGAs being transmitted.

[0056] Proceeding to block 908, the upshifted (in frequency) CGA is mixed with the game audio received in block 500. Proceeding to block 910, the resulting waveform superposition is transmitted to an output device using the receiver assembly of FIG.

[0057] Figure 10 shows in the top panel game audio 302 in the audible frequency range below the ultrasonic frequency range, and the second panel shows four audio assets 306 with original frequencies in the audible range (e.g., 0 KHz to 12 KHz) assigned to respective frequency bands that are compounded as shown in the third panel. The bottom panel of Figure 10 shows the compound channels shifted to their respective compound channels in the ultrasonic domain.

[0058] Figure 11 illustrates a second embodiment of exemplary logic that may be executed by the receiver assembly of Figure 4. A signal from the transmitter assembly shown in Figure 3 is received via a wireless and / or wired path at block 1100. Moving to block 1102, the LPF 404 is used by the DSP 402 to extract gameplay audio 302 in the audible portion of the signal received from the transmitter. Moving to block 1104, the bandpass filter 412 is used by the SP 402 to extract CGA from the ultrasonic portion of the signal received from the transmitter.

[0059] Continuing to block 1106, the frequencies in the ultrasound domain representing the extracted CGA are expanded to their full bandwidth. In block 1108, the expanded channels are shifted down to the original audible range as sources from the simulation 300. In block 1110, the CGA at the original frequencies is played back by one or more haptic generators in the case of haptic assets 304 or one or more speakers in the case of audio assets 306.

[0060] Figure 12 shows, in the top panel, gameplay audio 302 being received in the audible range and CGA 306 being received in the ultrasonic range of the signal received from the transmitter assembly. The second panel shows the use of LPF 404 to separate the gameplay audio from the audible portion of the signal, and the third through sixth panels of Figure 12 show the use of bandpass filters 412 to separate the ultrasonic versions of each CGA from their respective portions of the ultrasonic band, expanding the channels and shifting the channels down to their original frequencies for playback, respectively.

[0061] Thus, on the output device, the DSP applies multiple ultrasonic bandpass filters to extract each asset channel, expand it to restore its original bandwidth size, and frequency shift it back into the original audible frequency range.

[0062] The present principles can be applied to reduce the bandwidth requirements of multiple wireless audio channels to support game and haptic audio on computer game controllers such as the DualSense controller. The haptic audio channel can instead be frequency shifted into the ultrasonic range, mixed with regular game audio, transmitted on the game audio wireless channel, and then extracted and shifted back to the original audio frequency upon reception at the controller. This frees up wireless bandwidth for other uses, or for use in new peripheral devices for the computer game console.

[0063] Although particular embodiments have been shown and described in detail herein, it should be understood that the subject matter encompassed by the present invention is limited only by the scope of the claims.

Claims

1. 1. An apparatus including at least one processor, the at least one processor comprising: Identifying at least one computer simulation asset (CSA) associated with the at least one computer simulation; shifting at least one frequency of the CSA to at least one ultrasonic frequency bandwidth; mixing the CSA with audio associated with the computer simulation; The device configured to transmit the CSA along with audio associated with the computer simulation for presentation by at least one receiver assembly.

2. The apparatus of claim 1 , wherein the at least one CSA includes an audio asset.

3. The device of claim 1 , wherein the at least one CSA includes a haptic asset.

4. The processor: The apparatus of claim 1 , configured to frequency compand the at least one CSA before transmitting the at least one CSA.

5. the at least one CSA includes a first CSA; the processor: Identifying at least one second CSA; shifting the at least one frequency of the first CSA by at least a first ultrasonic frequency bandwidth; shifting at least one frequency of the second CSA to at least a second ultrasonic frequency bandwidth; 2. The apparatus of claim 1, configured to mix the first and second CSAs with the audio associated with the computer simulation before transmitting the CSAs and the audio associated with the computer simulation.

6. 10. The apparatus of claim 1, wherein the at least one ultrasonic frequency bandwidth comprises at least one bandwidth in the range of 12 to 24 kHz.

7. 1. A device including at least one computer storage device containing instructions executable by at least one processor, said instructions being other than transitory signals, said instructions comprising: receiving at least one computer simulation asset (CSA) associated with at least one computer simulation, the at least one CSA being mixed with audio associated with the computer simulation; extracting the audio associated with the computer simulation; playing the audio associated with the computer simulation on at least one speaker; downshifting the frequency of at least one of the CSAs; The device, after downshifting the at least one frequency of the CSA, presents the CSA to at least one output device.

8. The apparatus of claim 7 , wherein the at least one CSA includes an audio asset and the output device includes at least one speaker.

9. The apparatus of claim 7 , wherein the at least one CSA includes a haptic asset and the output device includes at least one haptic generator.

10. The apparatus of claim 7 , wherein the instructions are executable to expand a frequency of the at least one CSA before presenting the at least one CSA.

11. The at least one CSA includes a first CSA, and the instructions include: receiving at least a second CSA mixed with the audio associated with the computer simulation; downshifting the at least one frequency of the first CSA from at least one first ultrasonic frequency bandwidth; downshifting the at least one frequency of the second CSA from at least one second ultrasonic frequency bandwidth; executable to present the first CSA and the second CSA; 8. The apparatus of claim 7.

12. 8. The apparatus of claim 7, wherein the at least one frequency of the CSA comprises at least one bandwidth in the range of 12 to 24 kHz.

13. The device of claim 12 comprising the at least one processor.

14. mixing audio associated with at least one computer simulation with ultrasonic frequencies representative of at least a first computer simulation asset (CSA) associated with the at least one computer simulation; transmitting the audio and the first CSA; receiving the audio and the first CSA; playing the audio on at least one speaker; downshifting the ultrasonic frequency; presenting the first CSA to at least one output device after a downshift; A method comprising:

15. The method of claim 14 , wherein the first CSA includes an audio asset and the output device includes at least one speaker.

16. The method of claim 14 , wherein the first CSA includes a haptic asset and the output device includes at least one haptic generator.

17. The method of claim 14 , comprising companding a frequency associated with the first CSA.

18. 15. The method of claim 14, comprising extending a frequency of the first CSA before presenting the first CSA.

19. receiving at least a second CSA mixed with the audio associated with the computer simulation; downshifting the ultrasonic frequency of the first CSA from at least a first ultrasonic frequency bandwidth; downshifting at least one frequency of the second CSA from at least a second ultrasonic frequency bandwidth; submitting the first CSA and the second CSA; 15. The method of claim 14, comprising:

20. 15. The method of claim 14, wherein the ultrasonic frequencies include at least one bandwidth in the range of 12 to 24 kHz.

Images