Multi-track container for sound effects rendering

Abstract

Computer game sound effects (SFX) are provided to customers in a container that includes not only the audio files (such as .wav files) of the SFX, but also information underlying the SFX, so that the customer can change the SFX at runtime or offline without having to purchase new sounds or return to the sound provider to request that they make the desired changes.

Description

Technical Field

[0001] This application relates generally to sound effects ("SX") rendering in computer simulations and other applications. Background Art

[0002] Sound effects (SFX) are widely used in computer simulations (such as computer games) to provide realistic audio. SFX are not dialogue, and in some cases are not music, but rather artificially created sounds used to emphasize or enhance the game content. "Artificially created" includes SFX created using synthesizers and other devices, as well as recorded materials created in an "organic" way, such as sounds from nature, sounds from Foley sessions, etc. In certain cases, SFX may only include artificially created sounds and may exclude music.

[0003] Typically, game designers purchase the sound effects they wish to use from suppliers and then use SFX in their games as appropriate. SFX can also be used in any product or experience that includes interactive elements, including location-based entertainment, interactive movies, augmented reality (AR) / virtual reality (VR) and other extended reality (XR) experiences, in addition to computer simulations, as well as applications in industrial sectors including the automotive sector. Summary of the Invention

[0004] As understood herein, game designers and other designers of applications that utilize SFX cannot modify the inner workings of the SFX they purchase without either purchasing entirely new SFX or requesting that the vendor modify the SFX in the manner desired by the game designer. Therefore, the present principles provide a "multi-track container" to a customer, such as a game designer, as a digital audio workstation (DAW) session that lists the various tracks required to create a sound, as well as other information implicit in the sound. In other words, the multi-track container contains the "recipe" for each sound.

[0005] In this way, customers can modify SFX without having to purchase new speakers or return to the speaker vendor to request the desired changes, which can require a lot of back-and-forth. Instead, game designers can exercise greater flexibility in managing changes to the speakers themselves. Thus, by providing not only the audio files (such as .wav files) of the SFX but also all the underlying information via a DAW as a single product, game designers are given greater flexibility in game development.

[0006] It should be noted that, in addition, multi-track containers can be imported into SFX authoring tools / rendering engines (such as Wave Works Interactive Sound Engine (Wwise) from Audiokinetic), which can also be provided to game designers for a complete solution, allowing game designers to modify SFX "in the background".

[0007] Thus, in one aspect, an apparatus includes at least one processor programmed with instructions for: accessing at least a first sound effect (SFX) WAV file having at least a first audio track and a second audio track and provided by a provider of the SFX. The instructions are executable to: bundle the first SFX file with track modification information regarding the first and second audio tracks to create a multi-track container. The track modification information is used to modify the SFX file. The instructions are executable to: transmit the multi-track container to at least one computer of at least a first developer of a computer application so that the first developer can modify the first SFX file in a first manner for rendering in a first computer application developed by the first developer.

[0008] In some embodiments, the instructions are executable to: transmit the multi-track container to at least a second developer of a computer application to enable the second developer to modify the first SFX file in a second manner different from the first manner.

[0009] In a non-limiting example, the processor is implemented in a computer of the supplier of the SFX.

[0010] If desired, the computer of the first developer may be programmed with instructions for rendering the first SFX in the first computer application. The computer of the first developer may be programmed with instructions for modifying the first SFX using the track modification information while the first computer simulates runtime, and / or modifying the first SFX using the track modification information offline from the runtime of the first computer simulation. It should be noted that when providing offline modifications (which may be considered the production phase), all content is delivered and authored at the highest possible resolution (typically in pulse code modulation format (.wav)), whereas when delivered at runtime (where memory may be a limited resource at runtime), most SFX may be compressed using one of the codecs (CODECs) supported by the sound engine or platform's operating system (OS), such as Vorbis, Opus, ADPCM, ATRAC9 (PlayStation-specific CODEC), XMA (Xbox-specific CODEC), and AAC (Apple-specific).

[0011] The instructions may be executable to: transfer an SFX rendering engine to the computer of the first developer. The SFX rendering engine may include an SFX authoring tool.

[0012] In an exemplary implementation, the track modification information may include corresponding first track modification information and second track modification information for the first track and the second track, allowing the first developer to modify the first track separately from the second track. For example, the first track modification information may include one or more of the following: delay, volume, attenuation, compression, distortion, reverb, sound field positioning, filter, time stretching, and pitch shifting. The SFX file may include three or more tracks.

[0013] In another aspect, a method includes creating a multi-track container having a first multi-track sound effects (SFX) file and information for at least some of a plurality of tracks in the first multi-track SFX file, wherein track modification information is used to modify the corresponding tracks. The method includes providing the multi-track container to at least a first developer of a first computer application to allow the first developer to use the track modification information in a first manner to modify the first multi-track SFX file for rendering in the first computer application.

[0014] In another aspect, a game developer computer includes at least one processor configured with instructions executable to: receive at least a first data structure comprising: at least a first sound effects (SFX) file having at least a first audio track and a second audio track from a sound effects (SFX) provider, and audio track modification information about the first audio track and the second audio track, the instructions executable to: modify the first SFX file using the audio track modification information; and render the SFX file in a first computer application developed by a first developer.

[0015] The details of the present application, both as to its structure and operation, may be best understood with reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram of an exemplary system including an example according to the principles of the present invention;

[0017] Figure 2 is a block diagram of an example computer sound effects provisioning system;

[0018] Figure 3 An exemplary overall logic consistent with the principles of the invention is shown in an exemplary flow chart format;

[0019] Figures 4 to 6 is a screenshot showing the SFX user interface (UI);

[0020] Figure 7 Shows rendering of four tracks of an exemplary SFX into a single track;

[0021] Figure 8 shows rendering of four tracks of an exemplary SFX into multiple tracks;

[0022] Figure 9 shows multiple variations of the same SFX that can be produced by multiple game designers; and

[0023] Figure 10 Exemplary runtime rendering logic consistent with the principles of the invention is shown in exemplary flow chart format. DETAILED DESCRIPTION

[0024] The present disclosure generally relates to computer ecosystems, including various aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. The systems herein may include server and client components that may be connected via a network so that data may be exchanged between the client and server components. The client components may include one or more computing devices, including gaming consoles (such as Sony or game consoles manufactured by Microsoft or Nintendo or other manufacturers), virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart TVs, internet-enabled TVs), portable computers (such as laptops and tablets), and other mobile devices including smartphones and additional examples discussed below. These client devices can operate in a variety of operating environments. For example, some client computers may employ, for example, a Linux operating system, an operating system from Microsoft, or a Unix operating system, or an operating system produced by Apple or Google. These operating environments can be used to execute one or more browsing programs, such as browsers manufactured by Microsoft or Google or Mozilla, or other browser programs that can access websites hosted by internet servers discussed below. Moreover, an operating environment according to the principles of the present invention can be used to execute one or more computer game programs.

[0025] The server and / or gateway may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may be connected via a local intranet or a virtual private network. The server or controller may be powered by a game console such as a Sony ), personal computers, etc.

[0026] Information can be exchanged between the client and the server over the network. To this end, and for security reasons, the server and / or the client may include firewalls, load balancers, temporary storage devices and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method for providing a secure community (such as an online social networking site) to network members.

[0027] The processor may be a single chip or multi-chip processor that can execute logic through various lines such as address lines, data lines, and control lines, as well as registers and shift registers.

[0028] The components included in one embodiment can be used in any appropriate combination in other embodiments. For example, any of the various components described herein and / or depicted in the figures can be combined, interchanged, or excluded from other embodiments.

[0029] "A system having at least one of A, B, and C" (similarly, "a system having at least one of A, B, or C" and "a system having at least one of A, B, C") includes the following systems: having only A; having only B; having only C; having A and B at the same time; having A and C at the same time; having B and C at the same time; and / or having A, B, and C at the same time, etc.

[0030] Now specifically refer to Figure 1 , an exemplary system 10 is shown that may include one or more of the exemplary devices mentioned above and further described below in accordance with the principles of the present invention. A first of the exemplary devices included in system 10 is a consumer electronics (CE) device, such as an audio video device (AVD) 12, such as, but not limited to, an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). Alternatively, AVD 12 may also be a computerized Internet-enabled ("smart") phone, a tablet computer, a laptop computer, an HMD, a wearable computerized device, a computerized Internet-enabled music player, a computerized Internet-enabled headset, a computerized Internet-enabled implantable device (such as an implantable skin device), etc. In any event, it should be understood that AVD 12 is configured to employ the principles of the present invention (e.g., communicate with other CE devices to employ the principles of the present invention, execute the logic described herein, and perform any other functions and / or operations described herein).

[0031] Therefore, in order to adopt such a principle, the AVD 12 can be composed of Figure 1. For example, the AVD 12 may include one or more displays 14, which may be implemented as flat screens of high definition or ultra high definition ("4K") or higher, and may be touch-enabled for receiving user input signals via touch on the display. The AVD 12 may include one or more speakers 16 for outputting audio in accordance with the principles of the present invention, and at least one additional input device 18 (such as an audio receiver / microphone) for inputting audible commands to the AVD 12 to control the AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (such as the Internet, a WAN, a LAN, etc.) under the control of one or more processors 24. A graphics processor 24A may also be included. Thus, the interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that the processor 24 controls the AVD 12 to employ the principles of the present invention, including other elements of the AVD 12 described herein, such as controlling the display 14 to present images on the display and receive input from the display. In addition, it should be noted that the network interface 20 can be a wired or wireless modem or router or other suitable interface (such as a wireless telephone transceiver, or a Wi-Fi transceiver as mentioned above, etc.).

[0032] In addition to the foregoing, the AVD 12 may also include one or more input ports 26, such as a High-Definition Multimedia Interface (HDMI) port or a USB port for physically connecting to another CE device and / or a headphone port for connecting headphones to the AVD 12 so that audio from the AVD 12 is presented to the user via the headphones. For example, the input port 26 may be connected via wire or wirelessly to a wired or satellite source 26a of audio and video content. Thus, the source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, the source 26a may be a game console or disc player containing content. When implemented as a game console, the source 26a may include some or all of the components described below with respect to the CE device 44.

[0033] The AVD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices, which in some cases are embodied as separate devices within the AVD's housing, or as a personal video recorder (PVR) or video disk player for playing back AV programs inside or outside the AVD's housing, or as removable storage media. Furthermore, in some embodiments, the AVD 12 may include a position or location receiver (such as, but not limited to, a cell phone receiver, a GPS receiver, and / or an altimeter 30) configured to receive geographic location information from satellites or cell phone towers and provide the information to the processor 24 and / or determine, in conjunction with the processor 24, the altitude at which the AVD 12 is located. Component 30 may also be implemented by an inertial measurement unit (IMU), which typically includes a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions.

[0034] Continuing with the description of the AVD 12, in some embodiments, according to the principles of the present invention, the AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras (such as webcams), and / or cameras integrated into the AVD 12 and capable of being controlled by the processor 24 to collect pictures / images and / or videos. The AVD 12 may also include a Bluetooth transceiver 34 and other near field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technology, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0035] Furthermore, the AVD 12 may include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, gyrometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the processor 24. The AVD 12 may include a wireless TV broadcast port 38 for receiving OTA TV broadcasts that provide input to the processor 24. In addition to the foregoing, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVD 12, such as a kinetic energy harvester that can convert kinetic energy into electricity to charge the battery and / or power the AVD 12.

[0036] Still refer to Figure 1In addition to the AVD 12, the system 10 may also include one or more other CE device types. In one example, the first CE device 44 may be a computer game console that can be used to send computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or through a server described below, while the second CE device 46 may include components similar to the first CE device 44. In the example shown, the second CE device 46 may be configured as a computer game controller manipulated by a player or a head-mounted display (HMD) worn by a player 47. In the example shown, only two CE devices 44 and 46 are shown, and it should be understood that fewer or larger devices may be used. The devices herein may implement some or all of the components shown for the AVD 12. Any of the components shown in the following figures may be combined with some or all of the components shown in the context of the AVD 12.

[0037] Referring now to the aforementioned at least one server 50, it includes at least one server processor 52, at least one tangible computer-readable storage medium 54 (such as a disk-based storage device or a solid-state storage device), and at least one network interface 56 that, under the control of the server processor 52, allows communication with the server via the network 22. Figure 1 The network interface 56 may be a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as a wireless telephone transceiver, for example).

[0038] Thus, in some embodiments, the server 50 may be an Internet server or an entire server "farm" and may include and perform "cloud" functionality such that the devices of the system 10 may access the "cloud" environment via the server 50 in an exemplary embodiment, such as an online gaming application. Alternatively, the server 50 may be managed by a server that is associated with Figure 1 The other devices shown are implemented in the same room or nearby with one or more game consoles or other computers.

[0039] Figure 2 It shows that a sound effects (SFX) provider computer 200 can supply an SFX multi-track container 202 to a client computer 204 (such as a game developer's computer system), typically via a wired and / or wireless path of a wide area computer network (such as the Internet).

[0040] Multi-track container 202 includes a multi-track (two and typically more than two tracks) SFX audio file, for example, in WAV or MP3 format, and track modification information regarding the tracks. As discussed further below, the track modification information can be used to modify the SFX file. That is, a game developer operating client computer 204 can use the track modification information to modify the SFX file for rendering in a computer application (such as a computer simulation, such as a computer game developed by a first developer). Similarly, different developers can receive the same multi-track container 202 and use the track modification information to modify the same SFX file in different ways for different computer games. As discussed further below, the track modification information can be used to modify the SFX file while the computer game is running or offline from the runtime.

[0041] Figure 3 The exemplary overall logic summarized above is shown. Starting from block 300, the SFX track can be imported into a digital audio workstation (DAW) (such as Figure 2 200 ). Moving to block 302 , using a DAW, SFX files are created, each of which has multiple audio tracks. Track modification information (which may be referred to herein as “metadata”) is associated with the respective SFX files to produce a multi-track container.

[0042] Moving to block 304, a multi-track container is provided to the client, who can modify the SFX files within the container using the accompanying track modification information. If desired, an SFX rendering engine may also be provided to the client computer at block 306. The rendering engine may include an authoring tool. An example of such an engine is the WaveWorks Interactive Sound Engine (Wwise) from Audiokinetic. Using the authoring tool executed on the client computer 200, at block 308, the client game developer uses the corresponding track modification information accompanying the SFX file to make and manage changes to the SFX file as needed.

[0043] Now turn Figure 4 , which shows a screenshot that can be provided on a display 400 (such as any of the displays described herein). The screenshot shows track modification information associated with an SFX file (such as a WAV or MP3 file). As shown, the track modification information shows the multiple tracks that make up the SFX file, in the example shown, a bass track 402, an AC GTR track 404, an EG track 406, a PAD01 track 408, a Lead Vox Eng track 410, and a Back End track 412. A corresponding sound image 414 can be presented for each track, which represents the specific contribution of the track to the overall SFX file.

[0044] Furthermore, the screenshot shows that track parameters 416 can be presented for each track. These parameters include, but are not limited to, volume, delay, attenuation, compression, distortion and reverberation, sound field positioning, filters, time stretching, and pitch shifting. In short, all of the above information can be considered track modification information accompanying the SFX multi-track file, and application developers, such as game developers, can modify any of this information. Thus, game developers can modify the parameters of individual tracks, thereby modifying the individual tracks' sonic contributions to the SFX, and can delete individual tracks as needed and make other modifications to the SFX files created from them.

[0045] This is Figure 5 It is further shown in FIG, which shows that in the example shown, Wwise can be used to render, for example, Figure 2 500 of a screen shot on a client application developer computer 204 or other display disclosed herein. Figure 5 Four audio tracks 502 are shown for editing. An edit selector 504 is provided for each respective audio track to enable modification of individual audio tracks or deletion of certain audio tracks from the final SFX file, an exemplary audio diagram of which is shown in FIG. Figure 6 In the screenshot 600 , as indicated at 602 . Figure 6 It is also indicated that SFX output parameters 604 may be present, including, but not limited to, channel configuration, makeup gain, trim start, trim end, overlay WAV loop point and loop start and end times, crossfade duration and crossfade shape, marker input mode, marker detection sensitivity, fade-in duration, fade-in curve, and fade-out duration. It should be noted that, in a non-limiting example, these output parameters 604 may also be part of the track modification information for each track in the multi-track container 202.

[0046] Using any of the information discussed herein, game developers can modify multi-track SFX, such as by adding or removing high-frequency or low-frequency components of a track, to change the delay between tracks, the amplitude of any given track, the attenuation of any given track, and the filtering of any track. A track can be faded in or out or removed entirely. Compression, reverb, sound field positioning, filters, time stretching, pitch shifting, and distortion can be added to one or more tracks. Surround sound can be accommodated by designating some tracks to play on the front speakers and others on the rear speakers.

[0047] Figure 7A simplified SFX 700 is shown with four audio tracks 702 rendered into a single SFX sound 704. For example, assume the SFX 700 is for footsteps (shoe tapping). One audio track 702 may be for the sound of a shoe scraping against the road, another may be for the sound of a heel hitting the road, another may be for the sound of a toe hitting the road, and another may be for the rustling of clothing as the character walks.

[0048] Figure 8 It is shown that some audio tracks 800 can be combined to render a first layer of sound effects 802, while other audio tracks 804 can be combined to render a second layer of SFX 806. Rendering can be done offline, where the game designer can modify / mix the content to their liking before rendering to a single audio file, which is then imported into a sound engine (e.g., Wwise).

[0049] Or, as Figure 8 As shown, rendering can be done at runtime based on game calls. Essentially, a sound designer can import a multi-track session into a rendering engine (such as Wwise), then reassemble some of the tracks, adding real-time parameter controls (RTPCs) and states that are triggered by the game engine at runtime. RTPCs and states are sound designer-friendly abstractions for game engine application programming interface (API) calls.

[0050] Figure 9 A visual representation of SFX is shown, all derived from a generic multi-track container, to illustrate that different game designers can make many different modifications to the same SFX.

[0051] Figure 10 The runtime logic discussed above is shown. Beginning at block 1000, a multi-track container can be received at runtime. At block 1002, a game call driven by gameplay is received. Moving to block 1004, the called SFX files are accessed. Proceeding to block 1006, the SFX files in the container are modified using the track modification information, and the modified SFX files are rendered at block 1008.

[0052] It will be appreciated that while the principles of the invention have been described with reference to certain exemplary embodiments, these embodiments are not intended to be limiting and that various alternative arrangements may be used to implement the subject matter claimed herein.

Claims

1. A device for sound effect rendering, comprising: at least one processor programmed with instructions to: accessing at least a first sound effect SFX WAV file or MP3 file, the first SFX WAV file or MP3 file having at least a first audio track and a second audio track and provided by a supplier of SFX, wherein the first audio track and the second audio track at least partially overlap in the first SFX WAV file or MP3 file; bundling the first SFX WAV file or MP3 file with track modification information about the at least first and second audio tracks to create a multi-track container, the track modification information being used to modify the first SFX WAV file or MP3 file by modifying the first audio track in a first manner and modifying the second audio track in a second manner different from the first manner; and The multi-track container is transferred to at least one computer of at least a first developer of a computer application to enable the first developer to modify the first SFX WAV file or MP3 file in the first way for rendering in a first computer application developed by the first developer.

2. The apparatus of claim 1 , wherein the instructions are executable to: The multi-track container is transmitted to at least a second developer of a computer application to enable the second developer to modify the first SFX WAV file or MP3 file in the second manner.

3. The apparatus of claim 1, wherein the processor is implemented in a computer of a supplier of the SFX.

4. The apparatus of claim 1 , comprising: The computer of the first developer, the computer programmed with instructions for: The first SFX WAV file or MP3 file is rendered in the first computer application.

5. The apparatus of claim 4, wherein the computer of the first developer is programmed with instructions for: The first SFX WAV file or MP3 file is modified using the audio track modification information when the first computer application is running.

6. The apparatus of claim 4, wherein the computer of the first developer is programmed with instructions for: The first SFX WAV file or MP3 file is modified using the audio track modification information offline at runtime of the first computer application.

7. The apparatus of claim 1 , wherein the instructions are executable to: The SFX rendering engine is transferred to the computer of the first developer. The apparatus of claim 7 , wherein the SFX rendering engine comprises an SFX authoring tool.

9. The apparatus of claim 1, wherein the audio track modification information includes corresponding first audio track modification information and second audio track modification information for the first audio track and the second audio track, so that the first developer can modify the first audio track separately from the second audio track.

10. The apparatus of claim 9, wherein the first audio track modification information comprises at least one of the following items: delay, volume, attenuation, compression, distortion, reverberation, sound field positioning, filter, time stretching, and pitch shifting.

11. The apparatus of claim 1, wherein the first SFX WAV file or MP3 file includes at least a third audio track.

12. A method for sound effect rendering, comprising: creating a multi-track container, the multi-track container comprising a first multi-track sound effect (SFX) file and track modification information for at least some of a plurality of tracks in the first multi-track SFX file, the track modification information being used to modify the corresponding tracks by modifying the first track in a first manner and modifying the second track in a second manner different from the first manner, wherein the first track and the second track at least partially overlap in the first multi-track SFX file; as well as The multi-track container is provided to at least a first developer of a first computer game to allow the first developer to use the track modification information in the first manner to modify the first multi-track SFX file for rendering in the first computer game.

13. The method of claim 12, comprising: The multi-track container is provided to at least a second developer of a second computer game to allow the second developer to modify the first multi-track SFX file using the track modification information in the second manner for rendering in the second computer game.

14. The method of claim 12, comprising: The SFX rendering engine is transferred to the first developer. The method of claim 14 , wherein the SFX rendering engine comprises an SFX authoring tool.

16. The method of claim 12, wherein the track modification information includes corresponding first track modification information and second track modification information for at least a first track and a second track of the first multi-track SFX file, so that the first developer can modify the first track separately from the second track.

17. The method of claim 16, wherein the first audio track modification information comprises at least one of the following items: delay, volume, attenuation, compression, distortion, reverberation, sound field positioning, filter, time expansion, and pitch shift.

18. A game developer computer comprising: At least one processor configured with instructions executable to: receiving at least a first data structure, the at least first data structure comprising: at least a first SFX file having at least a first audio track and a second audio track from a provider of sound effects (SFX), and audio track modification information regarding the at least first audio track and the second audio track, wherein the first audio track and the second audio track at least partially overlap in the first SFX file; modifying the first SFX file using the audio track modification information by modifying the first audio track in a first manner and modifying the second audio track in a second manner different from the first manner; and The first SFX file is rendered in a first computer simulation developed by a first developer.

19. The game developer computer of claim 18, wherein the instructions are executable to modify and render the first SFX file while the first computer simulation is running.

20. The game developer computer of claim 18, wherein the instructions are executable to modify the first SFX file offline and then render the first SFX file while the first computer simulation is running.

Images