Audio processing method, device, equipment, medium and product

Abstract

The present application provides an audio processing method, device, equipment, medium and product, the method comprising: obtaining multi-channel media audio in a media playback scene, the multi-channel media audio comprising an audio signal collected through N channels; obtaining the target spatial position of a media viewing object in the media playback scene, and arranging a stereo speaker system around the target spatial position, the stereo speaker system comprising N virtual speakers, the layout position of each virtual speaker in the media playback scene being adapted to the spatial position of a channel in the media playback scene; using a stereo speaker system to perform multi-channel audio rendering processing on the multi-channel media audio, each virtual speaker performing audio rendering processing on the audio signal collected by the adapted channel respectively; playing the multi-channel media audio through the stereo speaker system. The present application can achieve the rendering effect of multi-channel spatial audio to improve the immersive experience in the media playback scene.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an audio processing method, apparatus, device, medium and product, and specifically to an audio processing method, an audio processing apparatus, a computer device, a computer-readable storage medium and a computer program product. Background Art

[0002] In the field of audio processing, spatial audio refers to the ability to use the directionality of sound to create an immersive experience, such as immersive movie and music viewing. In media playback devices, spatial audio components (such as speakers) can render and play audio resources.

[0003] Currently, if the format of the audio resource to be played is mono, only mono sound can be played; if the format of the audio resource to be played is stereo or multi-channel, all its channels will be mixed into a single channel before spatialization. In summary, the spatial audio component of the media playback device is monophonic. Since there is only one channel, that is, the sound comes from only one direction, the user experience is poor. Summary of the invention

[0004] The embodiments of the present application provide an audio processing method, apparatus, device, medium and product, which can achieve the rendering effect of multi-channel spatial audio to enhance the immersive experience in the media playback scene.

[0005] On the one hand, an embodiment of the present application provides an audio processing method, the method comprising:

[0006] Obtain multi-channel media audio in a media playback scene, where the multi-channel media audio includes audio signals collected through N channels, where each channel corresponds to a different spatial position in the media playback scene; N is an integer greater than 1;

[0007] Obtaining a target spatial position of a media viewing object in a media playback scene, and arranging a stereo speaker system around the target spatial position in the media playback scene; the stereo speaker system includes N virtual speakers, and the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene;

[0008] A stereo speaker system is used to perform multi-channel audio rendering processing on multi-channel media audio; wherein each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel;

[0009] The multi-channel media audio processed by the multi-channel audio rendering is played through the stereo speaker system.

[0010] On the one hand, an embodiment of the present application provides an audio processing device, the device comprising:

[0011] An acquisition unit, used to acquire multi-channel media audio in a media playback scene, where the multi-channel media audio includes audio signals collected through N channels, where each channel corresponds to a different spatial position in the media playback scene; N is an integer greater than 1;

[0012] A processing unit, used to obtain a target spatial position of a media viewing object in a media playback scene, and to arrange a stereo speaker system around the target spatial position in the media playback scene; the stereo speaker system includes N virtual speakers, and the arrangement position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene;

[0013] The processing unit is further used to perform multi-channel audio rendering processing on the multi-channel media audio using a stereo speaker system; wherein each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel;

[0014] The processing unit is also used to play the multi-channel media audio after the multi-channel audio rendering processing through the stereo speaker system.

[0015] In a possible implementation, the multi-channel media audio includes N audio data, and one audio data is an audio signal collected by one channel; the processing unit is further used to perform the following operations:

[0016] Obtaining the collected multi-channel audio signal source in the media playback scene;

[0017] Performing frame processing on a multi-channel audio signal source to obtain K multi-channel audio frames, each of which is composed of audio signals of N channels;

[0018] The multi-channel audio signal source is format-resampled by using the channel segmentation technology, so as to segment each multi-channel audio frame according to different channels to obtain N audio data; any audio data contains K audio frames, and the K audio frames contained in one audio data belong to the same channel, and K is a positive integer;

[0019] The channel segmentation technology includes: any one of audio editing software, audio segmentation tools, and audio processing framework.

[0020] In a possible implementation, the processing unit is further configured to perform the following operations:

[0021] Performing audio preprocessing on the multi-channel audio signal source to obtain a preprocessed multi-channel audio signal source; wherein the audio preprocessing includes: any one or more of echo cancellation processing, noise suppression processing, normalization processing, and voice activity detection;

[0022] The channel segmentation technology is used to perform format resampling processing on the preprocessed multi-channel audio signal source to obtain N audio data.

[0023] In a possible implementation, the multi-channel media audio is obtained by collecting audio signals through N physical speakers, and one physical speaker is used to collect an audio signal of one channel at a spatial position in a media playback scene;

[0024] The processing unit arranges a stereo speaker system around the target spatial position in the media playback scene to perform the following operations:

[0025] Analyze multi-channel media audio and determine a reference speaker layout of N physical speakers during signal acquisition, where the reference speaker layout includes: the distance and direction between each physical speaker and the reference spatial position in the media playback scene;

[0026] Based on the target spatial position and reference speaker layout, a stereo speaker system is arranged in the media playback scenario.

[0027] In a possible implementation, the target spatial position is represented by three-dimensional spatial coordinates (x, y, z), any one of the N physical speakers is represented as physical speaker a, a is a positive integer and 1≤a≤N; the distance between the physical speaker a and the reference spatial position includes: a first distance d1, a second distance d2, and a third distance d3;

[0028] The processing unit arranges a stereo speaker system in the media playback scene based on the target spatial position and the reference speaker layout, and performs the following operations:

[0029] Based on the target spatial position (x, y, z), and the first distance d1, the second distance d2, and the third distance d3 between the physical speaker a and the reference spatial position, calculate the candidate position (x', y', z') of the virtual speaker b to be arranged;

[0030] Based on the candidate position (x', y', z') of the virtual speaker b and the direction of the physical speaker a in the media playback scene, the virtual speaker b is arranged in the media playback scene.

[0031] In a possible implementation, the processing unit arranges the virtual speaker b in the media playback scene based on the candidate position (x', y', z') of the virtual speaker b and the direction of the physical speaker a in the media playback scene, and performs the following operations:

[0032] Based on the candidate position (x', y', z'), determine the spatial position of the virtual speaker b to be arranged in the media playback scene; wherein the distance between the spatial position to be arranged and the candidate position is less than a preset threshold;

[0033] The direction of the physical speaker a in the media playback scene is determined as the direction of the virtual speaker b to be arranged in the media playback scene;

[0034] Setting speaker properties for the virtual speaker b, where the speaker properties include: any one or more of geometric shape, material, and color;

[0035] According to the spatial position to be laid out, the direction to be laid out, and the speaker attributes, the virtual speaker b is laid out in the media playback scene.

[0036] In a possible implementation, the processing unit is further configured to perform the following operations:

[0037] Analyze the multi-channel media audio and determine the audio type of the multi-channel media audio;

[0038] Obtaining multiple multi-channel speaker layouts corresponding to the media playback scene; any multi-channel speaker layout is used to specify the spatial layout information of N physical speakers during the signal acquisition process;

[0039] Based on the audio type, a multi-channel speaker layout matching the audio type is determined from a plurality of multi-channel speaker layouts, and the determined multi-channel speaker layout is used as a reference speaker layout.

[0040] In a possible implementation, the multi-channel media audio includes N audio data, and one audio data includes K audio frames from the same channel; the processing unit uses a stereo speaker system to perform multi-channel audio rendering processing on the multi-channel media audio, and is used to perform the following operations:

[0041] Parse multi-channel media audio and determine the channel corresponding to each audio data;

[0042] Obtain a channel layout association rule, where the channel layout association rule defines that: each virtual speaker in the N virtual speakers is associated with one of the N channels;

[0043] Based on the channel layout association rule and the channels corresponding to each audio frame, N virtual speakers are used to perform audio rendering processing on N channels of N audio data; wherein one virtual speaker is used to render the audio data of one channel.

[0044] In a possible implementation, any one of the N virtual speakers is represented as the i-th virtual speaker, where i is a positive integer and 1≤i≤N; the processing unit uses the N virtual speakers to perform audio rendering processing of the N channels on the N audio data based on the channel layout association rule and the channels corresponding to each audio frame, and is used to perform the following operations:

[0045] Based on the channel layout association rule and the channels corresponding to each audio data, the target audio data to be processed by the i-th virtual speaker is obtained from the N audio data; the target audio data includes K audio frames whose channel type is channel i;

[0046] The i-th virtual speaker is used to perform audio rendering processing on the target audio data.

[0047] In a possible implementation, the processing unit uses the i-th virtual speaker to perform audio rendering processing on the target audio data to perform the following operations:

[0048] Construct an audio cache queue, where the audio cache queue is used to store a preset threshold number P of audio frames to be processed, where P is a positive integer;

[0049] Obtain an audio frame j to be processed from the K audio frames contained in the target audio data, and add the audio frame j to the audio cache queue; j is a positive integer and 1≤j≤K;

[0050] Let j=j+1, until the number of audio frames stored in the audio cache queue reaches a preset threshold value P, the i-th virtual speaker is used to perform audio rendering processing on the P audio frames in the audio cache queue.

[0051] In a possible implementation, the processing unit uses the i-th virtual speaker to perform audio rendering processing on the P audio frames in the audio buffer queue to perform the following operations:

[0052] Performing data analysis on any audio frame among the P audio frames, and constructing an audio structure description body corresponding to the audio cache queue; the audio description structure body is used to describe at least one type of attribute information of any audio frame;

[0053] Calling a system conversion interface to perform format conversion processing on the P audio frames respectively, so as to convert the P audio frames into a cache format;

[0054] Based on each audio frame in the buffered format, P coded data frames are created;

[0055] Based on the audio structure description body, the i-th virtual speaker is used to perform multiplexing rendering processing on P coded data frames.

[0056] In a possible implementation, before the processing unit adds the audio frame j to the audio buffer queue, it is further configured to perform the following operations:

[0057] Get the last audio frame at the end of the audio buffer queue;

[0058] Analyze the first attribute information of the audio frame j and the second attribute information of the tail audio frame, where the attribute information includes: any one or more of sampling rate, number of bytes, and audio format;

[0059] If the first attribute information of the audio frame j is the same as the second attribute information of the tail audio frame, the step of adding the audio frame j to the audio cache queue is triggered.

[0060] In a possible implementation, the processing unit is further configured to perform the following operations:

[0061] If the first attribute information of the audio frame j is different from the second attribute information of the tail audio frame, then data analysis is performed on the audio frame j to construct an audio description structure of the audio frame j;

[0062] Based on the audio description structure of audio frame j, the i-th virtual speaker is used to render audio frame j.

[0063] On the one hand, an embodiment of the present application provides a computer device, which includes a processor and a memory; a computer program is stored in the memory; when the computer program is executed by the processor, the above-mentioned audio processing method is executed.

[0064] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned audio processing method is executed.

[0065] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned audio processing method is performed.

[0066] In an embodiment of the present application, multi-channel media audio in a media playback scene is obtained, and the multi-channel media audio includes an audio signal obtained by collecting N channels, each channel corresponds to a different spatial position in the media playback scene, and N is an integer greater than 1; the target spatial position of a viewing object of media data in the media playback scene is obtained, and a stereo speaker system is arranged around the target spatial position in the media playback scene; the stereo speaker system includes N virtual speakers, and the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene; the multi-channel media audio is rendered by the stereo speaker system, and each virtual speaker in the stereo speaker system performs audio rendering on the audio signal collected by the adapted channel; and the multi-channel media audio after the multi-channel audio rendering is played through the stereo speaker system. It can be seen that in the process of laying out the stereo speaker system, the present application can virtually restore the spatial position of the multi-channel media audio during collection to determine the layout position of the virtual speakers, so that the actual sound environment of the media audio can be restored as much as possible in the media playback scene, thereby improving the authenticity of the audio playback effect; in addition, the present application uses N virtual speakers to render the audio signals collected by N channels separately, which can achieve the rendering effect of multi-channel spatial audio, improve the stereoscopic and spatial sense of audio playback, and thereby improve the immersive experience in the media playback scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 It is a schematic diagram of the principle of an audio processing solution provided in an embodiment of the present application;

[0069] Figure 2 is a structural schematic diagram of a stereo speaker system provided in an embodiment of the present application;

[0070] Figure 3 is a schematic diagram of the architecture of an audio processing system provided in an embodiment of the present application;

[0071] Figure 4 It is a flowchart of an audio processing method provided in an embodiment of the present application;

[0072] Figure 5a is a structural schematic diagram of a multi-channel audio signal source provided in an embodiment of the present application;

[0073] Figure 5b is a schematic diagram of the structure of a multi-channel media audio provided by an embodiment of the present application;

[0074] Figure 6a is a structural schematic diagram of a reference speaker layout provided in an embodiment of the present application;

[0075] Figure 6b It is a schematic diagram of the layout structure of a stereo speaker layout system provided in an embodiment of the present application;

[0076] Figure 7 is a schematic diagram of a multi-channel speaker layout provided in an embodiment of the present application;

[0077] Figure 8 is a flowchart of another audio processing method provided in an embodiment of the present application;

[0078] Fig. 9 It is a flowchart of an audio rendering provided by an embodiment of the present application;

[0079] Fig.10 is a schematic diagram of multi-channel rendering of a stereo speaker system provided in an embodiment of the present application;

[0080] Fig.11 is a schematic diagram of a media playback scenario provided by an embodiment of the present application;

[0081] Fig.12 is a structural schematic diagram of an audio processing device provided in an embodiment of the present application;

[0082] Fig.13 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0084] This application provides an audio processing solution that is suitable for multi-channel audio playback of any type of media audio obtained by multi-channel acquisition; the solution can achieve multi-channel spatial audio rendering effects in media playback scenarios, so as to virtually restore the real sound environment in the signal acquisition process as much as possible, thereby improving the authenticity of the audio playback effect while enhancing the immersive experience of the media viewing object. For details, please refer to Figure 1 , Figure 1This is a schematic diagram of the principle of an audio processing solution provided by an embodiment of the present application. Figure 1 The principle of the audio processing solution provided by this application is roughly explained:

[0085] (1) Obtain multi-channel media audio to be played in a media playback scene, where the multi-channel media audio includes audio signals collected through N (N is an integer greater than 1) channels (e.g., left channel, right channel, front channel, rear channel, etc.); each channel corresponds to a different spatial position in the media playback scene (e.g., the left channel corresponds to the position on the left side of an object in the media playback scene, and the right channel corresponds to the position on the right side of the object in the media playback scene); wherein the media playback scene refers to a scene in which a real spatial environment for playing multi-channel media audio is located, such as any type of three-dimensional spatial scene such as a movie playback scene and a TV playback scene.

[0086] (2) Obtaining the target spatial position of the media viewing object in the media playback scene; e.g. Figure 1 As shown, the media viewing object can be located at any position in the media playback scene (such as the center of the scene).

[0087] (3) Arrange a stereo speaker system around the target spatial position in the media playback scene. Specifically, since the multi-channel media audio is obtained by signal acquisition through N channels, N virtual speakers need to be arranged here to play the multi-channel media audio. Optionally, the present application can determine the layout information (such as layout position, direction, etc.) of each virtual speaker to be arranged based on the target spatial position, and then arrange N virtual speakers in the media playback scene according to the layout information, so that the stereo speaker system can be arranged around the media playback object. Among them, the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene (for example, the layout position is the same as the spatial position at the time of acquisition or the distance between the layout position and the spatial position is less than a preset threshold). That is, when arranging the virtual speakers, the spatial position of the speakers at the time of signal acquisition is restored as much as possible. In this way, the stereo speaker system can be arranged around the media viewing object according to the real-time position of the media viewing object.

[0088] (4) Using a stereo speaker system to perform multi-channel audio rendering processing on multi-channel media audio; wherein each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel, that is, each virtual speaker performs audio rendering on the audio signal under the corresponding channel, so that the multi-channel media audio after multi-channel audio rendering processing is played through the stereo speaker system to achieve the playback effect of multi-channel spatial audio. For example, Figure 1In the media playback scene shown, after multi-channel audio rendering is performed through a stereo speaker system, for the media viewing object, it is possible to achieve: the following: birds singing in the valley in front of the eyes 102, the breeze blowing overhead 101, the lingering sound of music behind the back 103, and the sound of zither and chimes 104, etc. This multi-directional or multi-channel audio playback effect, based on the audio-visual synchronization method, presents the media viewing object with a poetic, majestic and heavy media playback scene, and brings the user an immersive experience in the media playback scene.

[0089] As can be seen from the above, on the one hand, in the process of laying out the stereo speaker system, the present application can virtually restore the spatial position of the multi-channel media audio at the time of collection, and based on the real-time position of the media viewing object, lay out the stereo speaker system around the media viewing object, so that in the media playback scene, the real sound environment of the media audio can be restored as much as possible, thereby improving the authenticity of the audio playback effect; on the other hand, the present application uses N virtual speakers to render the audio signals collected by N channels separately, which can achieve the rendering effect of multi-channel spatial audio to improve the immersive experience in the media playback scene.

[0090] The key technical terms involved in this application are introduced below.

[0091] 1. Vocal channel.

[0092] Channel (also called track or audio track): In the field of audio processing, it refers to the transmission path of audio signals, and is usually used to describe the independent channels of different audio signals in audio processing systems; in simple terms, independent audio signals collected or played back at different spatial locations when recording or playing sound are called channels. The number of channels is referred to as the number of channels, and is usually expressed in the "XY" format, where "X" represents the number of main channels and "Y" represents the number of low-frequency special effect channels (usually called LFE, low-frequency effect channels). For example, 2.0 channels: represents two left and right channels, and there is no independent low-frequency channel (typical stereo configuration). Another example is 5.1 channels: represents five main channels and one low-frequency effect channel. Another example is 7.1 channels: represents seven main channels and one low-frequency effect channel. Therefore, according to the number of channels, it can be divided into: mono and multi-channel; the so-called mono channel means that all audio signals are transmitted and played through one channel. Mono channels are usually used in audio playback scenarios that do not require a high sense of space, such as telephones, radio broadcasts, etc. The so-called multi-channel means that all audio signals are transmitted and played through two or more channels. Therefore, the audio signals transmitted through multi-channels have a sense of stereo and space. For example, multi-channels include: left channel (L) and right channel (R).

[0093] 2. Multi-channel media audio.

[0094] Multi-channel media audio, as the name implies, refers to media audio or audio signals that are transmitted and played through multiple channels. It is an audio system that includes multiple independent channels and is suitable for complex audio playback scenarios, such as film production, game sound design, etc. Multi-channel media audio allows more detailed and precise sound positioning, usually including 5.1, 7.1, 9.1 or even more channels. In this application, multi-channel media audio includes an audio signal collected through N channels. For example, multi-channel media audio specifically includes N audio data, and one audio data refers to an audio signal collected through one channel. Therefore, multi-channel media audio allows audio transmission and playback through N channels (such as left channel, right channel, front and rear channels, upper and lower channels, etc.), so as to create an immersive sense of space and stereo.

[0095] 3. Stereo speaker system, virtual speaker.

[0096] A stereo speaker system refers to a speaker system that can play stereo sound or surround sound. The so-called stereo sound effect refers to an audio playback effect with a sense of three-dimensionality and space, using multiple channels to surround the audience, creating a stronger immersive sound experience, so that the audience can feel the effect of sound coming from different directions. In this application, the stereo speaker system is composed of N virtual speakers, and the stereo speaker system is arranged around the media viewing object, and the N virtual speakers are located in different spatial positions in the media playback scene. Please refer to Figure 2 , Figure 2 Schematic diagram of a stereo speaker system provided in an embodiment of the present application. Figure 2 As shown, the stereo speaker system includes five virtual speakers: speakers 1, 2, 3, 4, and 5; and the spatial positions of different virtual speakers in the media playback scene are arranged around the media viewing object, for example, speaker 1 is located to the left of the media viewing object, speaker 2 is located to the right of the media viewing object, speaker 3 is located directly in front of the media viewing object, speaker 4 is located directly behind the media viewing object, and speaker 5 is located directly above the media viewing object. Among them, different virtual speakers in the stereo speaker system are responsible for rendering audio signals of different channels. Since one channel corresponds to different spatial positions of speakers in the media playback scene, Figure 2As shown, the N channels specifically include: left channel (L), right channel (R), center channel (C), back channel (B), and upper channel (H). Then speaker 1 is responsible for rendering the audio signal of the left channel, speaker 2 is responsible for rendering the audio signal of the right channel... speaker 5 is responsible for rendering the audio signal of the upper channel, so that each virtual speaker renders the audio signal of its own channel respectively, so that the rendered audio signals come from different directions in the media playback scene, thereby jointly creating an immersive sound experience with a sense of stereo and space played from different directions and positions.

[0097] A virtual speaker (ModelEntity) refers to a virtual speaker (i.e., not a speaker that the user can actually perceive). A speaker refers to a device that has audio playback capabilities, i.e., a device that can make sounds. Therefore, the virtual speaker in this application refers to a virtual device that has audio playback capabilities, which can render and play the audio signals of the corresponding channels; wherein, the speaker properties (geometric shape, material, and color) of each virtual speaker in the stereo speaker system can be the same or different, for example, speaker 1 can be circular, and speaker 2 can be rectangular; or, all of the above speakers are circular.

[0098] 4. RealityKit technology

[0099] In this application, N virtual speakers can be arranged in the media playback scene and audio rendering processing can be performed through a rendering framework (such as the RealityKit framework). The so-called RealityKit framework is a high-performance rendering and animation framework that aims to simplify the development process of AR (Augmented Reality) applications while providing powerful tools and functions to achieve high-quality visual, auditory, and interactive experiences. In this application, RealityKit has the following functions:

[0100] (1) Custom materials and special effects:

[0101] Developers can use RealityKit to apply customized materials, textures, colors, and special effects to virtual speakers, allowing them to create more realistic and personalized visual effects.

[0102] (2) Environmental understanding:

[0103] RealityKit can obtain data from sensors of media playback devices, such as cameras, LIDAR, and gyroscopes, to help applications understand and interact with the real-world environment. In this application, the virtual speaker can detect the movement of the media viewing object in the media playback scene in real time based on RealityKit technology, and after the media viewing object moves, the spatial position of the virtual speaker is updated in real time to adapt it to the spatial position of the media viewing object, so that audio can be played according to the new spatial position, thereby achieving the ability to move audio playback with the user during audio playback.

[0104] It should be noted that the relevant data involved in the audio processing process of this application (such as multi-channel media audio, audio data, etc.). When the above embodiments of this application are applied to specific products or technologies, the permission or consent of the object must be obtained, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the region, comply with the principles of legality, legitimacy and necessity, and do not involve the acquisition of data types prohibited or restricted by laws and regulations. In some optional embodiments, the relevant data involved in the embodiments of this application are obtained after the object has been authorized separately. In addition, when obtaining the separate authorization of the object, the purpose of the relevant data involved is indicated to the object.

[0105] The audio processing system provided by this application is introduced in detail below.

[0106] See also Figure 3 , Figure 3 Schematic diagram of the architecture of an audio processing system provided by an embodiment of the present application. Figure 3 As shown, the architecture diagram of the audio processing system may at least include: a server 304 and a terminal device cluster; wherein the terminal device cluster includes: a first terminal device 301, a second terminal device 302, a third terminal device 303 and other terminal devices, and any terminal device has the ability to play audio; it should be understood that the number of terminal devices included in the terminal device cluster is only for example, and the embodiment of the present application does not limit the number and type of terminal devices. Among them, any terminal device in the terminal device cluster can be directly or indirectly connected to the server 304 through the network; the above network may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, WIFI (Wireless Fidelity, a standard wireless local area network), and other networks that implement wireless communication.

[0107] Each terminal device in the terminal device cluster may be: a mobile phone, a tablet computer, a laptop computer, a desktop computer, a gaming device, a vehicle-mounted device, an aircraft, a wearable device (such as a smart watch, a smart bracelet, a pedometer, etc.), a smart home appliance, an XR (Extended Reality) device, for example, an XR device includes: a virtual reality device (such as a VR (Virtual Reality) device, an AR (Augmented Reality) device), or an MR (Mixed Reality) device, etc. It is understandable that the types of each terminal device in the terminal device cluster may be the same or different, for example: the first terminal device 301 may be a laptop computer, the second terminal device 302 may be a desktop computer, and the third terminal device 303 may be a tablet computer. This application does not limit the number and type of terminal devices in the terminal device cluster.

[0108] Server 304 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0109] Taking any terminal device (such as the first terminal device 301) in the audio processing system as an example, the data interaction process between the first terminal device 301 and the server 304 in the media playback scenario is described accordingly:

[0110] ① In a media playback scene (such as a movie watching scene or a game watching scene, etc.), the first terminal device 301 can obtain a multi-channel audio signal source to be processed in the media playback scene, and send the multi-channel audio signal source to the server 304. The multi-channel audio signal source includes K multi-channel audio frames, each of which is composed of audio signals collected from N channels, each channel corresponds to a different spatial position in the media playback scene, N is an integer greater than 1, and K is a positive integer.

[0111] ② The server 304 performs format resampling processing (such as channel segmentation technology) on the multi-channel audio signal source to segment each multi-channel audio frame according to different channels, obtain multi-channel media audio to be played, and send the multi-channel media audio to the first terminal device 301.

[0112] ③ The first terminal device 301 obtains the target spatial position of the media viewing object in the media playback scene. For example, the target spatial position is the three-dimensional spatial coordinates of the media viewing object.

[0113] ④ The first terminal device 301 arranges a stereo speaker system around the target spatial position in the media playback scene based on the target spatial position. The stereo speaker system includes N virtual speakers, and the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene; that is, when arranging the virtual speakers, the spatial position of the speakers at the time of signal collection is restored as much as possible, so that the stereo speaker system can be arranged around the media viewing object according to the real-time position of the media viewing object.

[0114] ⑤ The first terminal device 301 uses a stereo speaker system to perform multi-channel audio rendering processing on the multi-channel media audio, and plays the multi-channel media audio after the multi-channel audio rendering processing through the stereo speaker system. Among them, each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel, so that each virtual speaker can independently be responsible for the rendering processing of the audio signal in the corresponding direction position, and jointly create an immersive audio playback effect for the media viewing object.

[0115] It should be noted that the above audio processing process is only used for example and does not limit the specific execution process of the terminal device and the server. Optionally, the terminal device can send the target spatial position and multi-channel media audio to the server, and the server determines the layout position of each virtual speaker in the stereo speaker system to be laid out, and then the server sends the specific layout position of each virtual speaker in the stereo speaker system in the media playback scene to the terminal device, and then the terminal device executes the layout process of the stereo speaker system in the media playback scene; or, the complete process of the above audio processing can be executed separately by any terminal device or server in the audio processing system.

[0116] In a possible implementation, the audio processing system provided in the embodiment of the present application can be deployed in the nodes of the blockchain. For example, the first terminal device 301, the second terminal device 302, the third terminal device 303, and the server 304 can all be regarded as node devices of the blockchain to jointly constitute a blockchain network. Therefore, the audio processing process executed in the present application can be executed on the blockchain, which can not only ensure the fairness and justice of the audio processing process, but also make the above process traceable, and at the same time ensure the data security during the audio processing process, thereby improving the security and reliability of the entire audio processing process.

[0117] Based on the audio processing system provided by the present application, on the one hand, in the process of laying out the stereo speaker system, the present application can virtually restore the spatial position of the multi-channel media audio at the time of collection, and based on the real-time position of the media viewing object, lay out the stereo speaker system around the media viewing object, so that in the media playback scene, the real sound environment of the media audio can be restored as much as possible, thereby improving the authenticity of the audio playback effect; on the other hand, the present application uses N virtual speakers to render the audio signals collected by N channels separately, which can achieve the rendering effect of multi-channel spatial audio to improve the immersive experience in the media playback scene.

[0118] It can be understood that the audio processing system described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided in the embodiment of the present application. It is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0119] The specific embodiments of the audio processing scheme of the present application are described in detail below with reference to the accompanying drawings.

[0120] See also Figure 4 , Figure 4 is a flow chart of an audio processing method provided by an embodiment of the present application. The audio processing method can be performed by a computer device (such as Figure 3 Any terminal device or server as shown. Figure 4 As shown, the audio processing method includes but is not limited to the following steps S401-S404:

[0121] S401: Acquire multi-channel media audio in a media playback scene.

[0122] Among them, multi-channel media audio includes audio signals collected through N channels, each channel corresponds to a different spatial position in the media playback scene, and N is an integer greater than 1. It should be understood that the audio signals of different channels are collected simultaneously during the signal collection process, that is, in the same time period, the lengths of the audio signals collected for different channels are the same. For example, in the process of collecting signals through the left channel and the right channel, within one minute, one minute of left channel audio signal can be collected, and at the same time, one minute of right channel audio signal can be collected.

[0123] The signal processing process of multi-channel media audio is described in detail below.

[0124] 1. Resample the collected multi-channel audio signal source.

[0125] In a possible implementation, the multi-channel media audio includes N audio data. Specifically, the process of obtaining the multi-channel media audio includes the following steps (1)-(3):

[0126] (1) Obtaining a collected multi-channel audio signal source in a media playback scene. It should be understood that a multi-channel audio signal source is generated by collecting signals from N channels, each channel corresponding to a different spatial position in the media playback scene. Therefore, the collected multi-channel audio signal source is a multi-channel mixed audio signal, that is, each audio frame contains audio signals from multiple channels.

[0127] (2) The multi-channel audio signal source is processed by framing to obtain K multi-channel audio frames, each of which is composed of N channels of audio signals, and K is a positive integer. Specifically, the multi-channel audio signal source can be processed by framing according to a preset frame length (such as 2.5ms~60ms) to obtain K multi-channel audio frames. Figure 5a , Figure 5a Schematic diagram of a multi-channel audio signal source provided by an embodiment of the present application. Figure 5a As shown, assuming that the N channels include: L (Left, left channel), R (Right, right channel), C (Center, center channel), Ls (Left surround, left surround), Rs (Right surround right surround channel). Since the audio signals of the N channels are collected simultaneously during the signal collection process, each multi-channel audio frame contains the audio signals of the N channels. For example, the multi-channel audio frame 1 (Frame1) contains: the audio signals of the five channels of L, R, C, Ls, and Rs; and the multi-channel audio frame 2 (Frame2) also contains: the audio signals of the five channels of L, R, C, Ls, and Rs.

[0128] (3) The multi-channel audio signal source is formatted and resampled by using the channel segmentation technology, so as to segment each multi-channel audio frame according to different channels to obtain N audio data; any audio data contains K audio frames, and the K audio frames contained in one audio data belong to the same channel, where K is a positive integer; wherein the channel segmentation technology includes: any one of audio editing software, audio segmentation tools, and audio processing frameworks. Specifically, ① the audio editing software may be, for example, the open source audio editor Audacity, which can be used to split the multi-channel audio signal source into tracks, thereby obtaining multiple mono audio data; ② the audio segmentation tool refers to a tool (such as the FFmpeg command) that has the function of segmenting the audio signal into channels, and the FFmpeg command can be called to segment the multi-channel audio signal source into tracks to obtain N audio data of channels; ③ the audio processing framework may be the Librosa framework, which is a Python library for audio analysis and can also realize the function of segmenting the audio signal into channels. It should be understood that in actual applications, appropriate channel segmentation technology can be selected according to specific scene requirements to improve the segmentation effect. Figure 5a The interleaved audio signal combination shown in the figure is not convenient for extracting the audio data of each channel separately, so it is necessary to resample the audio frame format and convert it into a planar combination, that is, to store the audio signals of each channel independently. Based on this, see Figure 5b , Figure 5b Schematic diagram of a multi-channel media audio structure provided by an embodiment of the present application. Figure 5b As shown, after the multi-channel audio signal source is format-resampled using the above-mentioned channel segmentation technology, the audio data of each channel can be easily obtained, so that the rendering and playback of each channel can be achieved separately without interfering with each other.

[0129] The process shown in the above steps (1) to (3) can store the audio signals of N channels separately after resampling the format of the multi-channel audio signal source, so that the audio signals of each channel can be processed independently later.

[0130] 2. Perform audio preprocessing on the collected multi-channel audio signal source.

[0131] In a possible implementation, audio preprocessing is performed on a multi-channel audio signal source to obtain a preprocessed multi-channel audio signal source; channel segmentation technology is used to perform format resampling processing on the preprocessed multi-channel audio signal source to obtain N audio data (the specific process of format resampling is shown in the above process). Among them, audio preprocessing includes: any one or more of echo cancellation processing, noise suppression processing, normalization processing, and voice activity detection. Specifically, ① Echo Cancellation: It is a technology for eliminating echoes. When the sound played by a loudspeaker (i.e., a speaker) is picked up again through a microphone and transmitted back to the other party, an echo will be generated; ② Noise Suppression: It is a technology for reducing background noise or environmental noise, which is often used to improve the quality of voice communication; ③ Voice Activity Detection (VAD): It is a technology for detecting whether there is an active audio signal in multi-channel media audio to improve the efficiency of audio processing. Through the above-mentioned audio preprocessing method, the collected multi-channel audio signal source can be optimized in advance, thereby improving the audio quality of the multi-channel media audio to be played, so as to enhance the audio playback effect.

[0132] S402: Acquire a target spatial position of a media viewing object in a media playback scene, and arrange a stereo speaker system around the target spatial position in the media playback scene.

[0133] It should be understood that since multi-channel media audio is generated by signal acquisition through N channels, each channel corresponds to a different spatial position in the media playback scene, it can be seen that multi-channel media audio is obtained by signal acquisition by N physical speakers located at N spatial positions in the media playback scene, so N virtual speakers need to be arranged in the stereo speaker system to virtually restore the speaker layout during the signal acquisition process. That is, the stereo speaker system includes N virtual speakers, and the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene. Among them, the adaptation here includes: the layout position of the virtual speaker is consistent with the corresponding spatial position during signal acquisition; or, the distance between the layout position of the virtual speaker and the corresponding spatial position during signal acquisition is within a specified preset threshold. In other words, when the present application is arranged in a stereo speaker system, it can virtually restore the speaker layout during signal acquisition as much as possible (i.e., the spatial position in the media playback scene), thereby improving the authenticity and accuracy of multi-channel audio playback.

[0134] In a specific implementation, the media viewing object in the media playback scene can be detected by a sensor, so as to obtain the target spatial position of the media viewing object in the media playback scene, where the target spatial position can be represented by three-dimensional coordinates (x, y, z). The number of media viewing objects in the media playback scene can be one or more. If there are multiple media viewing objects, it is necessary to obtain the spatial position of each media viewing object in the media playback scene. For the convenience of explanation, this application takes one media viewing object as an example for illustration.

[0135] In a specific implementation, the process of arranging a stereo speaker system around a target spatial position in a media playback scene is as follows: (1) Based on the target spatial position, determine the layout information of each virtual speaker in the stereo speaker system, where the layout information includes, for example, one or more of the layout position, direction, and speaker attributes; (2) Arrange the stereo speaker system in the media playback scene according to the layout information (such as the layout position and direction) of each virtual speaker. It should be understood that the stereo speaker system arranged in the present application is a virtual reality system (i.e., VR system) in a media playback scene. The so-called VR system is a computer simulation system that can create and experience a virtual world. The VR system generates a simulated environment in the media playback scene through VR technology, and provides users (such as media viewing objects) with simulations of senses such as vision, hearing, and touch, so that users can observe things in the three-dimensional space in a timely and unrestricted manner in the media playback scene, thereby providing users with an immersive experience that is indistinguishable from the real thing in the media playback scene. Although the stereo speaker system is a virtual reality system, the stereo speaker system has the ability to render and play audio for multi-channel media audio (including audio signals of multiple channels).

[0136] The specific process of how to layout a stereo speaker system is described in detail below.

[0137] Method 1: Arrange the stereo speaker system according to the preset unified speaker layout standard.

[0138] In a possible implementation, multi-channel media audio is obtained by collecting audio signals through N physical speakers; one physical speaker is used to collect an audio signal of one channel at a spatial position in the media playback scene. Specifically, a stereo speaker system is arranged around the target spatial position in the media playback scene, including the following steps (1)-(2):

[0139] (1) Analyze multi-channel media audio and determine the reference speaker layout of N physical speakers during the signal collection process. The reference speaker layout includes: the distance and direction between each physical speaker and the reference spatial position in the media playback scene. Specifically, during the signal collection process, you first need to set a reference spatial position, and then use the reference spatial position as the center and set the spatial positions and layout directions of the N physical speakers used for signal collection around the reference spatial position. Please refer to Figure 6a , Figure 6a is a schematic diagram of a reference speaker layout provided in an embodiment of the present application. Figure 6a As shown, the reference spatial position may be the spatial position of a spatial point S0 in a 3D (three-dimensional) space, and the signals of the N physical speakers arranged around the spatial point S1 are collected to obtain multi-channel media audio. Assume that the reference speaker layout is used to indicate: the signal is collected through 10 physical speakers, and the spatial positions of the 10 physical speakers during the signal collection process are as follows: Figure 6a As shown in: S1, S2, S3...S10, where S1 refers to the left speaker, S2 is the right speaker, S3 is the center speaker, S4 is the subwoofer speaker, S5 is the left surround speaker, S6 is the right surround speaker, S7 is the left rear surround speaker, S8 is the right rear surround speaker, S9 is the left top center speaker, and S10 is the right top center speaker. Moreover, according to the reference speaker layout, the distance and direction between each physical speaker and the reference spatial position (S0) can be determined. For example, the distance between the left speaker S1 and the reference spatial position (S0) is D1, and the direction is the first direction. For example, the left speaker S1 is located at a spatial position D1, 30° in the upper left direction of the reference spatial position S0; and the right speaker S2 is located at a spatial position D1, 30° in the upper right direction of the reference spatial position S0. Based on this, according to Figure 6a Multi-channel media audio can be obtained by collecting signals according to the reference preset layout shown. Then, after analyzing the multi-channel media audio, the detailed layout of each physical speaker of the multi-channel media audio during signal collection can be restored (reference speaker layout).

[0140] (2) Based on the target spatial position and the reference speaker layout, a stereo speaker system is arranged in the media playback scene. The target spatial position can be represented by three-dimensional spatial coordinates (x, y, z), and any one of the N physical speakers is represented as a physical speaker a, where a is a positive integer and 1≤a≤N. The distance between the physical speaker a and the reference spatial position includes: a first distance d1, a second distance d2, and a third distance d3. It should be noted that when the speakers are arranged, the specific arrangement can be made according to the three-dimensional spatial coordinates. The first distance here refers to the horizontal distance (x-axis distance) between the physical speaker a and the reference spatial position S0, the second distance refers to the longitudinal distance (y-axis distance) between the physical speaker a and the reference spatial position S0, and the third distance refers to the spatial distance (z-axis distance) between the physical speaker a and the reference spatial position S0. Then, the process of laying out any virtual speaker is as follows ①-②:

[0141] ① Based on the target spatial position (x, y, z), and the first distance d1, the second distance d2, and the third distance d3 between the physical speaker a and the reference spatial position, calculate the candidate position (x', y', z') of the virtual speaker b to be arranged. Figure 6a According to the reference loudspeaker layout standard shown, if the target spatial position and the distance between the virtual loudspeaker to be laid out and the target spatial position are known, the three-dimensional layout coordinates (x', y', z') of each virtual loudspeaker to be laid out can be determined.

[0142] See also Figure 6b , Figure 6b Schematic diagram of the layout structure of a stereo speaker layout system provided in an embodiment of the present application. Figure 6b As shown, Figure 6a Taking the 10-channel speaker layout shown in the figure as an example (i.e., 10 physical speakers), the coordinates of the target spatial position S0' are expressed as (x, y, z). Then, according to the distance between each virtual speaker and the target spatial position and the reference speaker layout, the three-dimensional spatial coordinates of each virtual speaker to be arranged can be calculated as follows: left speaker S1': (x-d1, y, z-d2), right speaker S2': (x+d1, y, z-d2), center speaker S3': (x, y, z-d2), bass Cannon speaker S4': (x-d1 / 2, y, z-d2), left surround speaker S5': (x-d3, y, z), right surround speaker S6': (x+d3, y, z), left rear surround speaker S7': (x-d5, y, z+d4), right rear surround speaker S8': (x+d5, y, z+d4), left top center speaker S9': (x-d7, y+y1, z-d6), right top center speaker S10': (x+d7, y, z-d6).

[0143] ② Based on the candidate position (x', y', z') of the virtual speaker b and the direction of the physical speaker a in the media playback scene, the virtual speaker b is laid out in the media playback scene. Optionally, the layout of a virtual speaker b needs to include the following layout information: spatial position, direction, and speaker attributes (such as geometric shape or material attributes). The process of determining the layout information of the virtual speaker is as follows:

[0144] I. Spatial position. The spatial position of the virtual speaker b to be arranged in the media playback scene includes the following two cases: Case 1, the candidate position (x', y', z') of the virtual speaker b can be directly used as the spatial position of the virtual speaker b to be arranged in the media playback scene; Case 2, the virtual speaker b can be arranged within a certain range around the candidate position (x', y', z'), that is, the spatial position of the virtual speaker b to be arranged in the media playback scene is determined based on the candidate position (x', y', z'); wherein the distance between the spatial position to be arranged and the candidate position is less than a preset threshold.

[0145] II. Direction: The direction of the physical speaker a in the media playback scene can be determined as the direction of the virtual speaker b to be arranged in the media playback scene, that is, the speaker layout direction remains unchanged.

[0146] III. Speaker properties: Speaker properties can be set for the virtual speaker b, where the speaker properties include: any one or more of geometric shapes (such as a circle, rectangle, ellipse, etc.), material (such as metal, plastic, etc.).

[0147] Finally, a virtual speaker b can be arranged in the media playback scene according to the spatial position to be arranged, the direction to be arranged, and the speaker attributes. For example, the virtual speaker b can be arranged in the media playback scene using RealityKit technology. Based on this, N virtual speakers can be arranged in the media playback scene to form a stereo speaker system (wherein the N virtual speakers can have the same or different attributes such as shape and material). Since each virtual speaker to be arranged can refer to Figure 6a The preset speaker layout standard shown is adaptively laid out, so the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene, thereby being able to restore the speaker layout standard during signal collection as much as possible.

[0148] Based on steps (1)-(2) shown in method one, the present application can virtually restore the speaker layout according to the real-time position of the media viewing object based on the preset layout standards of each physical speaker during signal collection, and then layout a stereo speaker system around the media viewing object to restore the layout standards in the real sound environment as much as possible, thereby improving the authenticity and accuracy of subsequent audio playback effects.

[0149] Method 2: Arrange a stereo speaker system that is suitable for the type of multi-channel media audio.

[0150] In a possible implementation, the specific process of arranging a stereo speaker system adapted to the type of multi-channel media audio is as follows (1)-(4):

[0151] (1) Analyze the multi-channel media audio and determine the audio type of the multi-channel media audio. For example, the audio types of the multi-channel media audio include: a first type, a second type, and a third type. The first type refers to a type corresponding to movie media, the second type refers to a type corresponding to game media, and the third type refers to a type corresponding to pure audio media.

[0152] (2) Obtain multiple multi-channel speaker layouts corresponding to the media playback scene; any multi-channel speaker layout is used to specify the spatial layout information of N physical speakers during the signal acquisition process. For example, this application can preset multiple speaker layout standards, see Figure 7 , Figure 7 Schematic diagram of the multi-channel speaker layout provided in the embodiment of the present application. Figure 7As shown, assuming that a four-channel speaker layout system is taken as an example (i.e., it includes four speakers), the present application provides three sets of multi-channel speaker layout standards: speaker layout standard 1, speaker layout standard 2, and speaker layout standard 3. Each set of speaker layout standards includes the layout positions and directions of N (four) physical speakers used for signal acquisition. For example, speaker layout standard 1 includes: four speakers S1, S2, S3, and S4 arranged at the upper left, upper right, lower left, and lower right; speaker layout standard 2 includes: four speakers S1, S2, S3, and S4 arranged at the left, right, upper, and lower; speaker layout standard 3 includes: four speakers S1, S2, S3, and S4 arranged at the left, right, upper, and upper right. It can be seen that the layout positions and directions of each speaker defined in different speaker layout standards are different. (3) Based on the audio type, a multi-channel speaker layout matching the audio type is determined from multiple multi-channel speaker layouts, and the determined multi-channel speaker layout is used as a reference speaker layout. For example, if the audio type is the first type, the above-mentioned speaker layout standard 1 is used as the reference speaker layout; if the audio type is the second type, the above-mentioned speaker layout standard 2 is used as the reference speaker layout; if the audio type is the third type, the above-mentioned speaker layout standard 3 is used as the reference speaker layout.

[0153] (4) Based on the target spatial position and the reference speaker layout, a stereo speaker system is arranged in the media playback scene. Specifically, the specific execution process of this step can be referred to the detailed process shown in step (2) in the above method 1, and will not be repeated here.

[0154] Based on steps (1)-(4) shown in method 2, the present application can select a most suitable speaker layout standard from multiple sets of speaker layout standards as a reference speaker layout according to the type of multi-channel media audio, and virtually restore the speaker layout according to the real-time position of the media viewing object. This method can further improve the accuracy of the speaker layout to achieve a more realistic speaker layout effect.

[0155] Based on this, based on the multi-channel media audio collected by N physical speakers, RealityKit technology can be used to abstract each physical speaker into a ModelEntity (model entity, i.e. virtual speaker). This virtual speaker ModelEntity can not only be used as the sound source in the media playback scene, but also determine the spatial position of the virtual speaker by setting the position three-dimensional coordinates of the ModelEntity. In addition, the speaker properties such as the geometric shape mesh, material, and color of the ModelEntity can be customized. For example, the physical speaker can be abstracted into a virtual speaker of a blue sphere, and after the spatial layout is performed according to the above placement algorithm, a virtual 5.1 spatial sound system (stereo speaker system) can be formed; among them, each ModelEntity virtual speaker is only responsible for the rendering of the audio signal of the corresponding direction position without affecting the rendering of other speakers, and together create a more spatially immersive listening experience.

[0156] S403: Perform multi-channel audio rendering processing on the multi-channel media audio using a stereo speaker system.

[0157] Among them, each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel. Specifically, multi-channel media audio includes N audio data, one audio data is an audio signal collected by one channel, that is, each audio data corresponds to one channel. Then, each virtual speaker in the stereo speaker system can perform audio rendering processing on the audio signal of one channel (that is, one audio data). For example, the left speaker is used to render the audio signal of the left channel (L), the right speaker is used to render the audio signal of the right channel (R), and the center speaker is used to render the audio signal of the middle channel (C).... And so on, the N virtual speakers in the stereo speaker system independently render the audio signals of their respective channels, without interfering with each other, and can jointly achieve multi-channel audio rendering effects.

[0158] In a possible implementation, the process of using a stereo speaker system (i.e., N virtual speakers) to perform multi-channel audio rendering processing on multi-channel media audio is as follows: (1) parsing the multi-channel media audio to determine a channel corresponding to each audio data; (2) obtaining a channel layout association rule, which defines that each virtual speaker in the N virtual speakers is associated with one of the N channels; (3) based on the channel layout association rule and the channels corresponding to each audio frame, using N virtual speakers to perform N-channel audio rendering processing on the N audio data; wherein one virtual speaker is used to render the audio data of one channel. In this implementation, each virtual speaker independently renders the audio signal of its own channel, without interfering with each other, while jointly achieving a multi-channel audio rendering effect.

[0159] S404: Play the multi-channel media audio after the multi-channel audio rendering process through the stereo speaker system.

[0160] Specifically, after each virtual speaker renders the audio signal of the corresponding channel it is responsible for rendering, it can play the audio signal after audio rendering at the corresponding direction position in the media playback scene; then N virtual speakers can simultaneously create a stereo surround sound effect after multi-channel rendering, that is, in the media playback scene, a stereo surround playback effect of multi-channel media audio can be achieved around the media viewing object. For example Figure 1 The media playback scene shown: birds singing in the valley in front of you, the breeze blowing overhead, the lingering sound of music behind you, the sound of the guzheng and chimes ringing together, and the synchronized audio and video bring a poetic, majestic and heavy traditional Chinese style scene, thus providing the media viewers with an immersive audio-visual experience.

[0161] In an embodiment of the present application, multi-channel media audio in a media playback scene is obtained, and the multi-channel media audio includes an audio signal obtained by collecting N channels, each channel corresponds to a different spatial position in the media playback scene, and N is an integer greater than 1; the target spatial position of a viewing object of media data in the media playback scene is obtained, and a stereo speaker system is arranged around the target spatial position in the media playback scene; the stereo speaker system includes N virtual speakers, and the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene; the multi-channel media audio is rendered by the stereo speaker system, and each virtual speaker in the stereo speaker system performs audio rendering on the audio signal collected by the adapted channel; and the multi-channel media audio after the multi-channel audio rendering is played through the stereo speaker system. It can be seen that in the process of laying out the stereo speaker system, the present application can virtually restore the spatial position of the multi-channel media audio during collection to determine the layout position of the virtual speakers, so that the actual sound environment of the media audio can be restored as much as possible in the media playback scene, thereby improving the authenticity of the audio playback effect; in addition, the present application uses N virtual speakers to render the audio signals collected by N channels separately, which can achieve the rendering effect of multi-channel spatial audio to improve the immersive experience in the media playback scene.

[0162] See also Figure 8 , Figure 8 is a flow chart of another audio processing method provided by an embodiment of the present application. The audio processing method can be performed by a computer device (such as Figure 3 Any terminal device or server as shown. Figure 8 As shown, the audio processing method includes but is not limited to the following steps S801-S807:

[0163] S801: Acquire multi-channel media audio in a media playback scene.

[0164] The multi-channel media audio includes audio signals collected through N channels, each channel corresponds to a different spatial position in the media playback scene, and N is an integer greater than 1.

[0165] In one possible implementation, the process of acquiring multi-channel media audio is as follows: acquiring a collected multi-channel audio signal source in a media playback scenario; performing frame processing on the multi-channel audio signal source to obtain K multi-channel audio frames, each multi-channel audio frame being composed of audio signals from N channels; using channel segmentation technology to perform format resampling processing on the multi-channel audio signal source to perform signal segmentation on each multi-channel audio frame according to different channels to obtain N audio data; any audio data contains K audio frames, and the K audio frames contained in one audio data belong to the same channel, and K is a positive integer; wherein the channel segmentation technology includes: any one of audio editing software, audio segmentation tools, and audio processing frameworks. It should be noted that for details of the specific processing process of multi-channel media audio in this application, please refer to Figure 4 The relevant process in step S401 in the embodiment will not be repeated here.

[0166] S802: Acquire a target spatial position of a media viewing object in a media playback scene, and determine layout information of each virtual speaker in a stereo speaker system based on the target spatial position.

[0167] Among them, the stereo speaker system includes N virtual speakers, and the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene. In addition, the adaptation here specifically includes: the layout position of the virtual speaker is consistent with the corresponding spatial position during signal collection; or, the distance between the layout position of the virtual speaker and the corresponding spatial position during signal collection is within a specified preset threshold. That is, when laying out the stereo speaker system, the present application virtually restores the speaker layout during signal collection (i.e., the spatial position in the media playback scene) as much as possible, thereby improving the authenticity and accuracy of multi-channel audio playback.

[0168] In a possible implementation, the layout information of each virtual speaker includes: any one or more of the layout position, direction, and speaker attributes, where the so-called speaker attributes include: geometric shape (such as circle, rectangle, polygon), material (such as metal, plastic), color (such as white, blue), etc. Specifically, based on the target spatial position, the layout information of each virtual speaker (such as virtual speaker b) in the stereo speaker system is determined, which specifically includes the following steps (1)-(3):

[0169] (1) Analyze the multi-channel media audio and determine the reference speaker layout of N physical speakers during the signal acquisition process. The reference speaker layout here includes: the distance and direction between each physical speaker and the reference space position in the media playback scene;

[0170] (2) Based on the target spatial position (x, y, z), and the first distance d1, the second distance d2, and the third distance d3 between the physical speaker a and the reference spatial position, calculate the candidate position (x', y', z') of the virtual speaker b to be arranged;

[0171] (3) Based on the candidate position (x', y', z') of the virtual speaker b and the direction of the physical speaker a in the media playback scene, the virtual speaker b is arranged in the media playback scene.

[0172] Based on the above steps (1)-(3), the present application can determine the layout information of each virtual speaker to be arranged. Since multi-channel media audio is obtained by collecting signals of N channels through N physical speakers in a media playback scene, it is necessary to arrange N virtual speakers, and the layout information of each virtual speaker can be determined and processed in the above manner, so that the layout information of each of the N virtual speakers can be obtained.

[0173] S803: Arrange a stereo speaker system in the media playback scene according to the layout information of each virtual speaker.

[0174] The process of determining the layout information of each virtual speaker can refer to the process shown in step S802 above, and then the layout information (such as layout position, direction, speaker attributes) of the N virtual speakers to be arranged can be determined. Optionally, the speaker attributes (such as geometric shape, material, color) of the N virtual speakers can be the same or different, for example, the N virtual speakers are all blue circular entities; for example, some virtual speakers are blue circular entities, and other virtual speakers are blue rectangular entities; for example, some virtual speakers are white circular entities, and other virtual speakers are blue rectangular entities, etc. This application does not limit the specific attributes of each virtual speaker. In actual applications, the corresponding speaker attributes can be customized as needed to meet the user's flexibility needs.

[0175] It should be noted that the detailed process of the steps shown in steps S802-S803 of the present application can refer to the relevant process in step S402 in the above embodiment, and the embodiment of the present application will not be repeated here.

[0176] The specific process of multi-channel audio rendering (steps S804-S806) is described in detail below.

[0177] S804: Analyze the multi-channel media audio and determine a channel corresponding to each audio data.

[0178] Among them, the multi-channel media audio includes N audio data. After parsing the multi-channel media audio, a channel corresponding to each audio data in the N audio data can be determined. For example, the audio format (AVAudioFormat) of each audio data in the multi-channel media audio can be parsed, and the audio format identifies the channel type of the corresponding audio data. For example, the channel types include: L (left channel), R (right channel), C (center channel), Ls (left surround channel), Rs (right surround channel), and so on.

[0179] S805: Acquire a channel layout association rule, where the channel layout association rule defines that each virtual speaker in the N virtual speakers is associated with one of the N channels.

[0180] Specifically, since one audio data in multi-channel media audio corresponds to one channel, and different channels correspond to different spatial positions in the media playback scene, and since different spatial positions correspond to the layout of corresponding virtual speakers, the channel layout association rule can define the channel to be associated with each virtual speaker. For example, the left speaker is used to associate with the left channel (L), the right speaker is used to associate with the right channel (R), and the center speaker is used to associate with the center channel (C).

[0181] S806: Based on the channel layout association rule and the channels corresponding to each audio frame, use N virtual speakers to perform audio rendering processing on the N audio data for the N channels; wherein one virtual speaker is used to render the audio data for one channel.

[0182] In a possible implementation, any one of the N virtual speakers is represented as the i-th virtual speaker, where i is a positive integer and 1≤i≤N. The computer device uses the N virtual speakers to perform audio rendering processing on the N audio data for the N channels based on the channel layout association rule and the channels corresponding to each audio frame, specifically including the following steps (1)-(2):

[0183] (1) Based on the channel layout association rule and the channels corresponding to each audio data, the target audio data to be processed by the i-th virtual speaker is obtained from the N audio data, and the target audio data includes K audio frames whose channel type is channel i. Specifically, since the channel layout association rule defines the channel associated with each virtual speaker, the target audio data to be processed by any virtual speaker (the i-th virtual speaker) can be determined from the N audio data, such as Figure 6bAs shown, assuming that N speakers include: left speaker S1', right speaker S2', center speaker S3', subwoofer speaker S4', left surround speaker S5', right surround speaker S6', left rear surround speaker S7', right rear surround speaker S8', left top center speaker S9', right top center speaker S10'; then, the left speaker S1' is used to process the audio data of the left channel (L), the right speaker S2' is used to process the audio data of the right sound (R), the center speaker S3' is used to process the audio data of the middle channel (C)... and so on, it can be determined that each virtual speaker needs to process the audio data, and one virtual speaker is used to process the audio data of one channel, as shown in the following Table 1:

[0184] Table 1. Audio data of the corresponding channels to be processed by each of the N virtual speakers

[0185]

[0186] (2) Using the i-th virtual speaker to perform audio rendering processing on the target audio data. In specific implementation, the audio rendering process of using the i-th virtual speaker to perform audio rendering processing on the target audio data is as follows:

[0187] ① Build an audio cache queue, which is used to store a preset threshold number of audio frames to be processed, P is a positive integer. It should be understood that a complete audio file, from being selected by the user to starting to play and hear the sound, mainly needs to go through complex processes such as audio and video demultiplexing, audio decoding, audio post-processing, receiving audio frames, building audio description structures, copying audio frame data, switching to the main thread to play audio (audio rendering), etc. Each audio frame needs to be rebuilt into an audio description structure and then copy the data to the PCM (Pulse Code Modulation) buffer and finally complete the audio rendering. During the audio rendering process, if the audio frame is too small, the processing frequency will increase, resulting in a large time consumption in the system; if the audio frame is too large, the total time consumption of the whole process will increase because the audio receiving frame takes too long. The above two situations are manifested in the experience as sound jamming or rustling sounds, so it is very important to reasonably set the size value of each audio frame. In practical applications, judging from statistical data and operating results, it is more reasonable to cache a certain number of original frames before processing them uniformly. Therefore, this application needs to pre-build a buffer (such as an audio cache queue) during the audio rendering process to accumulate multiple audio frames to improve processing efficiency.

[0188] ② Obtain the audio frame j to be processed from the K audio frames included in the target audio data, and add the audio frame j to the audio cache queue, where j is a positive integer and 1≤j≤K. Optionally, before adding the audio frame j (i.e., any audio frame among the K audio frames) to the audio cache queue, the following may also be included:

[0189] I. Obtain the tail audio frame at the tail position of the audio cache queue, where the so-called tail audio frame refers to the last audio frame in the audio cache queue.

[0190] II. Analyze the first attribute information of audio frame j and the second attribute information of the tail audio frame, the attribute information includes: sampling rate (i.e., the frequency sampleRate of signal sampling, such as 44100 Hz, 48000 Hz, etc.), number of bytes, audio format (such as PCM, AAC, etc.) any one or more. Therefore, here we can compare whether the audio format of audio frame j is the same as the audio format of the tail audio frame, whether the sampling rate of audio frame j is the same as the sampling rate of the tail audio frame, and whether the number of bytes of audio frame j is the same as the number of bytes of the tail audio frame.

[0191] III. If the first attribute information of audio frame j is the same as the second attribute information of the tail audio frame (it can be understood that audio frame j and the tail audio frame are two audio frames with the same attributes), the step of adding audio frame j to the audio cache queue is triggered; otherwise, if the first attribute information of audio frame j is different from the second attribute information of the tail audio frame, then audio frame j and the tail audio frame are two audio frames with different attributes, then data analysis is performed on the audio frame j to construct an audio description structure of audio frame j; and based on the audio description structure of audio frame j, the i-th virtual speaker is used to perform a separate rendering process on audio frame j. This method is to ensure that the formats or attributes of each audio frame added to the audio cache queue are the same, thereby laying the foundation for the subsequent reuse of a common audio description structure.

[0192] ③ Let j=j+1, repeat the above processes I-III until the number of audio frames stored in the audio cache queue reaches the preset threshold value P, and use the i-th virtual speaker to perform audio rendering processing on the P audio frames in the audio cache queue. Specifically, the audio rendering process performed on the P audio frames in the audio cache queue is as follows: I. Perform data analysis on any one of the P audio frames to construct an audio structure description body corresponding to the audio cache queue. II. Call the system conversion interface to perform format conversion processing on the P audio frames respectively to convert the P audio frames into a cache format. The cache format here can be understood as an audio format that the system can handle. III. Based on each audio frame in the cache format, create P encoded data frames (for example, PCM data frames). IV. Based on the audio structure description body, use the i-th virtual speaker to perform multiplexing rendering processing on the P encoded data frames.

[0193] In summary, in the process of audio rendering, multiple (such as P) audio frames can be uniformly rendered by building an audio cache queue, thereby improving rendering efficiency. Based on this, any virtual speaker is used to independently render the audio data of the corresponding channel, so that N virtual speakers can jointly achieve multi-channel audio rendering of N channels of audio data (multi-channel media audio). The following is a further description of the audio rendering process performed by any virtual speaker in conjunction with the attached figure, please refer to Fig. 9 , Fig. 9 is a flowchart of an audio rendering process provided by an embodiment of the present application. Fig. 9 As shown, the audio rendering process includes the following steps L1-L9:

[0194] L1. Receive an audio frame (such as the above-mentioned audio frame j, that is, any one of the K audio frames).

[0195] L2: Check whether the new audio frame is different from the last audio frame in the audio buffer queue (i.e., compare the attributes of audio frame j and the last audio frame to see if they are different). If yes, execute L5-L9; if no, execute L3.

[0196] L3. Add the received audio frame to the audio buffer queue.

[0197] L4: Check whether the preset number threshold of the audio buffer queue is reached. If not, execute L1; if yes, execute L5-L9.

[0198] L5. Construct an audio description structure.

[0199] Specifically, the audio description structure is used to describe at least one type of attribute information of any audio frame. The so-called audio description structure AudioStreamBasicDescription is a structure used to describe the basic attributes of an audio stream in the Core Audio framework. It defines the attribute information of audio data, including but not limited to: sampling rate, number of channels, number of bits per sample, etc. The following is an explanation and description of each field contained in the audio description structure AudioStreamBasicDescription:

[0200] 1.mSampleRate: sampling rate, which indicates the number of samples per second. Common sampling rates are 44100Hz (CD quality) and 48000Hz (professional audio and video).

[0201] 2.mFormatID: The identifier of the audio format; common formats include:

[0202] kAudioFormatLinearPCM: Linear Pulse Code Modulation (PCM), a lossless audio format.

[0203] kAudioFormatMPEG4AAC: Advanced Audio Coding (AAC), a lossy audio format.

[0204] 3.mFormatFlags: Format flags, used to describe the specific characteristics of the audio format. For example, for PCM format, you can specify whether it is a floating point number or an integer, big endian or little endian, etc.

[0205] 4.mBytesPerPacket: The number of bytes per packet. For uncompressed audio (such as PCM), this value is usually mBytesPerFrame × mFramesPerPacket.

[0206] 5.mFramesPerPacket: The number of frames per packet. For uncompressed audio (such as PCM), each packet usually contains one frame, so this value is usually 1; for compressed audio (such as AAC), this value may be 1024 or 2048.

[0207] 6.mBytesPerFrame: The number of bytes per frame. For uncompressed audio (such as PCM), this value is usually mChannelsPerFrame × (mBitsPerChannel / 8).

[0208] 7.mChannelsPerFrame: The number of channels per frame. For example, the value of mono audio is 1, the value of stereo audio is 2, and the value of N-channel audio is N.

[0209] 8.mBitsPerChannel: The number of bits per channel. For example, for 16-bit PCM audio, this value is 16.

[0210] 9.mReserved: Reserved field, usually set to 0.

[0211] Based on this, since the audio description structure can reflect the multi-dimensional attribute information of any audio frame, it provides a detailed description of the audio data. Therefore, the audio description structure is a very important data in audio programming, enabling audio processing and playback devices to correctly interpret and process audio data.

[0212] L6, build a cache format. The cache format AVAudioFormat here can be understood as an audio format that the system can recognize or process, so that the system can effectively process audio data.

[0213] L7, build the encoded data frame (such as PCM data frame). Here, you need to create AVAudioPCMBuffer, which is used to store PCM data frames that the system can recognize and process.

[0214] L8. Copy the data in the cache queue to the encoded data frame (such as a PCM data frame).

[0215] L9, audio rendering and clearing the cache queue.

[0216] Based on this, Fig. 9 As can be seen from the shown process, the embodiment of the present application constructs an audio cache queue, thereby adding P audio frames with the same format or attributes to the queue for unified rendering processing. In the rendering process of P audio frames in the audio cache queue, a common audio description structure can be reused. Only one common audio description structure needs to be constructed for the P audio frames, thereby avoiding repeated construction to improve audio rendering efficiency, and can reduce the time consumption of thread switching to improve the smoothness of audio playback.

[0217] S807: Play the multi-channel media audio after the multi-channel audio rendering process through the stereo speaker system.

[0218] Specifically, after each virtual speaker in the stereo speaker system renders the audio signal of the corresponding channel it is responsible for rendering, it can play the audio signal after audio rendering at the corresponding direction position in the media playback scene; then N virtual speakers can simultaneously create a stereo surround sound effect after multi-channel rendering, see Fig.10 , Fig.10 Schematic diagram of multi-channel rendering of a stereo speaker system provided in an embodiment of the present application. Fig.10 As shown, there are four virtual speakers (1, 2, 3, 4) arranged in the media playback scene. These four virtual speakers are responsible for rendering and processing the audio signals of their respective channels in the media playback scene, so that a stereo surround playback effect of multi-channel media audio can be achieved around the media viewing object in the media playback scene.

[0219] See also Fig.11 , Fig.11 is a schematic diagram of a media playback scenario provided by an embodiment of the present application. Fig.11 As shown, in the media playback scene, multi-channel media audio to be played is obtained (assuming that the multi-channel media audio is generated after signal collection through four channels), and the real-time spatial position of the media viewing object (such as position 1) can be located in the media playback scene. Based on the audio processing method provided in the embodiment of the present application, a stereo speaker system 1 can be arranged around position 1, that is, four virtual speakers need to be arranged. If the four virtual speakers (a, b, c, d) are respectively located on the left, right, front, and back of the media viewing object in the media playback scene; then the four virtual speakers a, b, c, d can independently render the audio signal of the left channel, the audio signal of the right channel, the audio signal of the front channel, and the audio signal of the back channel, respectively, to achieve a multi-channel audio rendering effect, so that the media viewing object can hear the sounds from the four directions of left, right, front, and back in the media playback scene, providing an immersive audio playback experience for the media viewing object. Furthermore, if the media viewing object moves from position 1 to position 2 in the media playback scene, a stereo speaker system 2 can be arranged around position 2, and the stereo speaker system 2 includes four virtual speakers: a', b', c', and d', and the layout positions of these four virtual speakers (a', b', c', and d') in the media playback scene change based on the change of the real-time position of the media viewing object, and each performs audio rendering and playback in the media playback scene based on the new position. Therefore, even if the media viewing object moves during the process of watching media, the present application can also arrange the stereo speaker system around the user (media viewing object) in real time according to the real-time position of the user, thereby providing an immersive audio playback experience for the media viewing object and dynamically changing to match the change of the user's position, so as to improve the spatial sense and stereoscopic sense of the audio playback.

[0220] In the embodiments of the present application, on the one hand, the multi-channel media audio is abstracted into virtual speakers (ModelEntity) in 3D space according to the spatial positions of the speakers in the media playback scene during signal collection, and audio rendering and playback are performed separately, thereby restoring the scene of the multi-channel sound system in the real scene, greatly improving the auditory experience of spatial audio; on the other hand, in the process of laying out the stereo speaker system, the stereo speaker system can be laid out around the media viewing object according to the real-time position of the media viewing object, and can change dynamically to match the changes in the user's position, so as to improve the spatial sense and stereoscopic sense of the audio playback, thereby providing an immersive audio playback experience for the media viewing object.

[0221] The following is an explanation of the devices related to the audio processing solution provided in the embodiments of the present application.

[0222] It should be noted that in the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0223] See also Fig.12 , Fig.12 1200 is a schematic diagram of the structure of an audio processing device provided in an embodiment of the present application. The audio processing device 1200 can be used to execute the corresponding steps in the audio processing method provided in an embodiment of the present application. In specific implementation, the audio processing device 1200 can specifically include:

[0224] The acquisition unit 1201 is used to acquire multi-channel media audio in the media playback scene, where the multi-channel media audio includes audio signals collected through N channels, each channel corresponding to a different spatial position in the media playback scene; N is an integer greater than 1;

[0225] The processing unit 1202 is used to obtain the target spatial position of the media viewing object in the media playback scene, and to arrange a stereo speaker system around the target spatial position in the media playback scene; the stereo speaker system includes N virtual speakers, and the arrangement position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene;

[0226] The processing unit 1202 is further configured to perform multi-channel audio rendering processing on the multi-channel media audio using a stereo speaker system; wherein each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel;

[0227] The processing unit 1202 is further configured to play the multi-channel media audio after the multi-channel audio rendering process through a stereo speaker system.

[0228] In a possible implementation, the multi-channel media audio includes N audio data, and one audio data is an audio signal collected by one channel; the processing unit 1202 is further configured to perform the following operations:

[0229] Obtaining the collected multi-channel audio signal source in the media playback scene;

[0230] Performing frame processing on a multi-channel audio signal source to obtain K multi-channel audio frames, each of which is composed of audio signals of N channels;

[0231] The multi-channel audio signal source is format-resampled by using the channel segmentation technology, so as to segment each multi-channel audio frame according to different channels to obtain N audio data; any audio data contains K audio frames, and the K audio frames contained in one audio data belong to the same channel, and K is a positive integer;

[0232] The channel segmentation technology includes: any one of audio editing software, audio segmentation tools, and audio processing framework.

[0233] In a possible implementation, the processing unit 1202 is further configured to perform the following operations:

[0234] Performing audio preprocessing on the multi-channel audio signal source to obtain a preprocessed multi-channel audio signal source; wherein the audio preprocessing includes: any one or more of echo cancellation processing, noise suppression processing, normalization processing, and voice activity detection;

[0235] The channel segmentation technology is used to perform format resampling processing on the preprocessed multi-channel audio signal source to obtain N audio data.

[0236] In a possible implementation, the multi-channel media audio is obtained by collecting audio signals through N physical speakers, and one physical speaker is used to collect an audio signal of one channel at a spatial position in a media playback scene;

[0237] The processing unit 1202 arranges a stereo speaker system around the target spatial position in the media playback scene to perform the following operations:

[0238] Analyze multi-channel media audio and determine a reference speaker layout of N physical speakers during signal acquisition, where the reference speaker layout includes: the distance and direction between each physical speaker and the reference spatial position in the media playback scene;

[0239] Based on the target spatial position and reference speaker layout, a stereo speaker system is arranged in the media playback scenario.

[0240] In a possible implementation, the target spatial position is represented by three-dimensional spatial coordinates (x, y, z), any one of the N physical speakers is represented as physical speaker a, a is a positive integer and 1≤a≤N; the distance between the physical speaker a and the reference spatial position includes: a first distance d1, a second distance d2, and a third distance d3;

[0241] The processing unit 1202 arranges a stereo speaker system in the media playback scene based on the target spatial position and the reference speaker arrangement, and is used to perform the following operations:

[0242] Based on the target spatial position (x, y, z), and the first distance d1, the second distance d2, and the third distance d3 between the physical speaker a and the reference spatial position, calculate the candidate position (x', y', z') of the virtual speaker b to be arranged;

[0243] Based on the candidate position (x', y', z') of the virtual speaker b and the direction of the physical speaker a in the media playback scene, the virtual speaker b is arranged in the media playback scene.

[0244] In a possible implementation, the processing unit 1202 arranges the virtual speaker b in the media playback scene based on the candidate position (x', y', z') of the virtual speaker b and the direction of the physical speaker a in the media playback scene, and performs the following operations:

[0245] Based on the candidate position (x', y', z'), determine the spatial position of the virtual speaker b to be arranged in the media playback scene; wherein the distance between the spatial position to be arranged and the candidate position is less than a preset threshold;

[0246] The direction of the physical speaker a in the media playback scene is determined as the direction of the virtual speaker b to be arranged in the media playback scene;

[0247] Setting speaker properties for the virtual speaker b, where the speaker properties include: any one or more of geometric shape, material, and color;

[0248] According to the spatial position to be laid out, the direction to be laid out, and the speaker attributes, the virtual speaker b is laid out in the media playback scene.

[0249] In a possible implementation, the processing unit 1202 is further configured to perform the following operations:

[0250] Analyze the multi-channel media audio and determine the audio type of the multi-channel media audio;

[0251] Obtaining multiple multi-channel speaker layouts corresponding to the media playback scene; any multi-channel speaker layout is used to specify the spatial layout information of N physical speakers during the signal acquisition process;

[0252] Based on the audio type, a multi-channel speaker layout matching the audio type is determined from a plurality of multi-channel speaker layouts, and the determined multi-channel speaker layout is used as a reference speaker layout.

[0253] In a possible implementation, the multi-channel media audio includes N audio data, and one audio data includes K audio frames from the same channel; the processing unit 1202 uses a stereo speaker system to perform multi-channel audio rendering processing on the multi-channel media audio, and is used to perform the following operations:

[0254] Parse multi-channel media audio and determine the channel corresponding to each audio data;

[0255] Obtain a channel layout association rule, where the channel layout association rule defines that: each virtual speaker in the N virtual speakers is associated with one of the N channels;

[0256] Based on the channel layout association rule and the channels corresponding to each audio frame, N virtual speakers are used to perform audio rendering processing on N channels of N audio data; wherein one virtual speaker is used to render the audio data of one channel.

[0257] In a possible implementation, any one of the N virtual speakers is represented as the i-th virtual speaker, where i is a positive integer and 1≤i≤N; the processing unit 1202 uses the N virtual speakers to perform audio rendering processing on the N channels of the N audio data based on the channel layout association rule and the channels corresponding to each audio frame, and is used to perform the following operations:

[0258] Based on the channel layout association rule and the channels corresponding to each audio data, the target audio data to be processed by the i-th virtual speaker is obtained from the N audio data; the target audio data includes K audio frames whose channel type is channel i;

[0259] The i-th virtual speaker is used to perform audio rendering processing on the target audio data.

[0260] In a possible implementation, the processing unit 1202 uses the i-th virtual speaker to perform audio rendering processing on the target audio data, and performs the following operations:

[0261] Construct an audio cache queue, where the audio cache queue is used to store a preset threshold number P of audio frames to be processed, where P is a positive integer;

[0262] Obtain an audio frame j to be processed from the K audio frames contained in the target audio data, and add the audio frame j to the audio cache queue; j is a positive integer and 1≤j≤K;

[0263] Let j=j+1, until the number of audio frames stored in the audio cache queue reaches a preset threshold value P, the i-th virtual speaker is used to perform audio rendering processing on the P audio frames in the audio cache queue.

[0264] In a possible implementation, the processing unit 1202 uses the i-th virtual speaker to perform audio rendering processing on the P audio frames in the audio buffer queue, and performs the following operations:

[0265] Performing data analysis on any audio frame among the P audio frames, and constructing an audio structure description body corresponding to the audio cache queue; the audio description structure body is used to describe at least one type of attribute information of any audio frame;

[0266] Calling a system conversion interface to perform format conversion processing on the P audio frames respectively, so as to convert the P audio frames into a cache format;

[0267] Based on each audio frame in the buffered format, P coded data frames are created;

[0268] Based on the audio structure description body, the i-th virtual speaker is used to perform multiplexing rendering processing on P coded data frames.

[0269] In a possible implementation, before the processing unit 1202 adds the audio frame j to the audio buffer queue, it is further configured to perform the following operations:

[0270] Get the last audio frame at the end of the audio buffer queue;

[0271] Analyze the first attribute information of the audio frame j and the second attribute information of the tail audio frame, where the attribute information includes: any one or more of sampling rate, number of bytes, and audio format;

[0272] If the first attribute information of the audio frame j is the same as the second attribute information of the tail audio frame, the step of adding the audio frame j to the audio cache queue is triggered.

[0273] In a possible implementation, the processing unit 1202 is further configured to perform the following operations:

[0274] If the first attribute information of the audio frame j is different from the second attribute information of the tail audio frame, then data analysis is performed on the audio frame j to construct an audio description structure of the audio frame j;

[0275] Based on the audio description structure of audio frame j, the i-th virtual speaker is used to render audio frame j.

[0276] In an embodiment of the present application, multi-channel media audio in a media playback scene is obtained, and the multi-channel media audio includes an audio signal obtained by collecting N channels, each channel corresponds to a different spatial position in the media playback scene, and N is an integer greater than 1; the target spatial position of a viewing object of media data in the media playback scene is obtained, and a stereo speaker system is arranged around the target spatial position in the media playback scene; the stereo speaker system includes N virtual speakers, and the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene; the multi-channel media audio is rendered by the stereo speaker system, and each virtual speaker in the stereo speaker system performs audio rendering on the audio signal collected by the adapted channel; and the multi-channel media audio after the multi-channel audio rendering is played through the stereo speaker system. It can be seen that in the process of laying out the stereo speaker system, the present application can virtually restore the spatial position of the multi-channel media audio during collection to determine the layout position of the virtual speakers, so that the actual sound environment of the media audio can be restored as much as possible in the media playback scene, thereby improving the authenticity of the audio playback effect; in addition, the present application uses N virtual speakers to render the audio signals collected by N channels separately, which can achieve the rendering effect of multi-channel spatial audio, improve the stereoscopic and spatial sense of audio playback, and thereby improve the immersive experience in the media playback scene.

[0277] See also Fig.13 , Fig.13 1300 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 1300 is used to execute the steps performed by the computer device in the aforementioned method embodiment. The computer device 1300 may include a device (for example, one or more of a server, a node, a terminal, etc.), or may include components inside the device (for example, a chip, a software module or a hardware module, etc.). The computer device may include at least one processor 1301 and a communication interface 1302. Further, optionally, the computer device may also include at least one memory 1303 and a bus 1304. In addition, the processor 1301, the communication interface 1302 and the memory 1303 are connected via a bus 1304. Wherein:

[0278] (1) The processor 1301 is a module that performs arithmetic operations and / or logical operations, and specifically may be one or a combination of multiple processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (assisting the central processing unit to complete corresponding processing and application), and a micro controller unit (MCU).

[0279] (2) The communication interface 1302 may be used to provide information input or output for at least one processor 1301. And / or, the communication interface 1302 may be used to receive data sent from the outside and / or send data to the outside, and may be a wired link interface including an Ethernet cable, etc., or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, vehicle-mounted short-range communication technology, and other short-range wireless communication technologies, etc.). The communication interface 1302 may be used as a network interface.

[0280] (3) Memory 1303 is used to provide storage space, and the storage space can store data such as operating system and computer program (computer program includes program instructions). Memory 1303 can be one or a combination of random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), portable read only memory (CD-ROM), etc.

[0281] In specific implementation, the processor 1301 runs the program instructions stored in the memory 1303 to enable the computer device to execute the method steps in the aforementioned embodiments of the present application; in addition, after the computer device executes the method steps in each embodiment through the processor, the effects achieved can be referred to the description of the aforementioned embodiments and will not be repeated here.

[0282] According to one aspect of the present application, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, and the computer program includes program instructions. When the processor executes the above program instructions, the method in the corresponding embodiment of the foregoing text can be executed, so it will not be repeated here. For technical details not disclosed in the computer-readable storage medium embodiment involved in the present application, please refer to the description of the method embodiment of the present application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located at one location, or, executed on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0283] According to one aspect of the present application, an embodiment of the present application further provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device can execute the method in the corresponding embodiment above, so it will not be repeated here. For technical details not disclosed in the computer program product embodiment involved in the present application, please refer to the description of the method embodiment of the present application.

[0284] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product, which includes one or more computer programs. When a computer program is loaded and executed on a computer device, the process or function described in the embodiment of the present application is generated in whole or in part; the computer device may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program may be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium; the computer program may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer device can access or a data processing device such as a server or data center that includes one or more available media integrated; wherein the available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0285] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An audio processing method, characterized in that: include: Obtain multi-channel media audio in a media playback scene, the multi-channel media audio comprising an audio signal collected through N channels, each of the channels corresponding to a different spatial position in the media playback scene; N is an integer greater than 1; the multi-channel media audio is obtained by collecting audio signals through N physical speakers; A physical speaker is used to collect an audio signal of one channel at a spatial position in the media playback scene; Obtaining a target spatial position of a media viewing object in the media playback scene, analyzing the multi-channel media audio, and determining a reference speaker layout of the N physical speakers during the signal acquisition process, wherein the reference speaker layout includes: a distance and direction between each of the physical speakers and a reference spatial position in the media playback scene; Based on the target spatial position and the reference speaker layout, the layout information of each virtual speaker in the stereo speaker system is determined; the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a physical speaker in the media playback scene; wherein the adaptation includes: the layout position of any one of the virtual speakers is consistent with the spatial position of the corresponding physical speaker during signal collection; or, the distance between the layout position of the virtual speaker and the spatial position of the corresponding physical speaker during signal collection is within a specified preset threshold; According to the layout information of each virtual speaker, a stereo speaker system is arranged in the media playback scene; wherein the layout of the stereo speaker system includes: arranging the stereo speaker system according to a preset unified speaker layout standard; or arranging a stereo speaker system adapted to the type of multi-channel media audio; The stereo speaker system is used to perform multi-channel audio rendering processing on the multi-channel media audio; wherein each of the virtual speakers in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel; The multi-channel media audio after the multi-channel audio rendering process is played through the stereo speaker system.

2. The method according to claim 1, characterized in that The multi-channel media audio includes N audio data, and one audio data is an audio signal collected by one channel; the method further includes: Obtaining the collected multi-channel audio signal source in the media playback scene; Performing frame processing on the multi-channel audio signal source to obtain K multi-channel audio frames, each of which is composed of audio signals of N channels; The multi-channel audio signal source is format-resampled by using a channel segmentation technology, so as to segment each multi-channel audio frame according to different channels to obtain N audio data; any audio data includes K audio frames, and the K audio frames included in one audio data belong to the same channel, where K is a positive integer; The channel segmentation technology includes: any one of audio editing software, audio segmentation tools, and audio processing framework.

3. The method according to claim 2, characterized in that The method further comprises: Performing audio preprocessing on the multi-channel audio signal source to obtain a preprocessed multi-channel audio signal source; wherein the audio preprocessing includes: any one or more of echo cancellation processing, noise suppression processing, normalization processing, and voice activity detection; The preprocessed multi-channel audio signal source is format resampled by using a channel segmentation technology to obtain N audio data.

4. The method according to claim 1, characterized in that The target spatial position is represented by three-dimensional spatial coordinates (x, y, z), and any one of the N physical speakers is represented as physical speaker a, where a is a positive integer and 1≤a≤N; the distance between the physical speaker a and the reference spatial position includes: a first distance d1, a second distance d2, and a third distance d3; the first distance d1 refers to the lateral distance between the physical speaker a and the reference spatial position, the second distance d2 refers to the longitudinal distance between the physical speaker a and the reference spatial position, and the third distance d3 refers to the spatial distance between the physical speaker a and the reference spatial position; The determining, based on the target spatial position and the reference speaker layout, layout information of each virtual speaker in the stereo speaker system includes: Based on the target spatial position (x, y, z), and the first distance d1, the second distance d2, and the third distance d3 between the physical speaker a and the reference spatial position, calculate the candidate position (x', y', z') of the virtual speaker b to be arranged; Based on the candidate position (x', y', z') of the virtual speaker b and the direction of the physical speaker a in the media playback scene, layout information of the virtual speaker b to be arranged in the stereo speaker system is determined.

5. The method according to claim 4, characterized in that The determining, based on the candidate position (x', y', z') of the virtual speaker b and the direction of the physical speaker a in the media playback scene, layout information of the virtual speaker b to be arranged in the stereo speaker system includes: Based on the candidate position (x', y', z'), determining the spatial position of the virtual speaker b to be arranged in the media playback scene; wherein the distance between the spatial position to be arranged and the candidate position is less than a preset threshold; Determine the direction of the physical speaker a in the media playback scene as the direction of the virtual speaker b to be arranged in the media playback scene; Setting speaker attributes for the virtual speaker b, wherein the speaker attributes include: any one or more of geometric shape, material, and color; According to the spatial position to be laid out, the direction to be laid out, and the speaker attributes, a virtual speaker b is laid out in the media playback scene.

6. The method according to claim 4 or 5, characterized in that The method further comprises: Analyze the multi-channel media audio to determine the audio type of the multi-channel media audio; Acquire a plurality of multi-channel speaker layouts corresponding to the media playback scene; any of the multi-channel speaker layouts is used to specify spatial layout information of N physical speakers during signal acquisition; Based on the audio type, a multi-channel speaker layout matching the audio type is determined from the multiple multi-channel speaker layouts, and the determined multi-channel speaker layout is used as a reference speaker layout.

7. The method according to claim 1, characterized in that The multi-channel media audio includes N audio data, and one audio data includes K audio frames from the same channel; The using the stereo speaker system to perform multi-channel audio rendering processing on the multi-channel media audio includes: Parsing the multi-channel media audio to determine a channel corresponding to each of the audio data; Acquire a channel layout association rule, where the channel layout association rule defines that: each virtual speaker in the N virtual speakers is associated with one of the N channels; Based on the channel layout association rule and the channels corresponding to each audio frame, N virtual speakers are used to perform audio rendering processing on the N channels of the N audio data; wherein one virtual speaker is used to render the audio data of one channel.

8. The method according to claim 7, characterized in that Any one of the N virtual speakers is represented as the i-th virtual speaker, where i is a positive integer and 1≤i≤N; the N virtual speakers are used to perform audio rendering processing of the N channels of the N audio data based on the channel layout association rule and the channels corresponding to each audio frame, including: Based on the channel layout association rule and the channels corresponding to each audio data, obtaining target audio data to be processed by the i-th virtual speaker from the N audio data; the target audio data includes K audio frames whose channel type is channel i; The target audio data is processed by using the i-th virtual speaker.

9. The method according to claim 8, characterized in that The using the i-th virtual speaker to perform audio rendering processing on the target audio data includes: Constructing an audio cache queue, wherein the audio cache queue is used to store a preset threshold number P of audio frames to be processed, where P is a positive integer; Obtaining an audio frame j to be processed from the K audio frames included in the target audio data, and adding the audio frame j to the audio cache queue; j is a positive integer and 1≤j≤K; When the number of audio frames stored in the audio cache queue reaches the preset number threshold P, the i-th virtual speaker is used to perform audio rendering processing on the P audio frames in the audio cache queue.

10. The method according to claim 9, characterized in that The using the i-th virtual speaker to perform audio rendering processing on the P audio frames in the audio cache queue includes: Performing data analysis on any one of the P audio frames to construct an audio structure description body corresponding to the audio cache queue; the audio structure description body is used to describe at least one type of attribute information of any one of the audio frames; Calling a system conversion interface to perform format conversion processing on the P audio frames respectively, so as to convert the P audio frames into a cache format; Based on each audio frame in the cache format, create P coded data frames; Based on the audio structure description body, the i-th virtual speaker is used to perform multiplexing rendering processing on the P encoded data frames.

11. The method according to claim 10, characterized in that Before adding the audio frame j to the audio buffer queue, the method further includes: Obtaining a tail audio frame at the tail position of the audio cache queue; Analyze the first attribute information of the audio frame j and the second attribute information of the tail audio frame, wherein the attribute information includes: any one or more of sampling rate, number of bytes, and audio format; If the first attribute information of the audio frame j is the same as the second attribute information of the tail audio frame, the step of adding the audio frame j to the audio cache queue is triggered.

12. The method according to claim 11, characterized in that The method further comprises: If the first attribute information of the audio frame j is different from the second attribute information of the tail audio frame, performing data analysis on the audio frame j to construct an audio structure description body of the audio frame j; Based on the audio structure description body of the audio frame j, the audio frame j is rendered using the i-th virtual speaker.

13. An audio processing device, characterized in that: include: An acquisition unit is used to acquire multi-channel media audio in a media playback scene, wherein the multi-channel media audio includes an audio signal acquired through N channels, each of the channels corresponding to a different spatial position in the media playback scene; N is an integer greater than 1; and the multi-channel media audio is acquired by acquiring audio signals through N physical speakers; A physical speaker is used to collect an audio signal of one channel at a spatial position in the media playback scene; A processing unit, configured to obtain a target spatial position of a media viewing object in the media playback scene, analyze the multi-channel media audio, and determine a reference speaker layout of the N physical speakers during signal acquisition, wherein the reference speaker layout includes: a distance and a direction between each of the physical speakers and a reference spatial position in the media playback scene; The processing unit is further used to determine the layout information of each virtual speaker in the stereo speaker system based on the target spatial position and the reference speaker layout; the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a physical speaker in the media playback scene; wherein the adaptation includes: the layout position of any one of the virtual speakers is consistent with the spatial position of the corresponding physical speaker during signal collection; or, the distance between the layout position of the virtual speaker and the spatial position of the corresponding physical speaker during signal collection is within a specified preset threshold; The processing unit is further used to arrange the stereo speaker system in the media playback scene according to the layout information of each virtual speaker; wherein the layout of the stereo speaker system includes: arranging the stereo speaker system according to a preset unified speaker layout standard; or arranging a stereo speaker system adapted to the type of the multi-channel media audio; The processing unit is further used to perform multi-channel audio rendering processing on the multi-channel media audio using the stereo speaker system; wherein each of the virtual speakers in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel respectively; The processing unit is further used to play the multi-channel media audio after the multi-channel audio rendering processing through the stereo speaker system.

14. A computer device, characterized in that: include: Memory and processor; One or more computer programs are stored in the memory; A processor, used to load the one or more computer programs to implement the audio processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio processing method according to any one of claims 1 to 12 is executed.

16. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the audio processing method according to any one of claims 1 to 12 is executed.

Images