Audio processing method and apparatus, device, medium, and product

Abstract

The present application provides an audio processing method and apparatus, a device, a medium, and a product. The method comprises: acquiring multi-channel media audio in a media playback scenario, the multi-channel media audio comprising audio signals acquired by means of N channels; acquiring a target spatial position of a media-viewing subject in the media playback scenario, and arranging a stereo speaker system around the target spatial position, wherein the stereo speaker system comprises N virtual speakers, and the arrangement position of each virtual speaker in the media playback scenario matches the spatial position of one channel in the media playback scenario; performing multi-channel audio rendering processing on the multi-channel media audio by using the stereo speaker system, wherein each virtual speaker performs audio rendering processing on audio signals acquired by the matching channel; and playing the multi-channel media audio via the stereo speaker system. The present application can achieve a rendering effect for multi-channel spatial audio, thereby enhancing the immersive experience in media playback scenarios.

Description

An audio processing method, apparatus, device, medium, and product

[0001] This application claims priority to Chinese Patent Application No. 2024111875941, filed on August 28, 2024, entitled "An Audio Processing Method, Apparatus, Device, Medium and Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to an audio processing method, apparatus, device, medium and product, specifically to an audio processing method, an audio processing apparatus, a computer device, a computer-readable storage medium and a computer program product. Background Technology

[0003] In the field of audio processing, spatial audio refers to the ability to create immersive experiences by utilizing the directionality of sound, such as immersive movie watching and music listening experiences. In media playback devices, spatial audio components (such as speakers) are used to render and play audio resources.

[0004] Currently, if the audio resource to be played is in mono format, it can only produce mono sound; if the audio resource to be played is in stereo or multi-channel format, all its channels will be mixed into a single channel before spatialization. In summary, the spatial audio component of a media playback device produces mono sound. Because there is only one channel, meaning the sound source direction is only one, the user experience is poor. Summary of the Invention

[0005] This application provides an audio processing method, apparatus, device, medium, and product that can achieve multi-channel spatial audio rendering effects to enhance the immersive experience in media playback scenarios.

[0006] On one hand, embodiments of this application provide an audio processing method, the method comprising:

[0007] Acquire multi-channel media audio from a media playback scenario. The multi-channel media audio consists of audio signals acquired through N channels, each channel corresponding to a different spatial location in the media playback scenario; N is an integer greater than 1.

[0008] Obtain the target spatial location of the media viewer in the media playback scene, and deploy a stereo sound system around the target spatial location in the media playback scene; the stereo sound system contains N virtual speakers, and the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene.

[0009] A stereo speaker system is used to perform multi-channel audio rendering processing on multi-channel media audio; wherein, each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel.

[0010] Multi-channel media audio, after multi-channel audio rendering processing, is played through a stereo speaker system.

[0011] On one hand, embodiments of this application provide an audio processing apparatus, the apparatus comprising:

[0012] The acquisition unit is used to acquire multi-channel media audio in the media playback scenario. The multi-channel media audio includes audio signals acquired through N channels, each channel corresponding to a different spatial position in the media playback scenario; N is an integer greater than 1.

[0013] The processing unit is used to obtain the target spatial position of the media viewing object in the media playback scene, and to deploy a stereo sound system around the target spatial position in the media playback scene; the stereo sound system includes N virtual speakers, and the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene.

[0014] The processing unit is also used to perform multi-channel audio rendering processing on multi-channel media audio using a stereo speaker system; wherein, each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel.

[0015] The processing unit is also used to play multi-channel media audio after multi-channel audio rendering processing through a stereo speaker system.

[0016] On one hand, embodiments of this application provide a computer device, which includes a processor and a memory; the memory stores a computer program; when the computer program is executed by the processor, it performs the above-described audio processing method.

[0017] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the aforementioned audio processing method.

[0018] On one hand, embodiments of this application provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it performs the above-described audio processing method.

[0019] In this embodiment, multi-channel media audio in a media playback scenario is acquired. This multi-channel media audio includes audio signals acquired through N channels, each channel corresponding to a different spatial location in the media playback scenario, where N is an integer greater than 1. The target spatial location of the media data's viewing object in the media playback scenario is acquired, and a stereo speaker system is deployed around the target spatial location in the media playback scenario. This stereo speaker system includes N virtual speakers, and the deployment position of each virtual speaker in the media playback scenario is adapted to the spatial location of one of the channels in the media playback scenario. It can be seen that in the process of deploying the stereo speaker system, this application can virtually restore the spatial location of the multi-channel media audio at the time of acquisition to determine the deployment position of the virtual speakers, thereby restoring the real sound environment of the media audio in the media playback scenario as much as possible and improving the realism of the audio playback effect. A stereo speaker system is used to perform multi-channel audio rendering processing on multi-channel media audio. Each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected from the corresponding channel. The multi-channel media audio after multi-channel audio rendering processing is played through the stereo speaker system. It can be seen that this application uses N virtual speakers to render the audio signals collected from N channels, which can realize the rendering effect of multi-channel spatial audio, improve the stereo and spatial sense of audio playback, and thus improve the immersive experience in the media playback scene. Attached Figure Description

[0020] Figure 1 is a schematic diagram of an audio processing scheme provided in an embodiment of this application;

[0021] Figure 2 is a structural schematic diagram of a stereo loudspeaker system provided in an embodiment of this application;

[0022] Figure 3 is a schematic diagram of the architecture of an audio processing system provided in an embodiment of this application;

[0023] Figure 4 is a flowchart illustrating an audio processing method provided in an embodiment of this application;

[0024] Figure 5a is a schematic diagram of the structure of a multi-channel audio signal source provided in an embodiment of this application;

[0025] Figure 5b is a schematic diagram of the structure of a multi-channel media audio provided in an embodiment of this application;

[0026] Figure 6a is a structural schematic diagram of a reference speaker layout provided in an embodiment of this application;

[0027] Figure 6b is a schematic diagram of the layout structure of a stereo loudspeaker layout system provided in an embodiment of this application;

[0028] Figure 7 is a schematic diagram of the multi-channel speaker layout provided in an embodiment of this application;

[0029] Figure 8 is a flowchart illustrating another audio processing method provided in an embodiment of this application;

[0030] Figure 9 is a schematic diagram of an audio rendering process provided in an embodiment of this application;

[0031] Figure 10 is a multi-channel rendering schematic diagram of a stereo speaker system provided in an embodiment of this application;

[0032] Figure 11 is a schematic diagram of a media playback scenario provided in an embodiment of this application;

[0033] Figure 12 is a schematic diagram of an audio processing device provided in an embodiment of this application;

[0034] Figure 13 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0036] This application provides an audio processing scheme applicable to multi-channel audio playback of any type of media audio acquired through multi-channel acquisition. This scheme can achieve multi-channel spatial audio rendering effects in media playback scenarios, virtually restoring the real sound environment during signal acquisition as much as possible, thereby improving the realism of the audio playback effect and enhancing the immersive experience of the media viewer. Specifically, please refer to Figure 1, which is a schematic diagram of the principle of an audio processing scheme provided in this application embodiment. The principle of the audio processing scheme provided in this application will be roughly described below with reference to Figure 1. The scheme principle mainly includes the following steps (1)-(4):

[0037] (1) Obtain the multi-channel media audio to be played in the media playback scene. The multi-channel media audio contains audio signals (such as left channel, right channel, front channel, rear channel, etc.) acquired through N (N is an integer greater than 1) channels. Each channel corresponds to a different spatial position in the media playback scene (for example, the left channel corresponds to the position to the left of the media viewing object in the media playback scene, and the right channel corresponds to the position to the right of the media viewing object in the media playback scene). The media playback scene refers to the scene in which the real spatial environment used to play multi-channel media audio is located, such as any type of three-dimensional spatial scene, such as movie playback scene, TV series playback scene, live broadcast scene, etc.

[0038] (2) Obtain the target spatial position of the media viewing object in the media playback scene; as shown in Figure 1, the media viewing object can be located at any position in the media playback scene (such as the center of the media playback scene).

[0039] (3) A stereo sound system is deployed around the target spatial location in the media playback scene. Specifically, since multi-channel media audio is obtained by acquiring signals from N channels, N virtual speakers need to be deployed to play the multi-channel media audio. Optionally, this application can determine the layout information (such as layout position, direction, etc.) of each virtual speaker to be deployed based on the target spatial location, and then deploy N virtual speakers in the media playback scene according to the layout information, thereby deploying a stereo sound system around the media playback object. Among them, the layout position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene (for example, the layout position is the same as the spatial position at the time of acquisition or the distance between the layout position and the spatial position is less than a preset threshold). That is, when deploying virtual speakers, the spatial position of the speakers at the time of signal acquisition is restored as much as possible. In this way, a stereo sound system can be deployed around the media viewing object according to the real-time position of the media viewing object.

[0040] (4) A stereo speaker system is used to perform multi-channel audio rendering processing on multi-channel media audio. Each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel. That is, each virtual speaker performs audio rendering on the audio signal under the corresponding channel, so as to play the multi-channel media audio after multi-channel audio rendering processing through the stereo speaker system to achieve the playback effect of multi-channel spatial audio. For example, in the media playback scene shown in Figure 1, after multi-channel audio rendering through the stereo speaker system, for the media viewer, it can achieve: birdsong in the valley in front of them 102, a gentle breeze overhead 101, lingering sound behind them 103, and the simultaneous sound of the guzheng and the chime bells 104, etc. This multi-directional or multi-channel audio playback effect, based on the audiovisual synchronization method, presents a picturesque and majestic media playback scene to the media viewer, and brings the user an immersive experience in the media playback scene.

[0041] In the audio processing solution provided in this application, on the one hand, during the layout of the stereo speaker system, this application can virtually restore the spatial position of multi-channel media audio during acquisition, and based on the real-time position of the media viewing object, deploy the stereo speaker system around the media viewing object, thereby restoring the real sound environment of the media audio as much as possible in the media playback scene and improving the realism of the audio playback effect; on the other hand, this application uses N virtual speakers to render the audio signals acquired from N channels separately, which can realize the rendering effect of multi-channel spatial audio, thereby improving the immersive experience in the media playback scene.

[0042] The following is a description of the key technical terms involved in this application.

[0043] I. Vocal tract.

[0044] Audio channel (also called audio track): In the field of audio processing, it refers to the transmission path of audio signals, usually used to describe the independent channels of different audio signals in an audio processing system; simply put, the independent audio signals collected or played back from different spatial locations during recording or playback are called audio channels. The number of audio channels is abbreviated as channel count, usually expressed in "XY" format, where "X" represents the number of main channels and "Y" represents the number of low-frequency effect channels (usually called LFE, low-frequency effects channels). For example, 2.0 channels: represents two channels, left and right, without a dedicated low-frequency channel (a typical stereo configuration). Another example is 5.1 channels: represents five main channels and one low-frequency effect channel. Yet another example is 7.1 channels: represents seven main channels and one low-frequency effect channel. Therefore, according to the number of channels, it can be divided into: mono and multi-channel; mono means that all audio signals are transmitted and played through a single channel. Mono is usually used in audio playback scenarios where spatial perception is not critical, such as telephone and radio broadcasts. Multichannel refers to the transmission and playback of all audio signals through two or more channels. Therefore, audio signals transmitted through multichannel have a sense of stereo and space. For example, multichannel includes: left channel (L) and right channel (R).

[0045] II. Multi-channel media audio.

[0046] Multichannel media audio, as the name suggests, refers to media audio or audio signals transmitted and played through multiple channels. It is an audio system containing multiple independent channels, suitable for complex audio playback scenarios such as film production and game sound design. Multichannel media audio allows for more detailed and precise sound positioning, typically including 5.1, 7.1, 9.1, or even more channels. In this application, multichannel media audio includes audio signals acquired through N channels. For example, multichannel media audio specifically includes N audio data, where one audio data refers to an audio signal acquired through one channel. Therefore, multichannel media audio allows audio transmission and playback through N channels (such as left channel, right channel, front and rear channels, top and bottom channels, etc.), thereby creating an immersive sense of space and stereo effect.

[0047] III. Stereo sound system and virtual loudspeakers.

[0048] A stereo speaker system refers to a speaker system capable of playing stereo or surround sound effects. Stereo sound effects refer to audio playback effects that possess a sense of depth and space, using multiple channels to surround the listener and create a more immersive audio experience, allowing the listener to perceive the sound coming from different directions. In this application, the stereo speaker system consists of N virtual speakers, which are arranged around the media viewing object. The N virtual speakers are located at different spatial positions within the media playback scene. Please refer to Figure 2, which is a structural schematic diagram of a stereo speaker system provided in an embodiment of this application. As shown in Figure 2, the stereo speaker system includes five virtual speakers: speakers 1, 2, 3, 4, and 5. Furthermore, the spatial positions of the different virtual speakers in the media playback scene are all arranged around the media viewing object. For example, speaker 1 is located to the left of the media viewing object, speaker 2 is located to the right of the media viewing object, speaker 3 is located directly in front of the media viewing object, speaker 4 is located directly behind the media viewing object, and speaker 5 is located directly above the media viewing object. In this stereo system, different virtual speakers are responsible for rendering audio signals from different channels. Since each channel corresponds to a different spatial position of a speaker in the media playback scene, the N channels shown in Figure 2 specifically include five channels: left channel (L), right channel (R), center channel (C), rear channel (B), and top channel (H). Speaker 1 is responsible for rendering the audio signal of the left channel, speaker 2 is responsible for rendering the audio signal of the right channel, and so on, speaker 5 is responsible for rendering the audio signal of the top channel. Thus, each virtual speaker renders the audio signal of its respective channel, so that the rendered audio signals come from different directions in the media playback scene, thereby creating an immersive sound experience with a sense of stereo and space played from different directions and positions.

[0049] A virtual speaker (ModelEntity) refers to a virtual speaker (i.e., a speaker that is not actually perceived by the user). A speaker is a device with audio playback capabilities, that is, a device that can emit sound. Therefore, the virtual speaker in this application refers to a virtual device with audio playback capabilities, capable of rendering and playing audio signals from the corresponding channels. The speaker attributes (geometry, material, and color) of each virtual speaker in the stereo system can be the same or different. For example, speaker 1 can be circular, and speaker 2 can be rectangular; or, all of the aforementioned speakers can be circular.

[0050] IV. RealityKit technology.

[0051] In this application, N virtual speakers can be deployed and audio rendering processing performed in a media playback scene using a rendering framework (such as the RealityKit framework). The RealityKit framework is a high-performance rendering and animation framework designed to simplify the development process of AR (Augmented Reality) applications while providing powerful tools and functions to achieve high-quality visual, auditory, and interactive experiences. In this application, RealityKit primarily possesses the following functions:

[0052] Feature 1: Custom Materials and Effects: Developers can use RealityKit to apply custom materials, textures, colors, and effects to virtual speakers, enabling them to create more realistic and personalized visual effects.

[0053] Function 2, Environmental Understanding: RealityKit can acquire data from sensors on media playback devices, such as cameras, LiDAR, and gyroscopes, to help applications understand and interact with the real-world environment. In this application, the virtual speaker, based on RealityKit technology, can detect the movement of media viewing objects in the media playback scene in real time. After the media viewing object moves, the virtual speaker's spatial position is updated in real time to adapt to the spatial position of the media viewing object, enabling audio playback according to the new spatial position. This allows for audio playback capabilities that follow the user's movement during audio playback.

[0054] In addition, the audio processing solution provided in this application requires further explanation of the following two points:

[0055] (1) Data related to audio processing in this application (e.g., multi-channel media audio, audio data, etc.). When the above embodiments of this application are applied to specific products or technologies, permission or consent from the target audience is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the region, conforming to the principles of legality, legitimacy, and necessity, and not involving the acquisition of data types prohibited or restricted by laws and regulations. In some optional embodiments, the related data involved in the embodiments of this application is obtained after separate authorization from the target audience. In addition, when obtaining separate authorization from the target audience, the purpose of the related data is explained to the target audience.

[0056] (2) It is understandable that the terms “first”, “second” and other similar terms are used to distinguish between identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between “first”, “second” and “nth”, nor is there any limitation on the quantity or execution order.

[0057] The audio processing system provided in this application will be described in detail below.

[0058] Please refer to Figure 3, which is a schematic diagram of the architecture of an audio processing system provided in an embodiment of this application. As shown in Figure 3, the architecture of the audio processing system may include at least: a server 304 and a terminal device cluster; wherein, the terminal device cluster includes: a first terminal device 301, a second terminal device 302, a third terminal device 303, and other multiple terminal devices, and any terminal device has audio playback capability; it should be understood that the number of terminal devices included in the terminal device cluster is only for example, and this embodiment of the application does not limit the number and type of terminal devices. Any terminal device in the terminal device cluster can be directly or indirectly connected to the server 304 through a network; the aforementioned network may include, but is not limited to: wired network, wireless network, wherein the wired network includes: local area network, metropolitan area network, and wide area network, and the wireless network includes: Bluetooth, WIFI (Wireless Fidelity, a standard wireless local area network), and other networks that implement wireless communication.

[0059] In the audio processing system shown in Figure 3, the terminal devices in the terminal device cluster can be: mobile phones, tablets, laptops, desktop computers, gaming devices, in-vehicle devices, aircraft, wearable devices (such as smartwatches, smart bracelets, pedometers, etc.), smart home appliances, XR (Extended Reality) devices, such as virtual reality devices (e.g., VR (Virtual Reality) devices, AR (Augmented Reality) devices), or MR (Mixed Reality) devices, etc. It is understood that the types of terminal devices in the terminal device cluster can be the same or different. For example, the first terminal device 301 can be a laptop, the second terminal device 302 can be a desktop computer, and the third terminal device 303 can be a tablet. This application does not limit the number or type of terminal devices in the terminal device cluster. A 304 error can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0060] The following uses any terminal device in the audio processing system (e.g., the first terminal device 301) as an example to illustrate the data interaction process between the first terminal device 301 and the server 304 in a media playback scenario:

[0061] ① In a media playback scenario (such as a movie viewing scenario or a game viewing scenario), the first terminal device 301 can acquire the multi-channel audio signal source to be processed in the media playback scenario and send the multi-channel audio signal source to the server 304. The multi-channel audio signal source contains K multi-channel audio frames, each composed of audio signals acquired from N channels. Each channel corresponds to a different spatial location in the media playback scenario, where N is an integer greater than 1 and K is a positive integer.

[0062] ② The server 304 performs format resampling processing (e.g., channel segmentation technology) on the multi-channel audio signal source to divide each multi-channel audio frame into signals according to different channels, obtain the multi-channel media audio to be played, and send the multi-channel media audio to the first terminal device 301.

[0063] ③ The first terminal device 301 obtains the target spatial position of the media viewing object in the media playback scene, for example, the target spatial position is the three-dimensional spatial coordinates of the media viewing object.

[0064] ④ The first terminal device 301 deploys a stereo sound system around the target spatial location in the media playback scene based on the target spatial location. This stereo sound system includes N virtual speakers, each virtual speaker's placement in the media playback scene is adapted to the spatial location of a channel in the media playback scene; that is, when deploying the virtual speakers, the spatial location of the speakers at the time of signal acquisition is reproduced as accurately as possible. In this way, the stereo sound system can be deployed around the media viewing object according to the real-time location of the media viewing object.

[0065] ⑤ The first terminal device 301 uses a stereo speaker system to perform multi-channel audio rendering processing on multi-channel media audio, and plays the multi-channel media audio after multi-channel audio rendering processing through the stereo speaker system. Each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected from its corresponding channel, so that each virtual speaker can independently handle the rendering processing of the audio signal in its corresponding direction, collectively creating an immersive audio playback effect for the media viewer.

[0066] It should be noted that the above audio processing procedure is for illustrative purposes only and does not limit the specific execution process of the terminal device and the server. Optionally, the terminal device can send the target spatial location and multi-channel media audio to the server, which then determines the placement of each virtual speaker in the stereo system to be deployed. The server then sends the specific placement of each virtual speaker in the stereo system within the media playback scene to the terminal device, which then executes the placement process of the stereo system within the media playback scene. Alternatively, the complete audio processing flow described above can be executed independently by any terminal device or server within the audio processing system.

[0067] In one possible implementation, the audio processing system provided in this application embodiment can be deployed on a blockchain node. For example, the first terminal device 301, the second terminal device 302, the third terminal device 303, and the server 304 can all be treated as blockchain node devices, jointly forming a blockchain network. Therefore, the audio processing flow executed in this application can be executed on the blockchain, which can ensure the fairness and impartiality of the audio processing flow, make the above flow traceable, and ensure data security during the audio processing, thereby improving the security and reliability of the entire audio processing flow.

[0068] The audio processing system provided in this application is used to execute the aforementioned audio processing scheme. On the one hand, in the process of setting up the stereo speaker system, this application can virtually restore the spatial position of multi-channel media audio during acquisition, and based on the real-time position of the media viewing object, deploy a stereo speaker system around the media viewing object, thereby restoring the real sound environment of the media audio as much as possible in the media playback scene and improving the realism of the audio playback effect. On the other hand, this application uses N virtual speakers to render the audio signals acquired from N channels separately, which can realize the rendering effect of multi-channel spatial audio, thereby improving the immersive experience in the media playback scene.

[0069] It is understood that the audio processing system described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0070] The specific embodiments of the audio processing scheme of this application will be described in detail below with reference to the accompanying drawings.

[0071] Please refer to Figure 4, which is a flowchart illustrating an audio processing method provided in an embodiment of this application. This audio processing method can be executed by a computer device (any terminal device or server shown in Figure 3). As shown in Figure 4, the audio processing method includes, but is not limited to, the following steps S401-S404:

[0072] S401: Obtain multi-channel media audio from a media playback scenario.

[0073] Multi-channel media audio comprises audio signals acquired through N channels, each channel corresponding to a different spatial location in the media playback scene, where N is an integer greater than 1. It should be understood that the audio signals from different channels are acquired simultaneously during the signal acquisition process; that is, within the same time period, the signal length of the audio signals acquired for each channel is the same. For example, during signal acquisition using the left and right channels, within one minute, a one-minute-long left channel audio signal can be acquired, and simultaneously, a one-minute-long right channel audio signal can be acquired.

[0074] Optionally, the multi-channel media audio acquired through N channels needs to undergo further signal processing. This signal processing includes, but is not limited to, at least one of: resampling processing and audio preprocessing. Based on this, the signal processing procedure for multi-channel media audio is described in detail below:

[0075] 1. Resample the acquired multi-channel audio signal source.

[0076] In one possible implementation, the multi-channel media audio contains N audio data. Specifically, the acquisition process of the multi-channel media audio includes the following steps (1)-(3):

[0077] (1) Acquire the acquired multi-channel audio signal source in the media playback scenario. It should be understood that the multi-channel audio signal source is generated by acquiring signals from N channels. Each channel corresponds to a different spatial position in the media playback scenario. Therefore, the acquired multi-channel audio signal source is a multi-channel mixed audio signal, that is, each audio frame contains audio signals from multiple channels.

[0078] (2) The multi-channel audio signal source is segmented into frames to obtain K multi-channel audio frames. Each multi-channel audio frame consists of audio signals from N channels, where K is a positive integer. Specifically, the multi-channel audio signal source can be segmented into frames according to a preset frame length (e.g., 2.5ms to 60ms) to obtain K multi-channel audio frames. Please refer to Figure 5a, which is a schematic diagram of the structure of a multi-channel audio signal source provided in an embodiment of this application. As shown in Figure 5a, assume that the N channels include five channels: L (Left), R (Right), C (Center), Ls (Left surround), and Rs (Right surround). Since the audio signals of the N channels are acquired simultaneously during the signal acquisition process, each multi-channel audio frame contains audio signals of the N channels. For example, multi-channel audio frame 1 (Frame1) contains audio signals of the five channels: L, R, C, Ls, and Rs; and multi-channel audio frame 2 (Frame2) also contains audio signals of the five channels: L, R, C, Ls, and Rs.

[0079] (3) Channel segmentation technology is used to perform format resampling processing on the multi-channel audio signal source to divide each multi-channel audio frame into N audio data according to different channels; each audio data contains K audio frames, and the K audio frames contained in an audio data all belong to the same channel, where K is a positive integer; among them, channel segmentation technology includes any one of the following: audio editing software, audio segmentation tools, and audio processing framework. Specifically, ① audio editing software can be, for example, the open-source audio editor Audacity, which can be used to split the multi-channel audio signal source into multiple audio data of a single channel; ② audio segmentation tools refer to tools that have channel segmentation function for audio signals (such as the FFmpeg command), which can be used to split the multi-channel audio signal source into multiple audio data of N channels; ③ the audio processing framework can be the Librosa framework, which is a Python library for audio analysis and can also realize the function of channel segmentation of audio signals. It should be understood that in practical applications, appropriate channel segmentation techniques can be selected according to specific scenario requirements to improve the segmentation effect.

[0080] Specifically, because the interleaved audio signal combination method shown in Figure 5a makes it inconvenient to extract the audio data of each channel individually, it is necessary to resample the audio frame format and convert it into a planar combination method, that is, to store the audio signals of each channel independently. Based on this, please refer to Figure 5b, which is a schematic diagram of the structure of a multi-channel media audio provided in an embodiment of this application. As shown in Figure 5b, after resampling the multi-channel audio signal source using the above-mentioned channel segmentation technology, the audio data of each channel can be easily obtained, thereby enabling each channel to be rendered and played without interference.

[0081] The process shown in steps (1)-(3) above can separate and store the audio signals of N channels by performing format resampling on the multi-channel audio signal source, so that the audio signals of each channel can be processed independently in the future.

[0082] 2. Perform audio preprocessing on the acquired multi-channel audio signal sources.

[0083] In one possible implementation, the multi-channel audio signal source undergoes audio preprocessing to obtain a preprocessed multi-channel audio signal source. Channel segmentation technology is then used to perform format resampling on the preprocessed multi-channel audio signal source to obtain N audio data (the specific process of format resampling is described in the above flowchart). The audio preprocessing includes one or more of the following: echo cancellation, noise suppression, normalization, and voice activity detection. Specifically, ① Echo Cancellation: This is a technique used to eliminate echoes. An echo occurs when the sound played by a loudspeaker is picked up again by a microphone and transmitted back to the other party; ② Noise Suppression: This is a technique used to reduce background noise or environmental noise, often used to improve the quality of voice communication; ③ Voice Activity Detection (VAD): This is a technique used to detect the presence of active audio signals in multi-channel media audio to improve the efficiency of audio processing. The above audio preprocessing method can optimize the acquired multi-channel audio signal source in advance, thereby improving the audio quality of the multi-channel media audio to be played, so as to enhance the audio playback effect.

[0084] S402: Obtain the target spatial location of the media viewing object in the media playback scene, and deploy a stereo speaker system around the target spatial location in the media playback scene.

[0085] It should be understood that since multi-channel media audio is generated by acquiring signals from N channels, each channel corresponds to a different spatial location in the media playback scene. Therefore, multi-channel media audio is obtained by acquiring signals from N physical speakers located at N spatial locations in the media playback scene. Consequently, a stereo system needs to deploy N virtual speakers to virtually recreate the speaker layout during signal acquisition. That is, the stereo system contains N virtual speakers, and the layout position of each virtual speaker in the media playback scene matches the spatial position of one of the channels in the media playback scene. This matching includes: the virtual speaker's layout position being consistent with the corresponding spatial position during signal acquisition; or, the distance between the virtual speaker's layout position and the corresponding spatial position during signal acquisition being within a specified preset threshold. In other words, this application, when deploying the stereo system, can virtually recreate the speaker layout (i.e., the spatial position in the media playback scene) during signal acquisition as much as possible, thereby improving the realism and accuracy of multi-channel audio playback.

[0086] In one possible implementation, the target spatial position of the media viewing object in the media playback scene can be obtained in two ways: Method 1: The media viewing object in the media playback scene can be detected by a sensor to obtain its target spatial position; Method 2: The media viewing object in the media playback scene can be detected by an infrared detector, and its target spatial position can be automatically obtained. Here, the target spatial position can be represented by three-dimensional coordinates (x, y, z). It should be understood that there can be one or more media viewing objects in the media playback scene. If there are multiple media viewing objects, it is necessary to obtain the target spatial position of each media viewing object in the media playback scene separately. For ease of explanation, this application uses one media viewing object as an example.

[0087] In specific implementation, the process of deploying a stereo speaker system around the target spatial location in the media playback scene is as follows: (1) Based on the target spatial location, determine the layout information of each virtual speaker in the stereo speaker system. The layout information here includes, for example, one or more of the following: layout position, direction, and speaker attributes; (2) Deploy the stereo speaker system in the media playback scene according to the layout information of each virtual speaker (such as layout position and direction). It should be understood that the stereo speaker system deployed in this application is a virtual reality system (i.e., VR system) in a media playback scene. A VR system is a computer simulation system that can create and experience a virtual world. The VR system generates a simulated environment in the media playback scene through VR technology, providing users (such as media viewers) with simulations of visual, auditory, and tactile senses. Thus, users can observe things in three-dimensional space in a timely and unrestricted manner in the media playback scene, thereby bringing users a realistic and immersive experience in the media playback scene. Although the stereo speaker system is a virtual reality system, it has the ability to render and play audio for multi-channel media audio (including audio signals from multiple channels).

[0088] In this application, a stereo speaker system can be deployed around the target spatial location in a media playback scenario. The deployment methods for this stereo speaker system mainly include the following two: 1. Deploying the stereo speaker system according to a preset unified speaker layout standard; 2. Deploying a stereo speaker system adapted to the type of multi-channel media audio. The two different deployment methods for the stereo speaker system are described in detail below:

[0089] Method 1: Arrange the stereo speaker system according to the preset unified speaker layout standard.

[0090] In one possible implementation, multi-channel media audio is obtained by acquiring audio signals through N physical loudspeakers; one physical loudspeaker is used to acquire the audio signal of one channel at a spatial location in the media playback scene. Specifically, a stereo speaker system is arranged around the target spatial location in the media playback scene, including the following steps (1)-(2):

[0091] (1) Analyze the multi-channel media audio to determine the reference speaker layout of N physical speakers during the signal acquisition process. The reference speaker layout includes the distance and direction between each physical speaker and the reference spatial position in the media playback scene. Specifically, during the signal acquisition process, a reference spatial position needs to be set first. Then, with this reference spatial position as the center, the spatial positions and layout directions of N physical speakers for signal acquisition are set around this reference spatial position. Please refer to Figure 6a, which is a schematic diagram of a reference speaker layout provided in an embodiment of this application. As shown in Figure 6a, the reference spatial position can be the spatial position of a spatial point S0 in 3D (three-dimensional) space, and the multi-channel media audio is obtained by acquiring signals from the N physical speakers arranged around this spatial point S1. Assume a reference speaker layout is used to indicate signal acquisition using 10 physical speakers. The spatial positions of these 10 speakers during signal acquisition are shown in Figure 6a: S1, S2, S3...S10. Here, S1 is the left speaker, S2 is the right speaker, S3 is the center speaker, S4 is the subwoofer speaker, S5 is the left surround speaker, S6 is the right surround speaker, S7 is the left rear surround speaker, S8 is the right rear surround speaker, S9 is the left top center speaker, and S10 is the right top center speaker. Furthermore, the distance and direction between each physical speaker and the reference spatial position (S0) can be determined according to the reference speaker layout. For example, the distance between the left speaker S1 and the reference spatial position (S0) is D1, and the direction is the first direction. Thus, the left speaker S1 is located at a spatial position 30° away from the upper left of the reference spatial position S0, at a distance of D1. Similarly, the right speaker S2 is located at a spatial position 30° away from the upper right of the reference spatial position S0, at a distance of D1. Based on this, after acquiring the signal according to the reference preset layout shown in Figure 6a, the multi-channel media audio can be obtained. Then, after analyzing the multi-channel media audio, the detailed layout of each physical speaker during the signal acquisition of the multi-channel media audio can be restored (reference speaker layout).

[0092] (2) Based on the target spatial location and reference speaker layout, a stereo speaker system is deployed in the media playback scenario. The target spatial location can be represented by three-dimensional spatial coordinates (x, y, z). Any one of the N physical speakers is represented as physical speaker a, where a is a positive integer and 1 ≤ a ≤ N. The distance between physical speaker a and the reference spatial location includes: a first distance d1, a second distance d2, and a third distance d3. It should be noted that the speaker layout can be based on three-dimensional spatial coordinates. Here, the first distance refers to the lateral distance (x-axis distance) between physical speaker a and the reference spatial location S0, the second distance refers to the longitudinal distance (y-axis distance) between physical speaker a and the reference spatial location S0, and the third distance refers to the spatial distance (z-axis distance) between physical speaker a and the reference spatial location S0. The process of deploying any virtual speaker is as follows: ①-②:

[0093] ① Based on the target spatial position (x, y, z) and the first distance d1, second distance d2, and third distance d3 between the physical speaker a and the reference spatial position, calculate the candidate positions (x', y', z') of the virtual speaker b to be placed. In specific implementation, according to the reference speaker layout standard shown in Figure 6a, if the target spatial position and the distance between the virtual speaker to be placed and the target spatial position are known, then the three-dimensional layout coordinates (x', y', z') of each virtual speaker to be placed can be determined.

[0094] Please refer to Figure 6b, which is a schematic diagram of the layout structure of a stereo speaker layout system provided in an embodiment of this application. As shown in Figure 6b, taking the 10-channel speaker layout shown in Figure 6a as an example (i.e., 10 physical speakers), the coordinates of the target spatial position S0' are represented as (x, y, z). Then, according to the distance between each virtual speaker and the target spatial position and the reference speaker layout, the three-dimensional spatial coordinates of each virtual speaker to be laid out can be calculated as follows: Left speaker S1': (x-d1, y, z-d2), Right speaker S2': (x+d1, y, z-d2), Center speaker S3': (x, y, z-d2), d2) Subwoofer Speaker S4': (x-d1 / 2, y, z-d2), Left Surround Speaker S5': (x-d3, y, z), Right Surround Speaker S6': (x+d3, y, z), Left Rear Surround Speaker S7': (x-d5, y, z+d4), Right Rear Surround Speaker S8': (x+d5, y, z+d4), Left Top Center Speaker S9': (x-d7, y+y1, z-d6), Right Top Center Speaker S10': (x+d7, y, z-d6).

[0095] ② Based on the candidate positions (x', y', z') of the virtual speaker b and the orientation of the physical speaker a in the media playback scene, the virtual speaker b is positioned within the media playback scene. Optionally, positioning a virtual speaker b requires the following layout information: spatial position, orientation, and speaker attributes (such as geometric shape or material attributes). The process for determining the layout information of the virtual speaker is as follows:

[0096] I. Spatial Location. The spatial location of the virtual speaker b in the media playback scenario includes the following two cases: Case 1: The candidate position (x', y', z') of the virtual speaker b can be directly used as the spatial location of the virtual speaker b to be placed in the media playback scenario; Case 2: The virtual speaker b can be placed within a certain range around the candidate position (x', y', z'), that is, the spatial location of the virtual speaker b to be placed in the media playback scenario is determined based on the candidate position (x', y', z'); wherein, the distance between the spatial location to be placed and the candidate position is less than a preset threshold.

[0097] II. Orientation. The orientation of the physical speaker a in the media playback scene can be determined as the orientation of the virtual speaker b to be placed in the media playback scene, that is, the speaker placement orientation remains unchanged.

[0098] III. Speaker Properties. You can set speaker properties for the virtual speaker b. The speaker properties here include one or more of the following: geometry (such as any shape like a circle, rectangle, ellipse, etc.) and material (such as metal, plastic, etc.).

[0099] Based on this, a virtual speaker b can be placed in the media playback scene according to the spatial location, orientation, and speaker attributes described above. For example, RealityKit technology can be used to place the virtual speaker b in the media playback scene. Following this method, N virtual speakers can be placed in the media playback scene to form a stereo sound system (wherein, the shape and material attributes of the N virtual speakers can be the same or different). Since each virtual speaker can be adapted to the preset speaker layout standard shown in Figure 6a, the placement position of each virtual speaker in the media playback scene matches the spatial position of a channel in the media playback scene, thus restoring the speaker layout standard during signal acquisition as closely as possible.

[0100] Based on steps (1)-(2) shown in Method 1, this application can virtually restore the speaker layout according to the real-time position of the media viewing object based on the preset layout standard of each physical speaker during signal acquisition, and then arrange a stereo sound system around the media viewing object to restore the layout standard in the real sound environment as much as possible, which can improve the authenticity and accuracy of the subsequent audio playback effect.

[0101] Method 2: Layout a stereo speaker system that is compatible with the type of multi-channel media audio.

[0102] In one possible implementation, the specific process of a stereo speaker system adapted to the type of multi-channel media audio is as follows (1)-(4):

[0103] (1) Analyze the multi-channel media audio to determine the audio type of the multi-channel media audio. For example, the audio types of multi-channel media audio include: Type 1, Type 2, and Type 3. Type 1 refers to the type corresponding to film media, Type 2 refers to the type corresponding to game media, and Type 3 refers to the type corresponding to pure audio media.

[0104] (2) Obtain the layout of multiple multi-channel speakers in the media playback scenario; any one multi-channel speaker layout is used to specify the spatial layout information of N physical speakers during the signal acquisition process. For example, this application can pre-set multiple speaker layout standards. Please refer to Figure 7, which is a schematic diagram of the multi-channel speaker layout provided in the embodiment of this application. As shown in Figure 7, assuming a four-channel speaker layout system (i.e., containing four speakers), this application provides three sets of multi-channel speaker layout standards: speaker layout standard 1, speaker layout standard 2, and speaker layout standard 3. Each set of speaker layout standards includes the layout positions and orientations of N (four) physical speakers used for signal acquisition. For example, speaker layout standard 1 includes: four speakers S1, S2, S3, and S4 arranged in the upper left, upper right, lower left, and lower right; speaker layout standard 2 includes: four speakers S1, S2, S3, and S4 arranged in the left, right, upper, and lower; speaker layout standard 3 includes: four speakers S1, S2, S3, and S4 arranged in the left, right, upper left, and upper right. It is evident that the layout positions and orientations of each speaker are defined differently in different speaker layout standards.

[0105] (3) Based on the audio type, determine the multi-channel speaker layout that matches the audio type from multiple multi-channel speaker layouts, and use the determined multi-channel speaker layout as the reference speaker layout. For example, if the audio type is type 1, then the speaker layout standard 1 mentioned above is used as the reference speaker layout; if the audio type is type 2, then the speaker layout standard 2 mentioned above is used as the reference speaker layout; and if the audio type is type 3, then the speaker layout standard 3 mentioned above is used as the reference speaker layout.

[0106] (4) Based on the target spatial location and reference speaker layout, deploy a stereo speaker system in the media playback scenario. Specifically, the detailed execution process of this step can be found in step (2) of Method 1 above, and will not be repeated here.

[0107] Based on steps (1)-(4) shown in Method 2, this application can select the most suitable speaker layout standard from multiple speaker layout standards as a reference speaker layout according to the type of multi-channel media audio, and virtually restore the speaker layout according to the real-time position of the media viewing object. This method can further improve the accuracy of the speaker layout to achieve a more realistic speaker layout effect.

[0108] Based on this, using RealityKit technology, each physical speaker can be abstracted into a ModelEntity (i.e., a virtual speaker) based on the multi-channel media audio collected by N physical speakers. This virtual speaker ModelEntity can not only serve as the sound source in the media playback scene, but its spatial position can also be determined by setting its 3D coordinates. Furthermore, the speaker's geometric shape (mesh), material, and color can be customized. For example, physical speakers can be abstracted into virtual speakers shaped like blue spheres, and after spatial arrangement according to the aforementioned placement algorithm, a virtual 5.1 spatial sound system (stereo speaker system) can be formed. Each ModelEntity virtual speaker is only responsible for rendering the audio signal in its corresponding direction without affecting the rendering of other speakers, collectively creating a more immersive spatial listening experience.

[0109] S403: Employs a stereo speaker system to perform multi-channel audio rendering processing on multi-channel media audio.

[0110] Multichannel audio rendering processing refers to the process of rendering multichannel media audio collected from N channels using N virtual speakers pre-positioned in a stereo speaker system. Each virtual speaker is used to render the audio signal collected from one channel; for example, the left speaker renders the audio signal collected from the left channel, and the right speaker renders the audio signal collected from the right channel. The left and right channels are determined relative to the position of the media viewer; the left channel corresponds to the spatial position to the left of the current media viewer, and similarly, the right channel corresponds to the spatial position to the right of the current media viewer. Furthermore, rendering processing includes, but is not limited to, sound effects processing, volume control, and digital-to-analog conversion (such as converting digital signals to analog signals). Only after the multichannel media audio has been rendered can it be played through the stereo speaker system.

[0111] In a stereo system, each virtual speaker performs audio rendering processing on the audio signal acquired from its corresponding channel. Specifically, multi-channel media audio includes N audio data points, each audio data point being an audio signal acquired from one channel; that is, each audio data point corresponds to one channel. Therefore, each virtual speaker in the stereo system can perform audio rendering processing on the audio signal of one channel (i.e., one audio data point). For example, the left speaker renders the audio signal of the left channel (L), the right speaker renders the audio signal of the right channel (R), the center speaker renders the audio signal of the center channel (C), and so on. The N virtual speakers in the stereo system independently render the audio signal of their respective channels without interference, while simultaneously achieving the multi-channel audio rendering effect.

[0112] In one possible implementation, the process of using a stereo speaker system (i.e., N virtual speakers) to perform multi-channel audio rendering on multi-channel media audio is as follows: (1) parse the multi-channel media audio and determine the channel corresponding to each audio data; (2) obtain the channel layout association rule, which defines that each of the N virtual speakers is associated with one channel of the N channels; (3) based on the channel layout association rule and the channel corresponding to each audio frame, use N virtual speakers to perform N-channel audio rendering on the N audio data; wherein, one virtual speaker is used to render the audio data of one channel. In this implementation, each virtual speaker performs independent rendering on the audio signal of its own channel, without interfering with each other, while still achieving the multi-channel audio rendering effect together.

[0113] S404: Plays multi-channel media audio after multi-channel audio rendering processing through a stereo speaker system.

[0114] Specifically, each virtual speaker processes the audio signal of its assigned channel and then plays the processed audio signal at the corresponding directional position in the media playback scene. Thus, N virtual speakers can simultaneously create a multi-channel (N-channel) rendered stereo surround sound effect, achieving a stereo surround sound effect of multi-channel media audio playing around the media viewer. For example, in the media playback scene shown in Figure 1: birdsong in the valley, a gentle breeze overhead, lingering sounds behind, the harmonious sounds of the guzheng and chimes—the synchronized audiovisual experience creates a picturesque and majestic traditional Chinese scene, providing the media viewer with an immersive audiovisual experience.

[0115] It should be understood that, since stereo sound systems are arranged around the media viewing object, the placement of virtual speakers is determined as much as possible by virtually recreating the spatial position of the multi-channel media audio during acquisition. For example, the left channel audio signal acquired through the left channel in multi-channel media audio will be placed as the virtual speaker for the left channel, and the placement of this virtual speaker for the left channel will match the spatial position of the left channel in the media playback scene. Similarly, the right channel audio signal acquired through the right channel in multi-channel media audio will be placed as the virtual speaker for the right channel, and the placement of this virtual speaker for the right channel will match the spatial position of the right channel in the media playback scene, and so on. In this way, the real sound environment of the media audio can be reproduced as much as possible in the media playback scene, improving the realism of the audio playback effect.

[0116] This application provides an audio processing method that aims to recreate the spatial location of the speakers during signal acquisition as much as possible when arranging a stereo speaker system, thereby achieving a multi-channel spatial audio rendering effect and enhancing the immersive experience in media playback scenarios. On one hand, during the arrangement of the stereo speaker system, this application can virtually recreate the spatial location of the multi-channel media audio during acquisition, and based on the real-time location of the media viewing object, arrange the stereo speaker system around the media viewing object, thus recreating the real sound environment of the media audio as much as possible in the media playback scenario and improving the realism of the audio playback effect. On the other hand, this application uses N virtual speakers to render the audio signals acquired from N channels individually, achieving a multi-channel spatial audio rendering effect and enhancing the immersive experience in media playback scenarios.

[0117] Please refer to Figure 8, which is a flowchart illustrating another audio processing method provided in an embodiment of this application. This audio processing method can be executed by a computer device (any terminal device or server shown in Figure 3). As shown in Figure 8, this audio processing method includes, but is not limited to, the following steps S801-S807:

[0118] S801: Acquire multi-channel media audio in a media playback scenario.

[0119] Among them, multi-channel media audio includes audio signals acquired through N channels, each channel corresponding to a different spatial position in the media playback scene, where N is an integer greater than 1.

[0120] In one possible implementation, the acquisition process of multi-channel media audio is as follows: A multi-channel audio signal source is acquired in the media playback scenario; the multi-channel audio signal source is processed into frames to obtain K multi-channel audio frames, each composed of audio signals from N channels; channel segmentation technology is used to resample the multi-channel audio signal source to divide each multi-channel audio frame into signals according to different channels, resulting in N audio data; each audio data contains K audio frames, and the K audio frames contained in one audio data all belong to the same channel, where K is a positive integer; wherein, the channel segmentation technology includes any one of: audio editing software, audio segmentation tools, or audio processing frameworks. It should be noted that the detailed processing procedure for multi-channel media audio in this application is detailed in step S401 of the embodiment in Figure 4, and will not be repeated here.

[0121] S802: Obtain the target spatial position of the media viewing object in the media playback scene, and determine the layout information of each virtual speaker in the stereo system based on the target spatial position.

[0122] The stereo system comprises N virtual speakers, each positioned in a location corresponding to a channel within the media playback scenario. Specifically, this matching includes: the virtual speaker's location matching its corresponding spatial location during signal acquisition; or, the distance between the virtual speaker's location and its corresponding spatial location during signal acquisition being within a specified preset threshold. In other words, this application aims to virtually recreate the speaker layout (i.e., its spatial location in the media playback scenario) during signal acquisition as much as possible when arranging the stereo system, thereby improving the realism and accuracy of multi-channel audio playback.

[0123] In one possible implementation, the layout information of each virtual speaker includes one or more of the following: layout position, orientation, and speaker attributes. The speaker attributes include: geometric shape (e.g., circle, rectangle, polygon), material (e.g., metal, plastic), color (e.g., white, blue), etc. Specifically, based on the target spatial location, the layout information of each virtual speaker (e.g., virtual speaker b) in the stereo system is determined, which specifically includes the following steps (1)-(3):

[0124] (1) Analyze the multi-channel media audio and determine the reference speaker layout of N physical speakers during the signal acquisition process. The reference speaker layout here includes: the distance and direction between each physical speaker and the reference spatial position in the media playback scene.

[0125] (2) Based on the target spatial position (x, y, z) and the first distance d1, the second distance d2 and the third distance d3 between the physical speaker a and the reference spatial position, calculate the candidate position (x', y', z') of the virtual speaker b to be laid out;

[0126] (3) Based on the candidate positions (x', y', z') of the virtual speaker b and the orientation of the physical speaker a in the media playback scene, the virtual speaker b is deployed in the media playback scene.

[0127] Based on the above steps (1)-(3), this application can determine the layout information of each virtual speaker to be laid out. Since multi-channel media audio is obtained by collecting signals from N channels through N physical speakers in a media playback scenario, it is necessary to lay out N virtual speakers, and the layout information of each virtual speaker can be determined and processed in the above manner, thereby obtaining the layout information of each of the N virtual speakers.

[0128] S803: Based on the layout information of each virtual speaker, deploy the stereo speaker system in the media playback scene.

[0129] The process of determining the layout information of each virtual speaker can refer to the process shown in step S802 above, thereby determining the layout information (such as layout position, orientation, and speaker attributes) of the N virtual speakers to be laid out. Optionally, the speaker attributes (such as geometry, material, and color) of the N virtual speakers can be the same or different. For example, all N virtual speakers may be blue circular entities; another example is that some virtual speakers are blue circular entities, and another example is that some virtual speakers are blue rectangular entities; yet another example is that some virtual speakers are white circular entities, and another example is that some virtual speakers are blue rectangular entities, and so on. This application does not limit the specific attributes of each virtual speaker. In practical applications, the corresponding speaker attributes can be customized according to requirements to meet the user's flexibility needs.

[0130] It should be noted that the detailed process of steps S802-S803 in this application can be referred to the relevant process in step S402 of the above embodiment, and will not be repeated here.

[0131] The following section provides a detailed explanation of the specific process of multi-channel audio rendering (steps S804-S806).

[0132] S804: Analyzes multi-channel media audio and determines the channel corresponding to each audio data.

[0133] Multichannel media audio comprises N audio data points. By parsing the multichannel media audio data, the channel corresponding to each of the N audio data points can be determined. For example, the audio format (AVAudioFormat) of each audio data point in the multichannel media audio can be parsed. This audio format identifies the channel type of the corresponding audio data, such as: L (left channel), R (right channel), C (center channel), Ls (left surround channel), Rs (right surround channel), etc.

[0134] S805: Obtain the channel layout association rule, which defines that each of the N virtual speakers is associated with one of the N channels.

[0135] Specifically, since one audio data point in multi-channel media corresponds to one channel, and different channels correspond to different spatial locations in the media playback scenario, and since different spatial locations have corresponding virtual speakers, the channel layout association rules can define the channels that each virtual speaker needs to be associated with. For example, the left speaker is used to associate with the left channel (L), the right speaker is used to associate with the right channel (R), and the center speaker is used to associate with the center channel (C).

[0136] S806: Based on the channel layout association rules and the channels corresponding to each audio frame, N virtual speakers are used to perform audio rendering processing on N audio data for N channels; wherein, one virtual speaker is used to render the audio data of one channel.

[0137] In one possible implementation, any one of the N virtual speakers is represented as the i-th virtual speaker, where i is a positive integer and 1 ≤ i ≤ N. Based on the channel layout association rules and the channels corresponding to each audio frame, the computer device uses N virtual speakers to perform N-channel audio rendering processing on N audio data, specifically including the following steps (1)-(2):

[0138] (1) Based on the channel layout association rules and the channels corresponding to each audio data, obtain the target audio data to be processed by the i-th virtual speaker from N audio data. The target audio data includes K audio frames with channel type i. Specifically, since the channel layout association rules define the associated channel for each virtual speaker, the target audio data to be processed by any virtual speaker (the i-th virtual speaker) can be determined from N audio data, as shown in Figure 6b. Assume the N speakers include: left speaker S1', right speaker S2', center speaker S3', subwoofer speaker S4', left surround speaker S5', right surround speaker S6', left rear surround speaker S7', right rear surround speaker S8', left top center speaker S9', and right top center speaker S10'; then, the left speaker S1' is used to process the audio data of the left channel (L), the right speaker S2' is used to process the audio data of the right channel (R), the center speaker S3' is used to process the audio data of the center channel (C), and so on. This allows us to determine the audio data to be processed by each virtual speaker, and one virtual speaker is used to process the audio data of one channel, as shown in Table 1 below.

[0139] Table 1. Audio data of each channel to be processed by each of the N virtual speakers

[0140] (2) The target audio data is rendered using the i-th virtual speaker. Specifically, the audio rendering process using the i-th virtual speaker is as follows:

[0141] ① Construct an audio buffer queue to store a preset threshold of P audio frames to be processed, where P is a positive integer. It should be understood that a complete audio file, from user selection to playback and hearing sound, mainly involves a complex process: audio / video demultiplexing, audio decoding, audio post-processing, receiving audio frames, constructing an audio description structure, copying audio frame data, and switching to the main thread to play the audio (audio rendering). Each audio frame needs to be reconstructed into an audio description structure, copied to a PCM (Pulse Code Modulation) buffer, and finally rendered. During audio rendering, if the audio frame is too small, the processing frequency will increase, resulting in significant system latency; if the audio frame is too large, the overall process time will also increase due to the excessive time spent receiving audio frames. These two situations manifest as stuttering or hissing sounds in the user experience. Therefore, setting the appropriate size of each audio frame is crucial. In practical applications, based on statistical data and operational results, it is more reasonable to cache the original frames to a certain number before processing them uniformly. Therefore, this application needs to pre-build a buffer (such as an audio cache queue) during the audio rendering process to accumulate and process multiple audio frames in order to improve processing efficiency.

[0142] ② Obtain the audio frame j to be processed from the K audio frames contained in the target audio data, and add audio frame j to the audio buffer queue, where j is a positive integer and 1≤j≤K. Optionally, before adding the audio frame j (i.e., any one of the K audio frames) to the audio buffer queue, the process may further include:

[0143] I. Retrieve the last audio frame located at the end of the audio buffer queue. Here, the so-called last audio frame refers to the last audio frame in the audio buffer queue.

[0144] II. Analyze the first attribute information of audio frame j and the second attribute information of the trailing audio frame. Attribute information includes: sampling rate (i.e., the frequency of signal sampling, such as 44100Hz, 48000Hz, etc.), number of bytes, and audio format (such as PCM, AAC, etc.) – any one or more of these. Therefore, we can compare whether the audio format of audio frame j is the same as that of the trailing audio frame, whether the sampling rate of audio frame j is the same as that of the trailing audio frame, and whether the number of bytes in audio frame j is the same as that of the trailing audio frame.

[0145] III. If the first attribute information of audio frame j is the same as the second attribute information of the trailing audio frame (which can be understood as audio frame j and the trailing audio frame being two audio frames with the same attributes), then the step of adding audio frame j to the audio buffer queue is triggered; otherwise, if the first attribute information of audio frame j is different from the second attribute information of the trailing audio frame, then audio frame j and the trailing audio frame are two audio frames with different attributes. In this case, data analysis is performed on audio frame j to construct its audio description structure; and based on the audio description structure of audio frame j, the i-th virtual speaker is used to perform separate rendering processing on audio frame j. This method is to ensure that the format or attributes of each audio frame added to the audio buffer queue are the same, thereby laying the foundation for the subsequent reuse of a general audio description structure.

[0146] ③ Let j = j + 1, and repeat the above processes I-III until the number of audio frames stored in the audio buffer queue reaches the preset threshold P. Then, use the i-th virtual speaker to perform audio rendering processing on the P audio frames in the audio buffer queue. Specifically, the audio rendering process performed on the P audio frames in the audio buffer queue is as follows: I. Perform data analysis on any one of the P audio frames to construct the audio structure description body corresponding to the audio buffer queue. II. Call the system conversion interface to perform format conversion processing on each of the P audio frames to convert the P audio frames into the buffer format. Here, the buffer format can be understood as the audio format that the system can process. III. Based on each audio frame in the buffer format, create P encoded data frames (e.g., PCM data frames). IV. Based on the audio structure description body, use the i-th virtual speaker to perform reuse rendering processing on the P encoded data frames.

[0147] In summary, as shown in steps ①-③ above, during the audio rendering process, multiple (e.g., P) audio frames can be rendered uniformly by constructing an audio buffer queue, thereby improving rendering efficiency. Based on this, each virtual speaker can independently render the audio data for its corresponding channel, allowing N virtual speakers to jointly achieve multi-channel audio rendering of N channels of audio data (multi-channel media audio). The audio rendering process executed by any virtual speaker will be further described below with reference to Figure 9, which is a schematic diagram of an audio rendering process provided in an embodiment of this application. As shown in Figure 9, the audio rendering process includes the following steps L1-L9:

[0148] L1, Receive audio frames (such as audio frame j above, which is any one of the K audio frames).

[0149] L2. Check if the new audio frame is different from the last audio frame in the audio buffer queue (i.e., compare the attributes of audio frame j with those of the last audio frame). If yes, execute L5-L9; otherwise, execute L3.

[0150] L3. Add the received audio frames to the audio buffer queue.

[0151] L4. Check if the preset threshold for the audio buffer queue has been reached. If not, execute L1; if yes, execute L5-L9.

[0152] L5. Construct the audio description structure.

[0153] Specifically, the audio description structure is used to describe at least one type of attribute information for any audio frame. The AudioStreamBasicDescription structure is a structure in the CoreAudio framework used to describe the basic properties of an audio stream. It defines the attribute information of the audio data, including but not limited to: sample rate, number of channels, bit depth per sample, etc. Below is an explanation and description of each field contained in the AudioStreamBasicDescription structure:

[0154] 1. mSampleRate: Sampling rate, representing the number of samples taken per second. Common sampling rates are 44100Hz (CD quality) and 48000Hz (professional audio and video).

[0155] 2. mFormatID: An identifier for the audio format; common formats include:

[0156] kAudioFormatLinearPCM: Linear Pulse Code Modulation (PCM), a lossless audio format.

[0157] kAudioFormatMPEG4AAC: Advanced Audio Coding (AAC), a lossy audio format.

[0158] 3. mFormatFlags: Format flags used to describe the specific characteristics of the audio format. For example, for PCM format, you can specify whether it is a floating-point number or an integer, whether it is big-endian or little-endian, etc.

[0159] 4. mBytesPerPacket: The number of bytes per data packet. For uncompressed audio (such as PCM), this value is typically mBytesPerFrame × mFramesPerPacket.

[0160] 5. mFramesPerPacket: The number of frames per packet. For uncompressed audio (such as PCM), each packet typically contains one frame, so this value is usually 1; for compressed audio (such as AAC), this value may be 1024 or 2048.

[0161] 6. mBytesPerFrame: Number of bytes per frame. For uncompressed audio (such as PCM), this value is typically mChannelsPerFrame × (mBitsPerChannel / 8).

[0162] 7.mChannelsPerFrame: The number of audio channels per frame. For example, the value is 1 for mono audio, 2 for stereo audio, and N for N-channel audio.

[0163] 8. mBitsPerChannel: The number of bits per channel. For example, for 16-bit PCM audio, this value is 16.

[0164] 9. mReserved: Reserved field, usually set to 0.

[0165] Therefore, since the audio description structure can reflect the multi-dimensional attribute information of any audio frame and provides a detailed description of the audio data, the audio description structure is a very important data in audio programming, enabling audio processing and playback devices to correctly interpret and process audio data.

[0166] L6. Construct the cache format. Here, the cache format AVAudioFormat can be understood as the audio format that the system can recognize or process, so as to facilitate the system's effective processing of audio data.

[0167] L7. Construct encoded data frames (such as PCM data frames). Here, an AVAudioPCMBuffer needs to be created, which is used to store PCM data frames that the system can recognize and process.

[0168] L8. Copy the data in the buffer queue to the encoded data frame (such as a PCM data frame).

[0169] L9. Render the audio and clear the cache queue.

[0170] Based on this, as shown in Figure 9, this embodiment of the application constructs an audio cache queue, thereby adding P audio frames with the same format or attributes to the queue for unified rendering processing. In the rendering process of the P audio frames in the audio cache queue, a general audio description structure can be reused. Only one general audio description structure needs to be constructed for the P audio frames, avoiding repeated construction to improve audio rendering efficiency and reducing the time spent on thread switching to improve the smoothness of audio playback.

[0171] S807: Plays multi-channel media audio after multi-channel audio rendering processing through a stereo speaker system.

[0172] Specifically, in a stereo system, each virtual speaker renders the audio signal of its corresponding channel, and then plays the rendered audio signal at the corresponding directional position in the media playback scene. Thus, N virtual speakers can simultaneously create a multi-channel rendered stereo surround sound effect. Please refer to Figure 10, which is a schematic diagram of multi-channel rendering in a stereo system according to an embodiment of this application. As shown in Figure 10, four virtual speakers (1, 2, 3, and 4) are arranged in the media playback scene. These four virtual speakers are responsible for rendering the audio signal of their respective channels in the media playback scene, thereby achieving a stereo surround sound effect for multi-channel media audio around the media viewing object in the media playback scene.

[0173] Please refer to Figure 11, which is a schematic diagram of a media playback scenario provided by an embodiment of this application. As shown in Figure 11, in the media playback scenario, the multi-channel media audio to be played is acquired (assuming that the multi-channel media audio is generated after signal acquisition from four channels), and the real-time spatial position of the media viewer (e.g., position 1) is known in the media playback scenario. Based on the audio processing method provided by this embodiment, a stereo speaker system 1 can be arranged around position 1, that is, four virtual speakers need to be arranged. If these four virtual speakers (a, b, c, d) are located on the left, right, front, and back of the media viewer in the media playback scenario, then these four virtual speakers a, b, c, d can independently render the audio signal of the left channel, the audio signal of the right channel, the audio signal of the front channel, and the audio signal of the rear channel, respectively, to achieve a multi-channel audio rendering effect. Thus, the media viewer can hear sounds from the left, right, front, and back directions in the media playback scenario, providing the media viewer with an immersive audio playback experience. Furthermore, if the media viewer moves from position 1 to position 2 in the media playback scene, a stereo sound system 2 can be arranged around position 2. The stereo sound system 2 includes four virtual speakers: a', b', c', and d'. The arrangement of these four virtual speakers (a', b', c', and d') in the media playback scene changes based on the real-time position of the media viewer, and each performs audio rendering and playback in the media playback scene based on its new position. Thus, even if the media viewer moves during the viewing of the media, this application can still arrange the stereo sound system around the user (media viewer) in real time according to the user's (media viewer's) real-time position. This not only provides the media viewer with an immersive audio playback experience, but also dynamically changes to match the user's position to improve the spatial and stereo effects of the audio playback.

[0174] In this embodiment, on the one hand, the multi-channel media audio is abstracted into virtual speakers (ModelEntities) in 3D space according to the spatial position of the speakers in the media playback scene at the time of signal acquisition, and the audio is rendered and played separately, thereby restoring the scene of the multi-channel sound system in the real scene and greatly improving the auditory experience of spatial audio; on the other hand, in the process of setting up the stereo speaker system, the stereo speaker system can be set up around the media viewing object according to the real-time position of the media viewing object, and can dynamically change to match the change of the user's position, so as to improve the spatial sense and stereo sense of audio playback, thereby providing the media viewing object with an immersive audio playback experience.

[0175] The following describes the relevant apparatus of the audio processing scheme provided in the embodiments of this application.

[0176] It should be noted that, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0177] Please refer to Figure 12, which is a schematic diagram of an audio processing apparatus provided in an embodiment of this application. This audio processing apparatus 1200 can be used to execute corresponding steps in the audio processing method provided in this embodiment. Specifically, the audio processing apparatus 1200 may include:

[0178] The acquisition unit 1201 is used to acquire multi-channel media audio in the media playback scene. The multi-channel media audio includes audio signals acquired through N channels, each channel corresponding to a different spatial position in the media playback scene; N is an integer greater than 1.

[0179] The processing unit 1202 is used to obtain the target spatial position of the media viewing object in the media playback scene, and to arrange a stereo sound system around the target spatial position in the media playback scene; the stereo sound system includes N virtual speakers, and the arrangement position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene.

[0180] The processing unit 1202 is also used to perform multi-channel audio rendering processing on multi-channel media audio using a stereo speaker system; wherein, each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel.

[0181] The processing unit 1202 is also used to play multi-channel media audio after multi-channel audio rendering processing through a stereo speaker system.

[0182] In one possible implementation, the multi-channel media audio contains N audio data, where each audio data is an audio signal acquired from one channel; the processing unit 1202 is also configured to perform the following operations:

[0183] Acquire the collected multi-channel audio signal source in a media playback scenario;

[0184] The multi-channel audio signal source is divided into frames to obtain K multi-channel audio frames, each of which consists of audio signals from N channels.

[0185] Channel segmentation technology is used to perform format resampling processing on multi-channel audio signal sources, so that each multi-channel audio frame is segmented according to different channels to obtain N audio data; each audio data contains K audio frames, and the K audio frames contained in an audio data all belong to the same channel, where K is a positive integer;

[0186] Channel segmentation technology includes any one of the following: audio editing software, audio segmentation tools, or audio processing frameworks.

[0187] In one possible implementation, the processing unit 1202 is further configured to perform the following operations:

[0188] Audio preprocessing is performed on a multi-channel audio signal source to obtain a preprocessed multi-channel audio signal source; wherein, audio preprocessing includes any one or more of the following: echo cancellation processing, noise suppression processing, normalization processing, and speech activity detection;

[0189] The preprocessed multi-channel audio signal source is resampled using channel segmentation technology to obtain N audio data.

[0190] In one possible implementation, multi-channel media audio is obtained by acquiring audio signals through N physical speakers, with one physical speaker used to acquire the audio signal of one channel at a spatial location in the media playback scene.

[0191] Processing unit 1202 deploys a stereo speaker system around the target spatial location in the media playback scene to perform the following operations:

[0192] Analyze multi-channel media audio to determine the reference speaker layout of N physical speakers during the signal acquisition process. The reference speaker layout includes the distance and direction between each physical speaker and the reference spatial position in the media playback scene.

[0193] Based on the target spatial location and reference speaker layout, a stereo speaker system is deployed in the media playback scenario.

[0194] In one possible implementation, the target spatial location is represented by three-dimensional spatial coordinates (x, y, z), and any one of the N physical loudspeakers is represented as physical loudspeaker a, where a is a positive integer and 1≤a≤N; the distance between physical loudspeaker a and the reference spatial location includes: a first distance d1, a second distance d2, and a third distance d3;

[0195] Based on the target spatial location and the reference speaker layout, the processing unit 1202 deploys a stereo speaker system in the media playback scenario to perform the following operations:

[0196] Based on the target spatial position (x, y, z) and the first distance d1, the second distance d2, and the third distance d3 between the physical speaker a and the reference spatial position, calculate the candidate position (x', y', z') of the virtual speaker b to be deployed;

[0197] Based on the candidate positions (x', y', z') of the virtual speaker b and the orientation of the physical speaker a in the media playback scene, the virtual speaker b is deployed in the media playback scene.

[0198] In one possible implementation, the processing unit 1202, based on the candidate position (x', y', z') of the virtual speaker b and the orientation of the physical speaker a in the media playback scene, positions the virtual speaker b in the media playback scene to perform the following operations:

[0199] Based on the candidate positions (x', y', z'), determine the spatial position of the virtual speaker b in the media playback scene to be placed; wherein the distance between the spatial position to be placed and the candidate position is less than a preset threshold.

[0200] The orientation of physical speaker a in the media playback scene is determined as the orientation of virtual speaker b to be placed in the media playback scene;

[0201] Set speaker properties for virtual speaker b. Speaker properties include one or more of the following: geometry, material, and color.

[0202] Based on the spatial location to be arranged, the direction to be arranged, and the speaker attributes, place virtual speakers b in the media playback scene.

[0203] In one possible implementation, the processing unit 1202 is further configured to perform the following operations:

[0204] Analyze multi-channel media audio to determine the audio type of the multi-channel media audio;

[0205] Obtain the layout of multiple multi-channel speakers in the media playback scenario; any one multi-channel speaker layout is used to specify the spatial layout information of N physical speakers during the signal acquisition process;

[0206] Based on the audio type, a multi-channel speaker layout that matches the audio type is determined from multiple multi-channel speaker layouts, and the determined multi-channel speaker layout is used as a reference speaker layout.

[0207] In one possible implementation, the multi-channel media audio contains N audio data, and each audio data contains K audio frames from the same channel; the processing unit 1202 uses a stereo speaker system to perform multi-channel audio rendering processing on the multi-channel media audio, for the following operations:

[0208] Analyze multi-channel media audio to determine the channel corresponding to each audio data;

[0209] Obtain the channel layout association rules, which define that each virtual speaker in N virtual speakers is associated with one of the N channels;

[0210] Based on the channel layout association rules and the channels corresponding to each audio frame, N virtual speakers are used to perform audio rendering processing on N audio data in N channels; among them, one virtual speaker is used to render the audio data of one channel.

[0211] In one possible implementation, any one of the N virtual speakers is represented as the i-th virtual speaker, where i is a positive integer and 1 ≤ i ≤ N; the processing unit 1202, based on the channel layout association rules and the channels corresponding to each audio frame, uses the N virtual speakers to perform N-channel audio rendering processing on the N audio data, and performs the following operations:

[0212] Based on the channel layout association rules and the channels corresponding to each audio data, the target audio data to be processed by the i-th virtual speaker is obtained from N audio data; the target audio data includes K audio frames with channel type i.

[0213] The target audio data is rendered using the i-th virtual speaker.

[0214] In one possible implementation, the processing unit 1202 uses the i-th virtual speaker to perform audio rendering processing on the target audio data, for the following operations:

[0215] Construct an audio buffer queue, which is used to store a preset number of P audio frames to be processed, where P is a positive integer;

[0216] Obtain the audio frame j to be processed from the K audio frames contained in the target audio data, and add the audio frame j to the audio buffer queue; j is a positive integer and 1≤j≤K;

[0217] Let j = j + 1, until the number of audio frames stored in the audio buffer queue reaches the preset threshold P, then use the i-th virtual speaker to perform audio rendering processing on the P audio frames in the audio buffer queue.

[0218] In one possible implementation, the processing unit 1202 uses the i-th virtual speaker to perform audio rendering processing on P audio frames in the audio buffer queue, for the following operations:

[0219] Perform data analysis on any one of the P audio frames and construct the audio structure description body corresponding to the audio buffer queue; the audio description structure is used to describe at least one type of attribute information of any audio frame;

[0220] The system conversion interface is called to perform format conversion on each of the P audio frames, so as to convert the P audio frames into a buffer format;

[0221] Based on each audio frame in the cache format, P encoded data frames are created;

[0222] Based on the audio structure description, the i-th virtual speaker is used to perform multiplexing and rendering of P encoded data frames.

[0223] In one possible implementation, before adding the audio frame j to the audio buffer queue, the processing unit 1202 is also configured to perform the following operations:

[0224] Retrieve the last audio frame located at the end of the audio buffer queue;

[0225] Analyze the first attribute information of audio frame j and the second attribute information of the tail audio frame. The attribute information includes one or more of the following: sampling rate, number of bytes, and audio format.

[0226] If the first attribute information of audio frame j is the same as the second attribute information of the tail audio frame, then the step of adding audio frame j to the audio buffer queue is triggered.

[0227] In one possible implementation, the processing unit 1202 is further configured to perform the following operations:

[0228] If the first attribute information of audio frame j is different from the second attribute information of the tail audio frame, then data analysis is performed on audio frame j to construct the audio description structure of audio frame j.

[0229] Based on the audio description structure of audio frame j, the i-th virtual speaker is used to render audio frame j.

[0230] In this embodiment, multi-channel media audio in a media playback scenario is acquired. This multi-channel media audio includes audio signals acquired through N channels, each channel corresponding to a different spatial location in the media playback scenario, where N is an integer greater than 1. The target spatial location of the media data's viewing object in the media playback scenario is acquired, and a stereo speaker system is deployed around the target spatial location in the media playback scenario. This stereo speaker system includes N virtual speakers, and the placement of each virtual speaker in the media playback scenario is adapted to the spatial location of a channel in the media playback scenario. The stereo speaker system is used to perform multi-channel audio rendering processing on the multi-channel media audio, where each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signals acquired by the corresponding channel. The multi-channel media audio after multi-channel audio rendering processing is played through the stereo speaker system. Therefore, it can be seen that in the process of arranging the stereo speaker system, this application can virtually restore the spatial position of the multi-channel media audio during acquisition to determine the layout position of the virtual speakers, thereby restoring the real sound environment of the media audio as much as possible in the media playback scenario and improving the realism of the audio playback effect. In addition, this application uses N virtual speakers to render the audio signals acquired from N channels separately, which can realize the rendering effect of multi-channel spatial audio, improve the stereo and spatial sense of audio playback, and thus enhance the immersive experience in the media playback scenario.

[0231] Please refer to Figure 13, which is a schematic diagram of the structure of a computer device provided in an embodiment of this application. This computer device 1300 is used to execute the steps performed by the computer device in the aforementioned method embodiments. The computer device 1300 may include one or more devices (e.g., a server, node, terminal, etc.) or internal components (e.g., a chip, software module, or hardware module). The computer device may include at least one processor 1301 and a communication interface 1302. Further optionally, the computer device may also include at least one memory 1303 and a bus 1304. Additionally, the processor 1301, communication interface 1302, and memory 1303 are connected via the bus 1304.

[0232] in:

[0233] (1) The processor 1301 is a module that performs arithmetic and / or logical operations. Specifically, it may be one or a combination of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (to assist the central processing unit in completing corresponding processing and applications), and a micro controller unit (MCU).

[0234] (2) The communication interface 1302 can be used to provide information input or output to at least one processor 1301. And / or, the communication interface 1302 can be used to receive data sent externally and / or send data externally, and can be a wired link interface including such as an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, vehicle short-range communication technology, and other short-range wireless communication technologies, etc.). The communication interface 1302 can serve as a network interface.

[0235] (3) The memory 1303 is used to provide storage space, in which data such as the operating system and computer programs (including program instructions) can be stored. The memory 1303 can be one or a combination of random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), etc.

[0236] In specific implementation, the processor 1301 executes the program instructions stored in the memory 1303 to cause the computer device to perform the method steps in the foregoing embodiments of this application; in addition, the effects achieved by the computer device after the processor executes the method steps in the various embodiments can be referred to the description of the foregoing embodiments, and will not be repeated here.

[0237] According to one aspect of this application, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions. When a processor executes the program instructions, it can perform the methods described in the corresponding embodiments above; therefore, further details will not be repeated here. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application. As an example, the program instructions can be deployed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network.

[0238] According to one aspect of this application, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads computer instructions from the computer-readable storage medium and executes the computer instructions, enabling the computer device to perform the methods described in the foregoing embodiments; therefore, further details will not be repeated here. For technical details not disclosed in the embodiments of the computer program product involved in this application, please refer to the description of the method embodiments of this application.

[0239] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product, which includes one or more computer programs. When the computer program is loaded and executed on a computer device, it generates, in whole or in part, the processes or functions described in the embodiments of this application; the computer device can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in or transmitted through a computer-readable storage medium; the computer program can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible to the computer device or a data processing device such as a server or data center that integrates one or more available media; wherein, the available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0240] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An audio processing method, characterized in that, include: Acquire multi-channel media audio in a media playback scenario. The multi-channel media audio includes audio signals acquired through N channels, each of which corresponds to a different spatial position in the media playback scenario; N is an integer greater than 1. The target spatial location of the media viewing object in the media playback scene is obtained, and a stereo sound system is arranged around the target spatial location in the media playback scene; the stereo sound system includes N virtual speakers, and the arrangement position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene; The stereo speaker system is used to perform multi-channel audio rendering processing on the multi-channel media audio; wherein, each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel. The multi-channel media audio, after multi-channel audio rendering processing, is played through the stereo speaker system.

2. The method as described in claim 1, characterized in that, The multi-channel media audio contains N audio data, where each audio data is an audio signal acquired from one channel; the method further includes: Acquire the collected multi-channel audio signal source in a media playback scenario; The multi-channel audio signal source is subjected to frame segmentation processing to obtain K multi-channel audio frames, each of which is composed of audio signals from N channels. The multi-channel audio signal source is resampled using channel segmentation technology to divide each multi-channel audio frame into N audio data according to different channels; each audio data contains K audio frames, and the K audio frames contained in an audio data all belong to the same channel, where K is a positive integer. The channel segmentation technology includes any one of the following: audio editing software, audio segmentation tools, and audio processing frameworks.

3. The method as described in claim 1 or 2, characterized in that, The method further includes: The multi-channel audio signal source is subjected to audio preprocessing to obtain a preprocessed multi-channel audio signal source; wherein, the audio preprocessing includes any one or more of the following: echo cancellation processing, noise suppression processing, normalization processing, and speech activity detection; The preprocessed multi-channel audio signal source is resampled using channel segmentation technology to obtain N audio data.

4. The method according to any one of claims 1-3, characterized in that, The multi-channel media audio is obtained by acquiring audio signals through N physical speakers; one physical speaker is used to acquire the audio signal of one channel at a spatial location in the media playback scene. The surrounding stereo speaker system deployed at the target spatial location in the media playback scenario includes: Analyze the multi-channel media audio to determine the reference speaker layout of the N physical speakers during the signal acquisition process. The reference speaker layout includes the distance and direction between each physical speaker and the reference spatial position in the media playback scenario. Based on the target spatial location and the reference speaker layout, a stereo speaker system is deployed in the media playback scenario.

5. The method according to any one of claims 1-4, characterized in that, The target spatial location is represented by three-dimensional spatial coordinates (x, y, z), and any one of the N physical loudspeakers is represented as physical loudspeaker a, where a is a positive integer and 1≤a≤N; the distance between physical loudspeaker a and the reference spatial location includes: a first distance d1, a second distance d2, and a third distance d3; The method of deploying a stereo speaker system in the media playback scenario based on the target spatial location and the reference speaker layout includes: Based on the target spatial position (x, y, z) and the first distance d1, second distance d2, and third distance d3 between the physical speaker a and the reference spatial position, calculate the candidate position (x', y', z') of the virtual speaker b to be deployed; Based on the candidate positions (x', y', z') of the virtual speaker b and the orientation of the physical speaker a in the media playback scene, the virtual speaker b is positioned in the media playback scene.

6. The method according to any one of claims 1-5, characterized in that, The step of deploying the virtual speaker b in the media playback scene based on the candidate positions (x', y', z') of the virtual speaker b and the orientation of the physical speaker a in the media playback scene includes: Based on the candidate positions (x', y', z'), the spatial position of the virtual speaker b in the media playback scene is determined; wherein the distance between the spatial position to be placed and the candidate position is less than a preset threshold. The orientation of the physical speaker a in the media playback scene is determined as the orientation of the virtual speaker b to be placed in the media playback scene; Set speaker attributes for the virtual speaker b, including any one or more of the following: geometry, material, and color; Based on the spatial location to be arranged, the direction to be arranged, and the speaker attributes, a virtual speaker b is arranged in the media playback scene.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Analyze the multi-channel media audio to determine the audio type of the multi-channel media audio; Obtain the layout of multiple multi-channel speakers in the media playback scenario; any one of the multi-channel speaker layouts is used to specify the spatial layout information of N physical speakers during the signal acquisition process; Based on the audio type, a multi-channel speaker layout that matches the audio type is determined from the plurality of multi-channel speaker layouts, and the determined multi-channel speaker layout is used as a reference speaker layout.

8. The method according to any one of claims 1-7, characterized in that, The multi-channel media audio contains N audio data, and each audio data contains K audio frames from the same channel. The step of using the stereo speaker system to perform multi-channel audio rendering processing on the multi-channel media audio includes: Analyze the multi-channel media audio to determine the channel corresponding to each audio data; Obtain the channel layout association rule, which defines that each virtual speaker in N virtual speakers is associated with one of the N channels; Based on the channel layout association rules and the channels corresponding to each audio frame, N virtual speakers are used to perform audio rendering processing on the N audio data for N channels; wherein, one virtual speaker is used to render the audio data for one channel.

9. The method according to any one of claims 1-8, characterized in that, Any one of the N virtual speakers is represented as the i-th virtual speaker, where i is a positive integer and 1 ≤ i ≤ N; the step of using N virtual speakers to perform N-channel audio rendering processing on the N audio data based on the channel layout association rules and the channels corresponding to each audio frame includes: Based on the channel layout association rules and the channels corresponding to each audio data, the target audio data to be processed by the i-th virtual speaker is obtained from the N audio data; the target audio data includes K audio frames of channel type i; The target audio data is rendered using the i-th virtual speaker.

10. The method according to any one of claims 1-9, characterized in that, The step of using the i-th virtual speaker to perform audio rendering processing on the target audio data includes: Construct an audio buffer queue, which is used to store a preset number threshold P audio frames to be processed, where P is a positive integer; Obtain the audio frame j to be processed from the K audio frames contained in the target audio data, and add the audio frame j to the audio buffer queue; j is a positive integer and 1≤j≤K; When the number of audio frames stored in the audio cache queue reaches the preset threshold P, the i-th virtual speaker is used to perform audio rendering processing on the P audio frames in the audio cache queue.

11. The method according to any one of claims 1-10, characterized in that, The step of using the i-th virtual speaker to perform audio rendering processing on P audio frames in the audio buffer queue includes: Data analysis is performed on any one of the P audio frames to construct an audio structure description body corresponding to the audio buffer queue; the audio description structure is used to describe at least one type of attribute information of any one audio frame; The system conversion interface is called to perform format conversion processing on the P audio frames respectively, so as to convert the P audio frames into a buffer format; Based on each audio frame of the cache format, P encoded data frames are created; Based on the audio structure description, the i-th virtual speaker is used to perform multiplexing rendering on the P encoded data frames.

12. The method according to any one of claims 1-11, characterized in that, Before adding the audio frame j to the audio buffer queue, the method further includes: Obtain the last audio frame located at the tail of the audio buffer queue; Analyze the first attribute information of the audio frame j and the second attribute information of the tail audio frame. The attribute information includes any one or more of the following: sampling rate, number of bytes, and audio format. If the first attribute information of the audio frame j is the same as the second attribute information of the tail audio frame, then the step of adding the audio frame j to the audio buffer queue is triggered.

13. The method according to any one of claims 1-12, characterized in that, The method further includes: If the first attribute information of the audio frame j is different from the second attribute information of the tail audio frame, then data analysis is performed on the audio frame j to construct the audio description structure of the audio frame j; Based on the audio description structure of the audio frame j, the i-th virtual speaker is used to render the audio frame j.

14. An audio processing apparatus, characterized in that, include: The acquisition unit is used to acquire multi-channel media audio in a media playback scenario. The multi-channel media audio includes audio signals acquired through N channels, each of which corresponds to a different spatial position in the media playback scenario; N is an integer greater than 1. The processing unit is used to obtain the target spatial position of the media viewing object in the media playback scene, and to arrange a stereo sound system around the target spatial position in the media playback scene; the stereo sound system includes N virtual speakers, and the arrangement position of each virtual speaker in the media playback scene is adapted to the spatial position of a channel in the media playback scene; The processing unit is further configured to perform multi-channel audio rendering processing on the multi-channel media audio using the stereo speaker system; wherein, each virtual speaker in the stereo speaker system performs audio rendering processing on the audio signal collected by the corresponding channel. The processing unit is also used to play the multi-channel media audio after multi-channel audio rendering processing through the stereo speaker system.

15. A computer device, characterized in that, include: Memory and processor; The memory stores one or more computer programs; A processor for loading one or more computer programs to implement the audio processing method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the audio processing method as described in any one of claims 1-13.

17. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs the audio processing method as described in any one of claims 1-13.

Images