Audio processing method, sound system, electronic equipment and vehicle
By separating the sound object and expanding the multi-dimensional direction of the original audio signal, the problem that audio processing in the existing technology cannot immersively restore the real sound situation, achieving better audio processing effects, and improving auditory immersion and spatial perception.
Patent Information
- Application Number
- CN202510751039.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-06
- Publication Date
- 2025-08-26
AI Technical Summary
In the prior art, audio processing methods for scenes such as home theaters and vehicle-mounted smart cockpits cannot effectively restore the real sound situation in immersively, and the overall auditory effect is poor, so the real sound spatial distribution cannot be simulated.
By separating the original audio signal with acoustic objects, combining the azimuth space orientation layout and multi-dimensional direction expansion of the acoustic field space, using deep learning network and spectrum analysis methods, multiple sound objects are laid out on speakers at different locations to expand the multi-dimensional direction of the acoustic field space.
It greatly improves the overall auditory effect of audio processing, realizes immersive restoration of the real sound situation, and improves the effects of tone restoration, sound image focus and sound field expansion.
Smart Images
Figure CN120547495A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio processing technology, and more specifically, to an audio processing method, electronic equipment, and vehicle. Background Art
[0002] In scenarios like home theaters and in-car smart cockpits, actual sound sources are converted into multi-channel sources and then played back through the audio system. However, related technologies suffer from poor overall auditory effects and fail to immersively restore realistic sound situations. Summary of the Invention
[0003] The embodiments of the present application provide an audio processing method, an audio system, an electronic device, and a vehicle, which improve the overall auditory effect of audio processing and immersively restore the real sound situation.
[0004] To achieve the above object, according to a first aspect of the present application, there is provided an audio processing method, comprising:
[0005] Separate the original audio signal to obtain multiple sound objects;
[0006] According to the multiple sound objects, the sound image space orientation layout and the sound field space multi-dimensional direction expansion are performed to obtain multi-directional channel sound sources, wherein the multi-directional channel sound sources are played through speaker modules at different positions.
[0007] Optionally, the multiple sound objects include a first sound object, a second sound object, and a third sound object, the multiple directions include a first direction, a second direction, a third direction, and a fourth direction, and performing a sound image spatial direction layout according to the multiple sound objects includes:
[0008] Arrange the first acoustic object in the first position;
[0009] Arrange the first acoustic object and the second acoustic object in the second position;
[0010] Arrange the second acoustic object and the third acoustic object in the third position;
[0011] Arrange the second acoustic object and the third acoustic object in the fourth position;
[0012] The first orientation, the second orientation, the third orientation and the fourth orientation are used to indicate different positions.
[0013] Optionally, the arranging the first acoustic object and the second acoustic object in the second orientation includes:
[0014] performing weighted processing on the first acoustic object according to a first coefficient to obtain a first weighted first acoustic object;
[0015] performing superposition processing on the first weighted first acoustic object and the second acoustic object to obtain a first superposition sound source;
[0016] The first superimposed sound source is arranged in the second orientation.
[0017] Optionally, the placing of the second acoustic object and the third acoustic object in the third position includes:
[0018] performing weighted processing on the third acoustic object according to the second coefficient to obtain a second weighted third acoustic object;
[0019] performing superposition processing on the second sound object and the second weighted third sound object to obtain a second superposition sound source;
[0020] The second superimposed sound source is arranged at the third position.
[0021] Optionally, the placing of the second acoustic object and the third acoustic object in the fourth position includes:
[0022] performing weighted processing on the third acoustic object according to a third coefficient to obtain a third weighted third acoustic object;
[0023] performing superposition processing on the second sound object and the third weighted third sound object to obtain a third superposition sound source;
[0024] The third superimposed sound source is arranged in the fourth orientation.
[0025] Optionally, the multi-dimensional directional expansion includes width expansion, depth expansion, and height expansion, and the multi-dimensional directional expansion of the sound field space according to the multiple sound objects includes:
[0026] performing a first tuning process on the first sound object and the second sound object to expand the depth of the sound field space;
[0027] performing a second tuning process on the second sound object to expand the width of the sound field space;
[0028] A third tuning process is performed on the second sound object to highly expand the sound field space.
[0029] Optionally, performing a first tuning process on the first sound object and the second sound object includes:
[0030] performing weighted processing on the first acoustic object located at the first position according to a fourth coefficient;
[0031] performing weighted processing on the first acoustic object and the second acoustic object arranged at the second position according to a fifth coefficient;
[0032] The fourth coefficient is smaller than the fifth coefficient.
[0033] Optionally, performing a second tuning process on the second sound object includes:
[0034] A decorrelation operation is performed on the second sound object arranged at the third position.
[0035] Optionally, performing a third tuning process on the second sound object includes:
[0036] A spectrum allocation operation is performed on the second sound objects arranged at the third position and the fourth position.
[0037] Optionally, the performing spectrum allocation operation processing on the second sound objects arranged in the third direction and the fourth direction includes:
[0038] performing a spectrum allocation operation on the second sound object to obtain a sound object in a first frequency band, a sound object in a second frequency band, and a sound object in a third frequency band, wherein the frequency ranges of the first frequency band, the second frequency band, and the third frequency band increase in sequence and do not overlap;
[0039] Arranging the second acoustic object and the third acoustic object in the third position includes:
[0040] Arranging the acoustic objects of the first frequency band, the acoustic objects of the second frequency band, and the third acoustic object at the third position;
[0041] Arranging the second acoustic object and the third acoustic object in the fourth position includes:
[0042] The acoustic objects in the second frequency band, the acoustic objects in the third frequency band, and the third acoustic object are arranged in the fourth direction.
[0043] Optionally, performing the third tuning process on the second sound object further includes:
[0044] A spectrum allocation operation is performed on the second sound object located at the fourth position.
[0045] Optionally, performing spectrum allocation operation processing on the second acoustic object arranged at the fourth position includes:
[0046] performing a spectrum allocation operation on the second sound object to obtain a sound object in a second frequency band and a sound object in a third frequency band, wherein the frequency ranges of the second frequency band and the third frequency band increase in sequence and do not overlap;
[0047] Arranging the second acoustic object and the third acoustic object in the fourth position includes:
[0048] The acoustic objects in the second frequency band, the acoustic objects in the third frequency band, and the third acoustic object are arranged in the fourth direction.
[0049] Optionally, the first sound object includes human voice, the second sound object includes background accompaniment, the third sound object includes ambient sound, the first position includes the center channel, the second position includes the left and right channels, the third position includes the surround channels, and the fourth position includes the sky channel.
[0050] According to a second aspect of the present application, an electronic device is provided, including:
[0051] Memory, on which computer programs / instructions are stored;
[0052] A processor is used to execute the computer program / instructions in the memory to implement the steps of the above-mentioned audio processing method.
[0053] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored, characterized in that the computer program / instruction implements the steps of the above-mentioned audio processing method when executed by a processor.
[0054] According to a fourth aspect of the present application, a computer program product is provided, comprising a computer program / instruction, wherein the computer program / instruction implements the steps of the above-mentioned audio processing method when executed by a processor.
[0055] According to a fifth aspect of the present application, a sound system is provided, comprising the electronic device as described above, and a plurality of speaker modules connected to the electronic device, wherein the plurality of speaker modules are located at different positions respectively.
[0056] According to a sixth aspect of the present application, a vehicle is provided, comprising the electronic device as described above, or the computer-readable storage medium as described above, or the audio system as described above.
[0057] This application improves the underlying technical thinking by performing spatial layout of the sound and image of the separated multiple sound objects and expanding the sound field in multiple dimensions. The final processing effect performs well in terms of timbre restoration, sound and image focusing, and sound field expansion. It greatly improves the overall auditory effect of audio processing and immersively restores the real sound environment.
[0058] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0060] Figure 1 A flowchart of an audio processing method provided by certain embodiments of the present application;
[0061] Figure 2 A schematic diagram of an audio processing algorithm flow provided by certain embodiments of the present application;
[0062] Figure 3 A schematic diagram of an audio processing architecture provided by certain embodiments of the present application;
[0063] Figure 4 Another schematic diagram of an audio processing architecture provided by certain embodiments of the present application;
[0064] Figure 5 A schematic diagram of an audio-visual spatial orientation layout and a multi-dimensional direction expansion of a sound field space provided by certain embodiments of the present application;
[0065] Figure 6 Certain embodiments of the present application provide a schematic diagram of spectrum allocation operation processing. DETAILED DESCRIPTION
[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application. In addition, it should be understood that the specific embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application.
[0067] In scenarios such as home theaters and in-car smart cockpits, after the actual sound source is converted into a multi-channel sound source and played back through the audio system, the original sound environment can be virtually reconstructed to create a sense of auditory surround and envelopment. However, due to the lack of overall architectural guidelines, how to arrange the spatial orientation of the sound and image? How to expand the sound field space? These are very realistic technical difficulties. Specifically: The fundamental principle of audio processing is to restore a more immersive and realistic sound field space, and it cannot be separated from the real world. For example, for the sake of fun, most sound objects are rendered behind the listener, or the drum kit is rendered above the head. This is also unacceptable in the professional audio field, resulting in a broken connection between auditory perception and the physical world, and perceptual confusion. This has led to poor overall auditory effects in related technologies, and an inability to immersively restore real sound situations.
[0068] In order to solve the above problems, the present invention provides an audio processing method. Figure 1 and Figure 2 As shown, including:
[0069] S10: Separate the original audio signal to obtain multiple sound objects;
[0070] S20: Performing spatial orientation layout of the sound image and multi-dimensional direction expansion of the sound field space according to the multiple sound objects to obtain multi-directional channel sound sources, wherein the multi-directional channel sound sources are played through speaker modules at different positions.
[0071] It can be understood that the separation of audio signals can include but is not limited to deep learning networks such as Demucs and MDX-NET, spectrum analysis and filtering, time-frequency masking technology, DNN (Deep Neural Networks, deep neural network), CNN (Convolutional Neural Network The invention also provides a method for separating sound objects by using methods such as convolutional neural networks (CNNs), frequency domain masking, and phase recovery. The multiple sound objects may include, but are not limited to, human voices, accompaniment, specific musical instruments, and environmental noise. The spatial orientation layout of the sound image may include, but is not limited to, re-arranging the orientation of a sound object or a combination of multiple sound objects in the sound image space through an algorithm, thereby converting the original two-dimensional stereo stage into a three-dimensional immersive sound stage. The multi-dimensional expansion of the sound field space may include, but is not limited to, expanding the sound objects in a certain direction in the multi-dimensional sound field space through an algorithm, such as expanding the width of the three-dimensional sound field space to obtain a wider auditory space. The multi-directional channel sound source may include, but is not limited to, the channel configuration orientation and output sound source of each speaker in the channel specifications such as 4.0 / 5.1 / 7.1 / 5.1.2 / 7.1.4 / 9.1.6, such as the center channel, front left and right channels, rear left and right surround channels, and the so-called 0.1 channel subwoofer channel in 5.1 channels. Or as in 5.1.2 channels, there are front left, front right, center, rear left, and rear right, plus 1 subwoofer (.1) and 2 sky channels (.2).
[0072] Specifically, the original audio signal is first subjected to sound object separation, and multiple sound objects are separated from the original audio signal. Then, the multiple sound objects are respectively subjected to sound image spatial orientation layout and sound field spatial multi-dimensional directional expansion. Specifically, the sound image spatial orientation layout can be performed first and then the sound field spatial multi-dimensional directional expansion can be performed, or the sound field spatial multi-dimensional directional expansion can be performed first and then the sound image spatial orientation layout can be performed, or the sound image spatial orientation layout and the sound field spatial multi-dimensional directional expansion can be processed simultaneously in parallel to obtain multi-directional channel sound sources. Finally, the multi-directional channel sound sources are played through the loudspeakers at different positions. By performing sound image spatial orientation layout and sound field spatial multi-dimensional directional expansion on the multiple sound objects obtained by separation, compared with the audio processing methods in the related art, this application is improved from the underlying technical ideas, and the final processing effect performs well in terms of timbre restoration, sound image focusing and sound field expansion. The overall auditory effect of audio processing is greatly improved, and the real sound situation is restored immersively.
[0073] In one specific embodiment, acoustic object separation is performed on the original audio signal, separating it into multiple acoustic objects. The separated acoustic objects are recorded in stereo format, preserving the original sound and image orientation information. An objective metric for evaluating acoustic object separation performance is the source-distortion ratio (SDR). Current mainstream deep learning networks can achieve separation performance of SDR>10. From a subjective listening perspective, this essentially eliminates issues such as sound leakage, artifacts, and distortion, achieving high-quality acoustic object separation.
[0074] In certain embodiments, combined Figure 3 and Figure 4 As shown, the multiple sound objects include a first sound object, a second sound object, and a third sound object, and the multiple directions include a first direction, a second direction, a third direction, and a fourth direction. The sound image spatial direction layout is performed according to the multiple sound objects, including:
[0075] Arrange the first sound object in the first position;
[0076] Arrange the first sound object and the second sound object in a second position;
[0077] Arrange the second and third sound objects in the third position;
[0078] Arrange the second sound object and the third sound object in the fourth position;
[0079] The first position, the second position, the third position and the fourth position are used to indicate different positions.
[0080] It can be understood that the first orientation, the second orientation, the third orientation and the fourth orientation may include but are not limited to the channel configuration orientations of each speaker in the channel specifications such as 4.0 / 5.1 / 7.1 / 5.1.2 / 7.1.4 / 9.1.6.
[0081] Specifically, the first sound object is placed in the first position for playback; the first and second sound objects are processed, such as by fusion, superposition, and weighting, and the resulting sound source is placed in the second position for playback; the second and third sound objects are processed, such as by fusion, superposition, and weighting, and the resulting sound sources are placed in the third and fourth positions for playback. By assigning different sound objects to different directional channels (such as left, right, center, and surround channels, or the vertical direction of a three-dimensional sound field), the spatial distribution of sound in a real environment (such as rain falling from overhead or footsteps coming from the left) is simulated, while also avoiding aliasing interference between the sound sources of the different sound objects, thereby improving playback quality.
[0082] In certain embodiments, combined Figure 3 and Figure 4As shown, the first sound object includes human voice, the second sound object includes background accompaniment, the third sound object includes ambient sound, the first position includes the center channel, the second position includes the left and right channels, the third position includes the surround channels, and the fourth position includes the sky channel.
[0083] Specifically, in a specific embodiment, the original audio signal is subjected to sound object separation, and the original audio signal is separated into three sound objects: human voice, background accompaniment (instruments, etc.) and ambient sound (reverberation, cheering, environment, etc.). The three sound objects are then arranged in the sound image space and expanded in multiple dimensions of the sound field space to obtain the channel sound sources in four directions: center channel, left and right channels, surround channels, and sky channel. For example, the center channel, left and right channels, and surround channels of the present application can be directly adapted to the center channel, left and right channels, and surround channels in the 5.1 channel specification, and the sky channel is not used (the audio information is still complete and does not affect the use). Alternatively, the center channel, left and right channels, surround channels, and sky channel of the present application can be directly adapted to the center channel, left and right channels, surround channels, and sky channel in the 5.1 channel specification.
[0084] In certain embodiments, combined Figure 3 and Figure 4 As shown, arranging the first acoustic object and the second acoustic object in the second position includes:
[0085] performing weighted processing on the first acoustic object according to the first coefficient to obtain a first weighted first acoustic object;
[0086] Performing superposition processing on the first weighted first sound object and the second sound object to obtain a first superposition sound source;
[0087] Arrange the first superimposed sound source at the second position.
[0088] It can be understood that weighted processing may include, but is not limited to, adjusting the contribution of a sound object in the comprehensive calculation by multiplying the sound object by a weight coefficient; superposition processing may include, but is not limited to, adding or mixing multiple sound objects to generate a new audio output, such as directly adding the multiple sound objects or adding the sound objects after adjusting the weights of the sound objects;
[0089] Specifically, the first sound object V1 is first weighted according to the first coefficient a (such as a < 1) to obtain the first weighted first sound object aV1. The first weighted first sound object aV1 is then superimposed with the second sound object V2 to obtain the first superimposed sound source aV1+V2. Finally, the first superimposed sound source aV1+V2 is arranged in the second direction for playback. In a specific embodiment, the left and right channels (second direction) are background accompaniment (second sound object) superimposed with a slightly smaller proportion of human voice (first sound object (a < 1)). This processing, firstly, ensures that the front sound stage contains basically all the sound source content, which is in line with the actual situation of the vast majority; secondly, it ensures that when there are multiple human voice objects such as chorus and dialogue, the original distance between each person is restored, rather than all the human voices being squeezed into the center channel.
[0090] In certain embodiments, combined Figure 3 and Figure 4 As shown, the second sound object and the third sound object are arranged in a third position, including:
[0091] performing weighted processing on the third acoustic object according to the second coefficient to obtain a second weighted third acoustic object;
[0092] performing superposition processing on the second sound object and the second weighted third sound object to obtain a second superposition sound source;
[0093] Arrange the second superimposed sound source in the third position.
[0094] Specifically, the third sound object V3 is first weighted according to the second coefficient b (such as b>1) to obtain the second weighted third sound object bV3. Then the second sound object V2 and the second weighted third sound object bV3 are superimposed to obtain the second superimposed sound source bV3+V2. Finally, the second superimposed sound source bV3+V2 is arranged in the third position for playback. In a specific embodiment, the surround sound channel (third position) is the background accompaniment (second sound object) superimposed with the ambient sound (third sound object (b>1)), which does not include human voices, and the ambient sound can be slightly more. By strengthening the ambient sound, the surround sound channel can create a sense of envelopment between multiple speakers, making the listener feel as if they are in a specific scene (such as a natural environment or a music scene), enhancing the overall sense of substitution and reality.
[0095] In certain embodiments, combined Figure 3 and Figure 4 As shown, the second sound object and the third sound object are arranged in the fourth position, including:
[0096] performing weighted processing on the third acoustic object according to the third coefficient to obtain a third weighted third acoustic object;
[0097] performing superposition processing on the second sound object and the third weighted third sound object to obtain a third superposition sound source;
[0098] Arrange the third superimposed sound source in the fourth direction.
[0099] Specifically, the third sound object V3 is first weighted according to the third coefficient c (such as c < 1) to obtain the third weighted third sound object cV3. Then the second sound object V2 and the second weighted third sound object cV3 are superimposed to obtain the third superimposed sound source cV3+V2. Finally, the third superimposed sound source bV3+V2 is arranged in the fourth direction for playback. In a specific embodiment, the sky channel (fourth direction) is the background accompaniment (second sound object) superimposed with the ambient sound (third sound object (c<1)), which does not include human voice, and the proportion of ambient sound should be slightly less than that of the surround channel. The background accompaniment is used as the main body to avoid confusion with the ambient sound, ensuring that the sky channel focuses on the sound effect expression of the height dimension, complementing the horizontal surround channel, and constructing a complete 3D sound field.
[0100] In certain embodiments, combined Figure 3 、 Figure 4 and Figure 5 As shown, multi-dimensional directional expansion includes width expansion, depth expansion and height expansion, and multi-dimensional directional expansion of the sound field space is performed according to multiple sound objects, including:
[0101] Performing a first tuning process on the first sound object and the second sound object to expand the depth of the sound field space;
[0102] performing a second tuning process on the second sound object to expand the width of the sound field space;
[0103] A third tuning process is performed on the second sound object to highly expand the sound field space.
[0104] It can be understood that the first tuning process can be adjusting the weight coefficients of different sound objects, the second tuning process can be a decorrelation operation process, and the third tuning process can be a spectrum allocation operation process. Of course, the first tuning process, the second tuning process, and the third tuning process can also include, but are not limited to, expanding the sound field space (width, depth, height) through methods such as panning, reverberation, delay and Haas effect, equalization and spectrum shaping, spatialization technology and three-dimensional audio.
[0105] Specifically, the first sound object and the second sound object are subjected to the first tuning process respectively, or the first sound object and the second sound object are superimposed and then subjected to the first tuning process, thereby expanding the sound field space in the depth direction. The second sound object is subjected to the second tuning process, thereby expanding the sound field space in the width direction. The second sound object is subjected to the third tuning process, thereby expanding the sound field space in the height direction. Expanding the width of the sound field space can make the sound from a single sound source expand to a wider sound field, avoiding the flat listening experience of "concentrating in the center"; expanding the depth of the sound field space can make the sense of distance of the sound source (such as near-field human voice and far-field ambient sound) form a multi-level spatial distribution in the front, middle and back; expanding the height of the sound field space can make the sound vertically distributed (such as overhead and ground sound sources), breaking the limitations of the planar sound field.
[0106] In certain embodiments, combined Figure 3 、 Figure 4 and Figure 5 As shown, performing a first tuning process on the first sound object and the second sound object includes:
[0107] performing weighted processing on a first acoustic object arranged at a first position according to a fourth coefficient;
[0108] performing weighted processing on the first acoustic object and the second acoustic object arranged at the second position according to the fifth coefficient;
[0109] Among them, the fourth coefficient is smaller than the fifth coefficient.
[0110] Specifically, the first acoustic object V1 positioned in the first direction is weighted using the fourth coefficient e, while the first acoustic object V1 and the second acoustic object V2 positioned in the second direction are weighted using the fifth coefficient f, where the fourth coefficient e needs to be smaller than the fifth coefficient f. In one specific embodiment, the front sound stage primarily relies on the left and right channels and the center channel. If the gain coefficient f of the left and right channels is 1, and the gain coefficient e of the center channel is set to 0.8, the sense of depth of the sound field is enhanced. Expanding the depth of the sound field space can enhance the sense of distance of the sound source (such as near-field human voice and far-field ambient sound) by forming a multi-layered spatial distribution in the front, center, and back.
[0111] In certain embodiments, combined Figure 3 、 Figure 4 and Figure 5 As shown, performing a second tuning process on the second sound object includes:
[0112] A decorrelation operation is performed on the second sound object arranged at the third position.
[0113] Specifically, in one embodiment, decorrelation processing is performed on the background accompaniment (second sound object) to expand the perceived width of the sound field. For stereo sound sources, the signal components of the sound object near the center of the sound stage are considered correlated, while the signal components near the left and right sides are considered uncorrelated. By removing the correlated components from the background accompaniment, crosstalk between surround channels is reduced, allowing instruments on the listener's surround side to be presented relatively independently, significantly broadening the actual listening experience.
[0114] In a specific embodiment of decorrelation processing, assuming the left and right time-domain signals of a stereo sound source are sL and sR, they are Fourier transformed to obtain frequency-domain amplitude spectrum signals ML and MR, respectively. This function is used to construct a sound image orientation mapping function φ. This function has a characteristic of varying monotonically with respect to the stereo image orientation within the range [-1, 1]. This example illustrates a method for constructing the sound image orientation mapping function φ; however, any method can be constructed based on the principles described above.
[0115]
[0116] The function φ is used to generate the correlation window function W. The window function takes the middle sound image orientation as the symmetry axis and adopts the Gaussian window function
[0117]
[0118] The parameter c0 controls the width of the window function.
[0119] P=W·(ML+MR)
[0120] The decorrelated component is
[0121] AL=ML-P
[0122] AR=MR-P
[0123] In certain embodiments, combined Figure 3 、 Figure 4 、 Figure 5 and Figure 6 As shown, performing a third tuning process on the second sound object includes:
[0124] A spectrum allocation operation is performed on the second sound objects arranged at the third and fourth positions.
[0125] Specifically, the spectrum allocation operation is performed on the second acoustic objects V2 arranged at the third and fourth positions, so that the second acoustic objects V2 with different frequency bands can be played at the third and fourth positions.
[0126] Further, combined with Figure 3 、 Figure 4 、 Figure 5 and Figure 6 As shown, the spectrum allocation operation is performed on the second sound objects arranged at the third and fourth positions, including:
[0127] performing a spectrum allocation operation on the second sound object to obtain a sound object in a first frequency band, a sound object in a second frequency band, and a sound object in a third frequency band, wherein the frequency ranges of the first frequency band, the second frequency band, and the third frequency band increase in sequence and do not overlap;
[0128] The layout of the second-tone object and the third-tone object in the third position includes:
[0129] Arranging the sound object of the first frequency band, the sound object of the second frequency band, and the third sound object in a third position;
[0130] Arranging the second sound object and the third sound object in the fourth position includes:
[0131] The acoustic object of the second frequency band, the acoustic object of the third frequency band, and the third acoustic object are arranged in a fourth direction.
[0132] Specifically, in a specific embodiment, after further spectrum allocation processing is performed on the background accompaniment (second sound object V2), the background accompaniment of the first frequency band, the background accompaniment of the second frequency band, and the background accompaniment of the third frequency band can be obtained. Then, in the surround sound channel (third position), the background accompaniment of the first frequency band, the background accompaniment of the second frequency band, and the ambient sound (third sound object) are arranged; in the sky channel (fourth position), the background accompaniment of the second frequency band, the background accompaniment of the third frequency band, and the ambient sound (third sound object) are arranged to expand the auditory height of the sound field. Following the "pitch-height" effect in psychoacoustics, the related document "Hyunkook Lee. 2D-to-3D Ambience Upmixing Based on Perceptual Band Allocation. Journal of the Audio Engineering Society. Vol. 63, No. 10, October 2015". It is generally believed that the positioning of the height direction of the sound field depends on the distribution of the spectrum, that is, the "pitch-height" effect, which refers to the phenomenon that there is a correlation between the frequency of the sound and the height positioning of the sound image perceived by the listener. Higher-frequency sounds are usually perceived by the listener as higher in the vertical direction, while lower-frequency sounds are perceived as lower, closer to the horizontal direction of the listener's ears. Based on the above principles, the sound content of the sky channel (background accompaniment and ambient sound) should appropriately weaken the mid-low frequencies or enhance the mid-high frequencies; the sound content of the surround channel (background accompaniment and ambient sound) should appropriately increase the mid-low frequencies or weaken the mid-high frequencies.
[0133] Combine Figure 6As shown in the figure, a specific embodiment of the spectrum allocation operation is listed, in which the sound object to be processed is divided into a first frequency band (f L ~f1), the second frequency band (f1~f2), the third frequency band (f2~f H ) and uses low-pass, band-pass, and high-pass filters for spectral decomposition. Spectral allocation is performed in conjunction with the surround channels, allocating the first and second frequency bands to the surround channels while allocating the second and third frequency bands to the sky channel, creating a more pronounced frequency band gradient.
[0134] In certain embodiments, combined Figure 3 、 Figure 4 、 Figure 5 and Figure 6 As shown, performing the third tuning process on the second sound object further includes:
[0135] A spectrum allocation operation is performed on the second sound object arranged at the fourth position.
[0136] Specifically, the spectrum allocation operation is performed on the second acoustic objects V2 arranged at the fourth position, so that the second acoustic objects V2 within the specified frequency band can be played at the fourth position.
[0137] Further, combined with Figure 3 、 Figure 4 、 Figure 5 and Figure 6 As shown, the spectrum allocation operation is performed on the second sound object arranged at the fourth position, including:
[0138] performing a spectrum allocation operation on the second sound object to obtain a sound object in a second frequency band and a sound object in a third frequency band, wherein the frequency ranges of the second frequency band and the third frequency band increase in sequence and do not overlap;
[0139] Arranging the second sound object and the third sound object in the fourth position includes:
[0140] The acoustic object of the second frequency band, the acoustic object of the third frequency band, and the third acoustic object are arranged in a fourth direction.
[0141] Specifically, in one embodiment, further spectrum allocation processing is performed on the background accompaniment (second sound object V2) to produce background accompaniment in the second and third frequency bands. The background accompaniment in the second and third frequency bands, along with the ambient sound (third sound object), is then arranged in the sky channel (fourth position) to expand the auditory depth of the sound field.
[0142] Combine Figure 6 As shown in the figure, a specific embodiment of the spectrum allocation operation is listed, in which the sound object to be processed is divided into a first frequency band (f L~f1), the second frequency band (f1~f2), the third frequency band (f2~f H ) and uses low-pass, band-pass, and high-pass filters to perform spectrum decomposition. Based on this, spectrum allocation is performed, and the second and third bands can be directly allocated to the sky channel.
[0143] To more clearly illustrate the implementation of this application, please refer to Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 as well as Figure 6 Please see the following example instructions.
[0144] Figure 2 This is a flowchart of the algorithm for the next-generation audio upmixer based on deep learning sound source separation. The input is a stereo sound source, and the output is a multi-channel sound source. This multi-channel sound source includes, but is not limited to, 4.0 / 5.1 / 7.1 / 5.1.2 / 7.1.4 / 9.1.6 channel formats. Figure 3 and Figure 4 This is the architecture diagram of the new generation of audio upmixer based on deep learning sound source separation in this invention. Figure 2 The following is an explanation of Figures 1 to 6 , specifically describes the overall architecture of the proposed audio upmixer:
[0145] First, the stereo sound source is separated into three sound objects. This can be achieved using deep learning networks such as Demucs and MDX-NET. The original sound source is separated into three sound objects: vocals, background accompaniment (instruments, etc.), and ambient sound (reverberation, cheers, environment, etc.). The separated sound objects are recorded in stereo format, and the original sound and image orientation information is retained. The objective indicator for evaluating sound object separation performance is the source-distortion ratio (SDR). The separation performance of current mainstream deep learning networks can reach SDR>10. From the perspective of subjective listening, this essentially eliminates problems such as sound leakage, artifacts, and distortion, achieving high-quality sound object separation.
[0146] Subsequently, the separated sound objects are spatially rendered, and the sound field is expanded on this basis. These two links are often intertwined, that is, spatial sound image rendering and sound field expansion can be processed in parallel. Spatial sound image rendering uses algorithms to rearrange the positioning of sound objects in the sound field, thereby converting the original two-dimensional stereo stage into a three-dimensional immersive sound stage; sound field expansion uses decorrelation and spectrum allocation algorithms to convert the width, depth and height ( Figure 3 ) to further expand and obtain a wider auditory space. Figure 2You can see the mapping relationship between each channel, including the center channel, left and right channels, surround channels, and sky channels. The subwoofer channel is ignored here. For simplicity and clarity, the formula is expressed as follows:
[0147] Center channel = e·Vocal
[0148] Left and right channels = f (a vocals + background accompaniment)
[0149] Surround channel = DR [background accompaniment] + b·ambience sound or surround channel = DR {FA [background accompaniment]} + b·ambience sound
[0150] Sky Channel = DR {FA [background accompaniment]} + c·Ambient sound
[0151] Wherein, DR[·] represents decorrelation operation, and FA[·] represents spectrum allocation operation.
[0152] like Figure 3 As shown, in a specific embodiment, the audio processing method includes:
[0153] S31: sound source separation;
[0154] The first sound object V1, the second sound object V2 and the third sound object V3 are separated. For example, the human voice, background accompaniment and ambient sound are separated.
[0155] S32: Multiply the first sound object V1 by the fourth coefficient e and arrange it in the first position for playback, such as:
[0156] Center channel = eV1 = e·Voice
[0157] S33: Multiply the first sound object V1 by the fourth coefficient a and then superimpose it with the second sound object V2 to obtain aV1+V2. Then multiply the superimposed sound source aV1+V2 by the fifth coefficient f and arrange it in the second direction for playback, such as:
[0158] Left and right channels = f(aV1+V2) = f(a·voice+background accompaniment)
[0159] S34: Decorrelation processing is performed on the second sound object V2 to obtain DR[V2]. The third sound object V3 is multiplied by the second coefficient b to obtain bV3. Then, the two sound sources DR[V2] and bV3 are superimposed and arranged at a third position for playback, such as:
[0160] Surround channel = DR[V2] + bV3 = DR[background accompaniment] + b·ambience sound
[0161] S35: Perform decorrelation and spectrum allocation on the second sound object V2 to obtain DR{FA[V2]}. Multiply the third sound object V3 by the third coefficient c to obtain cV3. Then, the two sound sources DR{FA[V2]} and cV3 are superimposed and arranged in the fourth direction for playback, as follows:
[0162] Sky channel = DR{FA[V2]} + cV3 = DR{FA[background accompaniment]} + c·ambience
[0163] like Figure 4 As shown, in a specific embodiment, the audio processing method includes:
[0164] S41: Sound source separation;
[0165] The first sound object V1, the second sound object V2 and the third sound object V3 are separated. For example, the human voice, background accompaniment and ambient sound are separated.
[0166] S42: Multiply the first sound object V1 by the fourth coefficient e and arrange it in the first position for playback, such as:
[0167] Center channel = eV1 = e·Voice
[0168] S43: Multiply the first sound object V1 by the fourth coefficient a and then superimpose it with the second sound object V2 to obtain aV1+V2. Then multiply the superimposed sound source aV1+V2 by the fifth coefficient f and arrange it in the second direction for playback, such as:
[0169] Left and right channels = f(aV1+V2) = f(a·voice+background accompaniment)
[0170] S44: Perform decorrelation and spectrum allocation on the second sound object V2 to obtain DR{FA[V2]}. Multiply the third sound object V3 by the second coefficient b to obtain bV3. Then, the two sound sources DR{FA[V2]} and bV3 are superimposed and arranged at a third position for playback, as follows:
[0171] Surround channel = DR{FA[V2]} + bV3 = DR{FA[background accompaniment]} + b·ambience sound
[0172] S45: Perform decorrelation and spectrum allocation on the second sound object V2 to obtain DR{FA[V2]}. Multiply the third sound object V3 by the third coefficient c to obtain cV3. Then, the two sound sources DR{FA[V2]} and cV3 are superimposed and arranged in the fourth direction for playback, as follows:
[0173] Sky channel = DR{FA[V2]} + cV3 = DR{FA[background accompaniment]} + c·ambience
[0174] The present application also provides an electronic device, including:
[0175] Memory, on which computer programs / instructions are stored;
[0176] A processor is used to execute the computer program / instructions in the memory to implement the steps of the above-mentioned audio processing method.
[0177] The embodiment of the present application further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above-mentioned audio processing method when the computer program / instruction is executed by a processor.
[0178] The embodiment of the present application further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned audio processing method when executed by a processor.
[0179] An embodiment of the present application further provides an audio system, comprising the electronic device as described above, and a plurality of speaker modules connected to the electronic device, wherein the plurality of speaker modules are located at different positions respectively.
[0180] An embodiment of the present application further provides a vehicle, characterized in that the vehicle includes the electronic device as described above, or the computer-readable storage medium as described above, or the audio system as described above.
[0181] In the description of this specification, the descriptions with reference to the terms "particularly", "further", "in particular", "may be understood as", "further" and the like are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms are not intended to refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0182] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.
[0183] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. An audio processing method, characterized in that: include: Separate the original audio signal to obtain multiple sound objects; According to the multiple sound objects, the sound image space orientation layout and the sound field space multi-dimensional direction expansion are performed to obtain multi-directional channel sound sources, wherein the multi-directional channel sound sources are played through speaker modules at different positions.
2. The method according to claim 1, characterized in that The multiple sound objects include a first sound object, a second sound object, and a third sound object, the multiple directions include a first direction, a second direction, a third direction, and a fourth direction, and performing a sound image spatial direction layout according to the multiple sound objects includes: Arrange the first acoustic object in the first position; Arrange the first acoustic object and the second acoustic object in the second position; Arrange the second acoustic object and the third acoustic object in the third position; Arrange the second acoustic object and the third acoustic object in the fourth position; The first orientation, the second orientation, the third orientation and the fourth orientation are used to indicate different positions.
3. The method according to claim 2, characterized in that The placing of the first acoustic object and the second acoustic object in the second orientation includes: performing weighted processing on the first acoustic object according to a first coefficient to obtain a first weighted first acoustic object; performing superposition processing on the first weighted first acoustic object and the second acoustic object to obtain a first superposition sound source; The first superimposed sound source is arranged in the second orientation.
4. The method according to claim 2, characterized in that Arranging the second acoustic object and the third acoustic object in the third position includes: performing weighted processing on the third acoustic object according to the second coefficient to obtain a second weighted third acoustic object; performing superposition processing on the second sound object and the second weighted third sound object to obtain a second superposition sound source; The second superimposed sound source is arranged at the third position.
5. The method according to claim 2, characterized in that Arranging the second acoustic object and the third acoustic object in the fourth position includes: performing weighted processing on the third acoustic object according to a third coefficient to obtain a third weighted third acoustic object; performing superposition processing on the second sound object and the third weighted third sound object to obtain a third superposition sound source; The third superimposed sound source is arranged in the fourth orientation.
6. The method according to claim 2, characterized in that The multi-dimensional directional expansion includes width expansion, depth expansion and height expansion, and the multi-dimensional directional expansion of the sound field space according to the multiple sound objects includes: performing a first tuning process on the first sound object and the second sound object to expand the depth of the sound field space; performing a second tuning process on the second sound object to expand the width of the sound field space; A third tuning process is performed on the second sound object to highly expand the sound field space.
7. The method according to claim 6, characterized in that The performing a first tuning process on the first sound object and the second sound object includes: performing weighted processing on the first acoustic object located at the first position according to a fourth coefficient; performing weighted processing on the first acoustic object and the second acoustic object arranged at the second position according to a fifth coefficient; The fourth coefficient is smaller than the fifth coefficient.
8. The method according to claim 6, characterized in that The performing a second tuning process on the second sound object includes: A decorrelation operation is performed on the second sound object arranged at the third position.
9. The method according to claim 6, characterized in that The performing the third tuning process on the second sound object includes: A spectrum allocation operation is performed on the second sound objects arranged at the third position and the fourth position.
10. The method according to claim 9, characterized in that The performing spectrum allocation operation processing on the second sound objects arranged in the third direction and the fourth direction includes: performing a spectrum allocation operation on the second sound object to obtain a sound object in a first frequency band, a sound object in a second frequency band, and a sound object in a third frequency band, wherein the frequency ranges of the first frequency band, the second frequency band, and the third frequency band increase in sequence and do not overlap; Arranging the second acoustic object and the third acoustic object in the third position includes: Arranging the acoustic objects of the first frequency band, the acoustic objects of the second frequency band, and the third acoustic object at the third position; Arranging the second acoustic object and the third acoustic object in the fourth position includes: The acoustic objects in the second frequency band, the acoustic objects in the third frequency band, and the third acoustic object are arranged in the fourth direction.
11. The method according to claim 6, characterized in that The performing the third tuning process on the second sound object further includes: A spectrum allocation operation is performed on the second sound object located at the fourth position.
12. The method according to claim 11, characterized in that The performing spectrum allocation operation processing on the second sound object arranged at the fourth position includes: performing a spectrum allocation operation on the second sound object to obtain a sound object in a second frequency band and a sound object in a third frequency band, wherein the frequency ranges of the second frequency band and the third frequency band increase in sequence and do not overlap; Arranging the second acoustic object and the third acoustic object in the fourth position includes: The acoustic objects in the second frequency band, the acoustic objects in the third frequency band, and the third acoustic object are arranged in the fourth direction.
13. The method according to any one of claims 2 to 12, characterized in that: The method comprises: The first sound object includes human voice, the second sound object includes background accompaniment, the third sound object includes ambient sound, the first position includes a center channel, the second position includes left and right channels, the third position includes surround channels, and the fourth position includes a sky channel.
14. An electronic device, characterized in that: include: Memory, on which computer programs / instructions are stored; A processor, configured to execute the computer program / instructions in the memory to implement the steps of the audio processing method according to any one of claims 1 to 13.
15. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the audio processing method according to any one of claims 1 to 13 are implemented.
16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the audio processing method according to any one of claims 1 to 13 are implemented.
17. A sound system, characterized in that The electronic device comprises the electronic device as claimed in claim 14, and a plurality of speaker modules connected to the electronic device, wherein the plurality of speaker modules are located at different positions respectively.
18. A vehicle, characterized in that: The vehicle includes the electronic device according to claim 14 , or the computer-readable storage medium according to claim 15 , or the audio system according to claim 17 .