Audio processing methods and related equipment

Abstract

This application discloses an audio processing method and related equipment. The method includes: acquiring initial azimuth parameters corresponding to each sound source signal in the audio to be adjusted, and the beat information of the audio to be adjusted; determining the movement parameters of different sound source signals based on the beat information; and changing each initial azimuth parameter accordingly based on the movement parameters and each preset azimuth difference to obtain spatial audio when each sound source signal is at a target azimuth, where the azimuth difference is the position difference between the target azimuth and the initial azimuth. This application can automatically adjust the playback position of the sound source signals according to the beat information of the audio, so that the musical information of different audios is preserved, while enhancing the spatial surround sound of the audio in the overall listening experience in a similar way to artificial effects, greatly saving the production cost of spatial audio; it can provide listeners with a continuous stereo listening effect, enhance the user's listening experience, and avoid the panoramic sound immersion that can only be produced for a moment in the traditional way.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to audio processing methods and related devices. Background Technology

[0002] Today, more and more listeners are favoring the playback quality of immersive sound. Immersive sound, also known as spatial audio, provides a sense of spatial surround sound compared to stereo sound with left and right channels. This not only greatly enhances the user's listening experience but also offers more creative possibilities for music streaming.

[0003] Currently, most audio on the market is recorded in traditional two-channel stereo form. Audio that has been released as a stereo template needs to be manually produced to create spatial audio, and the production process also has extremely high requirements for the monitoring environment. This makes the production of immersive sound time-consuming and expensive.

[0004] Therefore, it is necessary to provide an effective solution. Summary of the Invention

[0005] This application provides an audio processing method and related equipment for improving the panoramic listening effect of stereo audio.

[0006] The first aspect of this application provides an audio processing method, including:

[0007] Obtain the initial azimuth parameters corresponding to each sound source signal in the audio to be tuned, and the beat information of the audio to be tuned;

[0008] Based on the beat information, the movement parameters of different sound source signals are determined;

[0009] According to the movement parameters and each preset azimuth difference, the initial azimuth parameters are changed accordingly to obtain the spatial audio of each sound source signal when it is in the target azimuth position. The azimuth difference is the position difference between the target azimuth position and the initial azimuth position.

[0010] Optionally, the beat information includes the audio's BPM information and time signature information; determining the movement parameters of different sound source signals based on the beat information includes:

[0011] Based on the ratio between BPM information and time signature information, the time length corresponding to each measure in the audio to be tuned is calculated;

[0012] The travel time of each sound source signal from the initial orientation to the target orientation is calculated using the time length and the preset sound source moving speed.

[0013] The number of orientation updates during the movement period is determined based on the movement duration, and the number of orientation updates is used as the movement parameter.

[0014] Optionally, the step of changing each of the initial azimuth parameters according to the movement parameters and each preset azimuth difference includes:

[0015] Based on the proportional relationship between each preset azimuth difference and the number of azimuth updates, calculate the azimuth update step size for each change of the initial azimuth parameter;

[0016] For each sound source signal, starting from the initial time corresponding to the initial azimuth, the initial azimuth parameter is updated at each time interval until the sound source signal moves from the initial azimuth to the target azimuth; wherein, the initial time is determined according to the beat number information.

[0017] Optionally, the process of determining the start time includes:

[0018] The start time is aligned with the time point of the entire measure of the audio to be adjusted, and the distance between the start time and the end time point of the audio to be adjusted is greater than or equal to the movement duration.

[0019] Optionally, before changing each of the initial azimuth parameters according to the movement parameters and each preset azimuth difference, the method further includes:

[0020] The sound source signals are separated from the audio to be tuned using a neural network algorithm;

[0021] At least one of the sound source signals is subjected to sound field broadening processing, wherein the sound field broadening processing includes splicing a copy of the sound source signal at a preset segment of the sound source signal;

[0022] Based on the original loudness ratio of each of the sound source signals in the audio to be tuned, loudness scaling is performed on each of the sound source signals.

[0023] Frequency compensation is performed on the loudness-scaled sound source signal;

[0024] The signal weights between the sound sources are determined according to the original dissipation time of each sound source signal in the audio to be tuned, and the reverberation processing of all the sound source signals after frequency compensation is performed according to the signal weights to obtain a reverberation signal.

[0025] The azimuth parameters corresponding to all the sound source signals are used as the initial azimuth parameters.

[0026] Optionally, after performing reverberation processing on all frequency-compensated sound source signals according to the signal weights, and before changing each of the initial azimuth parameters according to the movement parameters and each preset azimuth difference, the method further includes:

[0027] The reverberation signal and all the sound source signals are subjected to azimuth initial modulation to obtain initial stereo sound that conforms to the preset placement.

[0028] The azimuth parameters corresponding to each sound source signal in the initial stereo are used as the initial azimuth parameters.

[0029] Optionally, the sound source signal includes an instrument signal, a human voice signal, and a reverberation signal; the initial azimuth modulation of the reverberation signal and all the sound source signals includes:

[0030] Using the listening position as a base point, the broadcast position between the instrument signal and the reverberation signal is set relative to each other, and the human voice signal is modulated around the listening position.

[0031] Optionally, the method further includes:

[0032] The sound source signals after the initial orientation change are superimposed and their loudness scaled to produce spatial audio with a distortion rate that conforms to a preset distortion range.

[0033] A second aspect of this application provides an electronic device, including:

[0034] Central processing unit, memory, and input / output interfaces;

[0035] The memory is either a short-term storage memory or a persistent storage memory;

[0036] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect of the embodiments of this application or any specific implementation thereof.

[0037] A third aspect of this application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect of this application or any specific implementation thereof.

[0038] A fourth aspect of this application provides a computer program product comprising instructions or a computer program, which, when run on a computer, causes the computer to perform the method described in the first aspect of this application or any specific implementation thereof.

[0039] As can be seen from the above technical solutions, the embodiments of this application have at least the following advantages:

[0040] This application embodiment can automatically adjust the playback position of the sound source signal according to the beat information of the audio, so that the musical information of different audio can be preserved, and the spatial surround sound of the audio can be enhanced in an artificial way, greatly saving the production cost of spatial audio. In addition, the implementation of this method can provide listeners with a continuous stereo sound effect, enhance the user's auditory experience, and avoid the traditional panoramic sound immersion that can only be produced for a moment. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0042] It should be noted that although the steps in the flowcharts (if any) involved in the embodiments are drawn sequentially according to the arrows, unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0043] Figure 1 This is a schematic diagram of a system architecture for a method according to an embodiment of this application;

[0044] Figure 2 This is a schematic flowchart of a method according to an embodiment of this application;

[0045] Figure 3 This is another schematic flowchart of the method in an embodiment of this application;

[0046] Figure 4 This is another schematic flowchart of the method in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of azimuth modulation of a method according to an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of another orientation modulation method according to an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0052] In the following description, expressions such as "one specific implementation" or "one specific example" describe a subset of all possible embodiments. However, it is understood that "one specific implementation" or "one specific example" can be the same or different subset of all possible embodiments and can be combined with each other without conflict. In the following description, the term "multiple" means at least two. When a certain value mentioned in this application reaches a threshold (if it exists), in some specific examples, it may include the former being greater than the latter. When "any" or "at least one" or similar expressions are mentioned, it specifically refers to any one of the listed examples or any combination of these examples.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0054] For ease of understanding and explanation, before providing a further detailed description of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0055] MIR (Music Information Retrieval): Music information retrieval, which may include BPM signal and time signature information.

[0056] BPM (Beat Per Minute), as the name suggests, is the total number of beats in a song per minute, which can express the speed of the song, i.e., the tempo.

[0057] VAD (Voice Activity Detection) is a technology used in speech processing to detect the presence of a speech signal.

[0058] The audio processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the audio in this application can specifically include songs containing dry vocals (including a cappella pure vocal works), although the audio can also be without dry vocals (dry vocals are human voices). Terminal 102 communicates with server 101 via a network. Data storage system 100 can store data that server 101 needs to process; data storage system 100 can be integrated onto server 101 or placed on the cloud or other network servers. Terminal 102 can acquire the audio to be tuned input by the user and send it to server 101. Server 101 can obtain the initial azimuth parameters corresponding to each sound source signal, the beat information (or MIR information) of the audio to be tuned based on the obtained audio, and determine the movement parameters of the sound source signals. Of course, this initial azimuth parameter and MIR information can also be obtained by terminal 102 and transmitted to server 101. Then, server 101 can change the spatial orientation of each sound source object according to the movement parameters and preset azimuth differences to modulate panoramic sound that meets the user's playback requirements.

[0059] The aforementioned terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 101 can be implemented using a standalone server or a server cluster composed of multiple servers. It should be noted that the method provided in this application embodiment can be implemented jointly by the terminal device and the server as described above, or it can be implemented entirely on the server side, or it can be implemented entirely on the terminal device side. The specific implementation can be determined according to the actual application scenario, and no limitation is made here.

[0060] The method described in this application will be further explained in detail below.

[0061] Please see Figure 2 The first aspect of this application provides a specific embodiment of an audio processing method, which includes the following operation steps:

[0062] 21. Obtain the initial orientation parameters and beat information corresponding to the audio signal to be tuned.

[0063] Obtain the initial azimuth parameters corresponding to each sound source signal in the audio to be tuned, as well as the tempo information of the audio to be tuned. In practical applications, the sound source signals include, but are not limited to, at least one sound source object such as vocal, drum, bass, piano, and guitar; the azimuth corresponding to the sound source signal can be understood as the point position of the point sound source object in virtual space, such as coordinates (Azimuth, Elevation, Radius). That is, the three azimuth parameters named azimuth, elevation, and radius can control or express the azimuth information of the sound source (such as a musical instrument like a piano); the order in which the above initial azimuth parameters and tempo information are obtained is not limited.

[0064] The tempo information of the above audio can be understood as MIR information, specifically the BPM information of the song and the time signature (time signature) such as 4 / 4 time. Of course, the above audio can also be other audio works with tempo characteristics besides songs, such as pure vocal humming works.

[0065] 22. Determine the movement parameters of different sound source signals based on beat information.

[0066] To create a spatial surround sound effect, the movement parameters of the sound source signal to undergo a pre-occurrence directional change can be determined based on the beat information of the audio to be tuned. These movement parameters can control the sound source signal to move from the initial directional position to the target directional position.

[0067] 23. Change each initial azimuth parameter according to the movement parameters and each preset azimuth difference to obtain spatial audio.

[0068] Immersive sound, compared to traditional two-channel stereo, has more playback channels and can be regarded as a type of stereo with a stronger sense of space. It can emphasize the reproduction of sound in more different locations, similar to the physical surround sound formed by multiple different speakers placed in different locations, and can virtually create a similar immersive surround effect.

[0069] Specifically, the initial azimuth parameters can be adjusted according to the movement parameters and preset azimuth differences to obtain the spatial audio of each sound source signal when it is in the target azimuth. The azimuth difference is the positional difference between the target azimuth and the initial azimuth. The target azimuth can be determined by the user's actual experience. The optimal spatial placement of drum and bass sounds in a song can be determined by a comprehensive analysis of music theory and listening experience. For example, according to the equal loudness curve of the human ear, the human ear is least sensitive to low frequencies. To ensure that the perceived volume of instruments in the low-frequency band is large enough in a musical work, instruments in this frequency band can be made to "sound bigger" according to their frequency characteristics to achieve frequency balance in the whole song.

[0070] In summary, the embodiments of this application can automatically adjust the playback position of the sound source signal according to the beat information of the audio, so as to preserve the musical information of different audio and enhance the spatial surround sound of the audio in the overall listening experience in a similar way to artificial effects, which greatly saves the production cost of spatial audio; in addition, the implementation of this method can provide listeners with a continuous stereo sound effect, enhance the user's auditory experience, and avoid the panoramic sound immersion that can only be produced for a moment in the traditional way.

[0071] Based on the examples above, some specific possible implementation examples will be provided below. In practical applications, the implementation details of these examples can be combined as needed according to the corresponding functional principles and application logic.

[0072] Please see Figures 3 to 6 This application provides another specific embodiment of an audio processing method, which includes the following operation steps:

[0073] 30. Separate the sound source signal and perform preprocessing.

[0074] In some specific examples, the original audio can be processed to separate the sound sources and preprocess the separated sound source signals; correspondingly, such as Figure 4 As shown, step 30 may specifically include the following operations (i) and (ii):

[0075] (a) Sound source separation.

[0076] The neural network algorithm separates the signals of each sound source from the audio to be tuned. For example, the left and right channels of the original stereo audio (such as the original song) can be input into the neural network algorithm, which can then separate the signals of each sound source from the audio to be tuned, such as signals of various common sound sources such as human voice, guitar, piano, bass, etc. Of course, it can also be other unlisted sound source objects such as pipa and guzheng.

[0077] As one possible implementation, the original loudness ratio (i.e., volume ratio) of the separated multi-source signals in the original stereo audio can be calculated so that the loudness of each source signal can be scaled later to preserve and restore the loudness ratio of each source signal in the original work, avoid sound quality loss, and enhance the fidelity of audio processing.

[0078] (ii) Preprocessing the separated sound source signals.

[0079] (1) Perform sound field widening processing on at least one sound source signal. The sound field widening processing includes splicing a copy of the sound source signal at a preset segment of the sound source signal.

[0080] Sound field widening can be achieved through two methods: delaying and adding reverb, or increasing the difference signal, thereby decorrelated between different sound source signals. Although the separated signal components offer greater freedom compared to directly adjusting stereo, they are still far from the track-by-track signals required for proper music production. Therefore, further processing is needed. Figure 4 Based on the six sound source signals, new sound signals, such as copy signals, are constructed. Here, multiple sound source signals that are most appealing to the listener can be selected for copying, such as signals with frequency indicators reaching a preset frequency, like guitar, piano, or vocal signals. Alternatively, each separated signal can be copied. Regarding the delay + reverberation method, copy signals can be spliced ​​at preset segments of the original sound source signals, allowing the copy signals to be played back with smaller intervals relative to the original signals, ensuring the listener still hears the original signal and thus extending the sound field width.

[0081] In terms of increasing the difference signal, the sound field of instruments with large perceived volume (such as piano) can be widened. For example, the difference signal between the two microphones recording the piano can be enhanced to increase the width of the piano's sound field. In the special processing of the piano, the maximum output of the difference signal can be limited to prevent the piano from being placed at the far left or far right in the original piece, which would cause the difference signal to be too large.

[0082] In this embodiment, the purpose of sound field broadening (also known as decorrelation) is to widen the sound field width of the signal, increase the perceived volume of the sound source in the sound field, and make the sound more realistic in terms of three-dimensional space and the sound field seem larger. In some examples, sufficiently low-frequency signals such as drum sounds and bass sounds may not require sound field broadening because such sound sources can themselves convey the surround sound effect of instruments placed in multiple directions. In other words, whether to perform decorrelation processing on each separated sound source signal can be determined by the actual situation or requirements. The aforementioned preset segment can refer to the end node (such as the end time) or the beginning node (such as the beginning time) of a signal. Accordingly, a copy of the signal can be spliced ​​at the end or the beginning of the signal. When splicing at the beginning, whether there is signal crossover between the two signals before and after the copy can be set as needed.

[0083] (2) Standardization or fidelity processing: Based on the original loudness ratio of each sound source signal in the audio to be adjusted, the loudness of each sound source signal is scaled to restore the volume of each sound source signal in the original work and avoid audio distortion.

[0084] It should be noted that the order of execution of the above-mentioned de-revision and standardization processes is not limited and can be determined by the user.

[0085] (3) Equalization compensation: Frequency compensation is performed on the loudness-scaled sound source signal.

[0086] After standardization, an equalizer can be used to compensate for the frequency of different sound source elements. The purpose of equalization adjustment here is twofold: First, to address some problems caused by separation, such as the distortion of vocal timbre due to the lack of high-frequency harmonics in vocals, which is specifically manifested as vocals being separated into drum sounds. Second, to compensate for the loss of certain frequency bands caused by the binaural rendering of spatial audio in advance.

[0087] Based on the above explanation, the frequency-compensated sound source signals can be superimposed to obtain the audio to be tuned in step 31. Alternatively, as another possible implementation, the frequency-compensated sound source signals can be further sidechained and reverb-ed, finally outputting the audio to be tuned in steps 32 and later. It should be noted that the audio to be tuned in steps 32 and later can be considered as a pair of left and right channels superimposed from at least multiple mono signals (such as instrument signals and / or vocal signals). Of course, this superposition process can also include a reverb signal, depending on the specific scenario.

[0088] (4) Side chain replication (which can be regarded as signal weighting) and reverberation: The signal weight between each sound source is determined according to the original dissipation time of each sound source signal in the audio to be tuned, and the reverberation processing is performed on all sound source signals after frequency compensation according to the signal weight to obtain a reverberation signal.

[0089] Because the dissipation times of different sound source signals in the original stereo audio may vary, in order to better simulate the sound field effect, after frequency compensation by the equalization module, weights can be assigned to all frequency-compensated signals (including copy signals), and finally weighted reverberation is used to create a new audio. For example, drum sounds are low-frequency signals and do not dissipate easily, so the weight of drum sounds can be assigned smaller, while vocal signals dissipate easily, so the weight can be assigned larger to make vocals more sustained; in other words, the input ratio for reverberation processing of different sound source signals can be flexibly adjusted. Figure 4 The use of a reverberation module can simulate the process of a sound source signal being dispersed onto a wall and then reflected onto a person's head.

[0090] After completing the above sound source separation and preprocessing operations, step 30 may further include: using the azimuth parameters corresponding to all sound source signals as initial azimuth parameters; in other words, all the aforementioned sound source signals can be superimposed to finally output the audio to be tuned in subsequent steps such as step 32.

[0091] The above operations can be regarded as completing the construction and pre-modulation of the required signal.

[0092] 31. Perform initial azimuth modulation on the reverberation signal and all sound source signals to obtain initial azimuth parameters.

[0093] In some specific examples, the sound source signals include instrument signals, vocal signals, and reverberation signals. Therefore, the sound source signals obtained in step 30, such as the equalized signals, including the vocal and vocal copies before and after replication, reverberation signals, etc., can be subjected to initial azimuth modulation (or static azimuth modulation) to obtain an initial stereo sound that conforms to the preset placement. The azimuth parameters corresponding to each sound source signal in the initial stereo sound (here, it does not necessarily refer specifically to the reverberation signal; that is, the reverberation signal may not participate in dynamic azimuth modulation. If it does participate, the azimuth of a certain instrument or vocal source in the reverberation can be used as the initial azimuth, which can be determined by actual conditions) are used as the initial azimuth parameters for subsequent steps such as step 32. For example, step 30 can finally obtain: 6 original sound source signals + 6 replicated sound source signals + 4 other instrument signals + 2 reverberation signals (reverb-2) = 18 mono signals. The azimuth of these 18 mono signals can be initially modulated, and the modulation effect can be as follows: Figure 5 As shown.

[0094] As one possible implementation, the specific operation process of "pre-modulating the reverberation signal and all sound source signals in the azimuth" mentioned above may include: using the listening position (such as the head position) as a base point, setting the broadcast position between the instrument signal and the reverberation signal (i.e., symmetrical virtual positioning), and modulating the human voice signal around the listening position.

[0095] like Figure 5 As shown, the standard two-channel music production standard can be continued, primarily modulating the two-channel signals to the left and right front of the listening position. Specifically, the vocal signal can be placed in the central area around the listening position (HEAD), such as the front and back or left and right sides in a direct direction (e.g., directly left and directly right). Instrumental signals are modulated to the left and right front of the listening position. The signals behind the HEAD can all be copy delay signals (marked with "copy"), which have decorrelation and can act as surround signals to increase the perception of height above the horizontal plane. The reverb signal is modulated to the left and right rear of the listening position to simulate the surround reverberation signal after the sound source is reflected by the room. After the initial directional modulation, a total of 18 mono signals, or 9 pairs of two-channel signals, can be symmetrically distributed around the ear, thus initially achieving the surround sound effect of panoramic sound. That is, the output of the initial directional modulation can be considered panoramic sound to a certain extent, but its spatial sense still needs to be enhanced. The above-mentioned static azimuth modulation process can be implemented using methods such as Head-Related Transfer Function (HRTF) convolution or HOA encoding / decoding, which will not be analyzed in detail here.

[0096] 32. Obtain the initial orientation parameters and beat information corresponding to the audio signal to be tuned.

[0097] It should be noted that the audio to be tuned mentioned in steps 32 to 34 can be the original input stereo audio, such as the original song before the correlation and normalization processes in step 30; or it can be the initial stereo audio obtained after processing in steps 30 and 31. In this case, the current orientation of each sound source in the initial stereo audio is the initial orientation used in subsequent steps such as step 32. Although the initial stereo audio can also present a panoramic sound spatial surround effect, it is preferable to further enhance the spatial listening experience by performing dynamic orientation modulation on the initial stereo audio (mainly implementing step 34). In other words, compared to static orientation modulation, dynamic orientation modulation is a finer-grained orientation adjustment, and the spatial sense output effect of dynamic orientation modulation is better. In addition, it should be mentioned that the beat information between the original stereo audio and the initial stereo audio remains unchanged. This is because the orientation modulation in this application is a quasi-physical modulation in virtual space, and the original beat (MIR) information of the original stereo audio will be retained if it is not changed.

[0098] 33. Determine the movement parameters of different sound source signals based on beat information.

[0099] In some specific examples, the beat information includes the BPM information and time signature information of the audio. Accordingly, the specific operation process of step 33 may include: calculating the time length corresponding to each measure in the audio to be tuned based on the proportional relationship between the BPM information and the time signature information; calculating the movement time of each sound source signal from the initial orientation to the target orientation by using the time length and the preset sound source movement speed; determining the number of orientation updates during the period based on the movement time, and using the number of orientation updates as the movement parameter.

[0100] For example, a song has a BPM of 120 and a time signature of 4 / 4, meaning the song's tempo is 120 beats per minute, with each measure consisting of 4 beats. Using this data, the duration T of each measure can be calculated.

[0101] T=(60s / BPM)*Time Signature=(60s / 120)*4=2s (seconds);

[0102] The period C (i.e., the duration of movement) of a point sound source (or simply the sound source) can be calculated by scaling T, as shown in the formula:

[0103] C = NT, where the value of N controls or expresses the speed at which the sound source moves. N can take any value from 0.5, 1, 2, and 3.

[0104] like Figure 6As shown, the sound source moves from the set starting coordinate position 1 (Azimuth1, Elevation1, Radius1) to the set ending coordinate position 2 (Azimuth2, Elevation2, Radius2), which is equivalent to an automated process of moving from the initial orientation to the target orientation (this can be referred to as a dynamic orientation modulation process). The movement time of this process is the cycle length C set above. Accordingly, in practical applications, the time step of coordinate (or orientation) update can be customized as step, with a start time of T1 and an end time of T2. The difference between T1 and T2 is the cycle length C mentioned above, and the following relationship can be obtained:

[0105] Location update count: update_count = (T1-T2) / step.

[0106] The values ​​of N for different sound sources can be the same or different. The reasons are as follows: It's understandable that a song is a mixture of numerous sound sources. The addition of different sound source signals is based on the overall rhythmic characteristics of the song. For example, the beat rate (BPM) and time signature of different sound source signals should correspond consistently to match and present the overall rhythm of the song. Therefore, the spatial orientation of each sound source can be uniformly adjusted based on the overall rhythmic information of the song. Under this consideration, the N values ​​for different sound sources can be the same. Alternatively, the N values ​​can be adjusted according to the different distributions of different sound sources in the song (such as dissipation time or loudness ratio). In this case, at least some of the N values ​​for different sound sources can be different, depending on the actual situation or requirements.

[0107] 34. Change each initial azimuth parameter according to the movement parameters and each preset azimuth difference to obtain spatial audio.

[0108] Based on the description of step 33, in some specific examples, the specific operation process of step 34 may include: calculating the azimuth update step size for each change of the initial azimuth parameter according to the proportional relationship between each preset azimuth difference and the number of azimuth updates; for each sound source signal, starting from the starting time corresponding to the initial azimuth of the sound source signal, updating the initial azimuth parameter at each time interval until the sound source signal moves from the initial azimuth to the target azimuth; wherein, the starting time is determined according to the beat number information.

[0109] The operations in steps 33 and 34 can be considered as processes related to dynamic azimuth modulation. For example, the coordinate update step size is as follows:

[0110] delta_azi=(Azimuth2-Azimuth1) / update_count,

[0111] delta_ele=(Elevation2-Elevation1) / update_count,

[0112] delta_r=(Radius2-Radius1) / update_cout,

[0113] Azimuth2-Azimuth1 represents the preset azimuth difference corresponding to the Azimuth parameter, and similarly, Radius2-Radius1 represents the preset azimuth difference corresponding to the Radius parameter. Since audio rendering is streamed, before the initial time T1, the sound source coordinates can be considered as (Azimuth1, Elevation1, Radius1). Upon reaching T1, automated operations begin, i.e., dynamic azimuth modulation starts. Specifically, the sound source coordinates can be updated every time step. The relationship of coordinate updates can be summarized as follows:

[0114] new_azi=Azimuth1+k*delta_azi,

[0115] new_ele=Elevation1+k*delta_ele,

[0116] new_r = Radius1 + k * delta_r,

[0117] In the above formula, k takes any value from 1 to update_count. In this way, the orientation of the sound source (new_azi, new_ele, new_r) can be updated in real time at intervals of step, thereby realizing the automatic movement of the sound source from coordinate 1 to coordinate 2.

[0118] In some specific examples, the process of determining the above-mentioned start time (T1) may include: aligning the start time with the full measure time point of the audio to be adjusted, and the distance between the start time and the end time point of the audio to be adjusted is greater than or equal to the movement duration.

[0119] The selection of automated segments can be defined by the song structure, such as segmenting by verse, chorus, vocal segment, and vocal silence segment. Taking the vocal silence segment (or non-vocal segment) as an example, VAD can be used to extract the silence segment from the vocal signal to obtain the non-vocal segment (such as a pure instrument segment) after removing the vocals. The start time (T1) is aligned with the strong beat time point in the MIR information, so that T1 is set to the full measure time point of the audio. In other words, setting T1 to the full measure time point of the audio can effectively ensure that the dynamic azimuth modulation is more in line with the audio rhythm and prevent audio playback from having timing errors or sound distortion. In order to avoid repeatedly modulating the sound source azimuth of the same sound segment, or wasting azimuth modulation resources on unnecessary sound segments, the distance between the start time (T1) and the end time point of the audio to be modulated can be set to be greater than or equal to the movement time (C). In other words, the end time (T2) of each sound source dynamic azimuth modulation should be at least one movement time (C) earlier than the end time point of the audio to be modulated.

[0120] Furthermore, the time step (step) that acts as an interval or period can be customized based on the movement duration (C). For example, step = C / n, where n can represent a proportional coefficient customized based on practical experience. In summary, the embodiments of this application can reset the three parameters Azimuth, Elevation, and Radius every fixed number of frames (steps), thereby achieving the purpose of automatically and accurately controlling the sound source trajectory.

[0121] As one possible implementation, a compression module can be added before the final signal output to standardize and protect the signal, preventing clipping distortion or signal distortion due to overload output; therefore, based on the above steps, the embodiments of this application may also include the following step 35:

[0122] 35. Audio rendering output.

[0123] Specifically, the sound source signals after the initial orientation change can be superimposed and their loudness scaled to obtain spatial audio with a distortion rate that conforms to a preset distortion range. Here, the sound source signals after the initial orientation change can specifically refer to the sound source signals obtained after static orientation modulation or dynamic orientation modulation. That is, step 35 can be selectively implemented once after step 31 or step 34, or it can be implemented once after each of the two steps, depending on the requirements. The detailed process of loudness scaling can be found in the relevant standardized instructions at step 30, and will not be repeated here. The preset distortion range can be determined based on the signal peak value and / or original loudness ratio of each sound source signal in the original stereo (e.g., the original track). For example, it can be... Figure 5 The 18 mono signals shown are superimposed and normalized into a pair of stereo (left, right) signals for output to meet the user's panoramic sound production needs.

[0124] Steps 32 to 34 are similar to steps 21 to 23, and will not be described in detail here. The execution order of steps 32 or 33 with any of steps 30 to 31 is not limited, and they can be executed simultaneously, depending on the actual operational scenario. It is evident that this method can separate different musical elements based on a neural network structure. Other neural network structures capable of achieving the same purpose can also be used in this method. Furthermore, the equalizers, reverbs, compressors, and other effects involved in this method are not limited to a specific algorithm; they only need to possess similar functionality. Similarly, directional modulation effects are not limited to HRTF, HOA+HRTF, or VBAP+HRTF methods.

[0125] In summary, this method can be mainly divided into an operational framework including neural network sound source separation, sound source object preprocessing and static azimuth modulation, audio rendering output, and dynamic azimuth modulation (e.g., Figure 4 This method can quickly convert stereo signals into surround sound signals, i.e., panoramic sound, according to different output requirements, thereby improving the user's listening experience. Among them, the automatic movement of sound sources based on the song's MIR information and in accordance with the song's rhythm and melody can enhance the overall sense of space in the listening experience and broaden the application scope of panoramic sound in car surround sound systems, live broadcasts, and other fields.

[0126] Comparative analysis reveals that traditional channel-based mixing methods, such as Dolby Atmos, require manual production, consuming significant time and demanding high-quality monitoring environments. This new method, however, automates Atmos production, greatly reducing costs and increasing efficiency. Traditional methods, like conventional mixing algorithms, calculate all signals based on the left and right channels, resulting in strong correlations between signals. Due to binaural hearing, these signals are often mistaken for the same signal, increasing the difficulty of expanding the perceived sound field. Furthermore, the difference in signal quality, while eliminating vocals, also eliminates in-phase instrument sounds, leading to a loss of musical information in the left and right surround channels. This new method, however, achieves source separation and provides automated tuning based on song MIR information. It retains all MIR information and applies different automated processing to different songs, resulting in a song-specific listening experience that mimics manual processing.

[0127] Among these methods, the neural network-based sound source separation achieves excellent decorrelation effects, ensuring that the sum of each signal yields the original auditory experience while maintaining good independence between the signals used to generate spatial audio. This effectively expands the sound field of the music itself. In short, this innovative method proposes a scheme for automated sound image movement based on music MIR information, enabling specific automated processing for each song, efficiently improving the conversion of stereo sound sources into immersive sound, and enhancing the listener's spatial auditory experience.

[0128] Please see Figure 7 The electronic device 700 of this application embodiment may include one or more central processing units (CPUs) 701 and a memory 705, wherein the memory 705 stores one or more applications or data.

[0129] The memory 705 can be volatile or persistent storage. The program stored in the memory 705 can include one or more modules, each module including a series of instruction operations on the electronic device. Furthermore, the central processing unit 701 can be configured to communicate with the memory 705 and execute the series of instruction operations stored in the memory 705 on the electronic device 700.

[0130] Electronic device 700 may also include one or more power supplies 702, one or more wired or wireless network interfaces 703, one or more input / output interfaces 704, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0131] The central processing unit 701 can perform the operations performed by the first aspect or any specific method embodiment of the first aspect, which will not be described in detail here.

[0132] This application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method as described in the first aspect or any specific implementation thereof.

[0133] This application provides a computer program product containing instructions or computer programs, which, when run on a computer, causes the computer to perform the method described in the first aspect or any specific implementation thereof.

[0134] It is understood that, in the various embodiments of this application, the sequence number of each step does not imply the order of execution. The execution order of each step should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0135] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system (if it exists) and device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system or apparatus, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product (computer program product) is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a business server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An audio processing method, characterized in that, include: Obtain the initial azimuth parameters corresponding to each sound source signal in the audio to be tuned, and the beat information of the audio to be tuned; Based on the beat information, the movement parameters of different sound source signals are determined; The beat information includes the BPM information and time signature information of the audio. Determining the movement parameters of different sound source signals based on the beat information includes: calculating the time length corresponding to each measure in the audio to be tuned according to the proportional relationship between the BPM information and the time signature information; calculating the movement duration of each sound source signal from the initial orientation to the target orientation using the time length and a preset sound source movement speed; determining the number of orientation updates during the period based on the movement duration, and using the number of orientation updates as the movement parameter. The initial azimuth parameters are changed accordingly based on the movement parameters and the preset azimuth differences to obtain the spatial audio of each sound source signal when it is in the target azimuth position. The azimuth difference is the position difference between the target azimuth position and the initial azimuth position.

2. The audio processing method according to claim 1, characterized in that, The step of changing each of the initial azimuth parameters according to the movement parameters and each preset azimuth difference includes: Based on the proportional relationship between each preset azimuth difference and the number of azimuth updates, calculate the azimuth update step size for each change of the initial azimuth parameter; For each sound source signal, starting from the initial time corresponding to the initial azimuth, the initial azimuth parameter is updated at each time interval until the sound source signal moves from the initial azimuth to the target azimuth; wherein, the initial time is determined according to the beat number information.

3. The audio processing method according to claim 2, characterized in that, The process of determining the start time includes: The start time is aligned with the time point of the entire measure of the audio to be adjusted, and the distance between the start time and the end time point of the audio to be adjusted is greater than or equal to the movement duration.

4. The audio processing method according to claim 1, characterized in that, Before changing the initial azimuth parameters according to the movement parameters and the preset azimuth differences, the method further includes: The sound source signals are separated from the audio to be tuned using a neural network algorithm; At least one of the sound source signals is subjected to sound field broadening processing, wherein the sound field broadening processing includes splicing a copy of the sound source signal at a preset segment of the sound source signal; Based on the original loudness ratio of each of the sound source signals in the audio to be tuned, loudness scaling is performed on each of the sound source signals; Frequency compensation is performed on the loudness-scaled sound source signal; The signal weights between the sound sources are determined according to the original dissipation time of each sound source signal in the audio to be tuned, and the reverberation processing of all the sound source signals after frequency compensation is performed according to the signal weights to obtain a reverberation signal. The azimuth parameters corresponding to all the sound source signals are used as the initial azimuth parameters.

5. The audio processing method according to claim 4, characterized in that, After performing reverberation processing on all frequency-compensated sound source signals according to the signal weights, and before changing the initial azimuth parameters according to the movement parameters and each preset azimuth difference, the method further includes: The reverberation signal and all the sound source signals are subjected to azimuth initial modulation to obtain initial stereo sound that conforms to the preset placement. The azimuth parameters corresponding to each sound source signal in the initial stereo are used as the initial azimuth parameters.

6. The audio processing method according to claim 5, characterized in that, The sound source signal includes instrument signal, human voice signal, and reverberation signal; the initial azimuth modulation of the reverberation signal and all sound source signals includes: Using the listening position as a base point, the broadcast position between the instrument signal and the reverberation signal is set relative to each other, and the human voice signal is modulated around the listening position.

7. The audio processing method according to claim 1, characterized in that, The method further includes: The sound source signals after the initial orientation change are superimposed and their loudness scaled to produce spatial audio with the distortion rate of each sound source signal conforming to a preset distortion range.

8. An electronic device, characterized in that, include: Central processing unit, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 7.

Images