Audio processing method, apparatus, device, and storage medium

Abstract

The application discloses an audio processing method, device and equipment and a storage medium, relates to the technical field of audio processing, and discloses an audio processing method, which comprises the following steps: inputting a stereo audio signal into a preset sound source separation model to obtain a plurality of independent audio objects; obtaining original spatial information of each audio object from the stereo audio signal, and obtaining music structure information of the stereo audio signal; generating target spatial parameters of the audio objects in a three-dimensional immersive sound field based on the original spatial information and the music structure information, wherein the target spatial parameters comprise position coordinates and / or movement trajectories; and rendering each audio object based on the target spatial parameters to obtain a multi-channel immersive audio signal. The application can improve the generation quality of the multi-channel immersive audio signal.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to audio processing methods, apparatus, devices and storage media. Background Technology

[0002] Currently, the main methods for upmixing stereo audio signals into multi-channel immersive audio signals rely on algorithms or fixed templates based on the principle of sound field expansion. These methods typically treat the stereo audio signal as a whole and estimate and assign a rough spatial impression by analyzing the relationship between the left and right channels.

[0003] However, such solutions usually fail to achieve precise separation of different sound sources such as human voices and instruments, resulting in blurred positioning and severe interference of each sound object in the sound field. At the same time, they cannot adapt personalized position coordinates or dynamic trajectories to the structural characteristics of the music itself, making it difficult to present a three-dimensional immersive effect that fits the musical expression, resulting in insufficient artistic expression of the processed audio sound field.

[0004] In summary, improving the generation quality of multi-channel immersive audio signals has become a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0005] The main objective of this application is to provide an audio processing method, apparatus, device, and storage medium, which aims to improve the generation quality of multi-channel immersive audio signals.

[0006] To achieve the above objectives, this application proposes an audio processing method, which includes: The stereo audio signal is input into a preset sound source separation model to obtain multiple independent audio objects; The original spatial information of each audio object is obtained from the stereo audio signal, and the musical structure information of the stereo audio signal is obtained. Based on the original spatial information and the music structure information, the target spatial parameters of the audio object in the three-dimensional immersive sound field are generated, wherein the target spatial parameters include position coordinates and / or movement trajectory; Based on the target space parameters, each audio object is rendered to obtain a multi-channel immersive audio signal.

[0007] In one embodiment, before the step of generating the target spatial parameters of the audio object in a three-dimensional immersive sound field based on the original spatial information and the music structure information, the method further includes: The activity information of the audio object in the time dimension is obtained through the sound source separation model. The step of generating the target spatial parameters of the audio object in a three-dimensional immersive sound field based on the original spatial information and the music structure information includes: The effective interval of the audio object on the timeline is determined based on the activity information; Within the effective range, the target spatial parameters of the audio object in the three-dimensional immersive sound field are generated based on the original spatial information and the music structure information.

[0008] In one embodiment, after the step of generating the target spatial parameters of the audio object in a three-dimensional immersive sound field based on the original spatial information and the music structure information, the method further includes: The visual scene of the icon elements corresponding to the audio object in the three-dimensional immersive sound field is displayed through a preset user interaction interface; In response to a user's touch command in the user interface, the target spatial parameters of the audio object in the three-dimensional immersive sound field are updated.

[0009] In one embodiment, the user interface further includes a timeline editor and simulation controls linked to the visualization scene. The step of updating the target spatial parameters of the audio object in the three-dimensional immersive sound field in response to a user's touch command in the user interface includes: In response to the user's drag operation on the icon element, the target spatial parameters of the audio object in the three-dimensional immersive sound field are adjusted; In response to the user's touch operation on the simulated control, the target spatial parameters of the audio object in the three-dimensional immersive sound field are adjusted; In response to the user's operation of selecting a target time interval for the audio object in the timeline editor, the system receives the user's adjustment command for the audio object within the target time interval in the visualization scene or the simulation control, and adjusts the target spatial parameters of the audio object in the three-dimensional immersive sound field according to the adjustment command.

[0010] In one embodiment, the step of obtaining the original spatial information of each of the audio objects from the stereo audio signal includes: Perform acoustic image analysis on the stereo audio signal to obtain the initial direction information of the stereo audio signal in each time-frequency unit; Obtain the object activation information of the weight of each audio object in each time-frequency unit; For each audio object, the initial direction information is weighted and fused according to the object activation information to obtain the original spatial information of the audio object.

[0011] In one embodiment, the step of rendering each of the audio objects based on the target spatial parameters to obtain a multi-channel immersive audio signal includes: If the audio output mode is headphone output, then the binaural rendering algorithm based on object audio is used to render each of the audio objects to obtain a multi-channel immersive audio signal. If the audio output mode is a multi-speaker system output, the layout information of the speakers in the listening space is obtained, and each audio object is rendered using a high-order high-fidelity stereo sound replication algorithm based on the layout information to obtain a multi-channel immersive audio signal.

[0012] In one embodiment, the audio source separation model is a neural network model based on a multi-task learning architecture. The audio source separation model includes an encoder, a bottleneck layer, and a decoder. The step of inputting the stereo audio signal into the preset audio source separation model to obtain multiple independent audio objects includes: The stereo audio signal is input into a preset sound source separation model so that the encoder can extract the shared features of the stereo audio signal. Core features are extracted from the shared features through the bottleneck layer; The core features are processed by the decoder to obtain multiple independent audio objects, wherein the decoder includes a vocal separation branch, an instrument separation branch, and a sound source activity detection branch.

[0013] Furthermore, to achieve the above objectives, this application also proposes an audio processing apparatus, which includes: The audio source separation module is used to input stereo audio signals into a preset audio source separation model to obtain multiple independent audio objects; The information acquisition module is used to acquire the original spatial information of each audio object from the stereo audio signal, and to acquire the musical structure information of the stereo audio signal. A spatial parameter generation module is used to generate target spatial parameters of the audio object in a three-dimensional immersive sound field based on the original spatial information and the music structure information, wherein the target spatial parameters include position coordinates and / or movement trajectory; The rendering module is used to render each of the audio objects based on the target space parameters to obtain a multi-channel immersive audio signal.

[0014] In addition, to achieve the above objectives, this application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the audio processing method described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the audio processing method described above.

[0016] This application proposes an audio processing method. The method involves inputting a stereo audio signal into a preset sound source separation model to obtain multiple independent audio objects; extracting the original spatial information of each audio object from the stereo audio signal, and extracting the musical structure information of the stereo audio signal; generating target spatial parameters of the audio objects in a three-dimensional immersive sound field based on the original spatial information and musical structure information, wherein the target spatial parameters include position coordinates and / or movement trajectories; and rendering each audio object based on the target spatial parameters to obtain a multi-channel immersive audio signal.

[0017] In summary, this application first separates the stereo audio signal into individual audio objects, and then combines the original spatial information and musical structure information of the stereo audio signal to make the target spatial parameters of the generated audio objects in the three-dimensional immersive sound field inherit the spatial design intent of the original stereo mix and dynamically arrange them according to the musical structure of the music itself, breaking the rigid mode of the traditional fixed template. Finally, through spatial rendering based on audio objects, a multi-channel immersive audio signal with better artistic expression is obtained. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the audio processing method of this application. Figure 2 This is a schematic diagram of the sound source separation model provided in Embodiment 1 of the audio processing method of this application; Figure 3 This is a schematic diagram of the implementation process of the audio processing method in Embodiment 2 of this application; Figure 4 This is a schematic diagram of automated trajectory generation provided in Embodiment 2 of the audio processing method of this application; Figure 5This is a schematic diagram of the user interface provided in Embodiment 2 of the audio processing method of this application; Figure 6 This is a schematic diagram of the module structure of the audio processing device according to an embodiment of this application; Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the audio processing method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] Currently, the main methods for upmixing stereo audio signals into multi-channel immersive audio signals rely on algorithms or fixed templates based on the principle of sound field expansion. These methods typically treat the stereo audio signal as a whole and estimate and assign a rough spatial impression by analyzing the relationship between the left and right channels.

[0024] However, such solutions usually fail to achieve precise separation of different sound sources such as human voices and instruments, resulting in blurred positioning and severe interference of each sound object in the sound field. At the same time, they cannot adapt personalized position coordinates or dynamic trajectories to the structural characteristics of the music itself, making it difficult to present a three-dimensional immersive effect that fits the musical expression, resulting in insufficient artistic expression of the processed audio sound field.

[0025] In summary, improving the generation quality of multi-channel immersive audio signals has become a pressing technical problem that needs to be solved in this field.

[0026] This application provides a solution that inputs a stereo audio signal into a preset sound source separation model to obtain multiple independent audio objects; obtains the original spatial information of each audio object from the stereo audio signal, and obtains the musical structure information of the stereo audio signal; generates target spatial parameters of the audio objects in a three-dimensional immersive sound field based on the original spatial information and musical structure information, wherein the target spatial parameters include position coordinates and / or movement trajectory; and renders each audio object based on the target spatial parameters to obtain a multi-channel immersive audio signal.

[0027] In summary, the embodiments of this application first perform source separation processing on the stereo audio signal to obtain independent audio objects. Then, by combining the original spatial information and music structure information of the stereo audio signal, the target spatial parameters of the generated audio objects in the three-dimensional immersive sound field can both inherit the spatial design intent of the original stereo mix and be dynamically arranged according to the music structure itself, breaking the rigid mode of the traditional fixed template. Finally, through spatial rendering based on audio objects, a multi-channel immersive audio signal with better artistic expression is obtained.

[0028] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device that can realize the above functions.

[0029] Based on this, embodiments of this application provide an audio processing method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the audio processing method of this application.

[0030] In this embodiment, the audio processing method includes steps S10 to S40: Step S10: Input the stereo audio signal into the preset sound source separation model to obtain multiple independent audio objects; It should be noted that a sound source separation model refers to an artificial intelligence model trained on large-scale music data, capable of identifying and separating different sound sources from mixed audio signals. It typically employs a deep neural network architecture, such as a model trained on U-Net (Convolutional Networks for Biomedical Image Segmentation, a convolutional neural network) or Transformer (a deep learning model). An audio object refers to an audio data stream representing a specific independent sound source (such as vocals, drums, bass, guitar, piano, etc.) obtained through model separation. Each audio object is usually a mono or stereo audio waveform file or data stream, preserving the timbre and dynamic characteristics of the corresponding sound source.

[0031] It receives stereo audio files (such as WAV or MP3 format) provided by the user or read from a storage device. First, it performs necessary preprocessing on the signal, such as converting it to a time-frequency domain representation. Then, this representation is input to a pre-loaded source separation model, which analyzes and decomposes the mixed spectrum of the input using learned intrinsic parameters, thereby outputting multiple independent audio objects. For example, for a pop music stereo file, the model might output five independent audio objects: a vocal track, a drum track, a bass track, a piano track, and a guitar track.

[0032] In one feasible embodiment, the sound source separation model is a neural network model based on a multi-task learning architecture. The sound source separation model includes an encoder, a bottleneck layer, and a decoder. Step S10 may include steps S101 to S103: Step S101: Input the stereo audio signal into the preset sound source separation model to extract the shared features of the stereo audio signal through the encoder; It should be noted that the encoder is the first half of the neural network, usually consisting of a series of convolutional layers and downsampling layers (such as pooling layers). Its function is to receive the time-frequency representation of the input audio (such as a spectrogram) and extract multi-scale, low-dimensional shared features through layer-by-layer abstraction. These features capture the low-level and mid-level patterns common to various sound components in the input signal.

[0033] Specifically, the input stereo audio signal is first converted into a suitable neural network input format, such as a dual-channel (left and right) amplitude spectrogram or Mel spectrogram. This spectrogram is fed into the encoder, where the first layer learns to detect basic time-frequency edges and textures. As the network deepens, the time-frequency resolution of the feature map gradually decreases through convolution and downsampling, but the number of feature channels increases. This allows deeper layers to capture more global and semantic information, such as pitch contours, rhythmic patterns, and abstract representations of different instrument timbres. Finally, the encoder outputs a highly condensed shared feature that contains all the basic information needed for sound source separation.

[0034] Step S102: Extract core features from shared features through the bottleneck layer; It should be noted that the bottleneck layer is located between the encoder and the decoder. Its function is to further integrate, filter and refine the shared features output by the encoder, and extract the core features most relevant to the task. The bottleneck layer is usually composed of several consecutive convolutional layers, fully connected layers or attention mechanism layers.

[0035] In the bottleneck layer, the network is forced to represent the input signal within an extremely limited dimension, thereby learning to discard irrelevant information and strengthen and distinguish key features related to different sound sources (such as human voice and guitar). These features correspond to higher-level semantic information, such as human voice dominance, drum rhythm, guitar harmonic structure, etc.

[0036] Step S103: The core features are processed by the decoder to obtain multiple independent audio objects. The decoder includes a vocal separation branch, an instrument separation branch, and a sound source activity detection branch.

[0037] It should be noted that the decoder is the latter half of the neural network, usually consisting of a series of upsampling layers and convolutional layers symmetrical to the encoder. Its function is to gradually reconstruct and map the core features extracted by the bottleneck layer back to the high-resolution output space. The vocal separation branch, instrument separation branch, and sound source activity detection branch refer to multiple sub-networks that branch off at the end of the decoder, each sub-network being specifically responsible for generating a specific type of output.

[0038] The decoder receives core features from the bottleneck layer and gradually recovers the temporal and frequency dimensions of the feature map through a series of upsampling (such as transposed convolution) and convolution operations, making it consistent with the size of the input spectrogram. At the end of the decoder, the network structure branches into three parallel task-specific branches: a voice separation branch, an instrument separation branch, and a sound source activity detection branch. The output layer of the voice separation branch generates a voice time-frequency mask. This mask is multiplied point-by-point with the input mixed spectrogram to obtain the estimated voice spectrum, which is then inversely transformed to obtain the voice audio object waveform. The instrument separation branch similarly obtains the instrument audio object waveform. The sound source activity detection branch generates the activity curve of each sound source (corresponding to the voice and each instrument) on the time axis. Its output can be a low-time-resolution sequence, with each time point containing multiple values, representing the activity level of each sound source within that time period.

[0039] For example, in a feasible implementation scenario, the sound source separation model is specifically a multi-task deep learning model (such as a variant of U-Net), and the structural diagram of this sound source separation model is shown below. Figure 2 As shown, a stereo sound source (i.e., a stereo audio signal) is input into a sound source separation model. The encoder in the model extracts the shared features of the stereo audio signal. The bottleneck layer extracts the core features from the shared features. The decoder processes the core features to obtain multiple independent audio objects. The decoder includes a vocal separation branch, an instrument separation branch, and a sound source activity detection branch. The vocal separation branch outputs the vocal track, the instrument separation branch outputs the instrument tracks, and the sound source activity detection branch outputs the sound source activity information.

[0040] Therefore, this embodiment adopts a codec structure based on multi-task learning, which enables the sound source separation model to complete high-fidelity audio separation and sound source activity detection in one stop during a forward propagation process, providing a high-quality and highly available front-end input for the entire immersive sound generation pipeline.

[0041] Step S20: Obtain the original spatial information of each audio object from the stereo audio signal, and obtain the musical structure information of the stereo audio signal; It should be noted that raw spatial information refers to the approximate orientation or sound image position of each audio object in the raw stereo field, usually one or more angular information representing the horizontal direction, such as azimuth. Musical structural information refers to the high-level semantic structure automatically identified from the audio signal through music information retrieval technology, including but not limited to the time points of musical paragraph divisions (such as verse, chorus, intro, interlude, and outro), and the identification of prominent events (such as the start of an instrumental solo, the entrance of vocals, and a powerful chord passage).

[0042] Acquiring raw spatial information and acquiring musical structure information can be two parallel or sequential analysis processes. For acquiring raw spatial information, a feasible approach is to perform fine-grained image analysis on the raw stereo audio signal: in the time-frequency domain, for each time-frequency unit, the level difference and / or time difference of the left and right channel signals within that unit is calculated, thereby estimating the horizontal azimuth angle of the energy source within that unit, forming an azimuth angle distribution map covering the entire piece. Subsequently, combining the activation information of each audio object obtained in step S10 (e.g., the time-frequency mask synchronously obtained from the sound source separation model), the parts of the azimuth angle distribution map highly correlated with a specific audio object are weighted and averaged to obtain more representative raw spatial information of that object, such as an average azimuth angle.

[0043] For acquiring music structure information, a feasible approach is to use a dedicated music structure analysis model. This model receives the original stereo audio signal or its extracted features (such as Mel spectrum), and through its internally trained algorithms (such as those based on recurrent neural networks or self-attention mechanisms), analyzes the temporal variation patterns of the melody, harmony, rhythm, and timbre of the audio signal, automatically labels the start and end times and type tags of different sections, and detects significant prominent events and their occurrence times.

[0044] Step S30: Generate target spatial parameters of the audio object in the three-dimensional immersive sound field based on the original spatial information and music structure information, wherein the target spatial parameters include position coordinates and / or movement trajectory; It should be noted that a 3D immersive sound field refers to an abstract model used to describe the position of sound in three-dimensional space. It typically uses a spherical coordinate system and includes three dimensions: azimuth (horizontal direction), elevation (vertical direction), and distance. Target space parameters refer to the final set of parameters defined for each audio object for positioning or movement within the 3D immersive sound field.

[0045] First, the original spatial information associated with each audio object (e.g., the original azimuth angle of the guitar object is "30 degrees to the left") and the musical structure information of the entire song are read (e.g., the time tag shows that the 60th to 90th seconds are the "interlude" section, and the guitar activity in this section is marked as a "solo event"). Based on this information, the target parameters in three-dimensional space are generated for each audio object through built-in mapping rules or intelligent algorithms.

[0046] For example, one feasible implementation is as follows: The initial position of the "voice" object is mapped to the center of the front of the 3D sound field by default (azimuth angle 0 degrees), and fine-tuned based on any minor offsets that may exist in the original spatial information. For the "guitar" object, its initial position is first set to the left front (e.g., azimuth angle -30 degrees) based on its original spatial information of "30 degrees to the left". Then, when the guitar object is marked as a "solo event" in the "interlude" section, a dynamic trajectory generation rule is triggered. This rule may predefine a "slow surround" trajectory template for solo instruments, thus generating a continuous movement trajectory for the guitar object within the time interval of 60 to 90 seconds, starting from the initial left front position and slowly moving to the right front, possibly accompanied by slight up-and-down fluctuations. For the "drum kit" object, based on its characteristics of containing multiple parts (e.g., snare drum, hi-hat, tom-toms) and the music genre (e.g., rock), it may be automatically split into multiple sub-objects and assigned static position coordinates spread out on a horizontal arc.

[0047] In one feasible embodiment, step a10 may be included before step S30: Step a10: Obtain the activity information of the audio object in the time dimension through the sound source separation model; It should be noted that activity information refers to the time-varying data sequence that is synchronously output by the sound source separation model, representing the sound intensity or probability of each audio object at each moment on the time axis. It is usually represented as a curve or time-frequency mask that changes with time and has a value between 0 and 1. The higher the value, the more significant or dominant the sound of the object is at the current moment.

[0048] Specifically, the pre-defined audio source separation model is designed with a multi-task learning architecture. Its output includes not only the waveforms of the separated audio objects but also an additional activity detection branch. When the model processes stereo audio signals, this activity detection branch analyzes the shared features extracted by the encoder and outputs a corresponding activity curve for each audio source category. For example, for a piece of music containing vocals and a guitar solo, the model will output vocal activity curves and guitar activity curves.

[0049] Based on this, step S30 may include steps S301 to S302: Step S301: Determine the effective interval of the audio object on the timeline based on the activity information; It should be noted that the effective interval refers to the time range within which an audio object, determined based on activity information, needs to be clearly perceived by the listener and processed independently in a spatial manner.

[0050] Read the activity curve corresponding to each audio object, scan the entire curve by setting a predefined activation threshold, and find all continuous time segments where the activity value exceeds the threshold. These continuous time segments are marked as the valid interval of the audio object.

[0051] For example, after analyzing the guitar activity curve, it was found that the values ​​of the guitar activity curve were consistently higher than the threshold during the two time periods from the 90th to the 120th second and from the 150th to the 180th second. Therefore, these two time periods were determined as the effective range of the guitar audio object.

[0052] Step S302: Within the effective range, generate the target spatial parameters of the audio object in the three-dimensional immersive sound field based on the original spatial information and music structure information.

[0053] For moments within the valid interval, based on the original spatial information and musical structure information, the target parameters in three-dimensional space are generated for each audio object through built-in mapping rules or intelligent algorithms.

[0054] For example, when a guitar object is within its valid range, and the musical structure information indicates that the range is an "interlude" or "solo event," a dynamic trajectory is generated for it, looping from the left to the front, based on its original spatial information and the event type. Outside the valid range, although the guitar object may exist in the data, its activity is low, and a default static position can be assigned to it, such as placing it at the back and reducing the gain, to avoid unnecessary interference from silent or weak sound objects in the main soundstage.

[0055] Therefore, by introducing and utilizing the activity information of audio objects, this embodiment can intelligently identify the actual significant time periods during which the sound source emits sound. Based on this determined effective interval, spatial design is carried out to ensure that dynamic sound imaging effects, prominent positioning, and other processing methods are accurately applied to the corresponding key sections of the music, such as solos and lead vocals. This greatly enhances the musical rationality and dramatic expressiveness of the spatial processing. At the same time, it avoids ineffective or interfering spatial processing of audio objects during inactive periods, optimizes computational resources, and makes the final generated sound field cleaner and clearer.

[0056] Step S40: Render each audio object based on the target space parameters to obtain a multi-channel immersive audio signal.

[0057] It should be noted that rendering refers to the process of calculating the signals allocated to each physical channel of the target playback system based on the target spatial parameters of the audio object using acoustic algorithms. Multichannel immersive audio signals refer to audio format signals with a three-dimensional spatial feel suitable for multi-speaker systems (such as Dolby Atmos layouts like 5.1.2 and 7.1.4) or headphones.

[0058] The rendering engine receives all audio objects and their target spatial parameters that change over time, and uses different rendering algorithms depending on the final output device.

[0059] Thus, by first separating the stereo audio signal into individual audio objects, and then combining the original spatial information and musical structure information of the stereo audio signal, the target spatial parameters of the generated audio objects in the three-dimensional immersive sound field can both inherit the spatial design intent of the original stereo mix and be dynamically arranged according to the musical structure of the music itself, breaking the rigid mode of the traditional fixed template. Finally, through spatial rendering based on audio objects, a multi-channel immersive audio signal with better artistic expression is obtained.

[0060] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, steps S50 to S60 may be included after step S30: Step S50: Display the visual scene of the icon elements corresponding to the audio object in the three-dimensional immersive sound field through a preset user interaction interface; It should be noted that the user interface refers to the interface of a software application that provides a graphical operating environment for users. Icon elements are graphical symbols used in the interface to intuitively represent different audio objects, such as a microphone icon for human voice, a guitar icon for guitar, and a drum icon for drum kit. Visualized scenes refer to the graphical simulation of a three-dimensional immersive sound field, usually represented by a sphere, cube, or room model to symbolize the listening space.

[0061] After the initial target space parameters are generated, the interactive editing mode will be launched or switched. In this mode, the main area of ​​the user interface will present a three-dimensional visualization scene. Based on the generated target space parameters, each audio object will be rendered and placed on the corresponding three-dimensional coordinates in the scene with corresponding icon elements. The visualization scene supports users to rotate and zoom the view so as to observe the sound field layout from different angles.

[0062] Step S60: In response to the user's touch command in the user interface, update the target space parameters of the audio object in the three-dimensional immersive sound field.

[0063] It should be noted that touch commands encompass all operation commands that users use to interact with interface elements through input devices such as mice, touch screens, and touchpads, including but not limited to clicking, dragging, long pressing, and drawing.

[0064] Users can directly interact with icon elements in the visualized scene to modify spatial parameters. For example, a user can click and hold an icon representing a specific audio object, then drag it to a new location in the visualized scene. This drag operation is detected in real time, the coordinates of the released position of the icon in the 3D sound field model are calculated, and these coordinates are immediately updated to the static position coordinates of the audio object. Simultaneously, the rendering engine can provide a real-time or near real-time audio preview based on the updated parameters, allowing users to instantly hear the effects of changes in the sound field layout, achieving a WYSIWYG interactive experience. In this way, users can freely and intuitively reshape the entire sound field, such as moving background harmonies to surround the listener from behind, or bringing vocal objects closer to create an intimate singing effect.

[0065] Thus, by providing a graphical user interface, this embodiment transforms the abstract adjustment of three-dimensional acoustic parameters into intuitive and easy-to-operate visual spatial editing, greatly reducing the professional threshold for immersive sound creation. This allows ordinary users to participate deeply in the construction of personalized sound fields through simple drag-and-drop operations without needing to understand complex audio engineering concepts, realizing a shift from passive acceptance to active creation and significantly improving user experience and creative freedom.

[0066] In one feasible embodiment, the user interface further includes a timeline editor and simulation controls that are linked to the visualization scene, and step S60 may include steps S601 to S603: Step S601: In response to the user's drag operation on the icon element, adjust the target space parameters of the audio object in the three-dimensional immersive sound field. Step S602: In response to the user's touch operation on the analog control, adjust the target spatial parameters of the audio object in the three-dimensional immersive sound field; It should be noted that analog controls refer to a series of parameter sliders or knobs provided in the interface to simulate physical laws or adjust audio properties, such as "gravity", "damping", and "elasticity" sliders, as well as "mute", "solo", and "gain" switches and sliders for individual audio tracks.

[0067] The interface sidebar or toolbar provides various simulation controls. For example, when the user adjusts the "Gravity" slider, a simulated downward gravitational acceleration is applied to the motion trajectory of one or more selected audio objects, causing their motion path to curve downwards, like the parabolic trajectory of an object after being launched. When the user adjusts the "Damping" slider, air resistance or friction in motion is simulated, causing the speed of the audio objects to gradually decrease over time. Adjustments to these physical parameters automatically and in real-time recalculate and correct the dynamic motion trajectory function of the affected audio objects, allowing users to quickly create complex motion effects that conform to the laws of natural physics with simple slider operations, without having to manually draw the path for each frame.

[0068] Step S603: In response to the user's operation of selecting a target time interval for the audio object in the timeline editor, the system receives the user's adjustment instructions for the audio object within the target time interval in the visualization scene or simulation control, and adjusts the target spatial parameters of the audio object in the three-dimensional immersive sound field according to the adjustment instructions.

[0069] It's important to note that timeline editors typically display the entire duration of an audio file in a horizontal timeline format, including tracks corresponding to each audio object, allowing users to perform fine-grained editing based on specific time points. A target time interval refers to a precisely defined period of time in the timeline editor, obtained by selecting areas, setting in / out points, etc.; it can also be a single frame.

[0070] In the timeline editor, when a user selects a specific time interval for an audio object track, this operation sets that time interval as the scope of the current editing operation. Subsequent user actions will then be limited to this target time interval. In this mode, the user can draw a motion path for the audio object only within this target time interval in the visual scene, while the trajectory outside the interval remains unchanged. Alternatively, the user can manipulate the simulation controls in this editing mode to adjust only the audio properties of the audio object within this target time interval.

[0071] Therefore, this embodiment integrates a timeline editor and simulation controls into the user interface. The timeline editor provides timing control capabilities, allowing users to perform time-based, sliced, and refined editing of spatial parameters. The simulation controls simplify the creation of complex dynamic trajectories into adjusting intuitive physical parameters, lowering the creative threshold and enabling users to achieve comprehensive immersive sound creation from macro-level global layout to micro-level timing control and natural motion simulation.

[0072] In one feasible embodiment, the step S20, "obtaining the original spatial information of each audio object from the stereo audio signal," includes steps S201 to S203: Step S201: Perform acoustic image analysis on the stereo audio signal to obtain the initial direction information of the stereo audio signal in each time-frequency unit; It should be noted that audio image analysis refers to the technique of estimating the direction of sound source by calculating the relationship between the left and right channel signals of stereo. The time-frequency unit is a basic unit in time-frequency analysis, corresponding to a specific time frame and frequency range. The initial direction information is the horizontal azimuth angle of the sound source estimated on each time-frequency unit based on the differences in intensity, phase, and other differences between the left and right channel data within that unit.

[0073] The original stereo audio signal is converted to the time-frequency domain, resulting in a two-dimensional complex spectrum. For each time-frequency unit in the spectrum, the intensity difference and / or phase difference between the left and right channel spectral values ​​are calculated. According to acoustic principles, the intensity difference primarily indicates the orientation of mid-to-high frequency sound sources, while the phase difference is more important for low-frequency orientation perception. Using a predetermined acoustic model or lookup table, these differences are mapped to an estimated horizontal azimuth angle, for example, -90 degrees to +90 degrees, with 0 degrees representing direct front. This process traverses all time-frequency units, resulting in an initial orientation information map covering the entire time and frequency range, recording the possible directions of each minute sound component.

[0074] Step S202: Obtain the object activation information of the weight of each audio object in each time-frequency unit; It should be noted that object activation information refers to the time-frequency mask corresponding to each audio object obtained from the audio source separation model. This mask is the same as the time-frequency grid used in audio-visual analysis. The value of the mask in each time-frequency unit (usually between 0 and 1) represents the proportion of energy or the probability of existence of the audio object in this unit.

[0075] The sound source separation model generates a corresponding mask matrix for each separated audio object, which is the object activation information. For example, a human voice audio object corresponds to a human voice mask, and a guitar audio object corresponds to a guitar mask. The higher the mask value, the more likely the energy in that time-frequency unit is to come from the corresponding audio object.

[0076] Step S203: For each audio object, the initial direction information is weighted and fused according to the object activation information to obtain the original spatial information of the audio object.

[0077] It should be noted that weighted fusion refers to the process of assigning and weighting the initial direction information on the same time-frequency unit by using the object activation information as the weight.

[0078] For a specific audio object whose raw spatial information needs to be calculated, iterate through all time-frequency units. For each unit, read its corresponding audio object activation mask value as a weight w, and read the initial orientation angle θ of that unit obtained through sound image analysis. Then, calculate the weight w for all time-frequency units. The product of θ is summed and then divided by the sum of all weights w to calculate a weighted average azimuth angle. This weighted average azimuth angle is considered the original spatial information of the audio object. Its physical meaning is that the direction is averaged only for those time-frequency units where the audio object components are dominant (i.e., the w value is high), so as to more accurately reflect the overall sound image tendency of the audio object in the original stereo and avoid the erroneous inclusion of the direction information of other sound sources or silent segments in the averaging calculation.

[0079] Therefore, this embodiment combines the audio image analysis results with the object activation information obtained through sound source separation. By using the object activation information as weights, it can accurately extract and attribute the spatial orientation of each individual audio object from the mixed stereo orientation information. This method is more accurate than performing overall audio image analysis on the completely separated audio track because it only estimates the direction in the frequency spectrum where the audio object is truly active and dominant, effectively eliminating interference.

[0080] In one feasible embodiment, step S40 may include steps S401-S402: Step S401: If the audio output mode is headphone output, then the binaural rendering algorithm based on object audio is used to render each audio object to obtain a multi-channel immersive audio signal. It should be noted that the object-based audio binaural rendering algorithm is a technology designed specifically for headphone playback. Its core is to use the Head Related Transfer Functions (HRTF) database for rendering. HRTF is a set of filters that describe the spectral modifications that sound undergoes as it travels from a point in space to the left and right eardrums of a person, which are related to the shape of the head, torso, and auricle.

[0081] When the output mode is detected or the user selects headphones, the rendering engine invokes the binaural renderer. This renderer has a built-in HRTF dataset that supports general or personalized measurements. For each audio object at each moment, the renderer selects a pair of filters in the corresponding direction from the HRTF dataset or generates them through interpolation based on its current target space parameters. Then, it convolves the mono signal of the audio object with the left and right ear HRTFs respectively to generate the left and right ear signals contributed by the object. After all audio objects have been processed, the left ear signals are summed to obtain the final left channel output, and the right ear signals are summed to obtain the final right channel output. The final output is a two-channel stereo file, but through HRTF processing, it can produce an accurate three-dimensional spatial auditory impression when wearing headphones.

[0082] Step S402: If the audio output mode is a multi-speaker system output, the layout information of the speakers in the listening space is obtained, and each audio object is rendered according to the layout information using a high-order high-fidelity stereo sound replication algorithm to obtain a multi-channel immersive audio signal.

[0083] It should be noted that the speaker layout information refers to the precise physical coordinates of each speaker actually existing in the listening room in three-dimensional space. The high-order high-fidelity stereo sound replication algorithm is an advanced spatial audio technology based on sound field principles. It encodes sound objects into spherical harmonic function coefficients, and then decodes them into individual speaker signals according to the specific speaker layout.

[0084] When the output mode is a multi-speaker system, the speaker layout needs to be obtained first. This can be achieved by the user manually selecting a standard template or by automatic reporting from a compatible amplifier. Then, the rendering engine uses the HOA (Higher Order Ambisonics) algorithm to output a multi-channel audio file or data stream, with each channel corresponding to a physical speaker. The advantages of the HOA method are its high sound field reconstruction quality, a certain degree of tolerance for listening position, and relatively relaxed requirements on the regularity of the speaker layout.

[0085] Therefore, this embodiment employs appropriate rendering algorithms for the two main types of immersive sound playback terminals—headphones and multi-speaker systems—ensuring that the final generated audio signal can reproduce a high-quality three-dimensional sound field effect on the target device.

[0086] For example, to help understand the implementation flow of the audio processing method obtained by combining this embodiment with the first embodiment described above, please refer to... Figure 3 , Figure 3 A simplified flowchart of an audio processing method is provided, specifically: First, the input stereo sound source (i.e., stereo audio signal) undergoes AI (artificial intelligence) sound source separation to isolate different components in the mixed signal, resulting in multiple independent audio objects. Then, spatial parameters are defined for each audio object, using two methods: initial position assignment and interactive trajectory definition. During initial position assignment, based on the original spatial information of each audio object analyzed from the original stereo signal and the music structure information obtained through a music structure analysis model, a suggested position and trajectory in a three-dimensional immersive sound field is intelligently assigned to each audio object. During interactive trajectory definition, users can manually draw the trajectory through a graphical interface or define complex dynamic movement trajectories for the audio objects using rule-based automation. Finally, the audio objects with defined spatial parameters are spatially rendered to obtain the output audio file, i.e., a multi-channel immersive audio signal.

[0087] The schematic diagram of automated trajectory generation is as follows: Figure 4 As shown, Figure 4 The horizontal axis represents the time progression of a song and is clearly divided into different musical structural segments: intro, verse, chorus, interlude, and back to the chorus. This segment division is based on the musical structure information obtained through a music structure analysis model. In the interlude segment, this segment is marked as a guitar solo event. Combining the time axis, in the intro, verse, and chorus segments, most audio objects such as vocals, drums, violin, and piano are assigned relatively fixed and stable positions. For example, vocals are always placed in the front center, the piano in the left front, the violin in the left rear, and the drums in the rear center. When the music progresses to the interlude segment and a guitar solo event is identified, a dynamic surround trajectory is generated for the guitar audio object, thus highlighting the solo segment in the three-dimensional sound field and enhancing the immersive experience of the music.

[0088] User interface such as Figure 5 As shown, the interface includes three main areas: a visual scene in the upper left corner, simulation controls in the upper right corner, and a timeline editor at the bottom. These areas allow users to manually adjust the target spatial parameters of each audio object. The visual scene uses a sphere to represent the listening space, containing icon elements corresponding to each audio object. The position of these icons visually displays the current coordinates of the audio object in the 3D sound field, and users can drag and drop these icons to change their spatial position in real time. The simulation space in the upper right corner includes an audio progress adjustment bar, sound effect adjustment bars for each audio object, motion speed adjustment bar, and smoothness adjustment bar, allowing users to manually adjust the sound effects of each audio object. The multi-track timeline editor at the bottom of the interface has a horizontal timeline and each track corresponds to an audio object vertically, allowing users to perform fine-grained time-based editing.

[0089] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the audio processing method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0090] This application also provides an audio processing device, please refer to... Figure 6 The audio processing device includes: The audio source separation module 10 is used to input stereo audio signals into a preset audio source separation model to obtain multiple independent audio objects; The information acquisition module 20 is used to acquire the original spatial information of each audio object from the stereo audio signal, and to acquire the musical structure information of the stereo audio signal. The spatial parameter generation module 30 is used to generate target spatial parameters of the audio object in the three-dimensional immersive sound field based on the original spatial information and music structure information. The target spatial parameters include position coordinates and / or movement trajectory. The rendering module 40 is used to render each audio object based on the target space parameters to obtain a multi-channel immersive audio signal.

[0091] Optionally, the audio source separation module 10 is also used for: By using a sound source separation model, the activity information of audio objects in the time dimension can be obtained; Optionally, the spatial parameter generation module 30 is also used for: Determine the valid interval of the audio object on the timeline based on activity information; Within the effective range, the target spatial parameters of the audio object in the three-dimensional immersive sound field are generated based on the original spatial information and music structure information.

[0092] Optionally, the spatial parameter generation module 30 is also used for: The visual scene of the icon elements corresponding to the audio object in the three-dimensional immersive sound field is displayed through a preset user interaction interface; In response to user touch commands in the user interface, update the target spatial parameters of the audio object in the three-dimensional immersive sound field.

[0093] Optionally, the user interface also includes a timeline editor and simulation controls that are linked to the visualization scene, and a spatial parameter generation module 30, which is also used for: In response to the user's dragging operation on the icon element, adjust the target space parameters of the audio object in the three-dimensional immersive sound field; In response to the user's touch operation on the analog controls, the target spatial parameters of the audio object in the three-dimensional immersive sound field are adjusted; In response to the user's selection of a target time interval for an audio object in the timeline editor, the system receives adjustment instructions from the user for the audio object within the target time interval in the visualization scene or simulation control, and adjusts the target spatial parameters of the audio object in the three-dimensional immersive sound field according to the adjustment instructions.

[0094] Optionally, the information acquisition module 20 is also used for: Perform acoustic image analysis on the stereo audio signal to obtain the initial direction information of the stereo audio signal in each time-frequency unit; Obtain the object activation information of the weight of each audio object in each time-frequency unit; For each audio object, the initial direction information is weighted and fused based on the object activation information to obtain the original spatial information of the audio object.

[0095] Optionally, the rendering module 40 is also used for: If the audio output mode is headphone output, then the binaural rendering algorithm based on object audio is used to render each audio object to obtain a multi-channel immersive audio signal. If the audio output mode is a multi-speaker system output, the layout information of the speakers in the listening space is obtained, and a high-order high-fidelity stereo sound replication algorithm is used to render each audio object based on the layout information to obtain a multi-channel immersive audio signal.

[0096] Optionally, the sound source separation model is a neural network model based on a multi-task learning architecture. The sound source separation model includes an encoder, a bottleneck layer, and a decoder. The sound source separation module 10 is also used for: The stereo audio signal is input into a preset sound source separation model so that the shared features of the stereo audio signal can be extracted by the encoder. Extract core features from shared features through a bottleneck layer; The core features are processed by the decoder to obtain multiple independent audio objects. The decoder includes a vocal separation branch, an instrument separation branch, and a sound source activity detection branch.

[0097] The audio processing apparatus provided in this application, employing the audio processing method described in the above embodiments, can improve the generation quality of multi-channel immersive audio signals. Compared with the prior art, the beneficial effects of the audio processing apparatus provided in this application are the same as those of the audio processing method provided in the above embodiments, and other technical features in the audio processing apparatus are the same as those disclosed in the audio processing method of the above embodiments, and will not be repeated here.

[0098] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the audio processing method in the first embodiment described above.

[0099] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of this application. The electronic devices in these embodiments may include, but are not limited to, mobile terminals such as laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0100] like Figure 7 As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the electronic device to exchange data with other devices wirelessly or via wired communication. Although the diagrams show electronic devices with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented alternatively.

[0101] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0102] The electronic device provided in this application, employing the audio processing method described in the above embodiments, can improve the generation quality of multi-channel immersive audio signals. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the audio processing method provided in the above embodiments, and other technical features of the electronic device are the same as those disclosed in the audio processing method of the previous embodiment, and will not be repeated here.

[0103] It should be understood that the various parts disclosed in the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0104] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0105] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the audio processing method in the above embodiments.

[0106] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0107] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0108] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: input a stereo audio signal into a preset sound source separation model to obtain multiple independent audio objects; obtain the original spatial information of each audio object from the stereo audio signal, and obtain the musical structure information of the stereo audio signal; generate target spatial parameters of the audio objects in a three-dimensional immersive sound field based on the original spatial information and the musical structure information, wherein the target spatial parameters include position coordinates and / or movement trajectories; and render each audio object based on the target spatial parameters to obtain a multi-channel immersive audio signal.

[0109] Computer program code for performing the operations of the embodiments of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0111] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0112] The readable storage medium provided in this application embodiment is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for performing the above-described audio processing method, which can improve the generation quality of multi-channel immersive audio signals. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the audio processing method provided in the above embodiments, and will not be repeated here.

[0113] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An audio processing method, characterized in that, The audio processing method includes: The stereo audio signal is input into a preset sound source separation model to obtain multiple independent audio objects; The original spatial information of each audio object is obtained from the stereo audio signal, and the musical structure information of the stereo audio signal is obtained. Based on the original spatial information and the music structure information, the target spatial parameters of the audio object in the three-dimensional immersive sound field are generated, wherein the target spatial parameters include position coordinates and / or movement trajectory; Based on the target space parameters, each audio object is rendered to obtain a multi-channel immersive audio signal.

2. The audio processing method as described in claim 1, characterized in that, Before the step of generating the target spatial parameters of the audio object in the three-dimensional immersive sound field based on the original spatial information and the music structure information, the method further includes: The activity information of the audio object in the time dimension is obtained through the sound source separation model. The step of generating the target spatial parameters of the audio object in a three-dimensional immersive sound field based on the original spatial information and the music structure information includes: The effective interval of the audio object on the timeline is determined based on the activity information; Within the effective range, the target spatial parameters of the audio object in the three-dimensional immersive sound field are generated based on the original spatial information and the music structure information.

3. The audio processing method as described in claim 1, characterized in that, After the step of generating the target spatial parameters of the audio object in the three-dimensional immersive sound field based on the original spatial information and the music structure information, the method further includes: The visual scene of the icon elements corresponding to the audio object in the three-dimensional immersive sound field is displayed through a preset user interaction interface; In response to a user's touch command in the user interface, the target spatial parameters of the audio object in the three-dimensional immersive sound field are updated.

4. The audio processing method as described in claim 3, characterized in that, The user interface also includes a timeline editor and simulation controls that are linked to the visualization scene. The step of updating the target spatial parameters of the audio object in the three-dimensional immersive sound field in response to the user's touch command in the user interface includes: In response to the user's drag operation on the icon element, the target spatial parameters of the audio object in the three-dimensional immersive sound field are adjusted; In response to the user's touch operation on the simulated control, the target spatial parameters of the audio object in the three-dimensional immersive sound field are adjusted; In response to the user's operation of selecting a target time interval for the audio object in the timeline editor, the system receives the user's adjustment command for the audio object within the target time interval in the visualization scene or the simulation control, and adjusts the target spatial parameters of the audio object in the three-dimensional immersive sound field according to the adjustment command.

5. The audio processing method as described in claim 1, characterized in that, The step of obtaining the original spatial information of each audio object from the stereo audio signal includes: Perform acoustic image analysis on the stereo audio signal to obtain the initial direction information of the stereo audio signal in each time-frequency unit; Obtain the object activation information of the weight of each audio object in each time-frequency unit; For each audio object, the initial direction information is weighted and fused according to the object activation information to obtain the original spatial information of the audio object.

6. The audio processing method as described in claim 1, characterized in that, The step of rendering each audio object based on the target spatial parameters to obtain a multi-channel immersive audio signal includes: If the audio output mode is headphone output, then the binaural rendering algorithm based on object audio is used to render each of the audio objects to obtain a multi-channel immersive audio signal. If the audio output mode is a multi-speaker system output, the layout information of the speakers in the listening space is obtained, and each audio object is rendered using a high-order high-fidelity stereo sound replication algorithm based on the layout information to obtain a multi-channel immersive audio signal.

7. The audio processing method as described in claim 1, characterized in that, The audio source separation model is a neural network model based on a multi-task learning architecture. The audio source separation model includes an encoder, a bottleneck layer, and a decoder. The step of inputting the stereo audio signal into the preset audio source separation model to obtain multiple independent audio objects includes: The stereo audio signal is input into a preset sound source separation model so that the encoder can extract the shared features of the stereo audio signal. The bottleneck layer extracts core features from the shared features; The core features are processed by the decoder to obtain multiple independent audio objects, wherein the decoder includes a vocal separation branch, an instrument separation branch, and a sound source activity detection branch.

8. An audio processing apparatus, characterized in that, The audio processing device includes: The audio source separation module is used to input stereo audio signals into a preset audio source separation model to obtain multiple independent audio objects; The information acquisition module is used to acquire the original spatial information of each audio object from the stereo audio signal, and to acquire the musical structure information of the stereo audio signal. A spatial parameter generation module is used to generate target spatial parameters of the audio object in a three-dimensional immersive sound field based on the original spatial information and the music structure information, wherein the target spatial parameters include position coordinates and / or movement trajectory; The rendering module is used to render each of the audio objects based on the target space parameters to obtain a multi-channel immersive audio signal.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the audio processing method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the audio processing method as described in any one of claims 1 to 7.

Images