Stereo synthesis method and device, stereo model construction method and device, equipment and storage medium

Abstract

The invention provides a stereo synthesis method and device, a model construction method and device, equipment and a storage medium. The audio processing model comprises a sound event positioning and classification network and a sound source separation network. The stereo synthesis method comprises the following steps: acquiring a to-be-processed audio stream; inputting the to-be-processed audio stream into an audio processing model, and performing sound event classification and sound source orientation estimation on the audio stream by the sound event positioning and classification network to obtain the category and orientation of at least one sound event, the sound source separation network performs separation processing on the mixed audio according to the category and the orientation of the at least one sound event to obtain an audio signal of each sound event, and outputs the orientation and the audio signal of each sound event in the at least one sound event; and performing sound mixing processing on the audio signals corresponding to the at least one sound event according to the orientation of each sound event in the at least one sound event, and synthesizing stereo sound.

Description

Technical Field

[0001] The present disclosure relates to the field of audio processing technology, and in particular to a method for constructing an audio processing model, a stereo synthesis method based on the audio processing model, a device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Stereo synthesis refers to the process of converting audio signals into stereo audio signals with a sense of space through computer technology. Stereo synthesis technology can provide listeners with a more realistic and lifelike auditory experience and is widely used in virtual reality, games, movies and other fields. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a method for constructing an audio processing model, a stereo synthesis method based on the audio processing model, a device, an electronic device and a computer-readable storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for constructing an audio processing model is provided, wherein the audio processing model includes a sound event localization and classification network and a sound source separation network; the method includes:

[0005] Acquire a training data set, wherein the training samples in the training data set include first audio signals, a first category, and a first orientation respectively corresponding to at least two sound events, and mixed audio obtained by mixing the first audio signals respectively corresponding to the at least two sound events;

[0006] Inputting the training data set into the audio processing model to be trained, so that the sound event localization and classification network classifies the sound events and estimates the sound source orientation according to the mixed audio to obtain a second category and a second orientation of at least one sound event, and the sound source separation network separates the mixed audio according to the second category and the second orientation of the at least one sound event to obtain a second audio signal of each sound event, and the audio processing model is trained with minimizing the error between the first audio signal and the second audio signal, the error between the first category and the second category, and the error between the first orientation and the second orientation as optimization goals;

[0007] The audio processing model is used to output the location and audio signal of at least one sound event contained in the audio stream based on the input audio stream, so as to be used for synthesizing stereo.

[0008] In the disclosed embodiment, on the one hand, a step-by-step processing method is adopted to split the audio processing task into multiple steps, and the category and orientation of the sound event are obtained by the sound event localization and classification network, and then the mixed audio is separated and processed by the sound source separation network. Each step is relatively simple and has low computational complexity, so that the computational complexity of the entire audio processing task will be greatly reduced. On the other hand, when the sound source separation network separates the mixed audio, it refers to the category and orientation of the sound event obtained by the sound event localization and classification network, so that the sound source separation network can accurately and quickly separate the audio signals of each sound event, which helps to reduce processing delay. In summary, the sound event localization and classification network and the sound source separation network in the audio processing model are optimized and improved for the audio processing task, and after training, they have high efficiency and performance in processing the tasks of sound event localization, classification and separation, so that the computational complexity of the whole process is low, and the processing can be completed in a shorter time.

[0009] According to a second aspect of an embodiment of the present disclosure, a stereo synthesis method based on an audio processing model is provided, wherein the audio processing model includes a sound event localization and classification network and a sound source separation network; the audio processing model is constructed based on the method described in the first aspect; the method includes:

[0010] Get the audio stream to be processed;

[0011] Input the audio stream to be processed into the audio processing model, so that the sound event localization and classification network performs sound event classification and sound source orientation estimation on the audio stream to obtain the category and orientation of at least one sound event, and the sound source separation network performs separation processing on the mixed audio according to the category and orientation of the at least one sound event to obtain audio signals of each sound event, and outputs the orientation and audio signal of each sound event in the at least one sound event;

[0012] According to the orientation of each sound event in the at least one sound event, audio signals respectively corresponding to the at least one sound event are mixed to synthesize stereo sound.

[0013] In the embodiment of the present disclosure, since the audio processing model can accurately output the direction and audio signal of each sound event in at least one sound event included in the audio stream, accurate stereo sound can be obtained through simple mixing processing with the direction of each sound event as a reference, thereby improving processing efficiency and ensuring the accuracy of stereo synthesis.

[0014] According to a third aspect of an embodiment of the present disclosure, a device for constructing an audio processing model is provided, wherein the audio processing model includes a sound event localization and classification network and a sound source separation network; the device includes:

[0015] A training data acquisition module is used to acquire a training data set, wherein the training samples in the training data set include first audio signals, a first category and a first orientation respectively corresponding to at least two sound events, and a mixed audio obtained by mixing the first audio signals respectively corresponding to the at least two sound events;

[0016] A model training module is used to input the training data set into the audio processing model to be trained, so that the sound event localization and classification network can classify sound events and estimate the sound source orientation according to the mixed audio to obtain a second category and a second orientation of at least one sound event, and the sound source separation network can separate the mixed audio according to the second category and the second orientation of the at least one sound event to obtain a second audio signal of each sound event, and train the audio processing model with the optimization goal of minimizing the error between the first audio signal and the second audio signal, the error between the first category and the second category, and the error between the first orientation and the second orientation; wherein the audio processing model is used to output the orientation and audio signal of at least one sound event contained in the audio stream based on the input audio stream for stereo synthesis.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a stereo synthesis device based on an audio processing model is provided, wherein the audio processing model includes a sound event localization and classification network and a sound source separation network; the audio processing model is constructed based on the device described in the third aspect; the device includes:

[0018] An audio stream acquisition module, used to acquire the audio stream to be processed;

[0019] An audio stream processing module, used for inputting the audio stream to be processed into an audio processing model, so that the sound event localization and classification network performs sound event classification and sound source orientation estimation on the audio stream to obtain the category and orientation of at least one sound event, and the sound source separation network performs separation processing on the mixed audio stream according to the category and orientation of the at least one sound event to obtain audio signals of each sound event, and outputs the orientation and audio signal of each sound event in at least one sound event;

[0020] The stereo synthesis module is used to perform mixing processing on the audio signals respectively corresponding to at least one sound event according to the orientation of each sound event in at least one sound event to synthesize stereo.

[0021] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0022] processor;

[0023] a memory for storing a computer program executable by the processor;

[0024] Wherein, when the processor executes the program, the steps of the method described in the first aspect or the second aspect are implemented.

[0025] According to a sixth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a schematic diagram of an application scenario shown in accordance with an exemplary embodiment of the present disclosure;

[0028] Figure 2 is a schematic diagram of another application scenario shown in the present disclosure according to an exemplary embodiment;

[0029] Figure 3 is a flowchart of a method for constructing an audio processing model according to an exemplary embodiment of the present disclosure;

[0030] Figure 4 is a structural schematic diagram of an audio processing model according to an exemplary embodiment of the present disclosure;

[0031] Figure 5 is a structural diagram of a sound event localization and classification network according to an exemplary embodiment of the present disclosure;

[0032] Figure 6 is a schematic diagram of the structure of another sound event localization and classification network according to an exemplary embodiment of the present disclosure;

[0033] Figure 7 is a structural schematic diagram of a sound source separation network according to an exemplary embodiment of the present disclosure;

[0034] Figure 8 is a schematic diagram of the structure of another sound source separation network according to an exemplary embodiment of the present disclosure;

[0035] Fig. 9 is a schematic diagram of a training sound event localization and classification network according to an exemplary embodiment of the present disclosure;

[0036] Fig.10 is a schematic diagram of a training sound source separation network according to an exemplary embodiment of the present disclosure;

[0037] Fig.11 is a flowchart of a stereo synthesis method based on an audio processing model according to an exemplary embodiment of the present disclosure;

[0038] Fig.12 is a flowchart of another stereo synthesis method based on an audio processing model according to an exemplary embodiment of the present disclosure;

[0039] Fig.13 is a structural block diagram of a device for constructing an audio processing model according to an exemplary embodiment of the present disclosure;

[0040] Fig.14 is a structural block diagram of a stereo synthesis device based on an audio processing model according to an exemplary embodiment of the present disclosure;

[0041] Fig.15 It is a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0043] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0044] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0045] Stereo synthesis technology can provide listeners with a more realistic and lifelike auditory experience and can be applied to electronic devices such as mobile phones, tablets, headphones, smart speakers, and vehicle audio, but is not limited to these. Application areas include but are not limited to:

[0046] 1. Virtual Reality (VR) and Augmented Reality (AR): Stereo synthesis technology can provide a more realistic and lifelike auditory experience for virtual reality and augmented reality applications, enhancing the user's sense of immersion and experience.

[0047] 2. Game scene: Provide more realistic and lifelike sound effects for the game, and enhance the immersiveness and experience of the game.

[0048] 3. Movies and Music: Provide more realistic and lifelike sound effects for movies and music, enhancing the sensory experience of viewers and listeners.

[0049] 4. Voice interaction: Provide more natural and realistic voice output for voice interaction applications to enhance the user's interactive experience.

[0050] 5. Education and Training: Provide a more vivid and realistic auditory experience for education and training applications, enhancing the learning effect and experience of students and trainers.

[0051] 6. Audio Advertising and Marketing: Provide more vivid and realistic sound effects for audio advertising and marketing applications, enhancing the effectiveness and appeal of advertising and marketing.

[0052] 7. Vehicle audio: By distributing audio signals to different speaker locations in the vehicle, music or sound can be delivered to the driver and passengers from different directions. This enhanced sense of space can make the music richer and more three-dimensional, creating a more immersive listening experience.

[0053] In summary, stereo synthesis technology has a wide range of applications in various fields, mainly providing users with a more realistic and lifelike auditory experience and enhancing users' sense of immersion and experience.

[0054] The stereo synthesis methods in the related art include methods based on deep learning, and common methods based on deep learning include methods based on convolutional neural networks (CNN), methods based on recurrent neural networks (RNN), methods based on generative adversarial networks (GAN), etc. These methods can automatically learn the time domain and frequency domain features of audio signals by learning a large amount of training data, and generate stereo audio signals with a sense of space; however, the deep learning methods in the related art directly output the synthesized stereo based on the mixed audio signals by the trained deep learning model, which has a high processing complexity, resulting in a high delay in stereo synthesis.

[0055] Based on this, the embodiments of the present disclosure provide a method for constructing an audio processing model, and a stereo synthesis method based on the audio processing model, an apparatus, an electronic device and a computer-readable storage medium.

[0056] In terms of the construction of the audio processing model, the audio processing model is trained using a training data set, and the audio processing model includes a sound event localization and classification network and a sound source separation network. The sound event localization and classification network is used to classify sound events and estimate the sound source orientation of the input audio stream to obtain the category and orientation of at least one sound event; the sound source separation network is used to separate and process the mixed audio according to the category and orientation of at least one sound event obtained by the sound event localization and classification network to obtain the audio signals of each sound event. This embodiment proposes the above-mentioned audio processing model, which is conducive to reducing computational complexity.

[0057] On the one hand, a step-by-step processing method is adopted to split the audio processing task into multiple steps. The sound event localization and classification network obtains the category and direction of the sound event, and then the sound source separation network separates the mixed audio. Each step is relatively simple and has low computational complexity, so the computational complexity of the entire audio processing task will be greatly reduced.

[0058] On the other hand, when the sound source separation network separates the mixed audio, it refers to the category and orientation of the sound event obtained by the sound event localization and classification network, which enables the sound source separation network to accurately and quickly separate the audio signals of each sound event, helping to reduce processing delays.

[0059] In summary, the sound event localization and classification network and the sound source separation network in the audio processing model are optimized and improved for audio processing tasks. After training, they have higher efficiency and performance in processing sound event localization, classification and separation tasks, which makes the computational complexity of the whole process lower and can complete the processing in a shorter time.

[0060] In actual application, after obtaining the audio stream to be processed, the audio stream can be input into the audio processing model, so that the sound event localization and classification network can classify the audio events and estimate the sound source orientation of the audio stream to obtain the category and orientation of at least one sound event, and the sound source separation network can separate the mixed audio according to the second category and the second orientation of at least one sound event to obtain the audio signals of each sound event, and output the orientation and audio signals of each sound event in at least one sound event; the audio signals of each sound event in at least one sound event are mixed according to the orientation of each sound event in at least one sound event to synthesize stereo. In this embodiment, since the audio processing model can accurately output the orientation and audio signal of each sound event in at least one sound event included in the audio stream, taking the orientation of each sound event as a reference, accurate stereo can be obtained through simple mixing processing, thereby improving the processing efficiency while ensuring the accuracy of stereo synthesis.

[0061] In some embodiments, the method for constructing an audio processing model and the method for stereo synthesis based on the audio processing model can be performed by an electronic device. The two methods can be performed by the same electronic device or by different electronic devices, and this embodiment does not impose any restrictions on this. Exemplarily, after the audio processing model is constructed by the first electronic device, the audio processing model can be deployed in the second electronic device, and the second electronic device can perform the stereo synthesis method based on the audio processing model.

[0062] Electronic devices include, but are not limited to, smart phones / cell phones, tablet computers, personal digital assistants (PDAs), laptop computers, desktop computers, media content players, video game stations / systems, virtual reality systems, augmented reality systems, wearable devices (e.g., watches, glasses, gloves, headwear (e.g., hats, helmets, virtual reality headsets, augmented reality headsets, head-mounted devices (HMDs), headbands), pendants, armbands, leg bands, shoes, vests), speakers, air conditioners, remote controls, vehicle stereos / speakers, or any other type of device.

[0063] In an exemplary application scenario, see Figure 1 , the electronic device 10 is deployed with an audio processing model, and the electronic device 10 is provided with a microphone 20 and two speakers 30. After acquiring the audio stream collected by the microphone 20, the electronic device 10 can perform the stereo synthesis method based on the audio processing model provided in the embodiment of the present disclosure based on the audio stream to obtain stereo sound, and then play the stereo sound through the two speakers 30. It can be understood that this embodiment does not impose any restrictions on the number of microphones and speakers, the number of microphones can be one or more than one, and the number of speakers can also be one or more than one.

[0064] In another exemplary application scenario, see Figure 2 The electronic device 10 is deployed with an audio processing model. The electronic device 10 can receive an audio stream transmitted by an external device 40, and execute a stereo synthesis method based on an audio processing model provided by an embodiment of the present disclosure based on the audio stream to obtain stereo sound, and then return the stereo sound to the external device 40.

[0065] It should be noted that this embodiment does not impose any restriction on the number of channels of the audio stream, and the audio stream can be a single-channel audio stream (such as Figure 1 The audio stream shown is collected by a single microphone), or it can be a multi-channel audio stream, such as an audio stream collected by at least two microphones.

[0066] Next, the construction process of the audio processing model is explained as follows:

[0067] In some embodiments, the construction process of the audio processing model can be: first, a model is represented by modeling, then the model is evaluated by constructing an evaluation function, and finally, the evaluation function is optimized according to sample data and optimization methods to adjust the model to the optimal level.

[0068] 1. Modeling: During the training process, we first need to select a model architecture that is suitable for the task. The model can be a neural network, decision tree, support vector machine, etc. The goal of modeling is to select a model that can effectively fit the data and has appropriate complexity.

[0069] 2. Evaluation Function: During the training process, an evaluation function needs to be defined to measure the difference or error between the model prediction result and the true label. This evaluation function is usually called the loss function, which can measure the performance of the model.

[0070] 3. Optimization Method: During the training process, you need to choose an optimization method to minimize the loss function. Common optimization methods include Stochastic Gradient Descent (SGD), Adam, RMSprop, etc. These methods adjust the model parameters to gradually reduce the loss function, thereby improving the model's predictive ability.

[0071] 4. Training Data: Model training requires a large amount of sample data. These data are usually divided into training set, validation set and test set. The training set is used to update parameters, the validation set is used to adjust model hyperparameters, and the test set is used to evaluate the generalization ability of the model on unseen data.

[0072] 5. Training Iterations: During the model training process, model parameters are gradually adjusted through iterative optimization. Each training iteration includes steps such as forward propagation, loss function value calculation, backward propagation, and parameter update. The number of training iterations is usually set by setting a fixed number of iterations or dynamically adjusted according to the performance of the model on the validation set.

[0073] Through the above steps, the model can be trained using sample data, and after multiple iterations of optimization, the optimal model that can accurately predict and generalize is finally obtained. Please note that during the model training process, technical means such as overfitting control, learning rate adjustment and regularization are also required to improve the training effect and generalization ability of the model.

[0074] like Figure 3 As shown, Figure 3 is a flowchart of a method for constructing an audio processing model according to an exemplary embodiment of the present disclosure, please refer to Figure 4 The audio processing model includes a sound event localization and classification network and a sound source separation network; the method for constructing the audio processing model of this embodiment can be executed in one electronic device or in multiple devices in a device cluster. The method includes:

[0075] In S101, a training data set is obtained, wherein training samples in the training data set include first audio signals, a first category and a first orientation respectively corresponding to at least two sound events, and mixed audio obtained by mixing the first audio signals respectively corresponding to the at least two sound events.

[0076] In S102, the training data set is input into the audio processing model to be trained, so that the sound event localization and classification network classifies sound events and estimates the sound source orientation according to the mixed audio to obtain a second category and a second orientation of at least one sound event, and the sound source separation network separates the mixed audio according to the second category and the second orientation of the at least one sound event to obtain a second audio signal of each sound event, and the audio processing model is trained with the optimization goal of minimizing the error between the first audio signal and the second audio signal, the error between the first category and the second category, and the error between the first orientation and the second orientation.

[0077] The audio processing model is used to output the location and audio signal of at least one sound event contained in the audio stream based on the input audio stream, so as to be used for synthesizing stereo.

[0078] In the present embodiment, the step-by-step processing method is adopted, and the audio processing task is split into multiple steps, and the category and orientation of the sound event are obtained by the sound event positioning and classification network, and then the mixed audio is separated and processed by the sound source separation network. Each step is relatively simple and the computational complexity is low, thereby reducing the computational complexity of the audio processing task. And the sound source separation network, when separating the mixed audio, refers to the category and orientation of the sound event obtained by the sound event positioning and classification network, so that the sound source separation network can accurately and quickly separate the audio signals of each sound event, which helps to reduce processing delay. In summary, the sound event positioning and classification network and the sound source separation network in the audio processing model are optimized and improved for the sound processing task, and are trained to have higher efficiency and performance in processing the tasks of sound event positioning, classification and separation, so that the computational complexity of the whole process is lower, and the processing can be completed in a shorter time.

[0079] (1) Preparation of training samples.

[0080] A training data set may be prepared in advance, and this embodiment does not limit the method for obtaining the training data set. This embodiment designs training samples to include: first audio signals corresponding to at least two sound events, a first category and a first orientation, and a mixed audio obtained by mixing the first audio signals corresponding to at least two sound events.

[0081] Sound events refer to sound-related events or phenomena occurring in the environment, such as various events or phenomena caused by sound waves, mechanical vibrations or other acoustic signals generated by humans or other organisms.

[0082] The audio signal of a sound event refers to the audio related to the sound event, such as the sound of music, instrumental music, and human speech.

[0083] Regarding the categories of sound events, by classifying sound events, sound events can be divided into multiple categories, such as speaking, music, car horns, animal calls and other different categories.

[0084] The orientation of a sound event refers to the direction and position in which the audio signal of the sound event occurs, which can be represented by angle information, for example.

[0085] The first audio signal, first category and first direction corresponding to at least two sound events contained in the training sample are calibrated real data, which are used as labels in the training process to evaluate the accuracy of the predicted data output by the audio processing model (such as the second audio data, second category and second direction mentioned later).

[0086] The first audio signals corresponding to at least two sound events can be mixed to obtain a mixed audio as an input of the audio processing model. Exemplarily, the first audio signals corresponding to at least two sound events can be mixed by addition operation to generate a mixed audio. During the mixing process, the amplitude, phase, frequency and other characteristics of each first audio signal will be superimposed on each other to form an overall audio signal.

[0087] The first audio signals corresponding to the at least two sound events may be single-channel audio signals or multi-channel audio signals. Accordingly, the mixed audio obtained by mixing the at least two first audio signals may be single-channel audio signals or multi-channel audio signals.

[0088] (2) About the specific structure of the audio processing model.

[0089] The audio processing model includes a sound event localization and classification network and a sound source separation network.

[0090] See also Figure 5 ,The sound event localization and classification network includes a first feature extraction layer, a feature mapping layer, a sound event classifier and a sound source orientation estimator.

[0091] The first feature extraction layer is used to extract a low-dimensional first audio feature from the mixed audio. Exemplarily, the low-dimensional first audio feature includes a logarithmic Mel spectrum of the mixed audio.

[0092] The feature mapping layer is used to map the low-dimensional first audio feature to the high-dimensional first audio feature. Exemplarily, the feature mapping layer includes a neural network based on the Transformer architecture, such as a convolutional recurrent neural network (CRNN) neural network based on the Transformer architecture.

[0093] A sound event classifier is used to predict the second category corresponding to at least one sound event based on the high-dimensional first audio feature. Exemplarily, the sound event classifier can use a neural network structure such as a convolutional neural network (CNN) or a recurrent neural network (RNN) to predict the category of the sound event based on the input high-dimensional first audio feature, and output the second category corresponding to at least one sound event.

[0094] A sound source direction estimator is used to predict the second direction corresponding to at least one sound event based on the high-dimensional first audio feature. Exemplarily, the sound source direction estimator can adopt a positioning algorithm based on DOA (Direction Of Arrival), such as an array signal processing method or a time difference measurement method, to infer the direction, position or angle of the sound event based on the input high-dimensional first audio feature, and output the second direction corresponding to at least one sound event.

[0095] For the sound event localization and classification network, the first feature extraction layer extracts low-dimensional features from the mixed audio, while the feature mapping layer converts the low-dimensional features into high-dimensional features. This hierarchical structure can help the network learn more representative audio features and extract more information about sound events, thereby improving the accuracy of classification and positioning. Sound event classification and sound source orientation estimation can also be performed simultaneously. By sharing the feature extraction layer and the feature mapping layer, the two tasks can promote each other and improve the overall performance and generalization ability of the network. At the same time, multi-task learning can reduce the number of parameters and save model storage space and computing resources. Furthermore, the network structure of the sound event localization and classification network has fewer layers, which can reduce the computational complexity and achieve higher real-time performance.

[0096] In some embodiments, in order to reduce the processing complexity of the audio processing model and improve the processing efficiency, the input sound event localization and classification network is a single-channel audio stream; if the mixed audio input to the audio processing model is a multi-channel audio stream, then before inputting the sound event localization and classification network, the mixed audio input to the audio processing model needs to be converted, such as converting the mixed audio into a single-channel mixed audio based on a beamforming method for signal processing, and then inputting the single-channel mixed audio into the sound event localization and classification network. This embodiment implements that the sound event localization and classification network can classify and locate the sound time included in the mixed audio even if there is no multi-channel audio data, thereby reducing the processing complexity and improving the processing efficiency.

[0097] See also Figure 6 ,The sound event localization and classification network also includes a judgment layer and a channel conversion layer.

[0098] The judgment layer is used to judge whether the mixed audio input to the audio processing model is single-channel audio; if so, the mixed audio is directly input to the first feature extraction layer; if not, the mixed audio is input to the channel conversion layer.

[0099] The channel conversion layer is used to convert the mixed audio into a single-channel mixed audio based on the beamforming method, and input the single-channel mixed audio into the first feature extraction layer. Exemplarily, based on the beamforming method, the multi-channel mixed audio can be filtered and weighted to obtain a single-channel mixed audio. For example, the maximum likelihood criterion can be used to estimate the source position and calculate the optimal weight coefficient to achieve beamforming; after beamforming, a weighted and filtered single-channel output signal can be obtained.

[0100] See also Figure 7 ,The sound source separation network includes a second feature extraction layer, a splicing layer and a sound source separation sub-network.

[0101] The second feature extraction layer is used to extract a second audio feature from the mixed audio. Exemplarily, the second audio feature includes a feature obtained by performing a short-time Fourier transform (STFT) process on the mixed audio.

[0102] The splicing layer is used to perform splicing processing on the second audio feature and the second position corresponding to the at least one sound event to obtain a splicing feature.

[0103] The sound source separation subnetwork is used to separate the second audio signal of each sound event according to the second category and splicing feature respectively corresponding to at least one sound event.

[0104] In this embodiment, when the sound source separation network separates the mixed audio, it combines the second orientation corresponding to at least one sound event through the splicing layer, and combines the second category corresponding to at least one sound event through the sound source separation sub-network. The category and orientation of the sound event obtained by the sound event localization and classification network can provide additional prior knowledge, so that the sound source separation network can better understand the different sound events in the mixed audio, thereby accurately and quickly separating the audio signals of each sound event, improving the separation effect and processing speed. Furthermore, the network structure of the sound source separation network has fewer levels, which can reduce the computational complexity and achieve higher real-time performance.

[0105] See also Figure 8, the sound source separation network also includes a conversion layer. The conversion layer is used to convert the second orientation corresponding to the sound event location and at least one sound event output by the classification network into an embedding vector (Embedding), and input it into the splicing layer. Converting high-dimensional discrete or continuous data into a dense, continuous vector representation helps to reduce storage and computing costs, and can extract the most relevant and meaningful features in the data, remove redundant information, and make the data more expressive. The splicing layer is specifically used to splice the second audio feature and the embedding vector to obtain a splicing feature.

[0106] See also Figure 6 The sound event localization and classification network is used to classify sound events and estimate the sound source direction of single-channel mixed audio. The second category of at least one sound event output is also one-dimensional data. In the case of a multi-channel audio stream of mixed audio, using the second category of at least one sound event of one-dimensional data as a reference is not accurate enough. Therefore, in order to improve the accuracy of sound source separation, please refer to Figure 8 , the sound source separation network also includes a matrix generation layer. If the mixed audio is single-channel audio, it is directly input into the sound source separation subnetwork; if the mixed audio is not single-channel audio, it needs to be further processed by the matrix generation layer. The matrix generation layer is used to copy the second categories corresponding to at least one sound event according to the number of channels of the mixed audio when the mixed audio is not single-channel audio, so as to obtain a two-dimensional matrix with the number of channels as the number of rows and the number of sound events as the number of columns. For example, the sound event localization and classification network outputs the second category corresponding to 5 sound events, and the number of channels of the mixed audio is 3. Then the second category corresponding to the 5 sound events can be copied into 3 copies, thereby obtaining a 3×5 two-dimensional matrix. The sound source separation subnetwork is specifically used to perform separation processing according to the two-dimensional matrix and splicing features to obtain the second audio signal of each sound event. In this embodiment, the second category corresponding to at least one sound event is copied to adapt to the number of channels of the mixed audio, which is conducive to improving the subsequent audio separation effect.

[0107] Exemplarily, the sound source separation subnetwork includes a U-Net neural network. In the U-Net neural network, the encoder is usually composed of multiple convolutional layers and pooling layers stacked alternately, which are used to extract features layer by layer and reduce the size at the same time. This can reduce the amount of calculation and memory consumption while retaining important features, and improve the training speed and effect; the decoder part uses upsampling or deconvolution operations to restore the low-resolution feature map generated by the encoder to a high-resolution output result with the same size as the original input data. At the same time, in order to overcome problems such as information loss and blurring, the U-Net neural network also introduces a skip connection mechanism, which enables the decoder to skip some levels from the encoder and directly use low-level feature maps for fine modeling.

[0108] Among them, the encoder and decoder in the U-Net neural network are respectively composed of multiple convolutional layers and multiple pooling layers; the input data of each neural network layer in the U-Net neural network includes at least one second category corresponding to each sound event. Exemplarily, the neural network layer includes a convolutional layer. In this embodiment, the second category corresponding to at least one sound event is input into each neural network layer in the U-Net neural network, such as a convolutional layer, which can introduce information of different scales for the convolutional layer, so that more comprehensive time and frequency features can be learned, which is helpful to accurately identify and separate the audio signals of different sound events in the mixed audio.

[0109] (3) About the training process of the audio processing model.

[0110] The training data set provided by the embodiment of the present disclosure includes a first training data set and a second training data set. The training samples in the first training data set include a first category and a first orientation corresponding to at least two sound events, and a mixed audio obtained by mixing the first audio signals corresponding to the at least two sound events. The training samples in the second training data set include a first audio signal corresponding to at least two sound events, and a mixed audio obtained by mixing the first audio signals corresponding to the at least two sound events.

[0111] Exemplarily, in order to improve the training effect, the training data set provided by the embodiment of the present disclosure also includes a third training data set; the training samples in the third training data set include first audio signals, first categories and first directions corresponding to at least two sound events, and mixed audio obtained by mixing the first audio signals corresponding to at least two sound events.

[0112] The training process of the audio processing model is mainly divided into three parts.

[0113] See also Fig. 9 , the first part is to train the sound event localization and classification network in the audio processing model. The electronic device inputs the first training data set into the sound event localization and classification network to be trained, so that the sound event localization and classification network classifies the sound events and estimates the sound source orientation according to the mixed audio in the first training data set, and obtains the second category and the second orientation corresponding to at least one sound event, and takes minimizing the error between the first category and the second category in the first training data set and the error between the first orientation and the second orientation in the first training data set as the optimization goal, and trains the sound event localization and classification network to obtain the trained sound event localization and classification network.

[0114] The loss functions used by the sound event localization and classification network during the training process include a position estimation error function and a sound event classification loss function. The optimization objective of the position estimation error function is used to minimize the error between the first position and the second position in the first training data set, and the optimization objective of the sound event classification loss function is used to minimize the error between the first category and the second category in the first training data set. The sum of the position estimation error function and the sound event classification loss function is used as the final loss of the sound event localization and classification network during the training process.

[0115] Training the sound event localization and classification network is a process of continuous iterative optimization. When the iteration end condition is met, the trained sound event localization and classification network can be obtained. The iteration end condition may be that the trained sound event localization and classification network reaches the optimization target, or the number of iterations reaches a preset number, etc.

[0116] See also Fig.10 The second part is to train the sound source separation network in the audio processing model. After completing the training of the sound event localization and classification network, the sound event localization and classification network can be frozen, and only the sound source separation network can be trained.

[0117] Exemplarily, the electronic device inputs a second training data set into the trained sound event localization and classification network, so that the sound event localization and classification network classifies sound events and estimates sound source orientations based on the mixed audio in the second training data set to obtain a second category and a second orientation corresponding to at least one sound event, obtains a first audio stream of at least one sound event based on the second category corresponding to at least one sound event obtained by the trained sound event localization and classification network, mixes the first audio stream of at least one sound event to obtain a target mixed audio, and uses a sound source separation network to be trained to separate and process the target mixed audio based on the second category and second orientation corresponding to at least one sound event obtained by the trained sound event localization and classification network to obtain second audio signals of each sound event, trains the sound source separation network with the optimization goal of minimizing the error between the first audio signal and the second audio signal in the second training data set, and obtains a trained sound source separation network. In this embodiment, the target mixed audio is separated and processed by the sound source separation network instead of the mixed audio input to the sound event localization and classification network. This can reduce noise interference. The mixed audio may contain other noise or interference sounds that are not related to the sound event, thereby improving the separation precision and accuracy. And if the input target mixed audio signal does not contain other noise or interference sounds that are not related to the sound event, it does not need to be additionally processed and calculated, which is beneficial to improving computing efficiency.

[0118] In order to improve the training efficiency of the sound source separation network, the second category and the second orientation corresponding to at least one sound event obtained by the sound event localization and classification network are considered to be real and accurate data, and then the first audio stream of at least one sound event is re-mixed based on the second category corresponding to at least one sound event to obtain the target mixed audio. The target mixed audio is separated and processed by the sound source separation network to be trained, which can effectively avoid the influence of unrecognized sound events in the mixed audio.

[0119] The optimization objective of the loss function used by the sound source separation network during the training process is used to minimize the error between the first audio signal and the second audio signal. Training the sound source separation network is a process of continuous iterative optimization. When the iteration end condition is met, a trained sound event localization and classification network can be obtained. Among them, the iteration end condition can be that the trained sound source separation network reaches the optimization goal, or the number of iterations reaches a preset number of times.

[0120] In this embodiment, the training process of the sound event localization and classification network and the training process of the sound source separation network are carried out in stages, and the entire training process is decomposed into two independent tasks, which can reduce the complexity of the network and the difficulty of training. In the training process of the sound event localization and classification network, the focus is on optimizing the loss function of sound event localization and classification, and in the training process of the sound source separation network, the focus is on optimizing the loss function of sound source separation.

[0121] The third part is to fine-tune the trained sound event localization and classification network and the trained sound source separation network to further improve the performance of the audio processing model.

[0122] Exemplarily, after obtaining the trained sound event localization and classification network and the trained sound source separation network, the electronic device can input the third training data set into the trained sound event localization and classification network, so that the trained sound event localization and classification network can classify sound events and estimate sound source orientations based on the mixed audio in the third training data set to obtain a second category and a second orientation corresponding to at least one sound event, and the trained sound source separation network can separate and process the mixed audio in the third training data set based on the second category and the second orientation corresponding to at least one sound event obtained by the trained sound event localization and classification network to obtain the second audio signal of each sound event, and optimize the trained sound event localization and classification network and the trained sound source separation network with the optimization objectives of minimizing the error between the first audio signal of the third training data set and the second audio signal obtained by the trained sound source separation network, the error between the first category of the third training data set and the second category obtained by the trained sound event localization and classification network, and the error between the first orientation of the third training data set and the second orientation obtained by the trained sound event localization and classification network.

[0123] Training the audio processing model is a process of continuous iterative optimization. When the iteration end condition is met, a trained audio processing model can be obtained. The iteration end condition may be that the trained audio processing model reaches the optimization target, or the number of iterations reaches a preset number, etc.

[0124] Among them, the trained audio processing model is used to output the position and audio signal of at least one sound event contained in the audio stream based on the input audio stream, so as to be able to mix the audio signal of at least one sound event according to the position of at least one sound event contained in the audio stream to synthesize stereo.

[0125] like Fig.11 and Fig.12 As shown, the embodiment of the present disclosure further provides a stereo synthesis method based on an audio processing model, wherein the audio processing model includes a sound event localization and classification network and a sound source separation network; the audio processing model is constructed based on any one of the construction methods described above; the method includes:

[0126] In S201, an audio stream to be processed is obtained.

[0127] In S202, the audio stream is input into an audio processing model, so that the sound event localization and classification network performs sound event classification and sound source orientation estimation on the audio stream to obtain the category and orientation of at least one sound event, and the sound source separation network performs separation processing on the mixed audio according to the second category and second orientation of the at least one sound event to obtain audio signals of each sound event, and outputs the orientation and audio signal of each sound event in at least one sound event.

[0128] In S203, audio signals of each sound event in the at least one sound event are mixed according to the orientation of each sound event in the at least one sound event to synthesize stereo sound.

[0129] In this embodiment, since the audio processing model can accurately output the direction and audio signal of each sound event in at least one sound event included in the audio stream, accurate stereo sound can be obtained through simple mixing processing with the direction of each sound event as a reference, thereby improving processing efficiency and ensuring the accuracy of stereo synthesis.

[0130] Exemplarily, the audio signals of each sound event can be time-aligned and mixed in the time domain or frequency domain according to the orientation of each sound event in at least one sound event. For example, the amplitude and delay during mixing can be adjusted according to the sound propagation characteristics of the sound source in different orientations to achieve a more accurate and realistic stereo effect. The audio signals after mixing are superimposed or mixed to generate a final stereo audio output.

[0131] In some embodiments, the sound event localization and classification network includes a first feature extraction layer, a feature mapping layer, a sound event classifier, and a sound source orientation estimator;

[0132] The first feature extraction layer is used to extract a low-dimensional first audio feature from the audio stream to be processed;

[0133] The feature mapping layer is used to map the low-dimensional first audio feature into a high-dimensional first audio feature;

[0134] The sound event classifier is used to predict the category corresponding to at least one sound event according to the high-dimensional first audio feature;

[0135] The sound source direction estimator is used to predict the direction corresponding to at least one sound event according to the high-dimensional first audio feature.

[0136] In some embodiments, the sound event localization and classification network further includes a judgment layer and a channel conversion layer;

[0137] The judgment layer is used to judge whether the audio stream to be processed is a single-channel audio; if so, the audio stream to be processed is input into the first feature extraction layer; if not, the audio stream to be processed is input into the channel conversion layer;

[0138] The channel conversion layer is used to convert the audio stream to be processed into a single-channel audio stream based on a beamforming method, and input the single-channel audio stream into the first feature extraction layer.

[0139] In some embodiments, the sound source separation network includes a second feature extraction layer, a concatenation layer, and a sound source separation sub-network;

[0140] The second feature extraction layer is used to extract a second audio feature from the audio stream to be processed;

[0141] The splicing layer is used to perform splicing processing on the second audio feature and the directions respectively corresponding to the at least one sound event to obtain a splicing feature;

[0142] The sound source separation sub-network is used to separate the audio signals of each sound event according to the categories corresponding to the at least one sound event and the splicing features.

[0143] In some embodiments, the sound source separation network further comprises a matrix generation layer;

[0144] The matrix generation layer is used for, when the audio stream to be processed is not single-channel audio, replicating the second categories corresponding to the at least one sound event according to the number of channels of the audio stream to be processed, to obtain a two-dimensional matrix with the number of channels as the number of rows and the number of sound events as the number of columns;

[0145] The sound source separation sub-network is used to perform separation processing according to the two-dimensional matrix and the splicing features to obtain the audio signals of each sound event.

[0146] In some embodiments, the sound source separation subnetwork includes a U-Net neural network; the input data of each neural network layer in the U-Net neural network includes at least the second category corresponding to the at least one sound event.

[0147] Corresponding to the embodiment of the aforementioned method for constructing an audio processing model, the present disclosure also provides a device for constructing an audio processing model.

[0148] like Fig.13 As shown, Fig.13 1 is a structural block diagram of a device for constructing an audio processing model according to an exemplary embodiment of the present disclosure. The audio processing model includes a sound event localization and classification network and a sound source separation network. The device includes:

[0149] The training data acquisition module 301 is used to acquire a training data set, wherein the training samples in the training data set include first audio signals, first categories and first directions corresponding to at least two sound events, and mixed audio obtained by mixing the first audio signals corresponding to the at least two sound events.

[0150] The model training module 302 is used to input the training data set into the audio processing model to be trained, so that the sound event localization and classification network can classify the sound events and estimate the sound source orientation according to the mixed audio to obtain the second category and second orientation of at least one sound event, and the sound source separation network can separate the mixed audio according to the second category and second orientation of the at least one sound event to obtain the second audio signal of each sound event, and train the audio processing model with the optimization goal of minimizing the error between the first audio signal and the second audio signal, the error between the first category and the second category, and the error between the first orientation and the second orientation; wherein the audio processing model is used to output the orientation and audio signal of at least one sound event contained in the audio stream based on the input audio stream for stereo synthesis.

[0151] In some embodiments, the sound event localization and classification network includes a first feature extraction layer, a feature mapping layer, a sound event classifier, and a sound source orientation estimator;

[0152] The first feature extraction layer is used to extract a low-dimensional first audio feature from the mixed audio;

[0153] The feature mapping layer is used to map the low-dimensional first audio feature into a high-dimensional first audio feature;

[0154] The sound event classifier is used to predict the second category corresponding to at least one sound event according to the high-dimensional first audio feature;

[0155] The sound source direction estimator is used to predict the second direction corresponding to at least one sound event according to the high-dimensional first audio feature.

[0156] In some embodiments, the sound event localization and classification network further includes a judgment layer and a channel conversion layer;

[0157] The judgment layer is used to judge whether the mixed audio is a single-channel audio; if so, the mixed audio is input to the first feature extraction layer; if not, the mixed audio is input to the channel conversion layer;

[0158] The channel conversion layer is used to convert the mixed audio into a single-channel mixed audio based on a beamforming method, and input the single-channel mixed audio into the first feature extraction layer.

[0159] In some embodiments, the low-dimensional first audio feature comprises a logarithmic Mel spectrum of the mixed audio; and / or the feature mapping layer comprises a neural network based on a Transformer architecture.

[0160] In some embodiments, the sound source separation network includes a second feature extraction layer, a concatenation layer, and a sound source separation sub-network;

[0161] The second feature extraction layer is used to extract a second audio feature from the mixed audio;

[0162] The splicing layer is used to perform splicing processing on the second audio feature and the second position corresponding to the at least one sound event to obtain a splicing feature;

[0163] The sound source separation subnetwork is used to separate the second audio signal of each sound event according to the second category corresponding to the at least one sound event and the splicing feature.

[0164] In some embodiments, the second audio feature includes a feature obtained by performing short-time Fourier transform processing on the mixed audio.

[0165] In some embodiments, the sound source separation network further comprises a conversion layer;

[0166] The conversion layer is used to convert the second orientation corresponding to the sound event location and the at least one sound event output by the classification network into an embedding vector;

[0167] The concatenation layer is used to concatenate the second audio feature and the embedding vector to obtain a concatenated feature.

[0168] In some embodiments, the sound source separation network further comprises a matrix generation layer;

[0169] The matrix generation layer is used for, when the mixed audio is not single-channel audio, replicating the second categories corresponding to the at least one sound event according to the number of channels of the mixed audio, to obtain a two-dimensional matrix with the number of channels as the number of rows and the number of sound events as the number of columns;

[0170] The sound source separation sub-network is used to perform separation processing according to the two-dimensional matrix and the splicing feature to obtain the second audio signal of each sound event.

[0171] In some embodiments, the sound source separation subnetwork includes a U-Net neural network; the input data of each neural network layer in the U-Net neural network includes at least the second category corresponding to the at least one sound event.

[0172] In some embodiments, the training data set includes a first training data set and a second training data set. The training samples in the first training data set include first categories and first directions corresponding to at least two sound events, and mixed audio obtained by mixing first audio signals corresponding to the at least two sound events; the training samples in the second training data set include first audio signals corresponding to at least two sound events, and mixed audio obtained by mixing first audio signals corresponding to the at least two sound events;

[0173] The model training module 302 includes a first training unit and a second training unit.

[0174] The first training unit is used to input the first training data set into the sound event localization and classification network to be trained, so that the sound event localization and classification network to be trained can classify sound events and estimate the direction of sound sources according to the mixed audio in the first training data set, and obtain a second category and a second direction respectively corresponding to at least one sound event, and train the sound event localization and classification network with the optimization goal of minimizing the error between the first category and the second category, and the error between the first direction and the second direction, to obtain a trained sound event localization and classification network.

[0175] A second training unit is used to input the second training data set into the trained sound event localization and classification network, so that the sound event localization and classification network classifies sound events and estimates sound source orientations according to the mixed audio in the second training data set to obtain a second category and a second orientation corresponding to at least one sound event, obtain a first audio stream of the at least one sound event according to the second category corresponding to the at least one sound event obtained by the trained sound event localization and classification network, mix the first audio stream of the at least one sound event to obtain a target mixed audio, and separate the target mixed audio according to the second category and the second orientation corresponding to the at least one sound event obtained by the trained sound event localization and classification network by the sound source separation network to be trained to obtain second audio signals of each sound event, and train the sound source separation network with the optimization goal of minimizing the error between the first audio signal and the second audio signal in the second training data set to obtain a trained sound source separation network.

[0176] In some embodiments, the training data set further includes a third training data set; the training samples in the third training data set include first audio signals, first categories and first directions respectively corresponding to at least two sound events, and mixed audio obtained by mixing the first audio signals respectively corresponding to the at least two sound events;

[0177] The model training module 302 also includes a third training unit, which is used to input the third training data set into the trained sound event localization and classification network after obtaining the trained sound event localization and classification network and the trained sound source separation network, so that the trained sound event localization and classification network classifies sound events and estimates sound source orientations according to the mixed audio in the third training data set to obtain a second category and a second orientation corresponding to at least one sound event, and the trained sound source separation network separates the mixed audio in the third training data set according to the second category and the second orientation corresponding to at least one sound event obtained by the trained sound event localization and classification network to obtain a second audio signal of each sound event, and optimizes the trained sound event localization and classification network and the trained sound source separation network with the optimization objectives of minimizing the error between the first audio signal of the third training data set and the second audio signal obtained by the trained sound source separation network, the error between the first category of the third training data set and the second category obtained by the trained sound event localization and classification network, and the error between the first orientation of the third training data set and the second orientation obtained by the trained sound event localization and classification network.

[0178] Corresponding to the above-mentioned embodiment of the stereo synthesis method based on the audio processing model, the present disclosure also provides a stereo synthesis device based on the audio processing model.

[0179] like Fig.14 As shown, Fig.14 The present disclosure is a block diagram of a stereo synthesis device based on an audio processing model according to an exemplary embodiment. The audio processing model includes a sound event localization and classification network and a sound source separation network. The audio processing model is constructed based on the construction device of the above-mentioned audio processing model; the stereo synthesis device includes:

[0180] The audio stream acquisition module 401 is used to acquire the audio stream to be processed.

[0181] The audio stream processing module 402 is used to input the audio stream to be processed into the audio processing model, so that the sound event localization and classification network can perform sound event classification and sound source orientation estimation on the audio stream to obtain the category and orientation of at least one sound event, and the sound source separation network can separate the mixed audio stream according to the category and orientation of the at least one sound event to obtain the audio signals of each sound event, and output the orientation and audio signal of each sound event in at least one sound event.

[0182] The stereo synthesis module 403 is used to perform mixing processing on the audio signals respectively corresponding to at least one sound event according to the orientation of each sound event in the at least one sound event to synthesize stereo.

[0183] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0184] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units.

[0185] Corresponding to the embodiments of the aforementioned method for constructing an audio processing model and a method for stereo synthesis based on an audio processing model, the present disclosure further provides an apparatus for constructing an audio processing model and an electronic device for stereo synthesis based on an audio processing model. The electronic device includes:

[0186] processor;

[0187] a memory for storing a computer program executable by the processor;

[0188] Among them, when the processor executes the program, it implements the method for constructing the audio processing model recorded in any of the above embodiments or the stereo synthesis method based on the audio processing model.

[0189] like Fig.15 As shown, Fig.15 1 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. The electronic device 500 may include one or more of the following components: a processing component 501, a memory 502, a power component 503, a multimedia component 504, an audio component 505, an input / output (I / O) interface 506, a sensor component 507, and a communication component 508.

[0190] The processing component 501 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 501 may include one or more processors 509 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 501 may include one or more modules to facilitate the interaction between the processing component 501 and other components. For example, the processing component 501 may include a multimedia module to facilitate the interaction between the multimedia component 504 and the processing component 501.

[0191] The memory 502 is configured to store various types of data to support operations on the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0192] The power supply component 503 provides power to various components of the electronic device 500. The power supply component 503 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 500.

[0193] The multimedia component 504 includes a screen that provides an output interface between the electronic device 500 and the user. The screen may include a touch panel (TP) implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 504 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0194] The audio component 505 is configured to output and / or input audio signals. For example, the audio component 505 includes a microphone (MIC), and when the electronic device 500 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 502 or sent via the communication component 508. In some embodiments, the audio component 505 also includes a speaker for outputting audio signals.

[0195] I / O interface 502 provides an interface between processing component 501 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0196] The sensor assembly 507 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 507 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 507 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of contact between the user and the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 507 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 507 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 507 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, a temperature sensor, a photoelectric sensor, or a GPS sensor.

[0197] The communication component 508 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G LTE, 5G NR (5G NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 508 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 508 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0198] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0199] The implementation process of the functions and effects of each component in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0200] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0201] Corresponding to the embodiments of the aforementioned skipping rope data processing method, the present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by the processor 509 of the above-mentioned electronic device, it implements the method for constructing the audio processing model recorded in any of the above-mentioned embodiments or the stereo synthesis method based on the audio processing model.

[0202] The present disclosure may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0203] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0204] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0205] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A method for constructing an audio processing model, characterized in that: The audio processing model includes a sound event localization and classification network and a sound source separation network; the method includes: Acquire a training data set, wherein the training samples in the training data set include first audio signals, a first category, and a first orientation respectively corresponding to at least two sound events, and mixed audio obtained by mixing the first audio signals respectively corresponding to the at least two sound events; Inputting the training data set into the audio processing model to be trained, so that the sound event localization and classification network classifies the sound events and estimates the sound source orientation according to the mixed audio to obtain a second category and a second orientation of at least one sound event, and the sound source separation network separates the mixed audio according to the second category and the second orientation of the at least one sound event to obtain a second audio signal of each sound event, and the audio processing model is trained with minimizing the error between the first audio signal and the second audio signal, the error between the first category and the second category, and the error between the first orientation and the second orientation as optimization goals; The audio processing model is used to output the location and audio signal of at least one sound event contained in the audio stream based on the input audio stream, so as to be used for synthesizing stereo.

2. The method according to claim 1, characterized in that: The sound event localization and classification network includes a first feature extraction layer, a feature mapping layer, a sound event classifier and a sound source orientation estimator; The first feature extraction layer is used to extract a low-dimensional first audio feature from the mixed audio; The feature mapping layer is used to map the low-dimensional first audio feature into a high-dimensional first audio feature; The sound event classifier is used to predict the second category corresponding to at least one sound event according to the high-dimensional first audio feature; The sound source direction estimator is used to predict the second direction corresponding to at least one sound event according to the high-dimensional first audio feature.

3. The method according to claim 2, characterized in that The sound event localization and classification network also includes a judgment layer and a channel conversion layer; The judgment layer is used to judge whether the mixed audio is a single-channel audio; if so, input the mixed audio into the first feature extraction layer; If not, input the mixed audio into a channel conversion layer; The channel conversion layer is used to convert the mixed audio into a single-channel mixed audio based on a beamforming method, and input the single-channel mixed audio into the first feature extraction layer.

4. The method according to claim 2, characterized in that The low-dimensional first audio feature comprises a logarithmic Mel spectrum of the mixed audio; and / or The feature mapping layer includes a neural network based on the Transformer architecture.

5. The method according to claim 1, characterized in that The sound source separation network includes a second feature extraction layer, a splicing layer and a sound source separation sub-network; The second feature extraction layer is used to extract a second audio feature from the mixed audio; The splicing layer is used to perform splicing processing on the second audio feature and the second position corresponding to the at least one sound event to obtain a splicing feature; The sound source separation subnetwork is used to separate the second audio signal of each sound event according to the second category corresponding to the at least one sound event and the splicing feature.

6. The method according to claim 5, characterized in that The second audio feature includes a feature obtained by performing short-time Fourier transform processing on the mixed audio.

7. The method according to claim 5, characterized in that The sound source separation network also includes a conversion layer; The conversion layer is used to convert the second orientation corresponding to the sound event location and the at least one sound event output by the classification network into an embedding vector; The concatenation layer is used to concatenate the second audio feature and the embedding vector to obtain a concatenated feature.

8. The method according to claim 5, characterized in that The sound source separation network also includes a matrix generation layer; The matrix generation layer is used for, when the mixed audio is not single-channel audio, replicating the second categories corresponding to the at least one sound event according to the number of channels of the mixed audio, to obtain a two-dimensional matrix with the number of channels as the number of rows and the number of sound events as the number of columns; The sound source separation sub-network is used to perform separation processing according to the two-dimensional matrix and the splicing feature to obtain the second audio signal of each sound event.

9. The method according to claim 5, characterized in that The sound source separation sub-network includes a U-Net neural network; The input data of each neural network layer in the U-Net neural network includes at least the second category corresponding to the at least one sound event.

10. The method according to claim 1, characterized in that The training data set includes a first training data set and a second training data set; The training samples in the first training data set include first categories and first directions corresponding to at least two sound events, and mixed audio obtained by mixing first audio signals corresponding to the at least two sound events; The training samples in the second training data set include first audio signals corresponding to at least two sound events, and mixed audio obtained by mixing the first audio signals corresponding to the at least two sound events; The training of the audio processing model comprises: Inputting the first training data set into a sound event localization and classification network to be trained, so that the sound event localization and classification network to be trained performs sound event classification and sound source orientation estimation according to the mixed audio in the first training data set, and obtains a second category and a second orientation respectively corresponding to at least one sound event, and training the sound event localization and classification network with minimizing the error between the first category and the second category and the error between the first orientation and the second orientation as the optimization goal, to obtain a trained sound event localization and classification network; The second training data set is input into the trained sound event localization and classification network, so that the sound event localization and classification network classifies sound events and estimates sound source orientations according to the mixed audio in the second training data set to obtain a second category and a second orientation corresponding to at least one sound event, obtains a first audio stream of the at least one sound event according to the second category corresponding to the at least one sound event obtained by the trained sound event localization and classification network, mixes the first audio stream of the at least one sound event to obtain a target mixed audio, and uses a sound source separation network to be trained to separate the target mixed audio according to the second category and the second orientation corresponding to the at least one sound event obtained by the trained sound event localization and classification network to obtain a second audio signal of each sound event, trains the sound source separation network with the optimization goal of minimizing the error between the first audio signal and the second audio signal in the second training data set to obtain a trained sound source separation network.

11. The method according to claim 10, characterized in that The training data set also includes a third training data set; The training samples in the third training data set include first audio signals, first categories and first directions respectively corresponding to at least two sound events, and mixed audio obtained by mixing the first audio signals respectively corresponding to the at least two sound events; The training of the audio processing model further includes: After obtaining the trained sound event localization and classification network and the trained sound source separation network, the third training data set is input into the trained sound event localization and classification network, so that the trained sound event localization and classification network classifies sound events and estimates sound source orientations based on the mixed audio in the third training data set to obtain a second category and a second orientation corresponding to at least one sound event, and the trained sound source separation network separates the mixed audio in the third training data set based on the second category and the second orientation corresponding to at least one sound event obtained by the trained sound event localization and classification network to obtain a second audio signal of each sound event, and optimizes the trained sound event localization and classification network and the trained sound source separation network with the optimization objectives of minimizing the error between the first audio signal of the third training data set and the second audio signal obtained by the trained sound source separation network, the error between the first category of the third training data set and the second category obtained by the trained sound event localization and classification network, and the error between the first orientation of the third training data set and the second orientation obtained by the trained sound event localization and classification network.

12. A stereo synthesis method based on an audio processing model, characterized in that: The audio processing model includes a sound event localization and classification network and a sound source separation network; The audio processing model is constructed based on the method described in any one of claims 1 to 11; the method comprises: Get the audio stream to be processed; Input the audio stream to be processed into the audio processing model, so that the sound event localization and classification network performs sound event classification and sound source orientation estimation on the audio stream to obtain the category and orientation of at least one sound event, and the sound source separation network performs separation processing on the mixed audio according to the category and orientation of the at least one sound event to obtain audio signals of each sound event, and outputs the orientation and audio signal of each sound event in the at least one sound event; According to the orientation of each sound event in the at least one sound event, audio signals respectively corresponding to the at least one sound event are mixed to synthesize stereo sound.

13. The method according to claim 12, characterized in that The sound event localization and classification network includes a first feature extraction layer, a feature mapping layer, a sound event classifier and a sound source orientation estimator; The first feature extraction layer is used to extract a low-dimensional first audio feature from the audio stream to be processed; The feature mapping layer is used to map the low-dimensional first audio feature into a high-dimensional first audio feature; The sound event classifier is used to predict the category corresponding to at least one sound event according to the high-dimensional first audio feature; The sound source direction estimator is used to predict the direction corresponding to at least one sound event according to the high-dimensional first audio feature.

14. The method according to claim 13, characterized in that The sound event localization and classification network also includes a judgment layer and a channel conversion layer; The judgment layer is used to judge whether the audio stream to be processed is a single-channel audio; if so, input the audio stream to be processed into the first feature extraction layer; If not, input the audio stream to be processed into a channel conversion layer; The channel conversion layer is used to convert the audio stream to be processed into a single-channel audio stream based on a beamforming method, and input the single-channel audio stream into the first feature extraction layer.

15. The method according to claim 12, characterized in that The sound source separation network includes a second feature extraction layer, a splicing layer and a sound source separation sub-network; The second feature extraction layer is used to extract a second audio feature from the audio stream to be processed; The splicing layer is used to perform splicing processing on the second audio feature and the directions respectively corresponding to the at least one sound event to obtain a splicing feature; The sound source separation sub-network is used to separate the audio signals of each sound event according to the categories corresponding to the at least one sound event and the splicing features.

16. The method according to claim 15, characterized in that The sound source separation network also includes a matrix generation layer; The matrix generation layer is used for, when the audio stream to be processed is not single-channel audio, replicating the second categories corresponding to the at least one sound event according to the number of channels of the audio stream to be processed, to obtain a two-dimensional matrix with the number of channels as the number of rows and the number of sound events as the number of columns; The sound source separation sub-network is used to perform separation processing according to the two-dimensional matrix and the splicing features to obtain the audio signals of each sound event.

17. The method according to claim 15, characterized in that The sound source separation subnetwork includes a U-Net neural network; the input data of each neural network layer in the U-Net neural network includes at least the second category corresponding to the at least one sound event.

18. A device for constructing an audio processing model, characterized in that: The audio processing model includes a sound event localization and classification network and a sound source separation network; the device includes: A training data acquisition module is used to acquire a training data set, wherein the training samples in the training data set include first audio signals, a first category and a first orientation respectively corresponding to at least two sound events, and a mixed audio obtained by mixing the first audio signals respectively corresponding to the at least two sound events; A model training module is used to input the training data set into the audio processing model to be trained, so that the sound event localization and classification network can classify sound events and estimate the sound source orientation according to the mixed audio to obtain a second category and a second orientation of at least one sound event, and the sound source separation network can separate the mixed audio according to the second category and the second orientation of the at least one sound event to obtain a second audio signal of each sound event, and train the audio processing model with the optimization goal of minimizing the error between the first audio signal and the second audio signal, the error between the first category and the second category, and the error between the first orientation and the second orientation; wherein the audio processing model is used to output the orientation and audio signal of at least one sound event contained in the audio stream based on the input audio stream for stereo synthesis.

19. A stereo synthesis device based on an audio processing model, characterized in that: The audio processing model includes a sound event localization and classification network and a sound source separation network; The audio processing model is constructed based on the device according to claim 18; the device comprises: An audio stream acquisition module, used to acquire the audio stream to be processed; An audio stream processing module, used for inputting the audio stream to be processed into an audio processing model, so that the sound event localization and classification network performs sound event classification and sound source orientation estimation on the audio stream to obtain the category and orientation of at least one sound event, and the sound source separation network performs separation processing on the mixed audio stream according to the category and orientation of the at least one sound event to obtain audio signals of each sound event, and outputs the orientation and audio signal of each sound event in at least one sound event; The stereo synthesis module is used to perform mixing processing on the audio signals respectively corresponding to at least one sound event according to the orientation of each sound event in at least one sound event to synthesize stereo.

20. An electronic device, characterized in that: include: processor; a memory for storing a computer program executable by the processor; Wherein, when the processor executes the program, the steps of the method according to any one of claims 1 to 17 are implemented.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images