Audio processing method and device, model training method and device, equipment and medium

Abstract

The present disclosure provides an audio processing method and device, a model training method and device, an electronic device and a medium, which relate to the field of artificial intelligence, and in particular to the field of speech technology. The implementation scheme is: for each of the multiple audio frames extracted from the audio data to be processed, the local feature information of the audio frame is determined in turn, wherein the audio data to be processed includes audio data from at least two sound sources; and any one of the multiple audio frames is determined as a target audio frame, and the following operations are performed for the target audio frame: based on the local feature information of each audio frame in the multiple audio frames, the global feature information of the target audio frame is determined; and based on the global feature information of the target audio frame, the sound source classification corresponding to the target audio frame is determined.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of speech technology, and specifically to an audio processing method and device, a model training method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware module-level technologies and software module-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, audio recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides an audio processing method, a model, a model training method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of the present disclosure, there is provided an audio processing method, comprising: determining local feature information of each audio frame in a plurality of audio frames extracted from audio data to be processed in turn, wherein the audio data to be processed includes audio data from at least two sound sources; and determining any one audio frame in the plurality of audio frames as a target audio frame, and performing the following operations for the target audio frame: determining global feature information of the target audio frame based on the local feature information of each audio frame in the plurality of audio frames; and determining a sound source classification corresponding to the target audio frame based on the global feature information of the target audio frame.

[0006] According to another aspect of the present disclosure, a training method for an audio processing model is provided, wherein the audio processing model includes a local feature extraction module, a global feature extraction module and an output module, and the method includes: for each sample audio frame extracted from the sample audio data in turn, using the local feature extraction module, obtaining first local feature information of the sample audio frame, wherein the sample audio data includes audio data from a first number of sound sources, and each of the multiple sample audio frames has a sound source label; based on at least the first local feature information of each sample audio frame in the multiple sample audio frames, using the global feature extraction module, obtaining global feature information of each sample audio frame in the multiple sample audio frames; for each sample audio frame in the multiple sample audio frames in turn, inputting the global feature information of the sample audio frame into the output module to obtain a first confidence of the sample audio frame output by the output module for each predicted sound source classification in the first number of predicted sound source classifications; and adjusting parameters of the audio processing model based on the first confidence of each sample audio frame in the multiple sample audio frames for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of each sample audio frame in the multiple sample audio frames.

[0007] According to another aspect of the present disclosure, an audio processing device is provided, including: a local feature extraction module, configured to determine, in turn, local feature information of each audio frame in a plurality of audio frames extracted from audio data to be processed, wherein the audio data to be processed includes audio data from at least two sound sources; a global feature extraction module, configured to determine any one of the plurality of audio frames as a target audio frame, and determine global feature information of the target audio frame based on the local feature information of each of the plurality of audio frames; and an output module, configured to determine a sound source classification corresponding to the target audio frame based on the global feature information of the target audio frame.

[0008] According to another aspect of the present disclosure, a training device for an audio processing model is provided, wherein the audio processing model includes a local feature extraction module, a global feature extraction module and an output module, and the device includes: a first acquisition module, configured to sequentially obtain first local feature information of each sample audio frame in a plurality of sample audio frames extracted from sample audio data by using the local feature extraction module, wherein the sample audio data includes audio data from a first number of sound sources, and each of the plurality of sample audio frames has a sound source label; a second acquisition module, configured to obtain first local feature information of each sample audio frame in the plurality of sample audio frames by using the local feature extraction module, wherein the sample audio data includes audio data from a first number of sound sources, and each of the plurality of sample audio frames has a sound source label; and A global feature extraction module is used to obtain global feature information of each sample audio frame in multiple sample audio frames; a third acquisition module is configured to input the global feature information of the sample audio frame into the output module in turn for each sample audio frame in the multiple sample audio frames, so as to obtain the first confidence of the sample audio frame output by the output module for each predicted sound source classification in the first number of predicted sound source classifications; and a first adjustment module is configured to adjust the parameters of the audio processing model based on the first confidence of each sample audio frame in the multiple sample audio frames for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of each sample audio frame in the multiple sample audio frames.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned methods.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any one of the above methods.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the above methods when executed by a processor.

[0012] According to one or more embodiments of the present disclosure, effective separation of audio data to be processed can be achieved according to different sound sources.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram showing an exemplary system in which the various methods described herein may be implemented according to an embodiment of the present disclosure;

[0016] Figure 2A and Figure 2B A flowchart of an audio processing method according to an embodiment of the present disclosure is shown;

[0017] Figure 3 A schematic diagram showing the confidence of sound source classification according to an embodiment of the present disclosure;

[0018] Figure 4 A flowchart of a method for training an audio processing model according to an embodiment of the present disclosure is shown;

[0019] Figure 5 A schematic diagram showing a method for training an audio processing model according to an embodiment of the present disclosure is shown;

[0020] Figure 6 A structural block diagram of an audio processing model according to an embodiment of the present disclosure is shown;

[0021] Figure 7 A structural block diagram of a training device for an audio processing model according to an embodiment of the present disclosure is shown;

[0022] Figure 8 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0024] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0025] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0026] For a section of unprocessed audio data collected by an audio acquisition device, it is often necessary to first separate the unprocessed audio data based on the sound sources from which each part of the unprocessed audio data comes. After that, subsequent targeted processing is performed on the separated audio data of the single sound source. For example, in scenarios such as intelligent customer service, conference discussions, interview dialogues, public security interrogations, etc., there are multiple speakers on a single channel in the unprocessed audio data collected by the audio acquisition device. It is necessary to first separate the human voices from the unprocessed audio data, that is, to separate the audio data from different speakers in the unprocessed audio data, and then perform targeted analysis on the audio data of each speaker in the unprocessed audio data. However, there is a lack of a simple and effective separation solution for the unprocessed audio data in the related art.

[0027] Based on this, the present disclosure proposes an audio processing method, for a target audio frame in the audio data to be processed, based on the local feature information of each audio frame in the multiple audio frames in the audio data to be processed, the global feature information of the target audio frame is determined; and based on the global feature information of the target audio frame, the sound source classification corresponding to the target audio frame is determined. Based on the information from the multiple audio frames in the audio data to be processed, the global feature information of the target audio frame that is different from other audio frames can be mined, thereby enabling the target audio frame and the audio frames from the same sound source in other audio frames to be classified into the same sound source classification based on the global feature information, and enabling the target audio frame and the audio frames from different sound sources in other audio frames to be classified into different sound source classifications, thereby enabling the entire audio data to be processed to be effectively separated according to the different sound sources.

[0028] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0030] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable the method of audio processing to be performed.

[0031] In some embodiments, server 120 may also provide other services or software applications that may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0032] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0033] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to obtain audio data to be processed. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0034] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0035] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0036] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0037] The computing unit in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0038] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0039] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0040] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0041] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0042] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0043] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0044] Figure 2A and Figure 2B is a flowchart showing an audio processing method according to an exemplary embodiment of the present disclosure, Figure 2A and Figure 2B As shown, an audio processing method includes: step S201, determining local feature information of each audio frame in a plurality of audio frames extracted from audio data to be processed in turn, wherein the audio data to be processed includes audio data from at least two sound sources; and step S202, determining any one audio frame in the plurality of audio frames as a target audio frame, and performing the following operations for the target audio frame: step S202-1, determining global feature information of the target audio frame based on the local feature information of each audio frame in the plurality of audio frames; and step S202-2, determining the sound source classification corresponding to the target audio frame based on the global feature information of the target audio frame.

[0045] Since the global feature information of the target audio frame that is different from other audio frames can be mined based on the information from multiple audio frames in the audio data to be processed, the target audio frame and audio frames from the same sound source in other audio frames can be classified into the same sound source classification based on the global feature information, and the target audio frame and audio frames from different sound sources in other audio frames can be classified into different sound source classifications, thereby making the entire audio data to be processed can be effectively separated according to different sound sources.

[0046] According to some embodiments, before extracting audio frames from the audio data to be processed, the audio data to be processed may be preprocessed, and the preprocessing may include but is not limited to removing noise (including environmental noise, busy tones, ringtones, etc.) to obtain clean audio data to be processed.

[0047] With respect to step S201 , the audio frames therein may be extracted from the audio data to be processed through a time window of a preset length.

[0048] According to some embodiments, multiple audio frames may be adjacent in the time domain of the audio data to be processed. For example, the time window of the preset length may be 40 ms, and the time window is moved in steps of 10 ms in the audio data to be processed, so that multiple audio frames adjacent in the time domain can be extracted from the audio data to be processed.

[0049] According to some embodiments, the local feature information of each audio frame may be one or more of Mel-frequency cepstral coefficients (MFCC), FBank features (Filter Banks), or perceptual linear prediction (PLP) features.

[0050] According to some embodiments, the local feature information of each audio frame can also be obtained by inputting each audio frame, or the time domain or frequency domain features of each audio frame, for example, one or more of the above-mentioned Mel-frequency cepstral coefficients (MFCC), FBank features (FilterBanks) or perceptual linear prediction (PLP) features, into a trained local feature extraction module, thereby obtaining the local feature information of the audio frame output by the local feature extraction module.

[0051] According to some embodiments, determining the local feature information of each audio frame in the multiple audio frames extracted from the audio data to be processed may include: determining the local feature information of the audio frame based on the audio frame and one or more audio frames close to the audio frame in the multiple audio frames. Thus, the local feature information of the audio frame can be determined based on the information in the local time range of the audio frame contained in the audio data to be processed, so that the local feature information of the audio frame can contain certain context information.

[0052] According to some embodiments, each of the at least two sound sources is a speaker, thereby enabling voice separation in scenarios such as intelligent customer service, conference discussions, interview dialogues, and public security interrogations.

[0053] Based on the local feature information of each audio frame in the multiple audio frames, step S202 can be further executed to determine the global feature information of the target audio frame, thereby mining the global feature information of the target audio frame that is different from other audio frames, so that the entire audio data to be processed can be effectively separated according to different sound sources.

[0054] According to some embodiments, the global feature information of the target audio frame may be determined based on information uniformly distributed in a plurality of audio frames in the entire audio data to be processed.

[0055] According to some embodiments, local feature information of a target audio frame may be modified based on local feature information of each audio frame in a plurality of audio frames, and global feature information of the target audio frame may be determined based on the modified local feature information.

[0056] According to some embodiments, the local feature information of each audio frame in the multiple audio frames can be input into a trained global feature extraction module to obtain the global feature information of the target audio frame output by the global feature extraction module. Then, the multiple audio frames are clustered by the extracted global feature information, in other words, the target audio frame and the audio frames from the same sound source in other audio frames are classified into the same sound source classification, and the target audio frame and the audio frames from different sound sources in other audio frames are classified into different sound source classifications.

[0057] According to some embodiments, based on the global feature information of the target audio frame, the confidence of the target audio frame for each of the multiple sound source classifications can be obtained, and the sound source classification corresponding to the target audio frame can be predicted based on the confidence.

[0058] In particular, the confidence of the target audio frame for each of the multiple sound source classifications can be obtained based on the trained output module.

[0059] For example, when the number of sound source categories is 2, Figure 3 is a schematic diagram showing the confidence of a plurality of audio frames for each of two sound source classifications according to an exemplary embodiment of the present disclosure.

[0060] like Figure 3 As shown, the confidence of the 1st to 4th audio frames for the first sound source classification is significantly higher than the confidence of the 1st to 4th audio frames for the second sound source classification. Therefore, it can be predicted that the 1st to 4th audio frames come from the same sound source and are classified into the first sound source classification.

[0061] The confidence of the 8th to 10th audio frames for the first sound source classification is significantly lower than the confidence of the 8th to 10th audio frames for the second sound source classification. Therefore, it can be predicted that the 8th to 10th audio frames come from the same sound source, which is different from the sound source from which the 1st to 4th audio frames come, and the 8th to 10th audio frames are classified into the second sound source classification.

[0062] The confidence of the 5th to 7th audio frames for the first sound source classification is comparable to the confidence of the second sound source classification, so it can be predicted that the 5th to 7th audio frames include audio data from both the first sound source classification and the second sound source classification.

[0063] In particular, when the difference between the confidence of the audio frame for the first sound source classification among multiple sound source classifications and the confidence of the audio frame for other sound source classifications is greater than a preset threshold, the audio frame is classified into the first sound source classification. When the difference between the confidences of the audio frame for any two sound source classifications is not greater than a preset threshold, the audio frame is simultaneously classified into the two sound source classifications.

[0064] It can be understood that in the above-mentioned process of performing sound source separation on the processed audio data, no prior information such as the number and characteristics of the sound sources is required. Instead, by relying on the differences and similarities between multiple audio frames reflected by the global feature information of each audio frame in the multiple audio frames, audio frames from the same sound source can be classified into the same sound source category, and audio frames from different sound sources can be classified into different sound source categories.

[0065] Figure 4 is a flowchart showing a method for training an audio processing model according to an exemplary embodiment of the present disclosure, Figure 4 As shown, a training method for an audio processing model, wherein the audio processing model includes a local feature extraction module, a global feature extraction module and an output module, and the method includes: step S401, sequentially for each sample audio frame in a plurality of sample audio frames extracted from sample audio data, using the local feature extraction module to obtain first local feature information of the sample audio frame, wherein the sample audio data includes audio data from a first number of sound sources, and each of the plurality of sample audio frames has a sound source label; step S402, based at least on the first local feature information of each sample audio frame in the plurality of sample audio frames, using the global feature extraction module to obtain first local feature information of the sample audio frame module, obtaining global feature information of each sample audio frame in a plurality of sample audio frames; step S403, inputting the global feature information of the sample audio frame into the output module in turn for each sample audio frame in the plurality of sample audio frames, so as to obtain a first confidence of the sample audio frame output by the output module for each predicted sound source classification in a first number of predicted sound source classifications; and step S404, adjusting the parameters of the audio processing model based on the first confidence of each sample audio frame in the plurality of sample audio frames for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of each sample audio frame in the plurality of sample audio frames.

[0066] The audio processing model trained in this way can, when predicting a target audio frame in the audio data to be processed, use the global feature extraction module therein to mine out the global feature information of the target audio frame that is different from other audio frames based on the information from multiple audio frames in the audio data to be processed, and based on the global feature information, classify the target audio frame and audio frames from the same sound source in other audio frames into the same sound source classification, and classify the target audio frame and audio frames from different sound sources in other audio frames into different sound source classifications, thereby effectively separating the entire audio data to be processed according to different sound sources.

[0067] According to some embodiments, the local feature extraction module, the global feature extraction module and the output module are connected in sequence in the audio processing model so that during the training process of the audio processing model, the output information of the local feature extraction module can be used as the input information of the global feature extraction module, and the output information of the global feature extraction module can be used as the input information of the output module.

[0068] According to some embodiments, before extracting sample audio frames from sample audio data, the sample audio data may be preprocessed, and the preprocessing may include but is not limited to removing noise (including environmental noise, busy tones, ringtones, etc.) to obtain clean sample audio data.

[0069] According to some embodiments, before extracting the sample audio frame from the sample audio data, data enhancement may be performed on the sample audio data, such as time domain warping, frequency domain masking, etc.

[0070] With respect to step S401 , the sample audio frame may be extracted from the sample audio data through a time window of a preset length.

[0071] According to some embodiments, a plurality of sample audio frames may be adjacent in sequence in the time domain of the sample audio data. For example, a time window of a preset length may be 40 ms, and the time window is moved in steps of 10 ms in the sample audio data, so that a plurality of sample audio frames adjacent in sequence in the time domain can be extracted from the sample audio data in sequence.

[0072] According to some embodiments, multiple sample audio frames can be selected for training from a first number of sample audio frames that are adjacent in sequence and obtained based on a time window according to preset rules. For example, a frame skipping strategy can be adopted to select a sample audio frame for training every certain number of sample audio frames, thereby effectively reducing the subsequent amount of calculation.

[0073] According to some embodiments, each sample audio frame may be sequentially input into a local feature extraction module to obtain local feature information of the sample audio frame output by the local feature extraction module.

[0074] According to some embodiments, the time domain or frequency domain features of each sample audio frame, for example, one or more of the above-mentioned Mel-frequency cepstral coefficients (MFCC), FBank features (Filter Banks) or perceptual linear prediction (PLP) features, etc., can be input into a local feature extraction module to obtain the local feature information of the sample audio frame output by the local feature extraction module.

[0075] According to some embodiments, determining the local feature information of each sample audio frame in the plurality of sample audio frames extracted from the sample audio data may include: determining the local feature information of the sample audio frame based on the sample audio frame and one or more sample audio frames close to the sample audio frame in the plurality of sample audio frames. Thus, the local feature information of the sample audio frame may be determined based on information within a local time range of the sample audio frame contained in the sample audio data.

[0076] For step S402, according to some embodiments, for each of the multiple sample audio frames extracted from the sample audio data, an additional local feature extraction module can be used to obtain the second local feature information of the sample audio frame in turn; wherein, based on at least the first local feature information of each of the multiple sample audio frames, using a global feature extraction module to obtain the global feature information of each of the multiple sample audio frames may include: based on the first local feature information and the second local feature information of each of the multiple sample audio frames, using a global feature extraction module to obtain the global feature information of each of the multiple sample audio frames.

[0077] Among them, an additional local feature extraction module is applied in the training process of the audio processing model. The additional local feature extraction module and the local feature extraction module in the audio processing model operate independently of each other. The local feature extraction module and the additional local feature extraction module can respectively extract different first local feature information and second local feature information of each sample audio frame. Multiple local feature information can help the subsequent construction of global feature information, and thus improve the training effect of the model.

[0078] For example, the local feature extraction module is a first multi-layer time dilation convolutional neural network (TDCN), and the additional local feature extraction module is a second multi-layer time dilation convolutional neural network, wherein the local feature extraction module and the additional local feature extraction module use the same network structure but have different parameters. Therefore, for the same sample audio frame, the sample audio frame is respectively input into the first multi-layer time dilation convolutional neural network and the second multi-layer time dilation convolutional neural network, and the first local feature information and the second local feature information can be respectively obtained, and the first local feature information and the second local feature information are different from each other based on different parameters in different networks. According to some embodiments, based on the first local feature information and the second local feature information of each sample audio frame in the plurality of sample audio frames, using the global feature extraction module, obtaining the global feature information of each sample audio frame in the plurality of sample audio frames may include: for each sample audio frame in the plurality of sample audio frames, fusing the first local feature information and the second local feature information of the sample audio frame to obtain the fused local feature information of the sample audio frame; and based on the fused local feature information of each sample audio frame in the plurality of sample audio frames, using the global feature extraction module, obtaining the global feature information of each sample audio frame in the plurality of sample audio frames.

[0079] According to some embodiments, fusing the first local feature information and the second local feature information of the sample audio frame may include performing splicing on the first local feature information and the second local feature information of the sample audio frame.

[0080] According to some embodiments, for each sample audio frame among multiple sample audio frames, the following operations can be performed: second local feature information of the sample audio frame is input into the first additional output module to obtain a second confidence of the sample audio frame output by the first additional output module for each predicted sound source classification in the first number of predicted sound source classifications; and based on the second confidence of the sample audio frame for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of the sample audio frame, the parameters of the additional local feature extraction module are adjusted.

[0081] Among them, the first additional output module is applied to the training process of the audio processing model, and the second confidence of the sample audio frame output by the first additional output module for each predicted sound source classification in the first number of predicted sound source classifications can be used to specifically adjust the parameters of the additional local feature extraction module, so that in the process of joint training, the additional local feature extraction module can have parameters different from the local feature extraction module, and thus the additional local feature extraction module can extract second local feature information different from the first local feature information.

[0082] It should be noted that the parameter adjustment of the additional local feature extraction module performed using the first additional output module can be performed based on each sample audio frame, that is, for each sample audio frame in the multiple sample audio frames sequentially input into the audio processing model, the parameter adjustment of the additional local feature extraction module can be performed based on the independent sample audio frame, without waiting for the second local feature extraction of each sample audio frame in the multiple sample audio frames sequentially input. In other words, for a sample audio data input, the parameters of the additional local feature extraction module can be adjusted multiple times based on multiple sample audio frames therein, and as multiple sample audio frames are sequentially input, the accuracy of the second local feature information extracted by the sample audio frame through the additional local feature extraction module can be continuously improved.

[0083] Different from this, since the global feature information of each of the multiple sample audio frames needs to rely on at least the first local feature information of each of the multiple sample audio frames, the adjustment of the parameters of the audio processing model needs to wait for the extraction of the first local features of each of the multiple sample audio frames input in sequence. Therefore, in the process of joint training, parameter adjustments of different frequencies can be implemented for different modules based on different mechanisms, so that the flexibility of training is improved, and the efficiency of training is improved on the basis of ensuring the training effect.

[0084] For step S403 and step S404, according to some embodiments, based on the first confidence of each predicted sound source classification in the first number of predicted sound source classifications for each sample audio frame in a plurality of sample audio frames and the sound source label of each sample audio frame in the plurality of sample audio frames, adjusting the parameters of the audio processing model may include: determining the correspondence between the first number of predicted sound source classifications and the first number of sound sources under the target mapping type, wherein, among the plurality of correspondences respectively corresponding to the plurality of mapping types between the first number of predicted sound source classifications and the first number of sound sources, the loss value between the first confidence corresponding to each sample audio frame in the plurality of sample audio frames calculated based on the target mapping type and the sound source label of the sample audio frame is the smallest; and adjusting the parameters of the audio processing model based on the loss value calculated based on the correspondence under the target mapping type.

[0085] Since the audio processing model can be used to realize clustering of multiple sample audio frames, it is not necessary to determine which sound source corresponds to each predicted sound source classification. Therefore, in order to realize the reverse parameter adjustment of the audio processing model, the PIT (Permutation invariant loss) strategy can be adopted, that is, the loss values ​​are calculated respectively under multiple mapping types between the first number of predicted sound source classifications and the first number of sound sources, and parameter adjustment is performed based on the smaller loss value.

[0086] According to some embodiments, each of the multiple corresponding relationships corresponding to the multiple mapping types between the first number of predicted sound source categories and the first number of sound sources is a one-to-one corresponding relationship.

[0087] For example, when there are two sound sources, sound source A and sound source B, the first confidence of the sample audio frame for the first predicted sound source classification among the two predicted sound source classifications is 0.1, and the first confidence of the sample audio frame for the second predicted sound source classification among the two predicted sound source classifications is 0.9, and the sound source label of the sample audio frame is sound source A.

[0088] First, the first predicted sound source is classified as sound source A, and the second predicted sound source is classified as sound source B. Based on the first confidence of the sample audio frame for sound source A, which is 0.1, and the first confidence of the sample audio frame for sound source B, which is 0.9, and the sound source label of the sample audio frame, that is, sound source A, the loss value A is calculated.

[0089] Secondly, the first predicted sound source is classified as sound source B, and the second predicted sound source is classified as sound source A. Based on the first confidence of the sample audio frame for sound source B, which is 0.1, and the first confidence of the sample audio frame for sound source A, which is 0.9, and the sound source label of the sample audio frame, that is, sound source A, the loss value B is calculated.

[0090] Obviously, the loss value B is smaller than the loss value A. Therefore, the correspondence between the predicted sound source classification and the sound source under the target mapping type is determined as the first predicted sound source classification as sound source B, and the second predicted sound source classification as sound source A. Based on the loss value B calculated based on the correspondence, the parameters of the audio processing model are adjusted.

[0091] According to some embodiments, the adjustment of the parameters of the audio processing model in step S404 may be performed based on a first confidence level corresponding to each of the plurality of sample audio frames.

[0092] For example, still taking the case where there are two sound sources, sound source A and sound source B, as an example, the multiple sample audio frames are sample audio frame a, sample audio frame b, and sample audio frame c. Among them, the first confidences of sample audio frame a, sample audio frame b, and sample audio frame c for the first predicted sound source classification in the two predicted sound source classifications are 0.1, 0.8, and 0.3, respectively, and the first confidences of sample audio frame a, sample audio frame b, and sample audio frame c for the second predicted sound source classification in the two predicted sound source classifications are 0.9, 0.2, and 0.7, and the sound source labels of the sample audio frame a, sample audio frame b, and sample audio frame c are sound source A, sound source B, and sound source A, respectively.

[0093] Obviously, the loss value calculated under the target mapping type in which the first predicted sound source classification is sound source B and the second predicted sound source classification is sound source A is smaller. Therefore, based on the smaller loss value, the parameters of the audio processing model are adjusted.

[0094] According to some embodiments, when an additional local feature extraction module is included in the training process, the parameters of the additional local feature extraction module can also be adjusted at the same time through step S404.

[0095] According to some embodiments, the loss value may be calculated by binary cross entropy (BCE).

[0096] According to some embodiments, the third confidence of the sample audio frame for each of the first number of predicted sound source classifications can be obtained in sequence based on the first local feature information of each sample audio frame in a plurality of sample audio frames using a second additional output module; and the parameters of the local feature extraction module in the audio processing model can be adjusted based on the third confidence of each of the sample audio frames in the plurality of sample audio frames for each of the first number of predicted sound source classifications and the sound source label of each of the plurality of sample audio frames.

[0097] Among them, the second additional output module is applied to the training process of the audio processing model. Through the second additional output module, the parameters of the local feature extraction module in the audio processing model can be adjusted based on the first local feature information of each sample audio frame, thereby further improving the training effect.

[0098] According to some embodiments, the adjustment of the parameters of the local feature extraction module performed by using the second additional output module can also be performed based on the third confidence level corresponding to each sample audio frame in a plurality of sample audio frames. The specific method is similar to the above-mentioned method of adjusting the parameters of the audio processing model based on the first confidence level corresponding to each sample audio frame in a plurality of sample audio frames, and will not be repeated here.

[0099] Figure 5is a schematic diagram showing a training method of an audio processing model according to an exemplary embodiment of the present disclosure, such as Figure 5 As shown, each of the multiple sample audio frames in the sample audio data is sequentially input into the local feature extraction module to obtain the first local feature information of each of the multiple sample audio frames output by the local feature extraction module.

[0100] Each of the multiple sample audio frames in the sample audio data is sequentially input into the additional local feature extraction module to obtain second local feature information of each of the multiple sample audio frames output by the additional local feature extraction module.

[0101] The first local feature information and the second local feature information of each sample audio frame are fused through a fusion module to obtain fused local feature information of the sample audio frame.

[0102] Until the fused local feature information of each sample audio frame in all the sample audio frames input sequentially is obtained, the fused local feature information of each sample audio frame in all the sample audio frames is input into the global feature extraction module to obtain the global feature information of each sample audio frame in all the sample audio frames output sequentially by the global feature extraction module.

[0103] For the global feature information of each sample audio frame, the output module is used to obtain the first confidence of the sample audio frame for each predicted sound source classification in the first number of predicted sound source classifications, and the first loss value is calculated based on the first confidence of each sample audio frame in the multiple sample audio frames for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of each sample audio frame in the multiple sample audio frames, and the parameters of the local feature extraction module, the global feature extraction module and the output module in the audio processing model, as well as the additional local feature extraction module are adjusted according to the first loss value. The adjustment of the parameters of the audio processing model is performed in units of the entire sample audio data.

[0104] For the second local feature information of each sample audio frame, the first additional output module is used to obtain the second confidence of the sample audio frame for each predicted sound source classification in the first number of predicted sound source classifications, and the second loss value is calculated based on the second confidence of the sample audio frame for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of the sample audio frame, and the parameters of the additional local feature extraction module are adjusted according to the second loss value. The adjustment of the parameters of the additional local feature extraction module is performed in units of each sample audio frame.

[0105] For the first local feature information of each sample audio frame, the second additional output module is used to obtain the third confidence of the sample audio frame for each predicted sound source classification in the first number of predicted sound source classifications, and the third loss value is calculated based on the third confidence of the sample audio frame for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of the sample audio frame, and the parameters of the local feature extraction module are adjusted according to the third loss value. The adjustment can be performed in units of each sample audio frame or in units of the entire sample audio data.

[0106] It should be noted that in Figure 5 The part in the dotted box is only used for the training process of the audio processing model, and does not participate in the application of the audio processing model.

[0107] Figure 6 is a block diagram showing a structure of an audio processing device according to an exemplary embodiment of the present disclosure, Figure 6 As shown, the audio processing model 600 includes: a local feature extraction module 601, configured to determine the local feature information of each audio frame in a plurality of audio frames extracted from audio data to be processed in turn, wherein the audio data to be processed includes audio data from at least two sound sources; a global feature extraction module 602, configured to determine any one audio frame in the plurality of audio frames as a target audio frame, and determine the global feature information of the target audio frame based on the local feature information of each audio frame in the plurality of audio frames; and an output module 603, configured to determine the sound source classification corresponding to the target audio frame based on the global feature information of the target audio frame.

[0108] According to some embodiments, the local feature extraction module includes a multi-layer temporal dilated convolutional neural network (TDCN).

[0109] According to some embodiments, the global feature extraction module may include one or more attention submodules, and at least one of the one or more attention submodules uses exponential linear operations, thereby effectively simplifying calculations and improving processing efficiency.

[0110] In order to improve the computational efficiency of the attention submodule, an exponential linear unit can be used to perform the calculation in the attention submodule. The processing result O of the attention submodule on the input feature can be expressed as:

[0111] O=φ(Q)(φ(K) T V).

[0112] Among them, Q represents the query vector, K represents the keyword vector, and V represents the value vector.

[0113] Among them, φ(x)=elu(x)+1, when x is greater than 0, elu(x) is equal to x, when x is less than or equal to 0, elu(x)=α(e x -1), where α is a hyperparameter and is usually set to 1.0.

[0114] According to some embodiments, the local feature extraction module includes: a submodule for determining local feature information of the audio frame based on the audio frame and one or more audio frames close to the audio frame among the multiple audio frames.

[0115] Figure 7 is a structural block diagram showing a training device for an audio processing model according to an exemplary embodiment of the present disclosure, such as Figure 7 As shown, a training device 700 for an audio processing model, wherein the audio processing model includes a local feature extraction module, a global feature extraction module and an output module, and the device 700 includes: a first acquisition module 701, configured to sequentially obtain, for each of a plurality of sample audio frames extracted from sample audio data, first local feature information of the sample audio frame using the local feature extraction module, wherein the sample audio data includes audio data from a first number of sound sources, and each of the plurality of sample audio frames has a sound source label; a second acquisition module 702, configured to obtain, based at least on the first local feature information of each of the plurality of sample audio frames, first local feature information of the sample audio frame using the global feature extraction module; A feature extraction module is used to obtain global feature information of each sample audio frame in a plurality of sample audio frames; a third acquisition module 703 is configured to input the global feature information of the sample audio frame into the output module in turn for each sample audio frame in the plurality of sample audio frames, so as to obtain a first confidence of the sample audio frame output by the output module for each predicted sound source classification in a first number of predicted sound source classifications; and a first adjustment module 704 is configured to adjust the parameters of the audio processing model based on the first confidence of each sample audio frame in a plurality of sample audio frames for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of each sample audio frame in the plurality of sample audio frames.

[0116] According to some embodiments, the first adjustment module includes: a determination submodule, configured to determine the correspondence between a first number of predicted sound source categories and a first number of sound sources under a target mapping type, wherein, among the multiple correspondences corresponding to the multiple mapping types between the first number of predicted sound source categories and the first number of sound sources, the loss value between the first confidence corresponding to each of the multiple sample audio frames calculated based on the target mapping type and the sound source label of the sample audio frame is the smallest; and an adjustment submodule, configured to adjust the parameters of the audio processing model based on the loss value calculated according to the correspondence under the target mapping type.

[0117] According to some embodiments, the device also includes: a fourth acquisition module, configured to sequentially obtain second local feature information of each sample audio frame in a plurality of sample audio frames extracted from the sample audio data using an additional local feature extraction module; wherein the second acquisition module includes: an acquisition submodule, configured to obtain global feature information of each sample audio frame in a plurality of sample audio frames using a global feature extraction module based on the first local feature information and the second local feature information of each sample audio frame in a plurality of sample audio frames.

[0118] According to some embodiments, the acquisition submodule includes: a unit for fusing, for each sample audio frame in a plurality of sample audio frames, first local feature information and second local feature information of the sample audio frame to obtain fused local feature information of the sample audio frame; and a unit for acquiring, based on the fused local feature information of each sample audio frame in a plurality of sample audio frames, the global feature information of each sample audio frame in a plurality of sample audio frames using a global feature extraction module.

[0119] According to some embodiments, the device also includes: a fifth acquisition module, configured to input the second local feature information of the sample audio frame into the first additional output module to obtain the second confidence of the sample audio frame output by the first additional output module for each predicted sound source classification in the first number of predicted sound source classifications; and a second adjustment module, configured to adjust the parameters of the additional local feature extraction module based on the second confidence of the sample audio frame for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of the sample audio frame.

[0120] According to some embodiments, the device also includes: a sixth acquisition module, configured to obtain, in sequence based on the first local feature information of each sample audio frame in a plurality of sample audio frames, using a second additional output module, a third confidence of the sample audio frame for each predicted sound source classification in a first number of predicted sound source classifications; and a third adjustment module, configured to adjust the parameters of the local feature extraction module in the audio processing model based on the third confidence of each sample audio frame in a plurality of sample audio frames for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of each sample audio frame in a plurality of sample audio frames.

[0121] According to an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above methods.

[0122] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to execute any one of the above methods.

[0123] According to an embodiment of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements any one of the above methods when executed by a processor.

[0124] refer to Figure 8 , a block diagram of an electronic device 800 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0125] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0126] Multiple components in the device 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 can be any type of device that can input information to the device 800. The input unit 806 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 807 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 808 can include but is not limited to a disk, an optical disk. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a 1302.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0127] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a processing method for audio data or a training method for an audio processing model. For example, in some embodiments, the processing method for audio data or the training method for an audio processing model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or a communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the processing method for audio data or the training method for an audio processing model described above may be executed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the method for processing audio data or the method for training an audio processing model in any other appropriate manner (for example, by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, audio input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0135] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. An audio processing method, comprising: sequentially determining local feature information of each audio frame of a plurality of audio frames extracted from audio data to be processed, wherein the audio data to be processed includes audio data from at least two sound sources; and Determine any one of the multiple audio frames as a target audio frame, and perform the following operations on the target audio frame: Determining global feature information of the target audio frame based on local feature information of each audio frame in the plurality of audio frames; and Based on the global feature information of the target audio frame, a sound source classification corresponding to the target audio frame is determined.

2. The method according to claim 1, wherein: The multiple audio frames are sequentially adjacent to each other in the time domain of the audio data to be processed.

3. The method according to claim 1 or 2, wherein: The step of sequentially determining the local feature information of each audio frame among the plurality of audio frames extracted from the audio data to be processed comprises: Based on the audio frame and one or more audio frames close to the audio frame among the multiple audio frames, local feature information of the audio frame is determined.

4. The method according to any one of claims 1 to 3, wherein: Each of the at least two sound sources is a speaker.

5. A method for training an audio processing model, wherein: The audio processing model includes a local feature extraction module, a global feature extraction module and an output module, and the method includes: For each of a plurality of sample audio frames extracted from sample audio data, using the local feature extraction module, obtaining first local feature information of the sample audio frame, wherein the sample audio data includes audio data from a first number of sound sources, and each of the plurality of sample audio frames has a sound source label; Based at least on the first local feature information of each of the multiple sample audio frames, using the global feature extraction module, obtain the global feature information of each of the multiple sample audio frames; For each of the plurality of sample audio frames, inputting the global feature information of the sample audio frame into the output module in sequence, so as to obtain a first confidence of the sample audio frame output by the output module for each of the first number of predicted sound source classifications; and Adjust the parameters of the audio processing model based on a first confidence of each of the multiple sample audio frames for each of the first number of predicted sound source classifications and a sound source label of each of the multiple sample audio frames.

6. The method according to claim 5, wherein: The adjusting the parameters of the audio processing model based on the first confidence of each sample audio frame in the plurality of sample audio frames for each predicted sound source classification in the first number of predicted sound source classifications and the sound source label of each sample audio frame in the plurality of sample audio frames comprises: Determining a correspondence between the first number of predicted sound source categories and the first number of sound sources under a target mapping type, wherein, among a plurality of correspondences respectively corresponding to a plurality of mapping types between the first number of predicted sound source categories and the first number of sound sources, a loss value between a first confidence corresponding to each of the plurality of sample audio frames calculated based on the target mapping type and a sound source label of the sample audio frame is minimum; and Based on the loss value calculated according to the corresponding relationship under the target mapping type, the parameters of the audio processing model are adjusted.

7. The method according to claim 5 or 6, further comprising: For each of the multiple sample audio frames extracted from the sample audio data, using the additional local feature extraction module, obtain second local feature information of the sample audio frame; The acquiring, based at least on the first local feature information of each of the multiple sample audio frames, the global feature extraction module, of the global feature information of each of the multiple sample audio frames comprises: Based on the first local feature information and the second local feature information of each sample audio frame in the multiple sample audio frames, the global feature information of each sample audio frame in the multiple sample audio frames is obtained by using the global feature extraction module.

8. The method according to claim 7, wherein: The acquiring, based on the first local feature information and the second local feature information of each sample audio frame in the multiple sample audio frames, the global feature extraction module comprises: For each sample audio frame of the multiple sample audio frames, fuse the first local feature information and the second local feature information of the sample audio frame to obtain fused local feature information of the sample audio frame; and Based on the fused local feature information of each sample audio frame in the multiple sample audio frames, the global feature information of each sample audio frame in the multiple sample audio frames is obtained by using the global feature extraction module.

9. The method according to claim 7 or 8, further comprising, for each of the plurality of sample audio frames, performing the following operations: Inputting the second local feature information of the sample audio frame into the first additional output module to obtain a second confidence level of the sample audio frame output by the first additional output module for each predicted sound source classification of the first number of predicted sound source classifications; and Adjust parameters of the additional local feature extraction module based on a second confidence of the sample audio frame for each of the first number of predicted sound source classifications and a sound source label of the sample audio frame.

10. The method according to any one of claims 5 to 9, further comprising: sequentially based on the first local feature information of each sample audio frame in the plurality of sample audio frames, using a second additional output module, obtaining a third confidence of the sample audio frame for each predicted sound source classification in the first number of predicted sound source classifications; as well as Adjust the parameters of the local feature extraction module in the audio processing model based on a third confidence of each of the multiple sample audio frames for each of the first number of predicted sound source classifications and a sound source label of each of the multiple sample audio frames.

11. An audio processing device, comprising: A local feature extraction module is configured to determine local feature information of each audio frame in turn for each of a plurality of audio frames extracted from audio data to be processed, wherein the audio data to be processed includes audio data from at least two sound sources; a global feature extraction module configured to determine any one of the multiple audio frames as a target audio frame, and determine global feature information of the target audio frame based on local feature information of each of the multiple audio frames; and The output module is configured to determine the sound source classification corresponding to the target audio frame based on the global feature information of the target audio frame.

12. The audio processing device according to claim 11, wherein: The local feature extraction module comprises: A submodule of local feature information of the audio frame is determined based on the audio frame and one or more audio frames close to the audio frame among the multiple audio frames.

13. The audio processing device according to claim 11 or 12, wherein: The global feature extraction module includes one or more attention sub-modules, and at least one of the one or more attention sub-modules adopts exponential linear operation.

14. A training device for an audio processing model, wherein: The audio processing model includes a local feature extraction module, a global feature extraction module and an output module, and the device includes: A first acquisition module is configured to sequentially obtain first local feature information of each sample audio frame of a plurality of sample audio frames extracted from sample audio data by using the local feature extraction module, wherein the sample audio data includes audio data from a first number of sound sources, and each sample audio frame of the plurality of sample audio frames has a sound source label; A second acquisition module is configured to acquire global feature information of each of the multiple sample audio frames using the global feature extraction module based on at least the first local feature information of each of the multiple sample audio frames; A third acquisition module is configured to input the global feature information of each sample audio frame of the plurality of sample audio frames into the output module in sequence, so as to acquire a first confidence of the sample audio frame output by the output module for each predicted sound source classification of the first number of predicted sound source classifications; and The first adjustment module is configured to adjust the parameters of the audio processing model based on a first confidence of each sample audio frame in the multiple sample audio frames for each predicted sound source classification in the first number of predicted sound source classifications and a sound source label of each sample audio frame in the multiple sample audio frames.

15. The device according to claim 14, wherein: The first adjustment module includes: a determination submodule configured to determine a correspondence between the first number of predicted sound source categories and the first number of sound sources under a target mapping type, wherein, among a plurality of correspondences respectively corresponding to a plurality of mapping types between the first number of predicted sound source categories and the first number of sound sources, a loss value between a first confidence corresponding to each of the plurality of sample audio frames calculated based on the target mapping type and a sound source label of the sample audio frame is minimum; and The adjustment submodule is configured to adjust the parameters of the audio processing model based on the loss value calculated according to the corresponding relationship under the target mapping type.

16. The device according to claim 14 or 15, further comprising: A fourth acquisition module is configured to sequentially obtain second local feature information of each sample audio frame of the plurality of sample audio frames extracted from the sample audio data by using the additional local feature extraction module; Wherein, the second acquisition module includes: The acquisition submodule is configured to acquire the global feature information of each sample audio frame in the multiple sample audio frames based on the first local feature information and the second local feature information of each sample audio frame in the multiple sample audio frames using the global feature extraction module.

17. The device according to claim 16, wherein: The acquisition submodule includes: a unit configured to fuse, for each sample audio frame of the plurality of sample audio frames, the first local feature information and the second local feature information of the sample audio frame to obtain fused local feature information of the sample audio frame; and A unit for obtaining global feature information of each of the multiple sample audio frames using the global feature extraction module based on the fused local feature information of each of the multiple sample audio frames.

18. The apparatus according to claim 16 or 17, further comprising: a fifth acquisition module, configured to input the second local feature information of the sample audio frame into the first additional output module to obtain a second confidence of the sample audio frame output by the first additional output module for each predicted sound source classification of the first number of predicted sound source classifications; as well as The second adjustment module is configured to adjust the parameters of the additional local feature extraction module based on the second confidence of the sample audio frame for each of the first number of predicted sound source classifications and the sound source label of the sample audio frame.

19. The device according to any one of claims 14 to 18, further comprising: a sixth acquisition module configured to sequentially acquire, based on the first local feature information of each sample audio frame in the plurality of sample audio frames, a third confidence of the sample audio frame for each predicted sound source classification in the first number of predicted sound source classifications by using the second additional output module; as well as A third adjustment module is configured to adjust the parameters of the local feature extraction module in the audio processing model based on a third confidence of each of the multiple sample audio frames for each of the first number of predicted sound source classifications and a sound source label of each of the multiple sample audio frames.

20. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

22. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images