Audio processing method and device, electronic equipment and storage medium

Abstract

The invention provides an audio processing method and device, electronic equipment and a storage medium, and the method comprises the steps: receiving a to-be-processed audio file which comprises at least one speaker; the to-be-processed audio file is input to a preset speaker segmentation model, speaker information corresponding to the to-be-processed audio file is obtained, and the speaker information comprises at least one speaker identifier and at least one duration corresponding to each speaker identifier; the preset speaker segmentation model is obtained by training a neural network model structure based on a self-attention mechanism. According to the invention, the received to-be-processed audio file is input into the preset speaker segmentation model obtained by training the neural network model structure based on the self-attention mechanism to obtain the corresponding speaker information, so that speaker segmentation processing can be carried out on the audio file for a long time at one time; and the accuracy of a speaker segmentation result in a multi-speaker scene is improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] Speech speaker segmentation and clustering is the task of finding “who spoke and when” from an audio stream with multiple speakers. This task is essential for automatically transcribing or understanding a conversation, a meeting, or a radio program.

[0003] The inventors found that the traditional speaker segmentation and clustering method first needs to segment the speech signal into different segments, then extract features from these segments, and finally use clustering algorithms to identify different speakers. This method has a low accuracy rate. In addition, the inventors also found that the existing end-to-end method based on deep learning is more effective in separating audio with fewer speakers, but as the number of speakers increases, the separation effect decreases rapidly. Summary of the invention

[0004] The embodiments of the present application provide an audio processing method, device, electronic device, and storage medium, which can improve the accuracy of speaker segmentation results of an audio file in a multi-speaking scenario.

[0005] In a first aspect, an embodiment of the present application provides an audio processing method, the method comprising:

[0006] receiving an audio file to be processed, wherein the audio file to be processed includes at least one speaker;

[0007] Inputting the audio file to be processed into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, wherein the speaker information includes at least one speaker identifier and at least one duration corresponding to each speaker identifier;

[0008] The preset speaker segmentation model is obtained by training a neural network model structure based on a self-attention mechanism.

[0009] In a second aspect, an embodiment of the present application further provides an audio processing device, the device comprising:

[0010] A receiving module, configured to receive an audio file to be processed, wherein the audio file to be processed includes at least one speaker;

[0011] A processing module, configured to input the audio file to be processed into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, wherein the speaker information includes at least one speaker identifier and at least one duration period corresponding to each of the speakers;

[0012] The preset speaker segmentation model is obtained by training a neural network model structure based on a self-attention mechanism.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-mentioned audio processing method when executed by the processor.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned audio processing method is implemented.

[0015] The embodiments of the present application include at least the following technical effects:

[0016] The technical solution of the embodiment of the present application obtains the corresponding speaker information by inputting the received audio file to be processed into a preset speaker segmentation model obtained by training a neural network model structure based on a self-attention mechanism. Compared with the speaker segmentation processing method including multiple separation steps, the preset speaker segmentation model uses the original speech signal in the audio file as the input of the model, which can reduce the complexity of the model's online deployment. Since the preset speaker segmentation model is not modeled by sequence, but by using an encoder within a feature segment and a decoder between feature segments to achieve modeling, the preset speaker segmentation model can perform speaker segmentation processing on an audio file of a longer duration at one time, and can also improve the accuracy of the speaker segmentation results in multiple speaker scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0018] Figure 1 This is one of the flowcharts of the audio processing method provided in the embodiment of the present application;

[0019] Figure 2 This is the second flowchart of the audio processing method provided in the embodiment of the present application;

[0020] Figure 3 is a structural diagram of an audio processing device provided in an embodiment of the present application;

[0021] Figure 4 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0024] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0025] like Figure 1 As shown, an embodiment of the present application provides an audio processing method, the method comprising:

[0026] Step 101: receiving an audio file to be processed, wherein the audio file to be processed includes at least one speaker.

[0027] In this embodiment, the audio content stored in the audio file to be processed may be a conversation, a meeting, or a broadcast program. The audio file to be processed includes at least one speaker, that is, the audio content included in the audio file to be processed is generated by the voice of at least one speaker, wherein each speaker corresponds to at least one duration in the audio file to be processed. For example, the audio content of a certain audio file is the conversation content of speaker A, speaker B, and speaker C. Then, when it is necessary to obtain the speech content of each speaker, when the corresponding duration of the audio file is short, the speech content of each speaker can be obtained by playing the audio file. However, when the corresponding duration of the audio file is long, the time cost of obtaining the speech content by playing the audio file is high. At this time, the audio file can be used as the audio file to be processed and processed by the preset speaker segmentation model in step 102.

[0028] Step 102: input the audio file to be processed into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, wherein the speaker information includes at least one speaker identifier and at least one duration corresponding to each speaker identifier.

[0029] The preset speaker segmentation model is obtained by training a neural network model structure based on a self-attention mechanism.

[0030] After receiving the audio file to be processed, the audio file to be processed is input into a preset speaker segmentation model, which is obtained by training a neural network model structure based on a self-attention mechanism. The preset speaker segmentation model can process the input audio to be processed to obtain speaker information corresponding to the audio file to be processed, wherein the speaker information includes at least one speaker identifier and at least one duration corresponding to each speaker identifier.

[0031] Here, the speaker identifier is used to distinguish different speakers. For example, when the model recognizes that the number of speakers corresponding to the audio to be processed is 3, a speaker identifier is assigned to each speaker to distinguish the three speakers.

[0032] Optionally, the loss function used in the training of the preset speaker segmentation model is a permutation invariant training (PIT) loss function. The attention mechanism corresponding to the TransformerDecoder module is a Mutli Head Attention multi-head attention mechanism.

[0033] In an embodiment of the present application, the received audio file to be processed is input into a preset speaker segmentation model obtained by training a neural network model structure based on a self-attention mechanism to obtain corresponding speaker information. Compared with a speaker segmentation processing method including multiple separation steps, the preset speaker segmentation model uses the original speech signal in the audio file as the input of the model, which can reduce the complexity of the model's online deployment. Since the preset speaker segmentation model is not modeled through sequences, but is modeled by using an encoder within a feature segment and a decoder between feature segments, the preset speaker segmentation model can perform speaker segmentation processing on an audio file of a longer duration at one time, and can also improve the accuracy of the speaker segmentation results in multiple speaker scenarios.

[0034] In an optional embodiment of the present application, the audio file to be processed is input into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, including:

[0035] The audio file to be processed is sampled by a convolutional neural network (CNN) module included in the preset speaker segmentation model to obtain a plurality of speech feature vectors;

[0036] Segmenting the plurality of speech feature vectors by using the preset speaker segmentation model to obtain a plurality of speech feature segments;

[0037] Processing the plurality of speech feature segments through a target module included in the preset speaker segmentation model to obtain speaker probability information corresponding to each speech feature segment, wherein the target module at least includes: an encoder module, a decoder module and a nonlinear activation module;

[0038] The speaker information corresponding to the audio file to be processed is determined according to the speaker probability information respectively corresponding to the multiple speech feature segments, wherein the speaker probability information includes at least one speaker identifier and the probability corresponding to each speaker identifier.

[0039] Specifically, when the audio file to be processed is input into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, the audio file to be processed is first sampled and processed by the CNN module included in the preset speaker segmentation model to obtain multiple speech feature vectors, wherein each semantic feature vector corresponds to a speech segment of a preset duration. For example, when the preset duration is 10ms, one frame of speech feature vector can be generated for every 10ms of audio.

[0040] Considering that the number of speech feature vectors is relatively large, for example, when one frame of speech feature vector can be generated for every 10ms of audio, 6000 frames of speech feature vectors will be generated for 1 minute of audio, and the speech feature vectors need to be segmented to save computing power and video memory in the subsequent processing. Specifically, multiple speech feature vectors can be segmented by a preset speaker segmentation model to obtain multiple speech feature segments. For example, 1000 frames of speech feature vectors can be divided into one segment, and then 1 minute of audio will generate 6 speech feature segments, where each speech feature segment includes 1000 frames of speech feature vectors.

[0041] After segmenting multiple speech feature vectors to obtain multiple speech feature segments, the multiple speech feature segments can be processed by a target module included in a preset speaker segmentation model to obtain speaker probability information corresponding to each speech feature segment. The target module here includes at least: an encoder module, a decoder module and a non-linear activation module, wherein the encoder module can be a Transformer Encoder module, the decoder module can be a Transformer Decoder module, and the non-linear activation module can be a Linear+Sigmoid module.

[0042] After obtaining the speaker probability information corresponding to each speech feature segment, the speaker information corresponding to the audio file to be processed can be determined.

[0043] The above implementation scheme of the present application, by segmenting the speech feature vector to obtain speech feature segments, and processing the speech feature segments through the target module to obtain the speaker probability information corresponding to the speech feature segments, can save the computing power and video memory used to process audio files in the preset speaker segmentation model.

[0044] In an optional embodiment of the present application, the target module included in the preset speaker segmentation model processes the multiple speech feature segments to obtain speaker probability information corresponding to each speech feature segment, including:

[0045] The encoder module processes the plurality of speech feature segments respectively to obtain a latent vector feature segment corresponding to each speech feature segment;

[0046] The multiple latent vector feature segments corresponding to the multiple speech feature segments are processed by the decoder module and the nonlinear activation module to obtain speaker probability information corresponding to each speech feature segment.

[0047] Specifically, when a plurality of speech feature segments are processed by a target module included in a preset speaker segmentation model to obtain speaker probability information corresponding to each speech feature segment, the plurality of speech feature segments are first processed separately by an encoder module included in the target module to obtain a latent vector feature segment corresponding to each speech feature segment, that is, through the processing of the encoder module, the plurality of speech feature segments can be encoded (i.e., transformed) into latent vector feature segments composed of vectors that are easy for a machine to learn, thereby facilitating subsequent processing. It should be noted that the encoder module (transformer encoder module) can use the information embedding vector of each position in the input speech feature vector segment as the initial input, and then, through a multi-layer self-attention mechanism and a fully connected layer, encode the information of each position in the speech feature vector segment into a fixed-length latent vector representation, thereby obtaining a latent vector feature segment corresponding to each speech feature segment.

[0048] Then, the decoder module and nonlinear activation module included in the target module process the multiple latent vector feature segments corresponding to the multiple speech feature segments to obtain the speaker probability information corresponding to each speech feature segment. That is, the output of the encoder module is used as the input of the decoder module and the nonlinear activation module. The decoder module can generate the speaker corresponding to the information of the next position until the processing of all latent vector feature segments is completed. The decoder is also composed of a multi-layer self-attention mechanism and a fully connected layer. The nonlinear activation module is used to map the output of the decoder to obtain the output speaker probability. Then, through the processing of the decoder and the nonlinear activation module, the speaker probability information corresponding to each speech feature segment can be obtained.

[0049] The above implementation scheme of the present application processes each speech feature segment through an encoder module. Compared with directly processing the speech feature vector, this can improve the processing efficiency of the encoder module and enhance the accuracy of the model.

[0050] In an optional embodiment of the present application, the decoder module and the nonlinear activation module process the multiple latent vector feature segments corresponding to the multiple speech feature segments to obtain speaker probability information corresponding to each speech feature segment, including:

[0051] Processing a first latent vector feature segment among the multiple latent vector feature segments through the nonlinear activation module to obtain speaker probability information of a speech feature segment corresponding to the first latent vector feature segment;

[0052] For each remaining latent vector feature segment except the first latent vector feature segment among the multiple latent vector feature segments, the remaining latent vector feature segment and the previous latent vector feature segment corresponding to the remaining latent vector feature segment are input into the decoder module together, and the obtained output result is processed by the nonlinear activation module to obtain the speaker probability information of the speech feature segment corresponding to the remaining latent vector feature segment.

[0053] Specifically, when processing multiple latent vector feature segments corresponding to multiple speech feature segments through the decoder module and the nonlinear activation module, the multiple latent vector feature segments need to be processed one by one to obtain speaker probability information corresponding to each speech feature segment.

[0054] Among them, for the first latent vector feature segment, it is directly processed by the nonlinear activation module to obtain the corresponding speaker probability information. For the second latent vector feature segment, the second latent vector feature segment and the previous latent vector feature segment, that is, the first latent vector feature segment, are input into the decoder module together. Specifically, the second latent vector feature segment is used as the input of the decoder module, and the first latent vector feature segment is used as the memory of the decoder module. Then, the output result of the decoder module is input into the nonlinear activation module to obtain the speaker probability information of the speech feature segment corresponding to the second latent vector feature segment. For the third latent vector feature segment, the third latent vector feature segment and the previous latent vector feature segment, that is, the second latent vector feature segment, are input into the decoder module together. Specifically, the third latent vector feature segment is used as the input of the decoder module, and the second latent vector feature segment is used as the memory of the decoder module. Then, the output result of the decoder module is input into the nonlinear activation module to obtain the speaker probability information corresponding to the third latent vector feature segment. For the Nth latent vector feature segment, the Nth latent vector feature segment and the previous latent vector feature segment, that is, the N-1th latent vector feature segment, are input into the decoder module together. Specifically, the Nth latent vector feature segment is used as the input of the decoder module, and the N-1th latent vector feature segment is used as the memory of the decoder module. Then, the output result of the decoder module is input into the nonlinear activation module to obtain the speaker probability information corresponding to the Nth latent vector feature segment, where N is an integer greater than or equal to 2. Through the above process, each remaining latent vector feature segment except the first latent vector feature segment in the multiple latent vector feature segments is processed in turn to obtain the corresponding speaker probability information.

[0055] In the above implementation scheme of the present application, when obtaining the speaker probability information corresponding to each latent vector feature segment, the latent vector feature segment having the previous latent vector feature segment is further processed by a decoder module and then processed by a nonlinear activation module, thereby improving the accuracy of the model.

[0056] In an optional embodiment of the present application, determining the speaker information corresponding to the audio file to be processed according to the speaker probability information respectively corresponding to the plurality of speech feature segments includes:

[0057] For each of the plurality of speech feature segments, confirming a speaker identifier whose probability corresponding to the speaker identifier in the speaker probability information corresponding to the speech feature segment is greater than a preset threshold as a valid speaker identifier;

[0058] Speaker information corresponding to the audio file to be processed is determined according to the valid speaker identifiers respectively corresponding to the multiple voice feature segments.

[0059] Specifically, when determining the speaker information corresponding to the audio file to be processed based on the speaker probability information corresponding to the multiple speech feature segments, first, for each speech feature segment, the speaker identifier corresponding to the speaker identifier in the speaker probability information corresponding to the speech feature segment whose probability is greater than a preset threshold is confirmed as a valid speaker identifier, and then the speaker information corresponding to the audio file to be processed can be obtained based on the valid speaker identifiers corresponding to the multiple speech feature segments. For example, for a certain speech feature segment, the speaker probability information is: the probability of speaker 1 is 0.6, the probability of speaker 2 is 0.4, and the probability of speaker 3 is 0.1, and the preset threshold is 0.5, then at this time, speaker 1 with a probability greater than 0.5 is confirmed as the valid speaker identifier corresponding to the speech feature segment. After determining the valid speaker identifier corresponding to each speech feature segment based on the above method, at least one duration information corresponding to each speaker can be obtained, and then the speaker information of the audio file to be processed can be obtained.

[0060] The above implementation scheme of the present application can improve the accuracy of the speaker information corresponding to the audio to be processed by judging the probability corresponding to the speaker identifier and confirming the corresponding speaker identifier as a valid speaker identifier when the probability is greater than a preset threshold.

[0061] The overall implementation process of the embodiment of the present application is introduced below. Figure 2 As shown, including:

[0062] Step 201, the speech signal is processed by a convolutional neural network (CNN) module to obtain a plurality of speech feature vectors, namely X1, X2, ...X T , where T is an integer.

[0063] Step 202, segmenting the plurality of speech feature vectors to obtain a plurality of speech feature segments, namely, speech feature segment 1, speech feature segment 2, ..., speech feature segment N, where N is an integer

[0064] Step 203, each of the multiple speech feature segments is processed by a Transformer Encoder module to obtain a latent vector feature segment corresponding to each speech feature segment, that is, feature segment 1, feature segment 2, ..., feature segment N.

[0065] Step 204: input feature segment 1 into a Linear+Sigmoid module to obtain corresponding speaker probability information.

[0066] Step 205: for feature segment M, input feature segment M and feature segment M-1 into the TransformerDecoder module to obtain an output result, and input the output result into the Linear+Sigmoid module to obtain speaker probability information corresponding to feature segment M, where M is an integer greater than or equal to 2 and less than or equal to N.

[0067] Step 206: Determine the speaker information corresponding to the speech signal according to the speaker probability information corresponding to the M feature segments.

[0068] It should be noted that the CNN model, Transformer Encoder model, Transformer Decoder model and Linear+Sigmoid module in the above steps are all part of the preset speaker segmentation model, which is obtained by training a neural network model based on the attention-only mechanism, namely, the Transformer model.

[0069] The present application obtains the corresponding speaker information by inputting the received audio file to be processed into a preset speaker segmentation model obtained by training a neural network model structure based on a self-attention mechanism. Compared with the speaker segmentation processing method including multiple separation steps, the preset speaker segmentation model uses the original speech signal in the audio file as the input of the model, which can reduce the complexity of the model's online deployment. Since the preset speaker segmentation model is not modeled through sequences, but is modeled by using an encoder within a feature segment and a decoder between feature segments, the preset speaker segmentation model can perform speaker segmentation processing on audio files of a longer duration at one time, and can also improve the accuracy of speaker segmentation results in multiple speaker scenarios.

[0070] The above describes the audio processing method provided in the embodiment of the present application. The following will describe the audio processing device provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0071] like Figure 3 As shown, an embodiment of the present invention further provides an audio processing device, the device comprising:

[0072] A receiving module 301 is used to receive an audio file to be processed, wherein the audio file to be processed includes at least one speaker;

[0073] A processing module 302 is used to input the audio file to be processed into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, wherein the speaker information includes at least one speaker identifier and at least one duration corresponding to each speaker;

[0074] The preset speaker segmentation model is obtained by training a neural network model structure based on a self-attention mechanism.

[0075] Optionally, inputting the audio file to be processed into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed includes:

[0076] The audio file to be processed is sampled by a convolutional neural network (CNN) module included in the preset speaker segmentation model to obtain a plurality of speech feature vectors;

[0077] Segmenting the plurality of speech feature vectors by using the preset speaker segmentation model to obtain a plurality of speech feature segments;

[0078] Processing the plurality of speech feature segments through a target module included in the preset speaker segmentation model to obtain speaker probability information corresponding to each speech feature segment, wherein the target module at least includes: an encoder module, a decoder module and a nonlinear activation module;

[0079] The speaker information corresponding to the audio file to be processed is determined according to the speaker probability information respectively corresponding to the multiple speech feature segments, wherein the speaker probability information includes at least one speaker identifier and the probability corresponding to each speaker identifier.

[0080] Optionally, the plurality of speech feature segments are processed by a target module included in the preset speaker segmentation model to obtain speaker probability information corresponding to each speech feature segment, including:

[0081] The encoder module processes the plurality of speech feature segments respectively to obtain a latent vector feature segment corresponding to each speech feature segment;

[0082] The multiple latent vector feature segments corresponding to the multiple speech feature segments are processed by the decoder module and the nonlinear activation module to obtain speaker probability information corresponding to each speech feature segment.

[0083] Optionally, the decoder module and the nonlinear activation module are used to process a plurality of latent vector feature segments corresponding to the plurality of speech feature segments to obtain a speaker probability corresponding to each speech feature segment, including:

[0084] Processing a first latent vector feature segment among the multiple latent vector feature segments through the nonlinear activation module to obtain speaker probability information of a speech feature segment corresponding to the first latent vector feature segment;

[0085] For each remaining latent vector feature segment except the first latent vector feature segment among the multiple latent vector feature segments, the remaining latent vector feature segment and the previous latent vector feature segment corresponding to the remaining latent vector feature segment are input into the decoder module together, and the obtained output result is processed by the nonlinear activation module to obtain the speaker probability information of the speech feature segment corresponding to the remaining latent vector feature segment.

[0086] Optionally, determining the speaker information corresponding to the audio file to be processed according to the speaker probability information respectively corresponding to the plurality of speech feature segments includes:

[0087] For each of the plurality of speech feature segments, confirming a speaker identifier whose probability corresponding to the speaker identifier in the speaker probability information corresponding to the speech feature segment is greater than a preset threshold as a valid speaker identifier;

[0088] Speaker information corresponding to the audio file to be processed is determined according to the valid speaker identifiers respectively corresponding to the multiple voice feature segments.

[0089] Optionally, the loss function used in training the preset speaker segmentation model is a permutation invariance training PIT Loss function.

[0090] Optionally, the attention mechanism corresponding to the decoder module is a Mutli Head Attention multi-head attention mechanism.

[0091] The audio processing device provided by the present application obtains corresponding speaker information by inputting the received audio file to be processed into a preset speaker segmentation model obtained by training a neural network model structure based on a self-attention mechanism. Compared with a speaker segmentation processing method including multiple separation steps, the preset speaker segmentation model uses the original speech signal in the audio file as the input of the model, which can reduce the complexity of the model's online deployment. Since the preset speaker segmentation model is not modeled through sequences, but is modeled by using an encoder within a feature segment and a decoder between feature segments, the preset speaker segmentation model can perform speaker segmentation processing on an audio file of a longer duration at one time, and can also improve the accuracy of the speaker segmentation results in multiple speaker scenarios.

[0092] An embodiment of the present application also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned audio processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0093] For example, Figure 4 A schematic diagram of the physical structure of an electronic device is shown.

[0094] like Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430, and the processor 410 is used to perform the following steps: receiving an audio file to be processed, wherein the audio file to be processed includes at least one speaker; inputting the audio file to be processed into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, wherein the speaker information includes at least one speaker identifier and at least one duration corresponding to each speaker identifier; wherein the preset speaker segmentation model is obtained by training a neural network model structure based on a self-attention mechanism.

[0095] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0096] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned audio processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0097] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0099] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0100] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0101] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0102] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0105] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0106] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An audio processing method, characterized in that: The method comprises: receiving an audio file to be processed, wherein the audio file to be processed includes at least one speaker; Inputting the audio file to be processed into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, wherein the speaker information includes at least one speaker identifier and at least one duration corresponding to each speaker identifier; The preset speaker segmentation model is obtained by training a neural network model structure based on a self-attention mechanism.

2. The audio processing method according to claim 1, characterized in that: Inputting the audio file to be processed into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, including: The audio file to be processed is sampled by a convolutional neural network module included in the preset speaker segmentation model to obtain a plurality of speech feature vectors; Segmenting the plurality of speech feature vectors by using the preset speaker segmentation model to obtain a plurality of speech feature segments; Processing the plurality of speech feature segments through a target module included in the preset speaker segmentation model to obtain speaker probability information corresponding to each speech feature segment, wherein the target module at least includes: an encoder module, a decoder module and a nonlinear activation module; The speaker information corresponding to the audio file to be processed is determined according to the speaker probability information respectively corresponding to the multiple speech feature segments, wherein the speaker probability information includes at least one speaker identifier and the probability corresponding to each speaker identifier.

3. The audio processing method according to claim 2, characterized in that: The plurality of speech feature segments are processed by a target module included in the preset speaker segmentation model to obtain speaker probability information corresponding to each speech feature segment, including: The encoder module processes the plurality of speech feature segments respectively to obtain a latent vector feature segment corresponding to each speech feature segment; The multiple latent vector feature segments corresponding to the multiple speech feature segments are processed by the decoder module and the nonlinear activation module to obtain speaker probability information corresponding to each speech feature segment.

4. The audio processing method according to claim 3, characterized in that: Processing the multiple latent vector feature segments corresponding to the multiple speech feature segments through the decoder module and the nonlinear activation module to obtain a speaker probability corresponding to each speech feature segment includes: Processing a first latent vector feature segment among the multiple latent vector feature segments through the nonlinear activation module to obtain speaker probability information of a speech feature segment corresponding to the first latent vector feature segment; For each remaining latent vector feature segment except the first latent vector feature segment among the multiple latent vector feature segments, the remaining latent vector feature segment and the previous latent vector feature segment corresponding to the remaining latent vector feature segment are input into the decoder module together, and the obtained output result is processed by the nonlinear activation module to obtain the speaker probability information of the speech feature segment corresponding to the remaining latent vector feature segment.

5. The audio processing method according to claim 2, characterized in that: Determining speaker information corresponding to the audio file to be processed according to speaker probability information respectively corresponding to the plurality of speech feature segments includes: For each of the plurality of speech feature segments, confirming a speaker identifier whose probability corresponding to the speaker identifier in the speaker probability information corresponding to the speech feature segment is greater than a preset threshold as a valid speaker identifier; Speaker information corresponding to the audio file to be processed is determined according to the valid speaker identifiers respectively corresponding to the multiple voice feature segments.

6. The audio processing method according to claim 2, characterized in that: The loss function used in training the preset speaker segmentation model is a permutation invariance training loss function.

7. The audio processing method according to claim 2, characterized in that: The attention mechanism corresponding to the decoder module is a multi-head attention mechanism.

8. An audio processing device, characterized in that: include: A receiving module, configured to receive an audio file to be processed, wherein the audio file to be processed includes at least one speaker; A processing module, configured to input the audio file to be processed into a preset speaker segmentation model to obtain speaker information corresponding to the audio file to be processed, wherein the speaker information includes at least one speaker identifier and at least one duration period corresponding to each of the speakers; The preset speaker segmentation model is obtained by training a neural network model structure based on a self-attention mechanism.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the audio processing method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the audio processing method according to any one of claims 1 to 7 are implemented.

Images