Multi-voice audio processing method, device, equipment and storage medium

Abstract

The present disclosure provides a multi-voice audio processing method, apparatus, device and storage medium, including: obtaining recorded audio data; identifying audio data with recognizable text in the recorded audio data, and establishing an audio timing table based on the audio data with recognizable text and the corresponding time period; extracting the voiceprint features of the speaker in the recorded audio data, matching them with the voiceprint features in a voiceprint feature library, and establishing a voiceprint timing table for the audio data generating the matching results and the corresponding time period; performing intersection processing on the time period of the audio timing table and the time period of the voiceprint timing table, and establishing an intersection timing table based on the intersection result and the corresponding audio data; receiving and responding to a play instruction, and playing the corresponding audio data according to the intersection timing table.

Description

Technical Field

[0001] The present disclosure relates to the field of speech technology, and in particular to a multi-voice audio processing method, apparatus, device, and storage medium. Background Art

[0002] During important online or offline meetings, recordings are often taken using voice recorders, recording software, or the recording function within conference software. The recordings are then played back, and manual effort is required to identify the current speaker and the content, and then edit and categorize them. This method not only suffers from poor recording accuracy, long recording times, and incomplete content, but also requires repeated playback for confirmation. Summary of the Invention

[0003] The present disclosure provides a multi-voice audio processing method, apparatus, device, and storage medium to at least solve the above technical problems existing in the prior art.

[0004] According to a first aspect of the present disclosure, a multi-voice audio processing method is provided, the method comprising: acquiring recorded audio data; identifying audio data with recognizable text in the recorded audio data, and establishing an audio timing table based on the audio data with recognizable text and the corresponding time period; extracting the voiceprint features of the speaker in the recorded audio data, matching them with the voiceprint features in a voiceprint feature library, and establishing a voiceprint timing table for the audio data generating the matching results and the corresponding time period; performing intersection processing on the time period of the audio timing table and the time period of the voiceprint timing table, and establishing an intersection timing table based on the intersection result and the corresponding audio data; receiving and responding to a play instruction, and playing the corresponding audio data according to the intersection timing table.

[0005] In one possible implementation, before obtaining the recorded audio data, the method includes extracting the voiceprint features of the registered user, and establishing the voiceprint feature library according to the voiceprint features of the registered user and the identification information of the registered user.

[0006] In one possible implementation manner, the identifying of audio data with recognizable text in the recorded audio data includes: extracting features from the recorded audio data; decoding the extracted features through a decoder to output the text in the recorded audio data; and outputting the audio data corresponding to the text to obtain the audio data with recognizable text.

[0007] In one possible implementation manner, the extracting of the voiceprint features of the speaker from the recorded audio data, matching the voiceprint features in a voiceprint feature library, and establishing a voiceprint time series table for the audio data generating the matching results and the corresponding time period includes: extracting the voiceprint features of each speaker from the recorded audio data, calculating a matching score between the voiceprint features of the speaker and the voiceprint features in the voiceprint feature library; determining identification information of the registered user corresponding to the voiceprint features in the voiceprint feature library whose matching score is greater than a preset threshold; marking the audio data corresponding to the voiceprint features of the speaker whose matching score is greater than the preset threshold with the corresponding identification information, and establishing a voiceprint time series table for the audio data and the corresponding time period.

[0008] In one possible implementation, the time period of the audio timing table is intersected with the time period of the voiceprint timing table, and an intersection timing table is established based on the intersection result and the corresponding audio data, including: obtaining the time period where the intersection occurs, and recording the time period and the corresponding audio data in the intersection timing table; obtaining the time period where no intersection occurs, marking the audio data corresponding to the time period as an unknown user, and recording the time period and the corresponding audio data in the intersection timing table.

[0009] In one possible implementation, the receiving and responding to the play instruction and playing the corresponding audio data according to the intersection timing table includes: the play instruction includes a selection instruction and a play audio instruction; in response to the selection instruction, obtaining at least one audio data of at least one speaker from the intersection timing table; and in response to the play audio instruction, playing the selected audio data in chronological order.

[0010] According to a second aspect of the present disclosure, a multi-voice audio processing device is provided, the device comprising: an acquisition module for acquiring recorded audio data; an identification module for identifying audio data with recognizable text in the recorded audio data, and establishing an audio timing table based on the audio data with recognizable text and the corresponding time period; an extraction module for extracting the voiceprint features of the speaker in the recorded audio data, matching them with the voiceprint features in a voiceprint feature library, and establishing a voiceprint timing table for the audio data generating the matching results and the corresponding time period; an intersection module for performing intersection processing on the time period of the audio timing table and the time period of the voiceprint timing table, and establishing an intersection timing table based on the intersection result and the corresponding audio data; and a playback module for receiving and responding to a playback instruction, and playing the corresponding audio data according to the intersection timing table.

[0011] In one embodiment, the device further includes: an establishment module for extracting the voiceprint features of the registered user before obtaining the recorded audio data, and establishing a voiceprint feature library based on the voiceprint features of the registered user and the identification information of the registered user.

[0012] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the multi-voice audio processing method described in the present disclosure.

[0016] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the multi-voice audio processing method described in the present disclosure.

[0017] The multi-voice audio processing method, apparatus, device, and storage medium disclosed herein establish an audio time series table by identifying audio data and time periods with identifiable text in recorded audio data. It also extracts voiceprint features from the recorded audio data and matches them with a voiceprint feature library. A voiceprint time series table is established for the audio data and time periods with matching results. The time periods in the audio time series table and the voiceprint time series table are then intersected to produce an intersection time series table. This enables automated processing of recorded audio data, and the audio data in the resulting intersection time series table accurately categorizes each speaker and the corresponding audio. Furthermore, the extraction of identifiable text audio data and voiceprint features is performed simultaneously, requiring only a single playback and recognition of the recorded audio to produce clear and accurate audio data. After generating the intersection time series table, the system can also receive and respond to instructions generated by user operations, obtaining and playing corresponding audio data from the intersection time series table, making it convenient for users to select the audio data they need to listen to based on their actual needs.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0020] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

[0021] Figure 1 A schematic diagram illustrating an implementation flow of a multi-voice audio processing method according to an embodiment of the present disclosure is shown;

[0022] Figure 2 A schematic diagram illustrating an implementation flow of a multi-voice audio processing method according to another embodiment of the present disclosure is shown;

[0023] Figure 3 A schematic diagram of the process of establishing a voiceprint feature library according to an embodiment of the present disclosure is shown;

[0024] Figure 4 A schematic diagram of a process for identifying audio data with recognizable text in recorded audio data according to an embodiment of the present disclosure is shown;

[0025] Figure 5 A schematic diagram of the process of establishing a voiceprint timing table according to an embodiment of the present disclosure is shown;

[0026] Figure 6 A schematic diagram of a process for establishing an intersection timing table according to an embodiment of the present disclosure is shown;

[0027] Figure 7 A schematic diagram of the process of playing selected audio data according to an embodiment of the present disclosure is shown;

[0028] Figure 8a A schematic diagram of an interactive interface for single-voice audio playback according to an embodiment of the present disclosure is shown;

[0029] Figure 8b A schematic diagram of an interactive interface for single-voice audio playback according to another embodiment of the present disclosure is shown;

[0030] Figure 9a A schematic diagram of an interactive interface for multi-voice audio playback according to an embodiment of the present disclosure is shown;

[0031] Figure 9b A schematic diagram of an interactive interface for multi-voice audio playback according to another embodiment of the present disclosure is shown;

[0032] Figure 10 A schematic diagram of a multi-voice audio processing device according to an embodiment of the present disclosure is shown;

[0033] Figure 11 A schematic diagram of the structure of an electronic device for implementing a multi-voice audio processing method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0034] To make the purposes, features, and advantages of the present disclosure more apparent and understandable, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative work shall fall within the scope of protection of the present disclosure.

[0035] Voiceprint recognition technology can generally be divided into two aspects according to different target requirements, including one-to-many speaker identification (Speaker Identification) and one-to-one speaker verification (Speaker Verification). Speaker identification involves searching for the voiceprint features in the voice in a voiceprint feature library to determine whether they are found in the voiceprint feature library.

[0036] Depending on whether specific speech content needs to be identified, voiceprint recognition technology can be divided into text-dependent recognition (text-dependent) and text-independent recognition (text-independent). Text-independent recognition does not limit the specific speech content, but only recognizes the speaker's voiceprint characteristics, and has a wide range of applications.

[0037] In one example, the present disclosure provides a multi-voice audio processing method based on the above-mentioned one-to-multi-person speaker recognition and text-incoherent recognition voiceprint recognition technology, such as Figure 1 As shown, the method includes:

[0038] Step 101: Obtain recorded audio data.

[0039] In this example, the recording can be performed using an audio acquisition device, such as a voice recorder, recorder, microphone, etc.; recording can be performed using recording software in a mobile terminal; or recording can be performed using the recording function in conference software. This disclosure does not limit the method for obtaining the recorded audio data.

[0040] Step 102: Identify audio data with recognizable text in the recorded audio data, and create an audio time sequence table based on the audio data with recognizable text and the corresponding time period.

[0041] In an actual recorded audio data, there are silent segments where the speaker pauses during the speech, or meaningless audio segments such as environmental noise and background music, which cause problems for speech recognition and segmentation. Therefore, it is necessary to eliminate silent or meaningless audio data in the recorded audio data.

[0042] The start time and end time of each audio data with recognizable text are obtained, and an audio time sequence table is established in chronological order according to the start and end time periods and the corresponding audio data.

[0043] Step 103: extract the voiceprint features of the speaker from the recorded audio data, match them with the voiceprint features in the voiceprint feature library, and create a voiceprint time series table for the audio data that generates the matching results and the corresponding time period.

[0044] A voiceprint is a sound wave spectrum that carries speech information, as detected by electroacoustic instruments. Modern scientific research shows that voiceprints are not only specific but also relatively stable. After adulthood, a person's voice remains relatively stable for a long time. Experiments have shown that regardless of whether a speaker deliberately imitates another person's voice and tone or whispers, even if the imitation is perfect, the voiceprint remains unique. Based on these two characteristics of a voiceprint, a pre-established voiceprint feature library can be used to match and identify the speaker's voiceprint features in recorded audio data.

[0045] The start and end times of the audio data with matching results are obtained, and a voiceprint time series table is established in chronological order according to the start and end time periods and the corresponding audio data.

[0046] Since both the audio timing table and the voiceprint timing table use the recorded audio data as the source audio, the extraction of audio data with recognizable text and the speaker's voiceprint features are carried out separately and simultaneously. Therefore, it is only necessary to recognize the recorded audio data once to simultaneously construct the audio timing table and the voiceprint timing table, saving the time of recognizing the recorded audio data.

[0047] Step 104: Intersect the time periods of the audio time series table with the time periods of the voiceprint time series table, and establish an intersection time series table based on the intersection result and the corresponding audio data.

[0048] Step 105: Receive and respond to the play instruction, and play the corresponding audio data according to the intersection timing table.

[0049] The present disclosure provides a method for processing multi-voice audio. The method establishes an audio timing table by identifying the audio data and time periods of recognizable text in the recorded audio data, extracts the voiceprint features in the recorded audio data and matches them with the voiceprint feature library, establishes a voiceprint timing table for the audio data and time periods with matching results, and performs intersection processing on the time periods of the audio timing table and the voiceprint timing table to obtain an intersection timing table. This achieves automated processing of the recorded audio data, and the audio data in the obtained intersection timing table accurately classifies each speaker and the corresponding audio. In addition, the extraction of the audio data of recognizable text and the voiceprint features is performed simultaneously, and clear and accurate audio data can be obtained by playing and identifying the recorded audio only once. After generating the intersection timing table, the method can also receive and respond to instructions generated by user operations, obtain and play corresponding audio data from the intersection timing table, and facilitate users to select the audio data they need to listen to according to actual needs.

[0050] In one example, in step 101, before obtaining the recorded audio data, Figure 2 As shown, it includes: Step 201: extracting the voiceprint features of the registered user, and establishing a voiceprint feature library based on the voiceprint features of the registered user and the identification information of the registered user. The process of establishing the voiceprint feature library is as follows: Figure 3 The steps shown are as follows:

[0051] Step 301: Acquire several audios provided by a registered user.

[0052] The audio provided by the registered user can be audio with specified text or audio with custom text.

[0053] In this example, all registered users can record an audio clip containing the same text content. In this disclosure, there are no restrictions on the content of the specified text. Furthermore, the audio clips provided by registered users can also contain user-defined text, with the content and length of the custom text being determined by the registered user. Of course, the more text content a registered user records, the more accurate the voiceprint recognition will be.

[0054] Step 302: Extract features from each audio file provided by the registered user.

[0055] In the field of voiceprint recognition, the extracted features can be acoustic features such as Power-Normalized Cepstral Coefficients (PNCC), Mel-Freguency Cepstral Coefficients (MFCC) or Perceptual Linear Prediction Cepstral Coefficients (PLP).

[0056] Specifically, taking MFCC coefficients as an example, the basic process of feature extraction is as follows: the audio signal preprocessed by Voice Activity Detection (VAD) is pre-emphasized through a high-pass filter; the pre-emphasized audio signal is then framed and each frame is windowed to reduce spectral leakage of the audio signal; the audio signal is then discrete Fourier transformed to obtain a frequency domain signal, and filtered through a Mel-scale filter bank to obtain a Mel spectrum, and finally the Mel spectrum is subjected to cepstrum analysis to obtain the MFCC coefficients.

[0057] Step 303: Modeling is performed based on the extracted features to obtain voiceprint features.

[0058] The features extracted from each audio are separately modeled for voiceprints. This means the extracted features are fed into a voiceprint encoder, which then outputs a voiceprint embedding code. The voiceprint embedding codes are aggregated to form a speaker model, i.e., the voiceprint features of the registered user.

[0059] Step 304: Create a voiceprint feature database based on the voiceprint features of the registered user and the identification information of the registered user.

[0060] Among them, the identification information can be a nickname, ID number, registration account, work number, etc., which can indicate the identity of the registered user.

[0061] In one example, the implementation process of step 102, identifying audio data with recognizable text in the recorded audio data, is as follows: Figure 4 The specific steps are as follows:

[0062] Step 401: Extract features from recorded audio data.

[0063] First, the recorded audio data is pre-processed with voice activity detection, and the audio data is divided into frames. Each frame is windowed to eliminate silent segments in the recorded audio data.

[0064] Then, feature extraction is performed on the audio data with silence removed. In the field of continuous speech recognition technology, in addition to the aforementioned MFCC coefficients and PLP coefficients, other features that can be extracted include linear prediction coefficients (LPC), cepstrum coefficients (CEP), and other acoustic features.

[0065] Step 402: Decode the extracted features through a decoder to output the text in the recorded audio data.

[0066] First, the decoder includes a dictionary, an acoustic model, and a language model. Decoding is the process of converting the acoustic features in the audio data into text using the acoustic model, dictionary, and language model.

[0067] An acoustic model (AM) is a model that classifies the acoustic features of audio data into corresponding phonemes. Commonly used acoustic models include the Hidden Markov-Gaussian Mixture Model (HMM) and the Hidden Markov-Deep Neural Network (HMN) model.

[0068] The dictionary is a mapping between phonemes and words and is used to connect the acoustic model and the language model.

[0069] A language model (LM) identifies the corresponding word or phoneme into text based on the probability of the association between individual words or characters. Common language models include the N-gram language model (N-Gram LM), the Big-gram language model (Big-Gram LM), and the Tri-gram language model (Tri-Gram LM).

[0070] Step 403 ： Output the audio data corresponding to the text to obtain audio data with recognizable text.

[0071] The ability to output text indicates that the audio data contains recognizable text and is audio data generated by the speaker. Through the above steps 401-403, the silent and meaningless audio data in the recorded audio data are deleted, and the remaining audio segments are all audio data with recognizable text.

[0072] In one example, the above step 103 extracts the voiceprint features of the speaker from the recorded audio data, matches them with the voiceprint features in the voiceprint feature library, and creates a voiceprint time series table for the audio data that generates the matching results and the corresponding time period, as shown in FIG. Figure 5 As shown, including:

[0073] Step 501: extract the voiceprint features of each speaker in the recorded audio data, and calculate the matching score between the speaker's voiceprint features and the voiceprint features in the voiceprint feature library.

[0074] The process of extracting the speaker's voiceprint features is consistent with the aforementioned process of step 302 to step 303, and will not be repeated here.

[0075] It should be noted that the voiceprint embedding code will be obtained during the voiceprint feature extraction process. The voiceprint embedding code of the speaker's voiceprint feature and the voiceprint embedding code of the voiceprint feature in the voiceprint feature library are used to calculate the matching score through cosine similarity, Euclidean distance matching or matching methods based on machine learning models.

[0076] Step 502: Determine the identification information of the registered user corresponding to the voiceprint feature in the voiceprint feature library whose matching score is greater than a preset threshold.

[0077] If the matching score between a certain voiceprint feature in the voiceprint feature library and the voiceprint feature of the speaker's audio data is greater than a preset threshold, the two are determined to match; if the matching scores between at least two voiceprint features in the voiceprint feature library and the voiceprint features of the speaker's audio data are greater than a preset threshold, the voiceprint feature with the highest matching score is selected as the matching voiceprint feature.

[0078] Step 503: The audio data corresponding to the voiceprint features of the speaker whose matching score is greater than a preset threshold is marked with corresponding identification information, and a voiceprint time series table is established for the audio data and the corresponding time period.

[0079] According to the above matching results, the audio data corresponding to the speaker's voiceprint features is marked with the identification information matched in the voiceprint feature library, the start time and end time of the marked audio data are obtained, and a voiceprint time series table is established in chronological order according to the beginning and end time periods and the corresponding identification information.

[0080] In complex conference scenarios, there will be multiple speakers throughout the entire conference process, and there will be various emergencies during the speaking process. It is possible that the speaker will be suddenly interrupted and refuted by another speaker, or other speakers will interrupt in an intervening manner, or there will be multiple people speaking at the same time. At this time, it is necessary to perform voice separation on the conference voice data, and separate the conference voice data into speaker voice data corresponding to at least two conference speakers, that is, the voice data of each speaker are independent of each other and do not interfere with each other. Preferably, based on the open source speech recognition framework (Auto Speech Recognition Tool SpeechRecognition), the overlapping audio data is separated, and the separated audio data is sorted in the order of the start time or end time of each speaker.

[0081] In one example, the above step 104 is to perform intersection processing on the time period of the audio time series table and the time period of the voiceprint time series table, and to establish the implementation process of the intersection time series table according to the intersection result and the corresponding audio data, as shown in FIG. Figure 6 As shown, including:

[0082] Step 601: Obtain the time period where the intersection occurs, and record the time period and the corresponding audio data in an intersection time sequence table.

[0083] If an intersection of time periods occurs, it means that the voiceprint features corresponding to the audio data corresponding to this period are recorded in the voiceprint feature library, and the registered user can be matched from the voiceprint feature library.

[0084] Step 602: Obtain a time period without an intersection, mark the audio data corresponding to the time period as an unknown user, and record the time period and the corresponding audio data in an intersection time series table.

[0085] If there is no intersection in the time period, it means that the voiceprint features corresponding to the audio data during this period are not recorded in the voiceprint feature library, and therefore cannot be matched to a registered user from the voiceprint feature library. Therefore, the voiceprint features corresponding to this audio data are identified as an unregistered user, and a new identification information is assigned to this audio data. The non-intersecting time period and the marked audio data are also recorded in the intersection time series table.

[0086] Therefore, by intersecting the time periods in the time series table and the identification information of the speaker, the corresponding audio data and the speaker can be determined.

[0087] In an example, the process of establishing the audio timing table, the voiceprint timing table, and the intersection timing table is described by way of the following examples.

[0088] For example, the audio data of a conference recording is deleting silent or meaningless audio through the above steps 401-403, outputting audio data with recognizable text, and establishing an audio timing table as shown in Table 1 below for each audio data segment and the corresponding time period in chronological order.

[0089] Table 1

[0090] Audio Sequence Time period First audio (00'05"，02'03") Second audio clip (02'06"，06'17") The third audio (06'18"，06'44") Fourth audio clip (06'45"，07'28")

[0091] Among them, "first audio segment" and "second audio segment" are only used to indicate the time sequence of audio data and have no special meaning.

[0092] While establishing the audio time series table, the voiceprint features of the speaker in the above-mentioned recorded audio data are extracted, and the matching score of the extracted voiceprint features and the voiceprint feature library is calculated. According to the matching results, the audio data is marked with the identification information of the matching voiceprint features in the voiceprint feature library, and a voiceprint time series table as shown in Table 2 below is established with the corresponding time period.

[0093] Table 2

[0094] Registered User Time period Manager Zhang (00'05"，02'03") Xiao Li (02'06"，03'01") Manager Zhang (02'59"，06'17") Xiao Chen (06'45"，07'28")

[0095] As shown in Table 2, the audio data time periods of the second speeches of "Xiao Li" and "Manager Zhang" overlap. Therefore, the overlapping audio parts can be separated through the open source speech recognition framework, and sorted according to the order in which the two started speaking, and recorded in the voiceprint time series table.

[0096] Table 2 shows that the audio data from the period after Manager Zhang's second speech and before Xiao Chen's speech is not recorded in the voiceprint time series table, indicating that the voiceprint feature database does not record the voiceprint features and identification information for this audio data. Therefore, by intersecting the time periods of the audio time series table and the voiceprint time series table, we obtain the intersection time series table shown in Table 3 below.

[0097] Table 3

[0098] Speaker Time period Manager Zhang (00'05"，02'03") Xiao Li (02'06"，03'01") Manager Zhang (02'59"，06'17") Unknown user (06'18"，06'44") Xiao Chen (06'45"，07'28")

[0099] Among them, "unknown user" is a non-registered user that is not recorded in the voiceprint feature library, and (06'18", 06'44") is a time period with no intersection. The audio data corresponding to the "unknown user" can be obtained through the "unknown user" and (06'18", 06'44") in the intersection time series table.

[0100] In one example, the above step 105 receives and responds to the play instruction, and plays the corresponding audio data according to the intersection time sequence table, such as Figure 7 As shown, including:

[0101] The play instruction includes a selection instruction and a play audio instruction.

[0102] Step 701: In response to a selection instruction, obtain at least one audio data of at least one speaker from the intersection time sequence table.

[0103] In this example, after all recorded audio data is processed and the intersection time sequence table is created, an interactive interface is generated based on the intersection time sequence table. In this interactive interface, speakers are sorted from top to bottom by the start time of their first audio data. Then, each audio data segment belonging to a speaker is organized into tracks in chronological order and displayed in the interactive interface.

[0104] The interactive interface is used to receive a user operation to generate a corresponding selection instruction, and to obtain corresponding audio data from the intersection time sequence table in response to the selection instruction.

[0105] Step 702: In response to the play audio instruction, the selected audio data is played in chronological order.

[0106] In an application scenario, when playing single-person audio, one or more audio data in a specified speaker's audio track are selected.

[0107] If you select one or part of the audio data in the audio track of a specified user, select the audio data to be played one by one and play them in chronological order. Figure 8a As shown, a section of audio data of user A is selected and played.

[0108] If all the audio data in the audio track of a specified speaker is played, for example, selecting the user's name or avatar in the interactive interface means that all the audio data of the user is selected and played in chronological order. Figure 8b As shown, select the avatar of user B and play all audio data of user B in chronological order.

[0109] In one application scenario, when playing multi-voice audio, multiple audio data of multiple speaker audio tracks are received and responded to.

[0110] If all audio data of multiple speakers' tracks are played, for example, if the names or avatars of multiple speakers are selected in the interactive interface, all audio data of the specified speakers' tracks will be selected and played in chronological order according to the start time of the audio data. Figure 9a As shown, select the avatars of user A and user B, and play all the audio data of user A and user B in chronological order.

[0111] If you want to play part of the audio data of multiple speakers, select the audio data to be played one by one and play them in chronological order. Figure 9b As shown, partial audio data of users A, B, C and D are selected one by one, and the selected audio data are played in chronological order.

[0112] In order to implement the above multi-voice audio processing method, the present disclosure also provides a multi-voice audio processing device, such as Figure 10 As shown, the device includes:

[0113] Acquisition module 1001, used to obtain recorded audio data;

[0114] Identification module 1002, for identifying audio data with recognizable text in the recorded audio data, and establishing an audio time sequence table based on the audio data with recognizable text and the corresponding time period;

[0115] Extraction module 1003, used to extract the voiceprint features of the speaker from the recorded audio data, match them with the voiceprint features in the voiceprint feature library, and create a voiceprint time series table based on the audio data that generates the matching results and the corresponding time period;

[0116] The intersection module 1004 is used to perform intersection processing on the time period of the audio time series table and the time period of the voiceprint time series table, and to establish an intersection time series table according to the intersection result and the corresponding audio data;

[0117] The playing module 1005 is configured to receive and respond to a playing instruction and play corresponding audio data according to the intersection timing table.

[0118] In one example, the apparatus further includes: an establishing module 1006 for extracting the voiceprint features of the registered user before acquiring the recorded audio data, and establishing a voiceprint feature library according to the voiceprint features of the registered user and the identification information of the registered user.

[0119] In one example, the identification module 1002 includes:

[0120] The first extraction submodule 10021 is used to extract features from the recorded audio data;

[0121] A decoding module 10022 is used to decode the extracted features through a decoder and output the text in the recorded audio data;

[0122] The output module 10023 is used to output the audio data corresponding to the text to obtain audio data with recognizable text.

[0123] In one example, the extraction module 1003 includes:

[0124] The second extraction submodule 10031 is used to extract the voiceprint features of each speaker in the recorded audio data and calculate the matching score between the speaker's voiceprint features and the voiceprint features in the voiceprint feature library;

[0125] Determination module 10032, configured to determine the identification information of the registered user corresponding to the voiceprint feature in the voiceprint feature library whose matching score is greater than a preset threshold;

[0126] The marking module 10033 is used to mark the audio data corresponding to the voiceprint features of the speaker whose matching score is greater than a preset threshold with corresponding identification information, and to establish a voiceprint time series table for the audio data and the corresponding time period.

[0127] In one example, the intersection module 1004 includes:

[0128] The recording module 10041 is used to obtain the time period where the intersection occurs, and record the time period and the corresponding audio data in the intersection time sequence table;

[0129] The recording module 10041 is further configured to obtain a time period in which no intersection occurs, mark the audio data corresponding to the time period as an unknown user, and record the time period and the corresponding audio data in an intersection time series table.

[0130] In one example, the playing module 1005 includes:

[0131] Play commands include selection commands and audio play commands;

[0132] The acquisition submodule 10051, in response to the selection instruction, acquires at least one audio data of at least one speaker from the intersection time sequence table;

[0133] The playing submodule 10052 responds to the play audio instruction and plays the selected audio data in chronological order.

[0134] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0135] Figure 11A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0136] like Figure 11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0137] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0138] The computing unit 1101 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the multi-voice audio processing method. For example, in some embodiments, the multi-voice audio processing method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the multi-voice audio processing method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the multi-voice audio processing method in any other appropriate manner (for example, by means of firmware).

[0139] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0140] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0141] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0142] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0143] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0144] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0145] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0146] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0147] The above description is merely a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for processing multi-voice audio, characterized in that: The method comprises: Get recorded audio data; Identifying audio data with recognizable text in the recorded audio data, and establishing an audio timing table according to the audio data with recognizable text and corresponding time periods; Extracting the voiceprint features of the speaker from the recorded audio data, matching them with the voiceprint features in the voiceprint feature library, and establishing a voiceprint time series table for the audio data that generates the matching results and the corresponding time period; Intersecting the time periods of the audio time series table with the time periods of the voiceprint time series table, and establishing an intersection time series table based on the intersection results and the corresponding audio data; receiving and responding to a play instruction, and playing corresponding audio data according to the intersection timing table; The step of performing intersection processing on the time period of the audio time sequence table and the time period of the voiceprint time sequence table, and establishing an intersection time sequence table according to the intersection result and the corresponding audio data, includes: Obtain the time period where the intersection occurs, and record the time period and the corresponding audio data in the intersection time series table; A time period without intersection is obtained, audio data corresponding to the time period is marked as an unknown user, and the time period and the corresponding audio data are recorded in the intersection time series table.

2. The method according to claim 1, characterized in that Before obtaining the recorded audio data, the following steps are included: The voiceprint features of the registered user are extracted, and the voiceprint feature library is established according to the voiceprint features of the registered user and the identification information of the registered user.

3. The method according to claim 1, characterized in that The identifying audio data having recognizable text in the recorded audio data includes: Performing feature extraction on the recorded audio data; Decode the extracted features through the decoder and output the text in the recorded audio data; The audio data corresponding to the text is output to obtain the audio data with recognizable text.

4. The method according to claim 2, characterized in that The extracting of the speaker's voiceprint features from the recorded audio data and matching them with the voiceprint features in the voiceprint feature library, and establishing a voiceprint time series table for the audio data generating the matching results and the corresponding time period, includes: Extracting the voiceprint features of each speaker in the recorded audio data, and calculating a matching score between the voiceprint features of the speaker and the voiceprint features in the voiceprint feature library; Determine the identification information of the registered user corresponding to the voiceprint feature in the voiceprint feature library whose matching score is greater than a preset threshold; The audio data corresponding to the voiceprint feature of the speaker whose matching score is greater than a preset threshold is marked with the corresponding identification information, and a voiceprint time series table is established for the audio data and the corresponding time period.

5. The method according to claim 1, wherein The receiving and responding to the play instruction and playing the corresponding audio data according to the intersection timing table includes: The play instructions include a selection instruction and a play audio instruction; In response to the selection instruction, obtaining at least one audio data of at least one speaker from the intersection time sequence table; In response to the play audio instruction, the selected audio data is played in chronological order.

6. A multi-voice audio processing device, characterized in that: The device comprises: Acquisition module, used to obtain recorded audio data; an identification module, configured to identify audio data with recognizable text in the recorded audio data, and establish an audio timing table according to the audio data with recognizable text and the corresponding time period; An extraction module is used to extract the voiceprint features of the speaker in the recorded audio data, match them with the voiceprint features in the voiceprint feature library, and establish a voiceprint time series table for the audio data that generates the matching results and the corresponding time period; An intersection module, configured to perform intersection processing on the time periods of the audio time sequence table and the time periods of the voiceprint time sequence table, and to establish an intersection time sequence table according to the intersection result and the corresponding audio data; A playing module, configured to receive and respond to a playing instruction and play corresponding audio data according to the intersection timing table; The intersection module includes: The recording module is used to obtain the time period where the intersection occurs, and record the time period and the corresponding audio data in the intersection time series table; The recording module is further configured to obtain a time period in which no intersection is generated, mark the audio data corresponding to the time period as an unknown user, and record the time period and the corresponding audio data in an intersection time series table.

7. The device according to claim 6, characterized in that The device further comprises: The establishment module is used to extract the voiceprint features of the registered user before obtaining the recorded audio data, and establish a voiceprint feature library based on the voiceprint features of the registered user and the identification information of the registered user.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

Images