Audio recognition method and system, electronic equipment and storage medium

Abstract

The invention provides an audio recognition method and system, electronic equipment and a storage medium, and the method comprises the steps: inputting the acoustic features of a to-be-recognized audio of a single channel into a conversion point detector in an audio recognition model, and obtaining a timestamp vector corresponding to a speaker conversion point in the to-be-recognized audio; inputting the acoustic features and timestamp vectors corresponding to the speaker conversion points into a fragment detector in an audio recognition model to obtain an audio fragment of a single speaker and overlapped audio fragments of multiple speakers in the audio to be recognized; and determining input information according to the voiceprint feature corresponding to the audio clip of the single speaker and the voiceprint features corresponding to the overlapped audio clips of the plurality of speakers, and inputting the input information to an estimator in the audio recognition model to obtain the number of speakers in the audio to be recognized. According to the invention, the accuracy of speaker number estimation is ensured, the configuration cost is reduced, and the adaptability of audio recognition is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an audio recognition method, system, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence and deep learning, the speaker estimation results obtained through audio recognition have been widely used in many speaker-related tasks such as speech separation and speaker separation. Therefore, how to efficiently and accurately implement audio recognition to obtain speaker estimation results is an important topic that needs to be studied in the industry.

[0003] In the related technology, it is usually necessary to collect multi-channel audio signals to extract amplitude spectrum and phase spectrum based on the multi-channel audio signals, and then use a deep neural network (DNN) model to extract amplitude features and phase features respectively and splice them. Finally, the spliced ​​features are sent to the fully connected layer of the neural network for classification. This method requires the configuration of multiple audio devices for multi-channel audio signal collection in order to realize audio recognition and obtain the speaker estimation result, which leads to high hardware configuration cost and poor adaptability. Summary of the invention

[0004] The present invention provides an audio recognition method, system, electronic device and storage medium, which are used to solve the defects of high hardware configuration cost and poor adaptability in the prior art, and achieve the goal of reducing configuration cost and improving the adaptability of audio recognition while ensuring the accuracy of speaker number estimation.

[0005] The present invention provides an audio recognition method, comprising: Inputting the acoustic features of the audio to be recognized of a single channel into a transition point detector in an audio recognition model to obtain a timestamp vector corresponding to a speaker transition point in the audio to be recognized; Inputting the acoustic features and the timestamp vector corresponding to the speaker switching point into a segment detector in the audio recognition model to obtain an audio segment of a single speaker and overlapping audio segments of multiple speakers in the audio to be recognized; Determining input information according to the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers, and inputting the input information into an estimator in the audio recognition model to obtain the number of speakers in the audio to be recognized; The audio recognition model is obtained by iterative training based on a sample data set, and the sample data set includes acoustic features of sample audio and labels of the number of speakers in the sample audio.

[0006] According to an audio recognition method provided by the present invention, the step of determining input information according to the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers includes: Inputting the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers into the long-term modeling unit in the audio recognition model to obtain the long-term voiceprint features corresponding to the audio to be recognized; the long-term modeling unit is used to concatenate and encode the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers; The input information is obtained according to the long-term voiceprint feature.

[0007] According to an audio recognition method provided by the present invention, obtaining the input information according to the long-term voiceprint feature includes: Calculating the similarity between every two elements in the long-term voiceprint feature to obtain a similarity matrix of the long-term voiceprint feature; The input information is obtained according to the similarity matrix.

[0008] According to an audio recognition method provided by the present invention, the voiceprint feature corresponding to the audio segment of the single speaker is obtained by inputting the audio segment of the single speaker into the first voiceprint encoder in the audio recognition model for voiceprint encoding; The voiceprint features corresponding to the overlapping audio segments of the multiple speakers are obtained by inputting the overlapping audio segments of the multiple speakers into a second voiceprint encoder in the audio recognition model for voiceprint encoding; The first voiceprint encoder and the second voiceprint encoder are both constructed based on a residual network; the last pooling layer of the residual network is a statistical pooling layer.

[0009] According to an audio recognition method provided by the present invention, the audio recognition model is trained based on the following steps: According to the sample data set, iteratively training the first initial voiceprint encoder and the second initial voiceprint encoder in the initial recognition model respectively to obtain the first voiceprint encoder and the second voiceprint encoder; According to the first voiceprint encoder and the second voiceprint encoder, the initial recognition model is updated to obtain a pre-trained recognition model; Randomly adjusting the order of the audio segments of each speaker in the sample audio to obtain a preprocessed sample audio; The pre-trained recognition model is trained in stages according to the acoustic features of the pre-processed sample audio, the acoustic features of the sample audio and the speaker number label to obtain the audio recognition model.

[0010] According to an audio recognition method provided by the present invention, the pre-trained recognition model is trained in stages according to the acoustic features of the pre-processed sample audio, the acoustic features of the sample audio and the speaker number label to obtain the audio recognition model, including: Iteratively training the network model to be trained in the pre-trained recognition model according to the acoustic features of the pre-processed sample audio and the speaker number label to obtain a first recognition model; the network model to be trained is a network model other than the first voiceprint encoder and the second voiceprint encoder; Iteratively training the to-be-trained network model in the first recognition model according to the acoustic features of the sample audio and the speaker number label to obtain a second recognition model; The acoustic features of the preprocessed sample audio, the acoustic features of the sample audio, and the speaker number label are combined to iteratively train the network model to be trained in the second recognition model to obtain the audio recognition model.

[0011] According to an audio recognition method provided by the present invention, the sample audio is obtained based on the following steps: Capture single channel raw audio; The original audio is preprocessed to obtain the sample audio; the preprocessing includes eliminating audio segments of conversation audio whose duration is less than a preset duration and / or eliminating audio segments whose speech duration ratio is less than a preset ratio.

[0012] The present invention also provides an audio recognition system, comprising: A first detection unit, configured to input the acoustic features of the audio to be recognized of a single channel into a transition point detector in the audio recognition model, and obtain a timestamp vector corresponding to a speaker transition point in the audio to be recognized; A second detection unit, configured to input the acoustic features and the timestamp vector corresponding to the speaker switching point into a segment detector in the audio recognition model to obtain an audio segment of a single speaker and overlapping audio segments of multiple speakers in the audio to be recognized; an identification unit, configured to determine input information according to the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers, and input the input information into an estimator in the audio recognition model to obtain the number of speakers in the audio to be identified; The audio recognition model is obtained by iterative training based on a sample data set, and the sample data set includes acoustic features of sample audio and labels of the number of speakers in the sample audio.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned audio recognition methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the audio recognition method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the audio recognition method described above is implemented.

[0016] The audio recognition method, system, electronic device and storage medium provided by the present invention input the acoustic features of the audio to be recognized of a single channel into an iteratively trained audio recognition model. The model first uses a transition point detector to determine the timestamp vector of the speaker's transition point. Then, a segment detector distinguishes the audio segments of a single speaker from the overlapping audio segments of multiple speakers based on the acoustic features and the transition point timestamp vector, and extracts voiceprint features of different durations based on these segments to compress the feature time length. The number of speakers is obtained through an estimator to ensure that the number of speakers in audio with a longer duration can be accurately estimated when there is only single-channel speech modal data. This achieves the goal of reducing configuration costs and improving the adaptability of audio recognition while ensuring the accuracy of speaker number estimation, effectively solving the high cost and low adaptability problems brought about by the previous reliance on multi-channel acquisition methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is one of the flow charts of the audio recognition method provided by the present invention.

[0019] Figure 2 This is the second flow chart of the audio recognition method provided by the present invention.

[0020] Figure 3It is a structural schematic diagram of the audio recognition system provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] With the development of artificial intelligence and deep learning, speaker number estimation technology has been widely used, such as speech separation and speaker separation. Therefore, it is of great significance to accurately estimate the number of speakers in audio.

[0024] The difficulty in estimating the number of speakers in long-duration conversation audio lies in extracting the voiceprint features of a single speaker and distinguishing the voiceprint features between multiple speakers in the long-duration conversation audio. Extracting the voiceprint features of a single speaker is the first step and a very important part. The quality of feature extraction is the key to the success or failure of subsequent speaker differentiation. In addition, long-duration conversation audio involves switching back and forth between multiple speakers and the total duration of the audio is relatively long, such as the total duration of the audio can reach thirty minutes to one hour. Therefore, how to build a model for long-term modeling to distinguish speakers is the key to improving the accuracy of speaker number estimation.

[0025] Most traditional methods for estimating the number of speakers use clustering algorithms, such as hierarchical clustering, spectral clustering, and k-means clustering. This method first detects speaker switching points and segments the speech data to achieve speaker segmentation, and then clusters the segmented speech data to predict the number of speakers. However, this method is difficult to cluster audio of different lengths and different numbers of people using a unified threshold. It is necessary to set corresponding thresholds or cluster center numbers for audio of different lengths and different numbers of people, resulting in the lack of accuracy and efficiency in estimating the number of speakers.

[0026] With the in-depth application of deep learning technology in the field of speech signal processing, systems based on deep neural network (DNN) models and machine learning classifiers such as support vector machine (SVM) have been applied to speaker number estimation, such as extracting speech data using several layers of neural network fully connected layers and then using SVM for classification. However, the simple DNN model in this method cannot extract rich speaker voiceprint representations, and compared with the current neural network model with powerful classification capabilities, SVM has difficulty in accurately classifying speech data features, and thus cannot achieve a sufficiently high accuracy rate.

[0027] In this regard, related technologies use multi-channel audio signals to estimate the number of speakers. This method extracts amplitude spectrum and phase spectrum based on multi-channel audio signals, and then uses a deep neural network (DNN) model to extract amplitude features and phase features respectively and splice them. Finally, the spliced ​​features are sent to the fully connected layer of the neural network for classification. However, this method requires the configuration of multiple audio devices for multi-channel audio signal acquisition in order to achieve audio recognition and obtain the speaker number estimation result. For example, some solutions have as many as 8 channels, but most devices in reality, such as mobile phones, computers, voice recorders, and office notebooks, are not equipped with multi-channel microphones, making it difficult to adapt to actual application scenarios. When additional multi-channel microphones are configured, the hardware configuration cost will be high.

[0028] In summary, it is of great significance to accurately estimate the number of people in long-term conversation audio with only single-channel speech signal data to provide prior information for subsequent speaker-related tasks.

[0029] In this regard, an embodiment of the present application provides an audio recognition method, which can accurately identify and extract the voiceprint features of an audio segment of a single speaker and the voiceprint features of overlapping audio segments of multiple speakers when only single-channel speech modal data is available, and can accurately determine the number of speakers in audio of longer duration based on the voiceprint features of the audio segment of a single speaker and the voiceprint features of overlapping audio segments of multiple speakers, thereby reducing configuration costs while improving the adaptability of audio recognition.

[0030] Figure 1 This is one of the flow charts of the audio recognition method provided by the present invention.

[0031] It should be noted that the execution subject of the method may be an audio recognition system, which may be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a voice recorder, a PDA, an in-vehicle electronic device, a wearable device, a super mobile personal computer, a netbook, or a personal digital assistant, etc., and the non-mobile electronic device may be a server, a network attached storage, a personal computer, etc., which is not specifically limited in the present invention.

[0032] like Figure 1 As shown, the method may be applied to a long-duration audio recognition scenario with a relatively long duration (such as a duration of thirty minutes to one hour), and the method includes: step 110 , step 120 , and step 130 .

[0033] Step 110, inputting the acoustic features of the audio to be recognized of a single channel into a transition point detector in an audio recognition model to obtain a timestamp vector corresponding to a speaker transition point in the audio to be recognized; Step 120, inputting the acoustic features and the timestamp vector corresponding to the speaker switching point into a segment detector in the audio recognition model to obtain an audio segment of a single speaker and overlapping audio segments of multiple speakers in the audio to be recognized; Step 130, determining input information according to the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers, and inputting the input information into an estimator in the audio recognition model to obtain the number of speakers in the audio to be recognized; The audio recognition model is obtained by iterative training based on a sample data set, and the sample data set includes acoustic features of sample audio and labels of the number of speakers in the sample audio.

[0034] It is understandable that the method of realizing speaker number estimation by multimodal feature fusion in the related art mainly relies on video data to obtain face data to estimate the number of speakers. The voice modality is not the main parameter that dominates the number of speakers. However, since video data is difficult to obtain and involves more privacy issues, the accuracy of the estimation result is difficult to guarantee. Moreover, in the existing technology, the number of speakers can only be estimated for short-term audio, that is, the audio duration is mostly within one minute, and it is difficult to make a relatively accurate estimate of the number of speakers for long-term audio (30 minutes to one hour); in addition, the existing technology usually does not perform special processing on overlapping segments, but uses the same processing method as the non-overlapping segments, which is easy to lead to over-estimation of the number of speakers.

[0035] In view of the above problems, the embodiment of the present application only requires a single-channel audio signal, and does not require multi-channel multi-modal data. Therefore, there is no need for specially set multi-microphone equipment, and there is no need to collect video data to reduce data costs and avoid excessive privacy issues. In addition, in the embodiment of the present application, an audio recognition model including a transition point detector, a fragment detector and an estimator is established, and in the audio recognition model, voiceprint features of different durations are extracted by the transition point detector and the fragment detector to compress the feature time length, and the voiceprint features of the main speaker in the overlapping fragments can be extracted to avoid the occurrence of overestimation of the number of speakers, thereby effectively achieving the accuracy of the number of speakers estimated under long-term audio while reducing the configuration cost and improving the adaptability of audio recognition.

[0036] Optionally, before executing step 110, it is necessary to pre-train an audio recognition model. Specifically, an initial recognition model can be constructed first; the initial recognition model here includes but is not limited to an initialization transition point detector, an initialization segment detector, an initialization voiceprint encoder, an initialization long-term modeling unit, and an initialization estimator. Among them, the initialization transition point detector, the initialization segment detector, the initialization voiceprint encoder, the initialization long-term modeling unit, and the initialization estimator The components can be models prepared for their respective functions after parameter initialization, or they can be pre-trained models that have been used to implement their respective functions. This embodiment does not specifically limit this.

[0037] In addition, the acoustic features of the sample audio of a single channel under various scenarios and various audio devices, as well as the speaker number label formed by marking the number of speakers in the sample audio, can also be collected. The acoustic features here are the acoustic features of each frame of the sample audio extracted by fast Fourier transform FFT (fast Fourier Transform) after framing and windowing the sample audio, such as Mel Frequency Cepstrum Coefficient (MFCC) features or perceptual linear prediction (PLP) features, filter bank (Fbank) features, etc., or hidden layer features reflecting local and / or global acoustic information obtained by further performing feature encoding operations on the basis of MFCC, PLP, Fbank and other features. The acoustic features referred to here can be single-scale features or multi-scale fused features, and the embodiments of the present invention do not specifically limit this.

[0038] Subsequently, the acoustic features of the sample audio can be input in sequence into the initial transition point detector, the initial segment detector, the initial voiceprint encoder, the initial long-term modeling unit and the initial estimator in the initial recognition model to obtain the estimated value of the number of speakers in the sample audio; after obtaining the estimated value of the number of speakers in the sample audio, the loss function value can be calculated based on the difference between the estimated value of the number of speakers in the sample audio and the speaker number label, and the initial recognition model can be frozen trained in stages or dynamically adjusted as a whole based on the loss function value to obtain an audio recognition model.

[0039] Figure 2 FIG. 2 is a flow chart of the audio recognition method provided by the present invention; Figure 2 As shown, according to the audio recognition model obtained by the above steps, in practical applications, the transition point detector (also called speaker transition detector) in the audio recognition model can first be used to detect the position of the speaker transition point in the audio to be recognized by applying the acoustic features of the audio to be recognized, so as to obtain the timestamp vector corresponding to the speaker transition point in the audio to be recognized. In this detection process, the detector can be expressed as ,in, is the input feature of the model, specifically the acoustic features of the audio to be recognized, and it can also be used as part of the output of the transition point detector; The timestamp vectors corresponding to the transition points of different speakers output by the transition detector; Transition point for speaker The corresponding timestamp vector, where ; is the total number of speaker transition points.

[0040] The transition point detector here may be constructed based on a neural network model that can detect the position of speaker transition points, such as being constructed by a transformer, etc., and the embodiments of the present application do not specifically limit this.

[0041] After the time stamp vector corresponding to the speaker switching point in the audio to be recognized is obtained by the switching point detector in the audio recognition model, the segment detector (or speaker overlapping segment detector) in the audio recognition model can be used to detect the overlapping segments of speakers in the long-term conversation audio by applying the acoustic features of the audio to be recognized and the time stamp vector corresponding to the speaker switching point, so as to detect the audio segments of a single speaker and the overlapping audio segments of multiple speakers in the audio to be recognized. In this detection process, the detector can be expressed as: ,in, is the input feature of the fragment detector, is the feature dimension; it is the acoustic feature of the audio to be identified output by the transition point detector and the timestamp vector corresponding to the speaker's transition point; It is the output representation of the segment detector, which marks the Fbank features of speaker transitions and overlapping segments, that is, it contains features for marking the labels of audio segments of a single speaker in the audio to be identified and the labels for marking the overlapping audio segments of multiple speakers.

[0042] The segment detector here can be constructed based on a neural network model that can detect overlapping segments of speakers in long-term conversation audio. Its structure can be the same as or different from that of the transition point detector, such as also being constructed by Transformer, etc. The embodiments of the present application do not specifically limit this.

[0043] The single-speaker audio segment here refers to a segment containing only audio of a single speaker's conversation; the overlapping audio segment of multiple speakers here refers to a segment containing audio of multiple speakers speaking at the same time.

[0044] After the audio segment of a single speaker and the overlapping audio segments of multiple speakers in the audio to be recognized are detected by the segment detector in the audio recognition model, voiceprint features can be extracted from the audio segment of the single speaker and the overlapping audio segments of multiple speakers respectively to obtain the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of multiple speakers. Input information of the estimator is obtained based on the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of multiple speakers.

[0045] Here, the voiceprint features corresponding to the audio clip of a single speaker and the voiceprint features corresponding to the overlapping audio clips of multiple speakers are implemented by encoding the voiceprint features through an integrated voiceprint encoder built into the audio recognition model that can encode the voiceprint features of the audio clip of a single speaker and the overlapping audio clips of multiple speakers; or encoding the voiceprint features of the audio clip of a single speaker based on the single-speaker voiceprint encoder built into the audio recognition model that can encode the voiceprint features of the audio clip of a single speaker to obtain the voiceprint features corresponding to the audio clip of the single speaker, and encoding the voiceprint features of the overlapping audio clips of multiple speakers based on the overlapping clip voiceprint encoder built into the audio recognition model that can encode the voiceprint features of the overlapping audio clips of multiple speakers to obtain the voiceprint features corresponding to the overlapping audio clips of multiple speakers. This embodiment does not specifically limit this.

[0046] The step of acquiring the input information here includes fusing the voiceprint features corresponding to the audio segment of a single speaker with the voiceprint features corresponding to the overlapping audio segments of multiple speakers to obtain the input information, or fusing the voiceprint features corresponding to the audio segment of a single speaker with the voiceprint features corresponding to the overlapping audio segments of multiple speakers and then processing them, such as similarity matrix calculation, to obtain the input information, which is not specifically limited in this embodiment.

[0047] After obtaining the input information, the estimator (or person estimator) in the audio recognition model can use the input information to estimate the number of speakers in the audio to be recognized for a long conversation, thereby obtaining the number of speakers in the audio to be recognized. In this estimation process, the estimator can be expressed as: ,in, is the input feature of the estimator, that is, the input information, is the dimension of the input information. is the output of the estimator, which is used to represent the estimated number of speakers in the audio to be recognized.

[0048] The estimator here can be constructed based on a neural network model for estimating the number of speakers in audio of long-term conversations. Its structure can be the same as or different from that of the transition point detector and the fragment detector. For example, it can be constructed by a residual network (Residual Network, ResNet), etc. The embodiments of the present application do not specifically limit this.

[0049] The method provided in this embodiment inputs the acoustic features of the audio to be recognized of a single channel into an iteratively trained audio recognition model. The model first uses a transition point detector to determine the timestamp vector of the speaker's transition point. Then, a segment detector distinguishes the audio segments of a single speaker from the overlapping audio segments of multiple speakers based on the acoustic features and the transition point timestamp vector, and extracts voiceprint features of different durations based on these segments to compress the feature time length. The number of speakers is obtained through an estimator to ensure that the number of speakers in audio with a longer duration can be accurately estimated when only single-channel speech modal data is available. This ensures that the accuracy of the speaker number estimation is guaranteed while reducing the configuration cost and improving the adaptability of audio recognition, effectively solving the high cost and low adaptability problems brought about by the previous reliance on multi-channel acquisition methods.

[0050] In some embodiments, step 130 specifically includes: Inputting the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers into the long-term modeling unit in the audio recognition model to obtain the long-term voiceprint features corresponding to the audio to be recognized; the long-term modeling unit is used to concatenate and encode the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers; The input information is obtained according to the long-term voiceprint feature.

[0051] like Figure 2 As shown, the audio recognition model also includes a long-term modeling unit, which is used to perform long-term modeling of long-term conversation voiceprint features. Its structure may be the same as or different from that of the transition point detector, the fragment detector, and the estimator. For example, the long-term modeling unit may be based on a Transformer encoder, etc. This embodiment does not specifically limit this.

[0052] Optionally, after obtaining the voiceprint features corresponding to the audio segment of a single speaker and the voiceprint features corresponding to the overlapping audio segments of multiple speakers, the voiceprint features corresponding to the audio segment of a single speaker and the voiceprint features corresponding to the overlapping audio segments of multiple speakers can be concatenated and then encoded by the long-term modeling unit in the audio recognition model to achieve long-term modeling, thereby obtaining the long-term voiceprint features corresponding to the audio to be recognized. In the long-term modeling process, the unit can be expressed as: ,in, The result of concatenating the voiceprint features corresponding to the audio clip of a single speaker and the voiceprint features corresponding to the overlapping audio clips of multiple speakers. It is the output representation of the long-term modeling unit, that is, the long-term voiceprint feature corresponding to the audio to be recognized.

[0053] After obtaining the long-term voiceprint features, the long-term voiceprint features can be directly used as input information of the estimator, or the long-term voiceprint features can be further processed, such as similarity matrix calculation, and then used as input information of the estimator. This embodiment does not specifically limit this.

[0054] Exemplarily, in some embodiments, the step of obtaining input information specifically includes: Calculating the similarity between every two elements in the long-term voiceprint feature to obtain a similarity matrix of the long-term voiceprint feature; The input information is obtained according to the similarity matrix.

[0055] Optionally, after the long-term voiceprint features are obtained, the similarities between any two elements in the long-term voiceprint features may be calculated to obtain a similarity matrix of the long-term voiceprint features, and the similarity matrix is ​​used as input information of the estimator.

[0056] The method provided in this embodiment splices and encodes the voiceprint features of a single speaker and overlapping audio segments through the long-term modeling unit in the audio recognition model, thereby extracting long-term voiceprint features to capture the characteristic information in the long-term speech data, and then achieve more accurate estimation of the number of speakers based on the characteristic information in the long-term speech data. The method is flexibly adaptable to long-term modeling units of different structures, effectively improving the accuracy and adaptability of audio recognition.

[0057] In some embodiments, the voiceprint feature corresponding to the audio segment of the single speaker is obtained by inputting the audio segment of the single speaker into a first voiceprint encoder in the audio recognition model for voiceprint encoding; The voiceprint features corresponding to the overlapping audio segments of the multiple speakers are obtained by inputting the overlapping audio segments of the multiple speakers into a second voiceprint encoder in the audio recognition model for voiceprint encoding; The first voiceprint encoder and the second voiceprint encoder are both constructed based on a residual network; the last pooling layer of the residual network is a statistical pooling layer.

[0058] like Figure 2 As shown, the audio recognition model also includes a first voiceprint encoder (or a single-speaker voiceprint encoder) and a second voiceprint encoder (or an overlapping segment voiceprint encoder); wherein the first voiceprint encoder is used to encode the audio segment of a single speaker marked by the segment detector to obtain a voiceprint representation of the audio segment of the single speaker, so as to obtain the voiceprint features corresponding to the audio segment of the single speaker; the second voiceprint encoder is used to encode the overlapping audio segments of multiple speakers marked by the segment detector to obtain the voiceprint features corresponding to the overlapping audio segments of multiple speakers.

[0059] The structures of the first voiceprint encoder and the second voiceprint encoder can be the same or different. For example, both the first voiceprint encoder and the second voiceprint encoder are constructed based on a residual network; the last pooling layer of the residual network is a statistical pooling layer, that is, it is obtained by changing the average pooling at the tail of the original ResNet to statistical pooling. The first voiceprint encoder and the second voiceprint encoder provided in the embodiment of the present application introduce statistical pooling, which can enable the voiceprint encoder to capture richer statistical information, such as variance, maximum value, etc. This information helps the model to better understand the distribution and changes of the input data, thereby improving the feature expression ability, and can extract more comprehensive and rich feature information, thereby enhancing the generalization ability of the model, enabling it to better adapt to different scenarios, improve the model evaluation ability, and thereby improve the accuracy and generalization of speaker estimation in audio.

[0060] The method provided in the embodiment of the present application, through the first voiceprint encoder and the second voiceprint encoder constructed based on the residual network (including the statistical pooling layer), respectively encodes the voiceprint of a single speaker and overlapping audio segments, thereby improving the extraction accuracy of the voiceprint features and the generalization ability of the model.

[0061] In some embodiments, the audio recognition model is trained based on the following steps: According to the sample data set, iteratively training the first initial voiceprint encoder and the second initial voiceprint encoder in the initial recognition model respectively to obtain the first voiceprint encoder and the second voiceprint encoder; According to the first voiceprint encoder and the second voiceprint encoder, the initial recognition model is updated to obtain a pre-trained recognition model; Randomly adjusting the order of the audio segments of each speaker in the sample audio to obtain a preprocessed sample audio; The pre-trained recognition model is trained in stages according to the acoustic features of the pre-processed sample audio, the acoustic features of the sample audio and the speaker number label to obtain the audio recognition model.

[0062] Optionally, the training steps of the audio recognition model include: First, get the sample dataset.

[0063] The sample data set here can be obtained by collecting voiceprint training data and long-term conversation audio data. The specific steps include: collecting sample audio data, wherein the audio segment containing only a single speaker with a speaker label in the sample audio data is used for training the first voiceprint encoder. In addition, the sample audio data also contains overlapping audio segments of multiple speakers with a certain signal-to-noise ratio for training the second voiceprint encoder, and the speaker label in the overlapping audio segment is the identity label of the main speaker. For long-term conversation audio data, there are mainly two characteristics of the speaker conversation: one is the situation where the speaker changes frequently, and the other is the situation where the speaker changes less frequently. Therefore, in the process of data collection, in order to take into account the above situation, in the process of model training, the collected single channel original audio data needs to be preprocessed to obtain sample audio data that meets the characteristics of the conversation. The preprocessing here includes but is not limited to the elimination of audio frequency bands where the speaker changes frequently and / or the elimination of audio frequency bands where the speaker changes less frequently.

[0064] Exemplarily, in some embodiments, the sample audio is obtained based on the following steps: Capture single channel raw audio; The original audio is preprocessed to obtain the sample audio; the preprocessing includes eliminating audio segments of conversation audio whose duration is less than a preset duration and / or eliminating audio segments whose speech duration ratio is less than a preset ratio.

[0065] Optionally, when obtaining sample audio, the original audio of a single channel may be first collected through an audio collector; then, the audio segments of the conversation audio in the original audio whose duration is less than a preset duration and / or the audio segments whose speech duration ratio is less than a preset ratio are eliminated, thereby obtaining the sample audio. The preset duration and the preset ratio here can be set according to actual needs, such as the preset duration is set to 0.2s, and the preset ratio is set to 0.2%. Accordingly, the speaker segments of less than 0.2s in the long-time conversation audio in the original audio and the dialogue segments of the speakers whose speech duration ratio is less than 0.2% can be deleted, thereby making the duration of the processed single long-time conversation audio range between thirty minutes and one hour, thereby improving the sample quality, so that the audio recognition model trained accordingly can better realize the estimation of the number of speakers in the long-time audio.

[0066] Next, after obtaining the sample data set, the first initial voiceprint encoder in the initial recognition model can be trained using the audio clips of a single speaker in the sample audio, and the second initial voiceprint encoder in the initial recognition model can be trained using the overlapping audio clips of multiple speakers in the sample audio to obtain the first voiceprint encoder and the second voiceprint encoder.

[0067] It should be noted that, in the training process of the first voiceprint encoder and the second voiceprint encoder, supervised classification training is adopted, and the loss function value can be realized by using loss functions such as Additive Angular Margin (AAM).

[0068] After the first voiceprint encoder and the second voiceprint encoder are trained, the first voiceprint encoder and the second voiceprint encoder can be used to update the initial recognition model to update the first initial voiceprint encoder and the second initial voiceprint encoder in the initial recognition model to the first voiceprint encoder and the second voiceprint encoder, thereby obtaining a pre-trained recognition model.

[0069] In addition, the order of the audio clips of each speaker in the sample audio can be randomly adjusted to obtain preprocessed sample audio to achieve the effect of data enhancement, and can better simulate the uncertainty of the speaker order in the real environment, so that the model can learn more generalized feature representations and richer feature information during the training process.

[0070] Next, the pre-trained recognition model is trained in stages by freezing the model parameters of the first voiceprint encoder and the second voiceprint encoder using the acoustic features of the preprocessed sample audio, the acoustic features of the sample audio, and the speaker number label, so as to gradually guide the model to learn more detailed speaker features and speaker number information, thereby obtaining an audio recognition model.

[0071] The stage-by-stage division here can be to construct multiple different training data sets based on the acoustic features of the sample audio, the acoustic features of the sample audio and the speaker number labels, so as to perform training at different stages based on different training data.

[0072] Exemplarily, in some implementations, the step of phased training includes: Iteratively training the network model to be trained in the pre-trained recognition model according to the acoustic features of the pre-processed sample audio and the speaker number label to obtain a first recognition model; the network model to be trained is a network model other than the first voiceprint encoder and the second voiceprint encoder; Iteratively training the to-be-trained network model in the first recognition model according to the acoustic features of the sample audio and the speaker number label to obtain a second recognition model; The acoustic features of the preprocessed sample audio, the acoustic features of the sample audio, and the speaker number label are combined to iteratively train the network model to be trained in the second recognition model to obtain the audio recognition model.

[0073] After loading and acquiring the pre-trained recognition model, the following three-stage training can be used to acquire the audio recognition model: In the first stage, the network model to be trained in the pre-trained recognition model is iteratively trained using the acoustic features and speaker number labels of the shuffled pre-processed sample audio to improve the generalization of the model.

[0074] In the second stage, the network model to be trained in the first recognition model trained in the first stage is further trained using the acoustic features of the sample audio in normal order and the speaker number labels to obtain a second recognition model.

[0075] In the third stage, the acoustic features of the shuffled preprocessed sample audio and the acoustic features of the normal order sample audio are mixed to continue training the second recognition model trained in the second stage using the acoustic features of the preprocessed sample audio, the acoustic features of the sample audio and the speaker number labels to obtain an audio recognition model.

[0076] The method provided in this embodiment enables the model to adapt to the speaker audio data in real scenes through data enhancement, and applies the enhanced data set for frozen training and phased training, which helps the model learn more generalized and robust feature representations, thereby maintaining stable recognition performance and accuracy when facing complex situations in real scenes.

[0077] Based on the description of the above embodiments, it can be seen that the method provided by the present application has the following advantages: First, the method proposed in the embodiment of the present application does not require a preset clustering threshold or clustering center, and does not require the collection of multi-channel audio data and video modality data, thereby reducing data collection costs and avoiding excessive privacy issues.

[0078] Second, the method proposed in the embodiment of the present application can not only accurately estimate the number of speakers in short-term audio, but also estimate the number of speakers in long-term audio (30 minutes to one hour). This is mainly achieved through two parts: first, the speaker conversion detector enables the voiceprint encoder to extract voiceprint features of different lengths to compress the feature time length; second, the long-term modeling unit captures the feature information in the long-term data.

[0079] Third, the method proposed in the embodiment of the present application can effectively extract the voiceprint features of the main speaker in the overlapping segments and avoid overestimation of the number of people by designing a speaker overlapping segment detector and an overlapping segment voiceprint encoder for overlapping segments.

[0080] The audio recognition system provided by the present invention is described below. The audio recognition system described below and the audio recognition method described above can be referenced to each other.

[0081] Figure 3Schematic diagram of the structure of the audio recognition system provided by the present invention; Figure 3 As shown, the system includes: a first detection unit 310 , a second detection unit 320 and an identification unit 330 .

[0082] The first detection unit 310 is used to input the acoustic features of the audio to be recognized of a single channel into a transition point detector in the audio recognition model to obtain a timestamp vector corresponding to a speaker transition point in the audio to be recognized; The second detection unit 320 is used to input the acoustic features and the timestamp vector corresponding to the speaker switching point into the segment detector in the audio recognition model to obtain the audio segment of a single speaker and the overlapping audio segments of multiple speakers in the audio to be recognized; The recognition unit 330 is used to determine input information according to the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers, and input the input information into the estimator in the audio recognition model to obtain the number of speakers in the audio to be recognized; The audio recognition model is obtained by iterative training based on a sample data set, and the sample data set includes acoustic features of sample audio and labels of the number of speakers in the sample audio.

[0083] The system provided by this embodiment inputs the acoustic features of the audio to be recognized of a single channel into an iteratively trained audio recognition model. The model first uses a transition point detector to determine the timestamp vector of the speaker's transition point. Then, a segment detector distinguishes the audio segments of a single speaker from the overlapping audio segments of multiple speakers based on the acoustic features and the transition point timestamp vector, and extracts voiceprint features of different durations based on these segments to compress the feature time length. The number of speakers is obtained through an estimator to ensure that the number of speakers in audio with a longer duration can be accurately estimated when there is only single-channel speech modal data. This ensures that the accuracy of the speaker number estimation is guaranteed while reducing the configuration cost and improving the adaptability of audio recognition, effectively solving the high cost and low adaptability problems brought about by the previous reliance on multi-channel acquisition methods.

[0084] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0085] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute the audio recognition method, which includes: inputting the acoustic features of the audio to be recognized of a single channel into a transition point detector in the audio recognition model to obtain a timestamp vector corresponding to the speaker transition point in the audio to be recognized; inputting the acoustic features and the timestamp vector corresponding to the speaker transition point into a segment detector in the audio recognition model to obtain an audio segment of a single speaker and overlapping audio segments of multiple speakers in the audio to be recognized; determining input information based on the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers, and inputting the input information into an estimator in the audio recognition model to obtain the number of speakers in the audio to be recognized; wherein the audio recognition model is obtained by iterative training based on a sample data set, and the sample data set includes the acoustic features of the sample audio and a label of the number of speakers in the sample audio.

[0086] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0087] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the audio recognition method provided by the above methods, the method comprising: inputting the acoustic features of the audio to be recognized of a single channel into a transition point detector in an audio recognition model to obtain a timestamp vector corresponding to a speaker transition point in the audio to be recognized; inputting the acoustic features and the timestamp vector corresponding to the speaker transition point into a segment detector in the audio recognition model to obtain an audio segment of a single speaker and overlapping audio segments of multiple speakers in the audio to be recognized; determining input information based on the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers, and inputting the input information into an estimator in the audio recognition model to obtain the number of speakers in the audio to be recognized; wherein the audio recognition model is obtained by iterative training based on a sample data set, and the sample data set includes the acoustic features of the sample audio and a label of the number of speakers in the sample audio.

[0088] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the audio recognition method provided by the above methods, the method comprising: inputting the acoustic features of the audio to be recognized of a single channel into a transition point detector in an audio recognition model to obtain a timestamp vector corresponding to a speaker transition point in the audio to be recognized; inputting the acoustic features and the timestamp vector corresponding to the speaker transition point into a segment detector in the audio recognition model to obtain an audio segment of a single speaker and overlapping audio segments of multiple speakers in the audio to be recognized; determining input information based on the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers, and inputting the input information into an estimator in the audio recognition model to obtain the number of speakers in the audio to be recognized; wherein the audio recognition model is obtained by iterative training based on a sample data set, and the sample data set includes the acoustic features of the sample audio and a label of the number of speakers in the sample audio.

[0089] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0090] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio recognition method, characterized in that: include: Inputting the acoustic features of the audio to be recognized of a single channel into a transition point detector in an audio recognition model to obtain a timestamp vector corresponding to a speaker transition point in the audio to be recognized; Inputting the acoustic features and the timestamp vector corresponding to the speaker switching point into a segment detector in the audio recognition model to obtain an audio segment of a single speaker and overlapping audio segments of multiple speakers in the audio to be recognized; Determining input information according to the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers, and inputting the input information into an estimator in the audio recognition model to obtain the number of speakers in the audio to be recognized; The audio recognition model is obtained by iterative training based on a sample data set, and the sample data set includes acoustic features of sample audio and labels of the number of speakers in the sample audio.

2. The audio recognition method according to claim 1, characterized in that: The determining the input information according to the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers includes: Inputting the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers into the long-term modeling unit in the audio recognition model to obtain the long-term voiceprint features corresponding to the audio to be recognized; the long-term modeling unit is used to concatenate and encode the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers; The input information is obtained according to the long-term voiceprint feature.

3. The audio recognition method according to claim 2, characterized in that: The acquiring the input information according to the long-term voiceprint feature includes: Calculating the similarity between every two elements in the long-term voiceprint feature to obtain a similarity matrix of the long-term voiceprint feature; The input information is obtained according to the similarity matrix.

4. The audio recognition method according to any one of claims 1 to 3, characterized in that: The voiceprint feature corresponding to the audio segment of the single speaker is obtained by inputting the audio segment of the single speaker into the first voiceprint encoder in the audio recognition model for voiceprint encoding; The voiceprint features corresponding to the overlapping audio segments of the multiple speakers are obtained by inputting the overlapping audio segments of the multiple speakers into a second voiceprint encoder in the audio recognition model for voiceprint encoding; The first voiceprint encoder and the second voiceprint encoder are both constructed based on a residual network; the last pooling layer of the residual network is a statistical pooling layer.

5. The audio recognition method according to any one of claims 1 to 3, characterized in that: The audio recognition model is trained based on the following steps: According to the sample data set, iteratively training the first initial voiceprint encoder and the second initial voiceprint encoder in the initial recognition model respectively to obtain the first voiceprint encoder and the second voiceprint encoder; According to the first voiceprint encoder and the second voiceprint encoder, the initial recognition model is updated to obtain a pre-trained recognition model; Randomly adjusting the order of the audio segments of each speaker in the sample audio to obtain a preprocessed sample audio; The pre-trained recognition model is trained in stages according to the acoustic features of the pre-processed sample audio, the acoustic features of the sample audio and the speaker number label to obtain the audio recognition model.

6. The audio recognition method according to claim 5, characterized in that: The step of training the pre-trained recognition model in stages according to the acoustic features of the pre-processed sample audio, the acoustic features of the sample audio and the speaker number label to obtain the audio recognition model comprises: Iteratively training the network model to be trained in the pre-trained recognition model according to the acoustic features of the pre-processed sample audio and the speaker number label to obtain a first recognition model; the network model to be trained is a network model other than the first voiceprint encoder and the second voiceprint encoder; Iteratively training the to-be-trained network model in the first recognition model according to the acoustic features of the sample audio and the speaker number label to obtain a second recognition model; The acoustic features of the preprocessed sample audio, the acoustic features of the sample audio, and the speaker number label are combined to iteratively train the network model to be trained in the second recognition model to obtain the audio recognition model.

7. The audio recognition method according to any one of claims 1 to 3, characterized in that: The sample audio is obtained based on the following steps: Capture single channel raw audio; The original audio is preprocessed to obtain the sample audio; the preprocessing includes eliminating audio segments of conversation audio whose duration is less than a preset duration and / or eliminating audio segments whose speech duration ratio is less than a preset ratio.

8. An audio recognition system, characterized in that: include: A first detection unit, configured to input the acoustic features of the audio to be recognized of a single channel into a transition point detector in the audio recognition model, and obtain a timestamp vector corresponding to a speaker transition point in the audio to be recognized; A second detection unit, configured to input the acoustic features and the timestamp vector corresponding to the speaker switching point into a segment detector in the audio recognition model to obtain an audio segment of a single speaker and overlapping audio segments of multiple speakers in the audio to be recognized; an identification unit, configured to determine input information according to the voiceprint features corresponding to the audio segment of the single speaker and the voiceprint features corresponding to the overlapping audio segments of the multiple speakers, and input the input information into an estimator in the audio recognition model to obtain the number of speakers in the audio to be identified; The audio recognition model is obtained by iterative training based on a sample data set, and the sample data set includes acoustic features of sample audio and labels of the number of speakers in the sample audio.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the audio recognition method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio recognition method according to any one of claims 1 to 7 is implemented.

Images