Speech recognition method and device, electronic equipment and storage medium

Abstract

The invention provides a voice recognition method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: acquiring target voice data containing a plurality of speakers; coding the target voice data based on a voice recognition model to obtain voice coding features, the voice recognition model comprising a voice recognition network and a speaker recognition network; performing content recognition on the voice coding features based on a voice recognition network to obtain voice text content corresponding to the target voice data; speaker recognition is carried out on the target voice data and the voice coding features based on a speaker recognition network, and speaker voiceprint features corresponding to the target voice data are obtained; and based on the voiceprint features of the speakers, each text character in the voice text content is classified to obtain target voice content generated by each speaker, and the target voice content is composed of a plurality of text characters. According to the method and the device, the recognition accuracy of voices containing a plurality of speakers can be improved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a speech recognition method and device, an electronic device and a storage medium. Background Art

[0002] In the field of modern communications and data processing, accurately identifying and processing target speech data containing multiple speakers is an important task. For example, when a financial business is processed, multi-speaker speech data will be generated in which a business person communicates with multiple business processing objects. For this kind of speech data, it is often necessary to accurately identify the speech content described by each speaker.

[0003] To meet this demand, speech recognition methods in related technologies usually rely on complex algorithms and models to extract useful information from speech signals. However, when the target speech data contains multiple speakers, traditional speech recognition systems often have difficulty distinguishing and identifying the voice characteristics of each speaker, resulting in a decrease in speech recognition accuracy. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a speech recognition method and device, an electronic device and a storage medium, aiming to improve the recognition accuracy of speech containing multiple speakers.

[0005] To achieve the above object, a first aspect of an embodiment of the present application provides a speech recognition method, the method comprising:

[0006] Acquire target speech data containing multiple speakers;

[0007] Encoding the target speech data based on a preset speech recognition model to obtain speech coding features, wherein the speech recognition model includes a speech recognition network and a speaker recognition network;

[0008] Performing content recognition on the speech coding features based on the speech recognition network to obtain speech text content corresponding to the target speech data;

[0009] Performing speaker recognition on the target voice data and the voice coding features based on the speaker recognition network to obtain speaker voiceprint features corresponding to the target voice data;

[0010] Based on the voiceprint features of the speaker, each text character in the voice text content is classified to obtain the target voice content generated by each speaker, wherein the target voice content is composed of a plurality of the text characters.

[0011] In some embodiments, the speech recognition network includes a first decoder, a second decoder, a third decoder, and a feature transformation network;

[0012] The performing content recognition on the speech coding feature based on the speech recognition network to obtain speech text content corresponding to the target speech data includes:

[0013] Performing feature alignment on the speech coding features based on the first decoder to obtain first speech decoding features;

[0014] Performing phoneme feature recognition on the speech coding feature based on the second decoder to obtain a second speech decoding feature;

[0015] Performing feature mapping on the speech coding feature based on the third decoder to obtain a third speech decoding feature;

[0016] Performing feature aggregation on the first speech decoding feature, the second speech decoding feature, and the third speech decoding feature to obtain a speech aggregation feature;

[0017] The speech aggregation features are subjected to feature transformation based on the feature transformation network to obtain the speech text content.

[0018] In some embodiments, the speaker recognition network includes a speaker encoder and a speaker decoder;

[0019] The performing speaker recognition on the target voice data and the voice coding features based on the speaker recognition network to obtain the speaker voiceprint features corresponding to the target voice data includes:

[0020] Performing feature splicing on the target speech data and the speech coding feature to obtain a target splicing feature;

[0021] Performing speaker feature extraction on the target concatenated features based on the speaker encoder to obtain speaker encoding features;

[0022] Speaker recognition is performed on the speaker encoding features based on the speaker decoder to obtain the speaker voiceprint features.

[0023] In some embodiments, the speaker voiceprint feature includes sub-voiceprint features corresponding to each text character in the speech text content;

[0024] The method of classifying each text character in the speech text content based on the voiceprint feature of the speaker to obtain the target speech content generated by each speaker includes:

[0025] Obtain reference voiceprint features and reference object identifiers of each reference speaking object;

[0026] For each of the text characters, calculating the voiceprint similarity between the sub-voiceprint feature and the reference voiceprint feature to obtain the voiceprint similarity between the sub-voiceprint feature and each of the reference voiceprint features;

[0027] Using the reference object identifier corresponding to the reference voiceprint feature with the greatest voiceprint similarity as the speaker identifier of the text character, wherein the speaker identifier is used to indicate the identity of the speaker;

[0028] Based on the speaker identifiers of the text characters, the text characters having the same speaker identifier are divided into the same text character group;

[0029] Based on the text character group, target speech content produced by each of the speakers is determined.

[0030] In some embodiments, the speech recognition network is trained by the following process:

[0031] Acquire sample audio data and annotated text data corresponding to the sample audio data;

[0032] Performing feature alignment on sample audio encoding features of the sample audio data based on a preset first original network to obtain a first sample decoding feature, and determining a first probability of generating the annotated text data based on the first sample decoding feature;

[0033] Performing phoneme feature recognition on the sample audio encoding feature based on the first original network to obtain a second sample decoding feature, and determining a second probability of generating the annotated text data based on the second sample decoding feature;

[0034] Performing feature mapping on the sample audio encoding feature based on the first original network to obtain a third sample decoding feature, and determining a third probability of generating the annotated text data based on the third sample decoding feature;

[0035] Based on the first probability, the second probability and the third probability, the first original recognition network is trained to obtain the speech recognition network.

[0036] In some embodiments, the speaker recognition network is trained by the following process:

[0037] Acquire a plurality of sample speech frames and a sample speaker voiceprint feature corresponding to each of the sample speech frames;

[0038] constructing sample pairs based on the sample speech frame and the sample speaker voiceprint features, the sample pairs comprising a positive sample pair consisting of the sample speech frame and the sample speaker voiceprint features corresponding to the sample speech frame, and a negative sample pair consisting of the sample speech frame and the sample speaker voiceprint features not corresponding to the sample speech frame;

[0039] Inputting the positive sample pair into a preset second original network for speaker recognition to obtain a first predicted voiceprint feature, and determining a first similarity between the first predicted voiceprint feature and the voiceprint feature of the sample speaker in the positive sample pair;

[0040] Inputting the negative sample pair into the second original network for speaker recognition to obtain a second predicted voiceprint feature, and determining a second similarity between the second predicted voiceprint feature and the voiceprint feature of the sample speaker in the negative sample pair;

[0041] Based on the first similarity and the second similarity of a plurality of the sample speech frames, the second original network is trained to obtain the speaker recognition network.

[0042] In some embodiments, obtaining target speech data containing multiple speakers includes:

[0043] Obtaining raw speech data containing multiple speakers;

[0044] Performing denoising processing on the original speech data to obtain denoised speech data;

[0045] Performing Fourier transform on the denoised speech data to obtain speech frequency domain data;

[0046] Mel filtering is performed on the speech frequency domain data to obtain the target speech data.

[0047] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech recognition device, the device comprising:

[0048] An acquisition unit, used for acquiring target speech data containing multiple speakers;

[0049] An encoding unit, configured to encode the target speech data based on a preset speech recognition model to obtain speech coding features, wherein the speech recognition model includes a speech recognition network and a speaker recognition network;

[0050] A content recognition unit, configured to perform content recognition on the speech coding features based on the speech recognition network to obtain speech text content corresponding to the target speech data;

[0051] A speaker recognition unit, configured to perform speaker recognition on the target speech data and the speech coding features based on the speaker recognition network to obtain a speaker voiceprint feature corresponding to the target speech data;

[0052] The character classification unit is used to classify each text character in the speech text content based on the voiceprint features of the speaker to obtain the target speech content generated by each speaker, wherein the target speech content is composed of a plurality of the text characters.

[0053] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the speech recognition method described in the first aspect is implemented.

[0054] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the speech recognition method described in the first aspect.

[0055] The speech recognition method and device, electronic device and storage medium proposed in the present application obtain target speech data containing multiple speakers; encode the target speech data based on a preset speech recognition model to obtain speech coding features, which can convert the original speech signal into a series of speech coding features, and represent the key attributes of the speech signal (such as pitch, sound intensity and timbre, etc.) through the speech coding features. Further, based on the speech recognition network, the speech coding features are subjected to content recognition to obtain the speech text content corresponding to the target speech data, and the speech coding features are mapped to the corresponding speech text content, thereby generating a text representation of the target speech data, which can achieve accurate recognition of the speech content of the target speech data. Further, based on the speaker recognition network, speaker recognition is performed on the target speech data and the speech coding features to obtain the speaker voiceprint features corresponding to the target speech data, which can capture the voiceprint characteristics of the target speaker used for the data. Finally, based on the speaker voiceprint features, each text character in the speech text content is classified to obtain the target speech content generated by each speaker, which can accurately identify and distinguish the target speech content generated by each speaker, and can improve the recognition accuracy of speech containing multiple speakers. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of a speech recognition method provided by an embodiment of the present application;

[0057] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0058] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0059] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0060] Figure 5 yes Figure 1 Flow chart of step S105 in FIG.

[0061] Figure 6 is a flow chart of a speech recognition network for training a speech recognition model in an embodiment of the present application;

[0062] Figure 7 is a flow chart of a speaker recognition network for training a speech recognition model in an embodiment of the present application;

[0063] Figure 8 is a structural diagram of a speech recognition device provided in an embodiment of the present application;

[0064] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0068] The speech recognition method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speech recognition method in the embodiments of the present application is described.

[0069] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0071] The speech recognition method provided in the embodiment of the present application relates to the fields of financial technology and artificial intelligence technology. The speech recognition method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech recognition method, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object behavior data, object historical data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0074] Figure 1 is an optional flow chart of the speech recognition method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0075] Step S101, obtaining target speech data containing multiple speakers;

[0076] Step S102, encoding the target speech data based on a preset speech recognition model to obtain speech coding features;

[0077] Step S103, performing content recognition on the speech coding features based on the speech recognition network to obtain the speech text content corresponding to the target speech data;

[0078] Step S104, performing speaker recognition on the target voice data and the voice coding features based on a speaker recognition network to obtain speaker voiceprint features corresponding to the target voice data;

[0079] Step S105, classifying each text character in the speech text content based on the speaker's voiceprint features to obtain the target speech content generated by each speaker.

[0080] The speech recognition model includes a speech recognition network and a speaker recognition network. The target speech content consists of multiple text characters.

[0081] Steps S101 to S105 shown in the embodiment of the present application are obtained by acquiring target speech data containing multiple speakers; encoding and processing the target speech data based on a preset speech recognition model to obtain speech coding features, which can convert the original speech signal into a series of speech coding features, and represent the key attributes of the speech signal (such as pitch, sound intensity and timbre, etc.) through the speech coding features. Further, based on the speech recognition network, the speech coding features are subjected to content recognition to obtain the speech text content corresponding to the target speech data, and the speech coding features are mapped to the corresponding speech text content, thereby generating a text representation of the target speech data, which can achieve accurate recognition of the speech content of the target speech data. Further, based on the speaker recognition network, speaker recognition is performed on the target speech data and the speech coding features to obtain the speaker voiceprint features corresponding to the target speech data, which can capture the voiceprint characteristics of the target speaker used for the data. Finally, based on the speaker voiceprint features, the various text characters in the speech text content are classified to obtain the target speech content generated by each speaker, which can accurately identify and distinguish the target speech content generated by each speaker, and can improve the recognition accuracy of the speech containing multiple speakers.

[0082] In step S101 of some embodiments, target speech data containing multiple speakers is acquired.

[0083] The target speech data refers to the Mel-cepstral feature data corresponding to the speech data generated by multiple speakers in a conversation.

[0084] Speakers refer to the various parties participating in the same conversation.

[0085] For example, in a financial business transaction, multiple parties are often involved in communication. At this time, the voice data retained for this conversation will record the communication content of multiple parties. For another example, in the field of digital medicine, when medical staff from multiple departments conduct a multi-department medical consultation on a certain medical problem, the voice data will record the opinions expressed by various parties in the medical consultation. Based on this, the specific speakers and specific content of voice data in multi-person communication scenarios are more complex than those in one-to-one communication scenarios.

[0086] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S204:

[0087] Step S201, obtaining original speech data containing multiple speakers;

[0088] Step S202, performing denoising processing on the original speech data to obtain denoised speech data;

[0089] Step S203, performing Fourier transform on the denoised speech data to obtain speech frequency domain data;

[0090] Step S204: Mel filtering is performed on the speech frequency domain data to obtain target speech data.

[0091] Steps S201 to S204 are described in detail below.

[0092] In step S201 of some embodiments, original voice data containing multiple speakers can be extracted from a preset audio database or downloaded from a network platform or multimedia device, etc., and original voice data containing multiple speakers can also be obtained in other ways, but not limited to this. The original voice data containing multiple speakers is waveform data, and the voice changes of the original voice data are reflected by the changes in the waveform.

[0093] In step S202 of some embodiments, the original speech data is subjected to denoising processing to eliminate the environmental noise in the target audio and obtain denoised speech data.

[0094] In step S203 of some embodiments, the denoised speech data (continuous-time audio signal) is converted into a discrete-time signal, and the discrete-time signal is truncated and multiplied by a window function to obtain a truncated audio signal. The window function includes but is not limited to a Hanning window, a Hamming window, and the like. Further, the truncated audio signal is Fourier transformed using a fast Fourier transform algorithm to provide frequency components in the truncated audio signal, and the target audio is converted from the time domain to the frequency domain to obtain speech domain data. The speech domain data is used to represent the amplitude and phase of different frequency components of the denoised speech data, and is used to indicate the frequency distribution information of the denoised speech data.

[0095] In step S204 of some embodiments, first, the speech frequency domain data is modulo-taken, and the modulus of the speech frequency domain data is bisected to obtain a target power spectrum. Then, with authorization, the Mel filter bank is called, and the target power spectrum is filtered using the Mel filter bank, and the conventional frequency is mapped to the Mel frequency scale to obtain a target audio filtering result. Then, the target audio filtering result is logarithmically compressed to simulate the nonlinear perception of the human ear to the sound intensity, and the target audio compression result is obtained. Further, the target audio compression result is discrete cosine transformed to obtain a discrete cosine transform result, wherein the discrete cosine transform result has multiple coefficients for characterizing the audio information of the denoised speech data. Then, the first predetermined number of coefficients in the discrete cosine transform result are selected as Mel cepstrum features. Among them, the predetermined number is not greater than 13, and the selected coefficients represent the important audio information of the target audio. By screening the coefficients in the discrete cosine transform result, redundant information can be well reduced. Finally, the selected Mel cepstrum feature is used as the target speech data.

[0096] It should be noted that the speech contents of the original speech data and the target speech data are basically the same, the difference is that the original speech data is time domain data, while the target speech data is frequency domain data.

[0097] The advantage of this embodiment is that, by using operations such as Fourier transform, Mel filtering and discretization mapping, the original speech data containing multiple speakers is converted from the time domain to the frequency domain, and important audio information and important speaker information of the original speech data in the frequency domain are extracted, which can improve the data quality of the target speech data used for speech recognition.

[0098] In step S102 of some embodiments, the target speech data is encoded based on a preset speech recognition model to obtain speech coding features, wherein the speech recognition model includes a speech recognition network and a speaker recognition network.

[0099] The speech recognition model refers to a neural network model that can be used to perform speech content recognition and speaker recognition on speech data corresponding to multiple speakers, as well as recognize the speech content corresponding to each speaker. The input of the speech recognition model is generally speech data in the form of Mel cepstrum, and the output of the speech recognition model is generally the speaker identity of the speaker of the speech data and the speech content produced by each speaker.

[0100] The speech recognition network is used to recognize the specific speech content of the input speech data and output the speech content text corresponding to the speech data.

[0101] The speaker recognition network is used to identify the specific identity of the speaker contained in the input voice data, and output the specific identity of the speaker and which part of the voice content text the content described by each speaker is.

[0102] The speech coding features are used to indicate the audio information representation of the target speech data in the latent vector space.

[0103] When this embodiment is implemented, a shared encoder based on a preset speech recognition model encodes the target speech data, maps the target speech data from the frequency domain space to the latent vector space, converts the speech data in the Mel-cepstral form into a vector form, and obtains speech coding features.

[0104] In step S103 of some embodiments, content recognition is performed on the speech coding features based on a speech recognition network to obtain speech text content corresponding to the target speech data.

[0105] The speech text content is used to reflect the specific speech content of the target speech data in the form of text. The speech text content can describe the specific overall communication content of multiple speakers in the conversation in the form of text (text characters).

[0106] In an embodiment of the present application, the speech recognition network includes a first decoder, a second decoder, a third decoder and a feature transformation network.

[0107] The first decoder is used to align the irregular time sequence structure of the speech signal in the input speech data, so that each feature in the output sequence (first speech decoding feature) is automatically aligned with each feature in the input sequence (speech encoding feature).

[0108] The second decoder is used to identify important voice information in the input voice data, identify feature information related to the current phoneme, and output key phoneme features corresponding to the voice data.

[0109] The third decoder is used to implement the mapping of the acoustic features of the input speech data to the phoneme features, and convert the speech data into a phoneme feature sequence while retaining the acoustic sequence feature information of the speech data.

[0110] The feature transformation network is used to convert the features of the latent vector space into a text sequence in text form to achieve feature decoding.

[0111] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S305:

[0112] Step S301, aligning speech coding features based on a first decoder to obtain first speech decoding features;

[0113] Step S302, performing phoneme feature recognition on the speech coding feature based on the second decoder to obtain a second speech decoding feature;

[0114] Step S303, performing feature mapping on the speech coding feature based on the third decoder to obtain a third speech decoding feature;

[0115] Step S304, performing feature aggregation on the first speech decoding feature, the second speech decoding feature and the third speech decoding feature to obtain a speech aggregation feature;

[0116] Step S305, performing feature transformation on the speech aggregation features based on the feature transformation network to obtain speech text content.

[0117] Steps S301 to S305 are described in detail below.

[0118] In step S301 of some embodiments, the first decoder may be a CTC decoder (Connectionist Temporal Classification Decoder). Specifically, first, feature mapping is performed on the speech coding features through the fully connected layer of the first decoder, and the speech coding features are mapped to a low-dimensional vector space to obtain a one-dimensional speech feature. Then, since the speech coding feature is a feature sequence and is composed of multiple features, the phoneme probability is calculated on the one-dimensional speech feature through the classification function of the first decoder to obtain the posterior probability that each feature in the speech coding feature belongs to each candidate phoneme in the preset phoneme table, and the candidate phoneme with the largest posterior probability is used as the target phoneme corresponding to the feature, and according to the feature order of each feature in the speech coding feature, multiple target phonemes are spliced ​​into a phoneme sequence to obtain the first speech decoding feature. Among them, the first speech decoding feature is a phoneme sequence that is completely aligned with the speech coding feature.

[0119] In step S302 of some embodiments, the second decoder may be a decoder based on an attention mechanism. Specifically, the voice coding feature is linearly projected through the query channel of the second decoder to obtain a query vector, the voice coding feature is linearly projected based on the key channel of the second decoder to obtain a key vector, and the voice coding feature is linearly projected through the value channel of the second decoder to obtain a value vector, wherein the projection parameters of the key channel, the query channel, and the value channel are different. Further, the query vector is multiplied by the transposed result of the key vector to obtain a product result, and the product result is divided by the square difference of the feature dimension of the key vector to obtain a division result. Finally, the division result is normalized to obtain an attention weight, and the attention weight is multiplied by the value vector to obtain a second voice decoding feature. Wherein, the second voice decoding feature is used to indicate the key phoneme feature information corresponding to the target voice data.

[0120] In step S303 of some embodiments, the third decoder may be a RNN-T decoder (Recurrent Neural Network Transducer). Specifically, the acoustic feature information in the speech coding feature is extracted and converted by the third decoder, and the speech coding feature is converted into a phoneme feature sequence while retaining the acoustic sequence feature information of the target speech data, and the converted phoneme feature sequence is used as the third speech decoding feature. The third speech decoding feature is used to indicate the overall phoneme feature information corresponding to the target speech data.

[0121] In step S304 of some embodiments, when the first speech decoding feature, the second speech decoding feature and the third speech decoding feature are subjected to feature aggregation, the first speech decoding feature, the second speech decoding feature and the third speech decoding feature are concatenated into a longer feature vector to obtain a speech aggregation feature. The speech aggregation feature contains both key speech information in the target speech data and overall speech information in the target speech data. At the same time, the speech aggregation feature is basically aligned with the target speech data (Mel-frequency cepstrum feature).

[0122] In step S305 of some embodiments, the feature transformation network can be based on a neural network with a transformer structure. Specifically, when performing feature transformation on the speech aggregation feature based on the feature transformation network, for each phoneme feature in the speech aggregation feature, first perform attention calculation on the phoneme feature, then perform classification probability calculation on the attention calculation result of each phoneme feature according to the preset character table, determine the probability of each phoneme feature being each candidate character in the preset vocabulary, and use the candidate character with the highest probability as the predicted text character corresponding to the phoneme feature. Furthermore, according to the feature sequence of each phoneme feature in the speech aggregation feature, the text characters corresponding to each phoneme feature are combined into a character sequence, and the character sequence is used as the speech text content corresponding to the target speech data.

[0123] The benefit of this embodiment is that, during speech recognition, the speech coding features are jointly decoded by the first decoder, the second decoder and the third decoder, so that the generated speech aggregation features (phoneme features / character features) can be automatically aligned with the original target speech data (acoustic features) at the feature level, and the key feature information and complete feature information of the target speech data can be accurately extracted, so that when the feature transformation network is used for feature decoding, text can be generated based on the overall phoneme feature information and key local phoneme feature information corresponding to the target speech data, thereby improving the content accuracy and content completeness of the speech text content generated in speech recognition.

[0124] In step S104 of some embodiments, speaker recognition is performed on the target speech data and speech coding features based on a speaker recognition network to obtain speaker voiceprint features corresponding to the target speech data.

[0125] The speaker voiceprint feature is used to indicate the voiceprint characteristics of each speaker in the target speech data.

[0126] In the embodiment of the present application, the speaker recognition network includes a speaker encoder and a speaker decoder.

[0127] The speaker encoder is used to extract speaker-related feature information from the target speech data, wherein the speaker-related feature information includes but is not limited to the speaker's accent, utterance frequency, timbre, etc.

[0128] The speaker decoder is used to project the speaker-related feature information extracted by the speaker encoder into a specific vector space to obtain the speaker embedding information corresponding to the target speech data. The speaker embedding information is used to indicate the expression form of the voiceprint characteristics of each speaker in the target speech data in a specific vector space.

[0129] It should be noted that the speaker encoder and speaker decoder in the embodiments of the present application can both be neural network structures composed of multiple self-attention mechanisms and feedforward neural networks.

[0130] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S403:

[0131] Step S401, performing feature concatenation on target speech data and speech coding features to obtain target concatenated features;

[0132] Step S402, performing speaker feature extraction on the target concatenated features based on the speaker encoder to obtain speaker encoding features;

[0133] Step S403: Perform speaker recognition on the speaker encoding features based on the speaker decoder to obtain the speaker voiceprint features.

[0134] Steps S401 to S403 are described in detail below.

[0135] In step S401 of some embodiments, since the target speech data is in the form of Mel-cepstral features and the speech coding features are in the form of vector features, the target speech data and the speech coding features are first mapped to the vector space of the same feature dimension to obtain the variable-dimensional speech features corresponding to the target speech data and the variable-dimensional coding features corresponding to the speech coding features. Then, the variable-dimensional speech features and the variable-dimensional coding features in the same feature dimension are spliced ​​into a longer feature vector to obtain the target spliced ​​features.

[0136] In step S402 of some embodiments, first, linearly project the target concatenated features through the query channel of the speaker encoder to obtain a query vector, linearly project the target concatenated features based on the key channel of the speaker encoder to obtain a key vector, and linearly project the target concatenated features based on the value channel of the speaker encoder to obtain a value vector, wherein the projection parameters of the key channel, the query channel, and the value channel are different. Further, the query vector and the transposed result of the key vector are multiplied to obtain a product result, and the product result is divided by the square difference of the feature dimension of the key vector to obtain a division result. Finally, the division result is normalized to obtain an attention weight, and the attention weight is multiplied by the value vector to obtain a speaker attention feature. The speaker attention feature is used to indicate key feature information related to the speaker in the target speech data. Next, the speaker attention feature is nonlinearly transformed to achieve feature mapping of the speaker attention feature to obtain a speaker encoding feature.

[0137] In step S403 of some embodiments, first, the speaker decoder performs self-attention calculation and feedforward processing on the speaker encoding features to obtain speaker representation features, and the specific process is similar to the above step S402. To save space, it is not repeated. Then, the speaker decoder performs embedding processing on the speaker representation features to obtain the speaker voiceprint features.

[0138] The benefit of this embodiment is that when performing speaker recognition on the target speech data, the speaker encoder is used to perform attention calculation and feedforward processing on the target concatenated features, so that the long-distance dependencies between various speech features in the target concatenated features can be captured, and the complex speech information in the target speech data can be learned, thereby improving the feature accuracy and information validity of the extracted speaker coding features. Furthermore, the speaker decoder is used to convert the speaker coding features into an embedded form, so that the voiceprint feature information in the speaker coding features can be obtained more conveniently, thereby improving the feature acquisition efficiency of the speaker voiceprint features corresponding to the target speech data.

[0139] In step S105 of some embodiments, each text character in the speech text content is classified based on the speaker's voiceprint features to obtain the target speech content generated by each speaker, wherein the target speech content consists of multiple text characters.

[0140] The target speech content is used to indicate the specific content expressed by a single speaker in the target speech data.

[0141] The text characters refer to the text representation of the phoneme features corresponding to each feature in the target speech data.

[0142] In the embodiment of the present application, the speaker's voiceprint features include sub-voiceprint features corresponding to each text character in the speech text content.

[0143] The sub-voiceprint feature is used to indicate the characteristics of the speaker's voiceprint that expresses the voice content of a single text character in the target voice data. Since each feature in the target voice data corresponds to a phoneme feature, and each phoneme feature corresponds to a text character, and each feature of the target voice data value also corresponds to a speaker voiceprint feature. Therefore, according to this series of mapping relationships, it can be determined that each text character has a corresponding speaker voiceprint feature (sub-voiceprint feature).

[0144] See also Figure 5 In some embodiments, step S105 may include but is not limited to steps S501 to S505:

[0145] Step S501, obtaining reference voiceprint features and reference object identifiers of each reference speaking object;

[0146] Step S502, for each text character, calculating the voiceprint similarity between the sub-voiceprint feature and the reference voiceprint feature, and obtaining the voiceprint similarity between the sub-voiceprint feature and each reference voiceprint feature;

[0147] Step S503, using the reference object identifier corresponding to the reference voiceprint feature with the greatest voiceprint similarity as the speaker identifier of the text character, wherein the speaker identifier is used to indicate the identity of the speaker;

[0148] Step S504, based on the speaker identifiers of the respective text characters, the text characters having the same speaker identifier are divided into the same text character group;

[0149] Step S505: determining the target speech content produced by each speaker based on the text character group.

[0150] Steps S501 to S505 are described in detail below.

[0151] In step S501 of some embodiments, the reference speaking object refers to each object legally registered on the network platform or application to which the speech recognition method is applied, and the reference voiceprint feature refers to the voiceprint feature extracted from the speech provided by the reference speaking object when registering, and the reference voiceprint feature can indicate the identity of the reference speaking object from the perspective of the voiceprint. The reference object identifier is used to identify the reference speaking object, and the reference voiceprint features and reference object identifiers of different reference speaking objects are different.

[0152] In the specific implementation of this embodiment, since the reference voiceprint features and reference object identifiers of each reference speaker are collected and recorded in the server background when the reference speaker is registered, based on this, the reference voiceprint features and reference object identifiers of each reference speaker can be extracted from the server background log with authorization.

[0153] In step S502 of some embodiments, for each text character, the sub-voiceprint feature and the reference voiceprint feature are calculated for voiceprint similarity using a cosine similarity algorithm to obtain the voiceprint similarity between the sub-voiceprint feature and each reference voiceprint feature. The voiceprint similarity is used to indicate the degree of closeness between the sub-voiceprint feature and each reference voiceprint feature.

[0154] In step S503 of some embodiments, since the greater the voiceprint similarity, the closer the sub-voiceprint feature is to the reference voiceprint feature. Based on this, the reference voiceprint feature with the greatest voiceprint similarity is considered to be from the same speaker as the sub-voiceprint feature. Therefore, the reference object identifier corresponding to the reference voiceprint feature with the greatest voiceprint similarity is used as the speaker identifier of the text character, and the speaker identifier is used to identify which reference speaker the text character comes from.

[0155] In step S504 of some embodiments, since the text characters with the same speaker identifier are all generated by the same speaker, based on this, the speaker identifiers of the respective text characters are compared and the text characters with the same speaker identifier are grouped into the same text character group.

[0156] In step S505 of some embodiments, for each text character group, each text character of each text character group is combined and spliced ​​to form multiple complete sentences, thereby obtaining the target speech content produced by each speaker. Among them, one text character group corresponds to the target speech content produced by one speech.

[0157] The benefit of this embodiment is that the sub-voiceprint features of each text character are compared with the reference voiceprint features of each reference speaking object collected in advance, and the reference object identifier corresponding to the reference voiceprint feature with the greatest voiceprint similarity is used as the speaker identifier of the text character, which can realize the conversion of voice signals into text output and distinguish different speakers, thereby realizing voice recognition, quickly switching between continuous voice inputs and accurately identifying the text content corresponding to each voice signal and the true identity of the speaker who generated the voice signal, which can improve the accuracy of voice recognition for multi-speaker voices and the comprehensiveness of voice recognition.

[0158] See also Figure 6 In some embodiments, the speech recognition network of the speech recognition model is trained by the following process:

[0159] Step S601, obtaining sample audio data and annotated text data corresponding to the sample audio data;

[0160] Step S602, aligning the sample audio encoding features of the sample audio data based on a preset first original network to obtain a first sample decoding feature, and determining a first probability of generating annotated text data based on the first sample decoding feature;

[0161] Step S603, performing phoneme feature recognition on the sample audio coding feature based on the first original network to obtain a second sample decoding feature, and determining a second probability of generating annotated text data based on the second sample decoding feature;

[0162] Step S604, performing feature mapping on the sample audio encoding feature based on the first original network to obtain a third sample decoding feature, and determining a third probability of generating annotated text data based on the third sample decoding feature;

[0163] Step S605: Based on the first probability, the second probability and the third probability, the first original recognition network is trained to obtain a speech recognition network.

[0164] Steps S601 to S605 are described in detail below.

[0165] In step S601 of some embodiments, sample audio data and annotated text data corresponding to the sample audio data can be extracted from a preset audio database, or downloaded from a network platform or multimedia device, etc. Sample audio data and annotated text data corresponding to the sample audio data can also be obtained in other ways, but are not limited to this.

[0166] In step S602 of some embodiments, the specific process of aligning the sample audio encoding features based on the preset first original network to obtain the first sample decoding features is similar to the above step S301, which will not be described in detail to save space.

[0167] Further, when determining the first probability of generating the annotated text data based on the first sample decoding feature, firstly, the first sample decoding feature is feature decoded, and the first sample decoding feature is converted from a feature sequence to a text sequence to obtain the predicted text data corresponding to the sample audio data, and the specific process is similar to the above step S305. To save space, it will not be repeated.

[0168] Furthermore, the predicted text data and the annotated text data are compared for text similarity, similarity is calculated between the predicted text data and the annotated text data to obtain text similarity, and the text similarity is used as the first probability.

[0169] In step S603 of some embodiments, the specific process of performing phoneme feature recognition on the sample audio encoding feature based on the first original network to obtain the second sample decoding feature is similar to the above step S302, which will not be described in detail to save space.

[0170] Furthermore, the specific process of determining the second probability of generating the annotated text data based on the second sample decoding feature is similar to the specific process of determining the first probability of generating the annotated text data based on the first sample decoding feature, which will not be described in detail to save space.

[0171] In step S604 of some embodiments, the specific process of performing feature mapping on the sample audio encoding features based on the first original network to obtain the third sample decoding features is similar to the above step S303, which will not be described in detail to save space.

[0172] Furthermore, the specific process of determining the third probability of generating the annotated text data based on the third sample decoding feature is similar to the specific process of determining the first probability of generating the annotated text data based on the first sample decoding feature, which will not be described in detail to save space.

[0173] In step S605 of some embodiments, first, a first loss function is calculated based on the negative logarithm of the first probability. Then, a second loss function is calculated based on the negative logarithm of the second probability. Further, a third loss function is calculated based on the negative logarithm of the third probability. Then, a total loss function is obtained based on the weighted sum of the first loss function, the second loss function and the third loss function. Finally, the model parameters of the first original network are updated according to the output value of the total loss function in the current training round, and the above steps S601 to S605 are repeated to implement iterative training of the first original network until the total loss function is less than a preset loss threshold in a certain training round, then the model parameters of the training round are used as the final model parameters, and the first original network with the final model parameters is used as the trained speech recognition network.

[0174] Among them, the first loss function of the first original network in the current training round can be expressed as shown in formula (1):

[0175] L ctc = -logP ctc (Y|X) formula (1);

[0176] Among them, L ctc is the first loss function, X refers to the sample audio data, Y refers to the annotated text data, P ctc (Y|X) is inputting the sample audio data into the first decoder, and obtaining the first probability of the labeled text data according to the output result of the first decoder.

[0177] The second loss function can be expressed as shown in formula (2):

[0178] L att = -logP att (Y|X) formula (2);

[0179] Among them, L att is the second loss function, X refers to the sample audio data, Y refers to the labeled text data, P att (Y|X) is inputting the sample audio data into the second decoder, and obtaining the second probability of the labeled text data according to the output result of the second decoder.

[0180] The third loss function can be expressed as shown in formula (3):

[0181] L rnnt = -logP rnnt (Y|X) formula (3);

[0182] Among them, L rnnt is the third loss function, X refers to the sample audio data, Y refers to the labeled text data, P rnnt(Y|X) is inputting the sample audio data into the third decoder, and obtaining the third probability of the labeled text data according to the output result of the third decoder.

[0183] The total loss function can be expressed as shown in formula (4):

[0184] L=λ ctc L ctc +λ att L att +λ rnnt L rnnt Formula (4);

[0185] Among them, L is the total loss function, λ ctc is the training weight of the first decoder, λ att is the training weight of the second decoder, λ rnnt is the training weight of the third decoder. ctc , att , rmnt The specific value can be preset according to actual business needs without restriction.

[0186] The benefit of this embodiment is that when training the speech recognition network of the speech recognition model, three different decoders (the first decoder CTC, the second decoder Attention and the third decoder RNN-T) are 3D jointly modeled, and the performance of automatic speech recognition is improved through joint training and optimization. During the training process, the model can simultaneously learn how to align input and output sequences, focus on important parts, and use effective acoustic features for decoding. This joint modeling method helps to improve the performance of the speech recognition system and can achieve accurate speech recognition in the case of irregular timing and complex background noise. This training method can better handle problems such as overlapping speech, error correction and task trade-offs by integrating the advantages and characteristics of different decoders, thereby improving the accuracy and robustness of speech recognition of the speech recognition model.

[0187] See also Figure 7 In some embodiments, the speaker recognition network of the speech recognition model is trained by the following process:

[0188] Step S701, obtaining a plurality of sample speech frames and a sample speaker voiceprint feature corresponding to each sample speech frame;

[0189] Step S702, constructing a sample pair based on the sample speech frame and the sample speaker's voiceprint features;

[0190] Step S703, inputting the positive sample pair into a preset second original network to perform speaker recognition, obtaining a first predicted voiceprint feature, and determining a first similarity between the first predicted voiceprint feature and the voiceprint feature of the sample speaker in the positive sample pair;

[0191] Step S704, inputting the negative sample pair into the second original network for speaker recognition to obtain a second predicted voiceprint feature, and determining a second similarity between the second predicted voiceprint feature and the voiceprint feature of the sample speaker in the negative sample pair;

[0192] Step S705: Based on the first similarity and the second similarity of the plurality of sample speech frames, a second original network is trained to obtain a speaker recognition network.

[0193] Steps S701 to S705 are described in detail below.

[0194] In step S701 of some embodiments, when obtaining multiple sample speech frames, the sample audio data of step S601 can be obtained first, and the sample audio data can be regarded as a sample speech sequence and divided into frames to obtain multiple speech frames, wherein each sample speech frame corresponds to a text character.

[0195] For example, a sample audio data is "Speaker A: What are your interests? Speaker B: Skiing; Speaker C: Swimming". Based on this, the speech coding features corresponding to the sample audio data have 11 features, the number of text characters of the text content obtained by the speech recognition network of the speech recognition model is 11, the number of sample speech frames is 11, the sample speaker voiceprint features corresponding to the 1st to 7th sample speech frames are the voiceprint features of speaker A, the sample speaker voiceprint features corresponding to the 8th to 9th sample speech frames are the voiceprint features of speaker B, and the sample speaker voiceprint features corresponding to the 10th to 11th sample speech frames are the voiceprint features of speaker C.

[0196] In step S702 of some embodiments, the sample pairs include a positive sample pair consisting of a sample speech frame and a sample speaker voiceprint feature corresponding to the sample speech frame, and a negative sample pair consisting of a sample speech frame and a sample speaker voiceprint feature not corresponding to the sample speech frame.

[0197] In the specific implementation of this embodiment, when constructing sample pairs, the sample speech frame and the sample speaker voiceprint features corresponding to the sample speech frame are constructed into a positive sample pair, and the sample speech frame and the sample speaker voiceprint features of other sample speech frames are constructed into a negative sample pair.

[0198] In step S703 of some embodiments, first, the positive sample pair is input into a preset second original network. Then, the speaker recognition is performed on the sample speech frame in the positive sample pair through the second original network to obtain the first predicted voiceprint feature corresponding to the sample speech frame, and the specific process is similar to the above steps S401-S402. Further, when determining the first similarity between the first predicted voiceprint feature and the sample speaker voiceprint feature in the positive sample pair, the process is similar to the above step S502. To save space, it will not be repeated.

[0199] In step S704 of some embodiments, the specific process of step S704 is similar to the above step S703, and will not be described in detail to save space.

[0200] In step S705 of some embodiments, first, for each sample speech frame, an exponential calculation is performed with a natural constant as the base and a first similarity as the exponent to obtain a first value, and then an exponential calculation is performed with a natural constant as the base and a second similarity as the exponent to obtain a second value, and the first value and the sum of multiple second values ​​are used as the denominator, and the first value is used as the numerator to obtain a first fraction. Further, the negative logarithm of the first fraction of all sample speech frames is summed to obtain the speaker loss function of the second original network in the current training round. Further, the second original network is trained according to the speaker loss function to obtain a speaker recognition network, and the specific process is similar to the above step S605. To save space, it will not be repeated.

[0201] Among them, the speaker loss function of the second original network in the current training round can be expressed as shown in formula (5):

[0202]

[0203] Among them, L spk is the speaker loss function. u is the character index of the text character (i.e., speaker ID), and is the first predicted voiceprint feature e corresponding to the text character (sample speech frame) whose speaker ID is u. u , is the sample speaker voiceprint feature d corresponding to the text character (sample speech frame) whose speaker identifier is u u . ′ The sample speaker voiceprint features that do not correspond to the text characters (sample speech frames) whose speaker identifier is u, Φ(d) is the space composed of the speaker voiceprint features of all reference speakers. cos(e u ,d u ) is the first similarity, is the second similarity. u It refers to the speaker voiceprint features (output vector set, including sub-voiceprint features corresponding to each text character) output by the second original network for the speech frame sequence corresponding to the current text character u. is the first score, is the first value. <cc>Refers to a plural set.

[0204] The advantage of this embodiment is that it takes minimizing the speaker classification error corresponding to the sample speech frame, that is, maximizing the correct probability of speaker prediction of the sample frame speech as the training goal, introduces a comparative training method to train the speaker network of the speech recognition model, can effectively handle the speaker changes in multi-speaker scenarios, and provide more accurate speaker recognition results, thereby improving the accuracy of speaker recognition of the speech recognition model.

[0205] See also Figure 8 The present application also provides a speech recognition device that can implement the above-mentioned speech recognition method. The device includes:

[0206] An acquisition unit 801 is used to acquire target speech data containing multiple speakers;

[0207] The encoding unit 802 is used to encode the target speech data based on a preset speech recognition model to obtain speech coding features, wherein the speech recognition model includes a speech recognition network and a speaker recognition network;

[0208] The content recognition unit 803 is used to perform content recognition on the speech coding features based on the speech recognition network to obtain the speech text content corresponding to the target speech data;

[0209] The speaker recognition unit 804 is used to perform speaker recognition on the target speech data and speech coding features based on the speaker recognition network to obtain the speaker voiceprint features corresponding to the target speech data;

[0210] The character classification unit 805 is used to classify each text character in the speech text content based on the speaker's voiceprint features to obtain the target speech content generated by each speaker, wherein the target speech content is composed of multiple text characters.

[0211] In some embodiments, the acquisition unit 801 is used to:

[0212] Obtaining raw speech data containing multiple speakers;

[0213] Performing denoising processing on the original speech data to obtain denoised speech data;

[0214] Perform Fourier transform on the denoised speech data to obtain speech frequency domain data;

[0215] Mel filtering is performed on the speech frequency domain data to obtain the target speech data.

[0216] In some embodiments, the speech recognition network includes a first decoder, a second decoder, a third decoder, and a feature transformation network;

[0217] The content identification unit 803 is used to:

[0218] Performing feature alignment on the speech coding features based on the first decoder to obtain first speech decoding features;

[0219] Performing phoneme feature recognition on the speech coding feature based on the second decoder to obtain a second speech decoding feature;

[0220] Performing feature mapping on the speech coding feature based on the third decoder to obtain a third speech decoding feature;

[0221] Performing feature aggregation on the first speech decoding feature, the second speech decoding feature, and the third speech decoding feature to obtain a speech aggregation feature;

[0222] The speech aggregation features are transformed based on the feature transformation network to obtain the speech text content.

[0223] In some embodiments, the speaker recognition network includes a speaker encoder and a speaker decoder;

[0224] The speaker recognition unit 804 is used for:

[0225] Perform feature concatenation on the target speech data and the speech coding features to obtain target concatenated features;

[0226] Based on the speaker encoder, speaker features are extracted from the target concatenated features to obtain speaker encoding features;

[0227] Speaker recognition is performed on the speaker encoding features based on the speaker decoder to obtain the speaker voiceprint features.

[0228] In some embodiments, the speaker voiceprint feature includes sub-voiceprint features corresponding to each text character in the speech text content;

[0229] The character classification unit 805 is used to:

[0230] Obtain reference voiceprint features and reference object identifiers of each reference speaking object;

[0231] For each text character, the voiceprint similarity between the sub-voiceprint feature and the reference voiceprint feature is calculated to obtain the voiceprint similarity between the sub-voiceprint feature and each reference voiceprint feature;

[0232] The reference object identifier corresponding to the reference voiceprint feature with the greatest voiceprint similarity is used as the speaker identifier of the text character, wherein the speaker identifier is used to indicate the identity of the speaker;

[0233] Based on the speaker identifiers of the respective text characters, the text characters having the same speaker identifier are divided into the same text character group;

[0234] Based on the text character groups, the target speech content produced by each speaker is determined.

[0235] In some embodiments, the speech recognition network is trained by the following process:

[0236] Obtaining sample audio data and annotated text data corresponding to the sample audio data;

[0237] Performing feature alignment on sample audio encoding features of the sample audio data based on a preset first original network to obtain a first sample decoding feature, and determining a first probability of generating annotated text data based on the first sample decoding feature;

[0238] Performing phoneme feature recognition on the sample audio encoding feature based on the first original network to obtain a second sample decoding feature, and determining a second probability of generating annotated text data based on the second sample decoding feature;

[0239] Performing feature mapping on the sample audio encoding feature based on the first original network to obtain a third sample decoding feature, and determining a third probability of generating annotated text data based on the third sample decoding feature;

[0240] Based on the first probability, the second probability and the third probability, the first original recognition network is trained to obtain a speech recognition network.

[0241] In some embodiments, the speaker recognition network is trained by the following process:

[0242] Acquire multiple sample speech frames and sample speaker voiceprint features corresponding to each sample speech frame;

[0243] Constructing sample pairs based on sample speech frames and sample speaker voiceprint features, the sample pairs including positive sample pairs consisting of sample speech frames and sample speaker voiceprint features corresponding to the sample speech frames, and negative sample pairs consisting of sample speech frames and sample speaker voiceprint features not corresponding to the sample speech frames;

[0244] Inputting the positive sample pair into a preset second original network for speaker recognition to obtain a first predicted voiceprint feature, and determining a first similarity between the first predicted voiceprint feature and the voiceprint feature of the sample speaker in the positive sample pair;

[0245] Inputting the negative sample pair into the second original network for speaker recognition to obtain a second predicted voiceprint feature, and determining a second similarity between the second predicted voiceprint feature and the voiceprint feature of the sample speaker in the negative sample pair;

[0246] Based on the first similarities and the second similarities of the plurality of sample speech frames, the second original network is trained to obtain a speaker recognition network.

[0247] The specific implementation of the speech recognition device is substantially the same as the specific implementation of the speech recognition method described above, and will not be described in detail herein.

[0248] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned speech recognition method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0249] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0250] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0251] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speech recognition method of the embodiment of this application;

[0252] Input / output interface 903, used to implement information input and output;

[0253] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0254] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0255] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0256] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the above-mentioned speech recognition method.

[0257] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0258] The speech recognition method, speech recognition device, electronic device and computer-readable storage medium provided by the embodiment of the present application obtain target speech data containing multiple speakers; encode the target speech data based on a preset speech recognition model to obtain speech coding features, which can convert the original speech signal into a series of speech coding features, and represent the key attributes of the speech signal (such as pitch, sound intensity and timbre, etc.) through the speech coding features. Further, based on the speech recognition network, the speech coding features are subjected to content recognition to obtain the speech text content corresponding to the target speech data, and the speech coding features are mapped to the corresponding speech text content, thereby generating a text representation of the target speech data, which can achieve accurate recognition of the speech content of the target speech data. Further, based on the speaker recognition network, speaker recognition is performed on the target speech data and the speech coding features to obtain the speaker voiceprint features corresponding to the target speech data, which can capture the voiceprint characteristics of the speaker used for the target data. Finally, based on the speaker voiceprint features, each text character in the speech text content is classified to obtain the target speech content generated by each speaker, which can accurately identify and distinguish the target speech content generated by each speaker, and can improve the recognition accuracy of speech containing multiple speakers.

[0259] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0260] It can be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0261] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0262] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0263] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0264] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0265] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0266] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0267] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0268] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0269] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.< / cc>

Claims

1. A speech recognition method, characterized in that: The speech recognition method comprises: Acquire target speech data containing multiple speakers; Encoding the target speech data based on a preset speech recognition model to obtain speech coding features, wherein the speech recognition model includes a speech recognition network and a speaker recognition network; Performing content recognition on the speech coding features based on the speech recognition network to obtain speech text content corresponding to the target speech data; Performing speaker recognition on the target voice data and the voice coding features based on the speaker recognition network to obtain speaker voiceprint features corresponding to the target voice data; Based on the voiceprint features of the speaker, each text character in the voice text content is classified to obtain the target voice content generated by each speaker, wherein the target voice content is composed of a plurality of the text characters.

2. The method according to claim 1, characterized in that The speech recognition network includes a first decoder, a second decoder, a third decoder and a feature transformation network; The performing content recognition on the speech coding feature based on the speech recognition network to obtain speech text content corresponding to the target speech data includes: Performing feature alignment on the speech coding features based on the first decoder to obtain first speech decoding features; Performing phoneme feature recognition on the speech coding feature based on the second decoder to obtain a second speech decoding feature; Performing feature mapping on the speech coding feature based on the third decoder to obtain a third speech decoding feature; Performing feature aggregation on the first speech decoding feature, the second speech decoding feature, and the third speech decoding feature to obtain a speech aggregation feature; The speech aggregation features are subjected to feature transformation based on the feature transformation network to obtain the speech text content.

3. The method according to claim 1, characterized in that The speaker recognition network includes a speaker encoder and a speaker decoder; The performing speaker recognition on the target voice data and the voice coding features based on the speaker recognition network to obtain the speaker voiceprint features corresponding to the target voice data includes: Performing feature splicing on the target speech data and the speech coding feature to obtain a target splicing feature; Performing speaker feature extraction on the target concatenated features based on the speaker encoder to obtain speaker encoding features; Speaker recognition is performed on the speaker encoding features based on the speaker decoder to obtain the speaker voiceprint features.

4. The method according to claim 1, characterized in that: The speaker's voiceprint features include sub-voiceprint features corresponding to each text character in the speech text content; The method of classifying each text character in the speech text content based on the voiceprint feature of the speaker to obtain the target speech content generated by each speaker includes: Obtain reference voiceprint features and reference object identifiers of each reference speaking object; For each of the text characters, calculating the voiceprint similarity between the sub-voiceprint feature and the reference voiceprint feature to obtain the voiceprint similarity between the sub-voiceprint feature and each of the reference voiceprint features; Using the reference object identifier corresponding to the reference voiceprint feature with the greatest voiceprint similarity as the speaker identifier of the text character, wherein the speaker identifier is used to indicate the identity of the speaker; Based on the speaker identifiers of the text characters, the text characters having the same speaker identifier are divided into the same text character group; Based on the text character group, target speech content produced by each of the speakers is determined.

5. The method according to claim 1, characterized in that The speech recognition network is trained by the following process: Acquire sample audio data and annotated text data corresponding to the sample audio data; Performing feature alignment on sample audio encoding features of the sample audio data based on a preset first original network to obtain a first sample decoding feature, and determining a first probability of generating the annotated text data based on the first sample decoding feature; Performing phoneme feature recognition on the sample audio encoding feature based on the first original network to obtain a second sample decoding feature, and determining a second probability of generating the annotated text data based on the second sample decoding feature; Performing feature mapping on the sample audio encoding feature based on the first original network to obtain a third sample decoding feature, and determining a third probability of generating the annotated text data based on the third sample decoding feature; Based on the first probability, the second probability and the third probability, the first original recognition network is trained to obtain the speech recognition network.

6. The method according to claim 1, characterized in that The speaker recognition network is trained by the following process: Acquire a plurality of sample speech frames and a sample speaker voiceprint feature corresponding to each of the sample speech frames; constructing sample pairs based on the sample speech frame and the sample speaker voiceprint features, the sample pairs comprising a positive sample pair consisting of the sample speech frame and the sample speaker voiceprint features corresponding to the sample speech frame, and a negative sample pair consisting of the sample speech frame and the sample speaker voiceprint features not corresponding to the sample speech frame; Inputting the positive sample pair into a preset second original network for speaker recognition to obtain a first predicted voiceprint feature, and determining a first similarity between the first predicted voiceprint feature and the voiceprint feature of the sample speaker in the positive sample pair; Inputting the negative sample pair into the second original network for speaker recognition to obtain a second predicted voiceprint feature, and determining a second similarity between the second predicted voiceprint feature and the voiceprint feature of the sample speaker in the negative sample pair; Based on the first similarity and the second similarity of a plurality of the sample speech frames, the second original network is trained to obtain the speaker recognition network.

7. The method according to any one of claims 1 to 6, characterized in that: The obtaining of target speech data containing multiple speakers includes: Obtaining raw speech data containing multiple speakers; Performing denoising processing on the original speech data to obtain denoised speech data; Performing Fourier transform on the denoised speech data to obtain speech frequency domain data; Mel filtering is performed on the speech frequency domain data to obtain the target speech data.

8. A speech recognition device, characterized in that: The speech recognition device comprises: An acquisition unit, used for acquiring target speech data containing multiple speakers; An encoding unit, configured to encode the target speech data based on a preset speech recognition model to obtain speech coding features, wherein the speech recognition model includes a speech recognition network and a speaker recognition network; A content recognition unit, configured to perform content recognition on the speech coding features based on the speech recognition network to obtain speech text content corresponding to the target speech data; A speaker recognition unit, configured to perform speaker recognition on the target speech data and the speech coding features based on the speaker recognition network to obtain a speaker voiceprint feature corresponding to the target speech data; The character classification unit is used to classify each text character in the speech text content based on the voiceprint features of the speaker to obtain the target speech content generated by each speaker, wherein the target speech content is composed of a plurality of the text characters.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech recognition method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech recognition method according to any one of claims 1 to 7 is implemented.

Images