Speech recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides a speech recognition method and device, an electronic device and a storage medium, and belongs to the technical field of financial technology. The method comprises the following steps: obtaining target speech data containing multiple speakers; performing encoding processing on the target speech data based on a speech recognition model to obtain speech coding features, wherein the speech recognition model comprises a speech recognition network and a speaker recognition network; performing content recognition on the speech coding features based on the speech recognition network to obtain speech text content corresponding to the target speech data; performing speaker recognition on the target speech data and the speech coding features based on the speaker recognition network to obtain speaker voiceprint features corresponding to the target speech data; and classifying each text character in the speech text content based on the speaker voiceprint features to obtain target speech content generated by each speaker, wherein the target speech content is composed of multiple text characters. The application can improve the recognition accuracy of speech containing multiple speakers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial technology, and particularly relates to a speech recognition method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In the field of modern communication and data processing, accurately recognizing and processing target speech data containing multiple speakers is an important task. For example, in a financial business process, multiple-speaker speech data generated by a business personnel communicating with multiple business objects is often needed to accurately identify the speech content described by each speaker.

[0003] To meet this demand, the speech recognition method in the related art usually relies on complex algorithms and models to extract useful information from speech signals. However, when the target speech data contains multiple speakers, the traditional speech recognition system often has difficulty in distinguishing and recognizing the voice features of each speaker, which can lead to a decrease in speech recognition accuracy. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a speech recognition method and device, electronic equipment and a storage medium, which aims to improve the recognition accuracy of speech containing multiple speakers.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application provides a speech recognition method, which comprises:

[0006] obtaining target speech data containing multiple speakers;

[0007] encoding and processing the target speech data based on a preset speech recognition model to obtain speech coding features, wherein the speech recognition model comprises a speech recognition network and a speaker recognition network;

[0008] performing content recognition on the speech coding features based on the speech recognition network to obtain speech text content corresponding to the target speech data;

[0009] performing speaker recognition on the target speech data and the speech coding features based on the speaker recognition network to obtain speaker voiceprint features corresponding to the target speech data;

[0010] classifying each text character in the speech text content based on the speaker voiceprint features to obtain target speech content generated by each speaker, wherein the target speech content is composed of multiple text characters.

[0011] In some embodiments, the speech recognition network comprises a first decoder, a second decoder, a third decoder, and a feature transformation network;

[0012] The content recognition of the speech coding features based on the speech recognition network comprises:

[0013] The feature alignment of the speech coding features based on the first decoder comprises a first speech decoding feature;

[0014] The phoneme feature recognition of the speech coding features based on the second decoder comprises a second speech decoding feature;

[0015] The feature mapping of the speech coding features based on the third decoder comprises a third speech decoding feature;

[0016] The feature aggregation of the first speech decoding feature, the second speech decoding feature, and the third speech decoding feature comprises a speech aggregation feature;

[0017] The feature transformation of the speech aggregation feature based on the feature transformation network comprises the speech text content.

[0018] In some embodiments, the speaker recognition network comprises a speaker encoder and a speaker decoder;

[0019] The speaker recognition of the target speech data and the speech coding features based on the speaker recognition network comprises a speaker voiceprint feature corresponding to the target speech data, which comprises:

[0020] The feature splicing of the target speech data and the speech coding features comprises a target splicing feature;

[0021] The speaker feature extraction of the target splicing feature based on the speaker encoder comprises a speaker coding feature;

[0022] The speaker recognition of the speaker coding feature based on the speaker decoder comprises the speaker voiceprint feature.

[0023] In some embodiments, the speaker voiceprint feature comprises a sub-voiceprint feature corresponding to each text character in the speech text content;

[0024] The classification of each text character in the speech text content based on the speaker voiceprint feature comprises target speech content produced by each speaker, which comprises:

[0025] The reference voiceprint features and reference object identifiers of each reference speaker object are obtained;

[0026] performing a voiceprint similarity calculation on the sub voiceprint feature and the reference voiceprint feature to obtain a voiceprint similarity between the sub voiceprint feature and each reference voiceprint feature;

[0027] identifying the reference object corresponding to the reference voiceprint feature with the largest voiceprint similarity as a speaker identifier of the text character, wherein the speaker identifier is used to indicate the identity of the speaker;

[0028] based on the speaker identifier of each text character, dividing the text characters with the same speaker identifier into the same text character group;

[0029] based on the text character group, determining a target speech content generated by each speaker.

[0030] In some embodiments, the speech recognition network is trained by the following process:

[0031] obtaining sample audio data and labeled text data corresponding to the sample audio data;

[0032] performing feature alignment on a sample audio encoding feature of the sample audio data based on a preset first original network to obtain a first sample decoding feature, and determining a first probability of generating the labeled text data based on the first sample decoding feature;

[0033] performing phoneme feature recognition on the sample audio encoding feature based on the first original network to obtain a second sample decoding feature, and determining a second probability of generating the labeled text data based on the second sample decoding feature;

[0034] performing feature mapping on the sample audio encoding feature based on the first original network to obtain a third sample decoding feature, and determining a third probability of generating the labeled text data based on the third sample decoding feature;

[0035] training the first original recognition network based on the first probability, the second probability and the third probability to obtain the speech recognition network.

[0036] In some embodiments, the speaker recognition network is trained by the following process:

[0037] obtaining a plurality of sample speech frames and a sample speaker voiceprint feature corresponding to each sample speech frame;

[0038] constructing a sample pair based on the sample speech frame and the sample speaker voiceprint feature, the sample pair including a positive sample pair constituted by the sample speech frame and the sample speaker voiceprint feature corresponding to the sample speech frame, and a negative sample pair constituted by the sample speech frame and the sample speaker voiceprint feature not corresponding to the sample speech frame;

[0039] inputting the positive sample pair into a preset second original network to perform speaker recognition, obtaining a first predicted voiceprint feature, and determining a first similarity between the first predicted voiceprint feature and the sample speaker voiceprint feature in the positive sample pair;

[0040] inputting the negative sample pair into the second original network to perform speaker recognition, obtaining a second predicted voiceprint feature, and determining a second similarity between the second predicted voiceprint feature and the sample speaker voiceprint feature in the negative sample pair;

[0041] training the second original network based on the first similarity and the second similarity of a plurality of sample speech frames, to obtain the speaker recognition network.

[0042] In some embodiments, the obtaining the target speech data containing a plurality of speakers includes:

[0043] obtaining original speech data containing a plurality of speakers;

[0044] performing denoising processing on the original speech data to obtain denoised speech data;

[0045] performing Fourier transform on the denoised speech data to obtain speech frequency domain data;

[0046] performing Mel filtering on the speech frequency domain data to obtain the target speech data.

[0047] To achieve the above object, a second aspect of the embodiment of the present application proposes a speech recognition device, the device comprising:

[0048] an obtaining unit configured to obtain target speech data containing a plurality of speakers;

[0049] an encoding unit configured to perform encoding processing on the target speech data based on a preset speech recognition model to obtain speech encoding features, wherein the speech recognition model includes a speech recognition network and a speaker recognition network;

[0050] a content recognition unit configured to perform content recognition on the speech encoding features based on the speech recognition network to obtain speech text content corresponding to the target speech data;

[0051] The speaker recognition unit is configured to perform speaker recognition on the target voice data and the speech coding features based on the speaker recognition network, to obtain speaker voiceprint features corresponding to the target voice data.

[0052] The character classification unit is configured to classify each text character in the speech text content based on the speaker voiceprint features, to obtain target speech content generated by each speaker, wherein the target speech content is composed of a plurality of text characters.

[0053] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the speech recognition method of the first aspect when executing the computer program.

[0054] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the speech recognition method of the first aspect.

[0055] The speech recognition method and device, electronic device and storage medium provided by the present application can obtain target voice data containing a plurality of speakers, perform coding processing on the target voice data based on a preset speech recognition model to obtain speech coding features, convert the original voice signal into a series of speech coding features, and represent the key attributes (such as pitch, intensity and timbre) of the voice signal through the speech coding features. Further, the speech recognition network is used to perform content recognition on the speech coding features to obtain speech text content corresponding to the target voice data, map the speech coding features to the corresponding speech text content, thereby generating the text representation of the target voice data, and accurately recognizing the speech content of the target voice data. Further, the speaker recognition network is used to perform speaker recognition on the target voice data and the speech coding features to obtain speaker voiceprint features corresponding to the target voice data, which can capture the voiceprint characteristics of the speaker of the target voice data. Finally, the speaker voiceprint features are used to classify each text character in the speech text content to obtain target speech content generated by each speaker, which can accurately recognize and distinguish the target speech content generated by each speaker, and improve the recognition accuracy of the speech containing multiple speakers. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a flowchart of the speech recognition method provided by the embodiments of the present application;

[0057] Figure 2 is Figure 1 the flowchart of step S101 in

[0058] Figure 3 is a flowchart of step S103 in Figure 1

[0059] Figure 4 is a flowchart of step S104 in Figure 1

[0060] Figure 5 is a flowchart of step S105 in Figure 1

[0061] is a flowchart of a speech recognition network for training a speech recognition model according to an embodiment of the present application; Figure 6

[0062] is a flowchart of a speaker recognition network for training a speech recognition model according to an embodiment of the present application; Figure 7

[0063] is a structural schematic diagram of a speech recognition apparatus according to an embodiment of the present application; Figure 8

[0064] is a hardware structural schematic diagram of an electronic device according to an embodiment of the present application. Figure 9 DETAILED DESCRIPTION

[0065] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0066] It should be noted that although the functional modules are divided in the apparatus schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the apparatus or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0068] The speech recognition method and apparatus, electronic device and storage medium provided by the embodiments of the present application are specifically described by the following embodiments. First, the speech recognition method in the embodiments of the present application is described.

[0069] ​​​The embodiments of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain optimal results.

[0070] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0071] The voice recognition method provided by the embodiments of the present application relates to the field of financial technology and artificial intelligence. The voice recognition method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application for implementing the voice recognition method, but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0073] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the object, such as object information, object behavior data, object historical data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the object, the necessary object-related data for enabling the embodiments of the present application to function normally will be obtained.

[0074] Figure 1 is an optional flowchart of a speech recognition method provided by the embodiments of the present application, Figure 1 The method in can include but is not limited to including steps S101 to S105.

[0075] Step S101, obtaining target speech data containing a plurality of speakers;

[0076] Step S102, performing encoding processing on the target speech data based on a preset speech recognition model to obtain speech encoding features;

[0077] Step S103, performing content recognition on the speech encoding features based on a speech recognition network to obtain speech text content corresponding to the target speech data;

[0078] Step S104, performing speaker recognition on the target speech data and the speech encoding features based on a speaker recognition network to obtain speaker voiceprint features corresponding to the target speech data;

[0079] Step S105, classifying each text character in the speech text content based on the speaker voiceprint features to obtain target speech content generated by each speaker.

[0080] The speech recognition model includes a speech recognition network and a speaker recognition network. The target speech content is composed of a plurality of text characters.

[0081] The steps S101 to S105 shown in the embodiments of the present application can obtain target speech data containing multiple speakers, encode the target speech data based on a preset speech recognition model to obtain speech coding features, convert the original speech signal into a series of speech coding features, and represent the key attributes (such as pitch, intensity and timbre) of the speech signal through the speech coding features. Further, the speech coding features are subjected to content recognition based on a speech recognition network to obtain the speech text content corresponding to the target speech data, map the speech coding features to the corresponding speech text content, thereby generating the text representation of the target speech data, and accurately recognizing the speech content of the target speech data. Further, the target speech data and the speech coding features are subjected to speaker recognition based on a speaker recognition network to obtain the speaker voiceprint features corresponding to the target speech data, and the voiceprint characteristics of the speaker of the target speech data can be captured. Finally, the speaker voiceprint features are used to classify each text character in the speech text content to obtain the target speech content generated by each speaker, accurately recognize and distinguish the target speech content generated by each speaker, and improve the recognition accuracy of the speech containing multiple speakers.

[0082] In step S101 of some embodiments, target speech data containing multiple speakers is obtained.

[0083] The target speech data refers to the mel-frequency cepstrum feature data corresponding to the speech data generated by multiple speakers in a conversation.

[0084] The speaker refers to each object participating in the same conversation.

[0085] For example, in a financial service, multiple objects often communicate together, and the speech data retained for this conversation will record the communication content of multiple objects. For another example, in the field of digital medicine, medical staff from multiple departments hold a medical conference to discuss a medical problem, and the speech data will record the opinions of each object from different departments in the medical conference. Based on this, the specific speaker and specific content of the speech data in the multi-person communication scenario are more complex than those in the one-to-one communication scenario.

[0086] Please refer to Figure 2 In some embodiments, step S101 can include but is not limited to steps S201 to S204:

[0087] Step S201, obtaining original speech data containing multiple speakers;

[0088] Step S202, denoising the original speech data to obtain denoised speech data;

[0089] Step S203, Fourier transform is performed on the de-noised voice data to obtain voice frequency domain data.

[0090] Step S204, Mel filtering is performed on the voice frequency domain data to obtain target voice data.

[0091] The steps S201 to S204 are described in detail as follows.

[0092] In step S201 of some embodiments, the original voice data containing multiple speakers can be extracted from a preset audio database, or downloaded from a network platform or a multimedia device, or obtained by other means, which is not limited. The original voice data containing multiple speakers is waveform data, and the voice change of the original voice data is reflected by the change of the waveform.

[0093] In step S202 of some embodiments, the original voice data is de-noised to eliminate environmental noise in the target audio to obtain de-noised voice data.

[0094] In step S203 of some embodiments, the de-noised voice data (continuous time audio signal) is converted into a discrete time signal, the discrete time signal is truncated and multiplied by a window function to obtain a truncated audio signal. The window function includes but is not limited to Hanning window, Hamming window, etc. Further, the truncated audio signal is Fourier transformed by using fast Fourier transform algorithm to provide frequency components in the truncated audio signal, and the target audio is converted from time domain to frequency domain to obtain voice frequency domain data. The voice frequency domain data is used to represent the amplitude and phase of different frequency components of the de-noised voice data, and is used to indicate the frequency distribution information of the de-noised voice data.

[0095] In step S204 of some embodiments, first, the voice frequency domain data is taken modulo, and the modulo of the voice frequency domain data is evenly divided to obtain target power spectrum. Then, under the condition of authorized permission, the mel filter bank is called, and the target power spectrum is filtered by using the mel filter bank to map the regular frequency to the mel frequency scale to obtain the target audio filtering result. Then, the target audio filtering result is logarithmically compressed to simulate the nonlinear perception of the human ear to the sound intensity to obtain the target audio compression result. Further, the target audio compression result is subjected to discrete cosine transform to obtain a discrete cosine transform result, wherein the discrete cosine transform result has a plurality of coefficients for representing audio information of the denoised voice data. Then, the first predetermined number of coefficients in the discrete cosine transform result are selected as mel cepstrum features. Wherein, the predetermined number is not greater than 13, and the selected coefficients represent important audio information of the target audio. By screening the coefficients in the discrete cosine transform result, redundant information can be reduced. Finally, the selected mel cepstrum features are used as target voice data.

[0096] It should be noted that the voice content of the original voice data and the target voice data is basically the same, and the difference is that the original voice data is time domain data, and the target voice data is frequency domain data.

[0097] The advantage of this embodiment is that by using Fourier transform, mel filtering and discrete mapping operations, the original voice data containing multiple speakers is converted from time domain to frequency domain, and important audio information and important speaker information of the original voice data in the frequency domain are extracted, which can improve the data quality of the target voice data for voice recognition.

[0098] In step S102 of some embodiments, the target voice data is encoded based on a preset voice recognition model to obtain voice encoding features, wherein the voice recognition model includes a voice recognition network and a speaker recognition network.

[0099] The voice recognition model refers to a neural network model that can be used for voice content recognition and speaker recognition of voice data corresponding to multiple speakers, and recognition of voice content corresponding to each speaker.

[0100] The voice recognition network is used to identify the specific voice content of the input voice data and output the voice content text corresponding to the voice data.

[0101] The speaker recognition network is used to identify the specific identity of a speaker contained in the input voice data, and output the specific identity of the speaker and which part of the voice content text is described by each speaker.

[0102] The voice coding feature is used to indicate the audio information representation of the target voice data in the latent vector space.

[0103] In the implementation of this embodiment, the target voice data is encoded by the shared encoder of the preset voice recognition model, and the target voice data is mapped from the frequency domain space to the latent vector space, the voice data in the mel-frequency cepstrum form is converted into a vector form, and the voice coding feature is obtained.

[0104] In step S103 of some embodiments, the voice coding feature is content-identified based on the voice recognition network, and the voice text content corresponding to the target voice data is obtained.

[0105] The voice text content is used to reflect the specific voice content of the target voice data in the form of text. The voice text content can describe what the overall communication content is in the communication of multiple speakers in the form of words (text characters).

[0106] In the embodiments of the present application, the voice recognition network includes a first decoder, a second decoder, a third decoder, and a feature transformation network.

[0107] The first decoder is used to align the irregular time sequence structure of the voice signal in the input voice data, so that each feature (first voice decoding feature) in the output sequence is automatically aligned with each feature (voice coding feature) in the input sequence.

[0108] The second decoder is used to identify important voice information in the input voice data, and identify feature information related to the current phoneme, and output the key phoneme feature corresponding to the voice data.

[0109] The third decoder is used to realize the mapping of the acoustic features of the input voice data to the phoneme features, and convert the voice data into a phoneme feature sequence while preserving the acoustic sequence feature information of the voice data.

[0110] The feature transformation network is used to convert the features in the latent vector space into a text sequence in the form of text, and realize feature decoding.

[0111] Please refer to Figure 3 In some embodiments, step S103 can include but is not limited to steps S301 to S305:

[0112] Step S301, aligning the voice coding feature based on the first decoder to obtain the first voice decoding feature;

[0113] Step S302, performing phoneme feature recognition on the speech coding features based on a second decoder to obtain second speech decoding features;

[0114] Step S303, performing feature mapping on the speech coding features based on a third decoder to obtain third speech decoding features;

[0115] Step S304, performing feature aggregation on the first speech decoding features, the second speech decoding features and the third speech decoding features to obtain speech aggregation features;

[0116] Step S305, performing feature transformation on the speech aggregation features based on a feature transformation network to obtain speech text content.

[0117] The steps S301 to S305 are described in detail as follows.

[0118] In step S301 of some embodiments, the first decoder can be a CTC decoder (Connectionist Temporal Classification Decoder). Specifically, the speech coding features are first mapped to a low-dimensional vector space by a fully connected layer of the first decoder to obtain one-dimensional speech features. Then, since the speech coding features are a feature sequence composed of multiple features, the one-dimensional speech features are subjected to phoneme probability calculation by a classification function of the first decoder to obtain the posterior probability of each feature in the speech coding features belonging to each candidate phoneme in a preset phoneme table. The candidate phoneme with the maximum posterior probability is taken as the target phoneme corresponding to the feature, and multiple target phonemes are concatenated into a phoneme sequence according to the feature order of each feature in the speech coding features to obtain the first speech decoding features. The first speech decoding features are a phoneme sequence completely aligned with the speech coding features.

[0119] In step S302 of some embodiments, the second decoder can be an attention mechanism-based decoder. Specifically, the speech coding features are linearly projected by a query channel of the second decoder to obtain a query vector, linearly projected by a key channel of the second decoder to obtain a key vector, and linearly projected by a value channel of the second decoder to obtain a value vector, wherein the projection parameters of the key channel, the query channel and the value channel are different. Further, the query vector is multiplied by the transpose of the key vector to obtain a product result, and the product result is divided by the square difference of the feature dimension of the key vector to obtain a division result. Finally, the division result is normalized to obtain an attention weight, and the attention weight is multiplied by the value vector to obtain the second speech decoding features. The second speech decoding features are used to indicate the key phoneme feature information corresponding to the target speech data.

[0120] In step S303 of some embodiments, the third decoder can be a RNN-T decoder (Recurrent Neural Network Transducer). Specifically, the acoustic feature information in the speech coding features is extracted and converted by the third decoder, and the speech coding features are converted into a phoneme feature sequence while retaining the acoustic sequence feature information of the target speech data, and the converted phoneme feature sequence is taken as the third speech decoding feature. The third speech decoding feature is used to indicate the overall phoneme feature information corresponding to the target speech data.

[0121] In step S304 of some embodiments, when the first speech decoding feature, the second speech decoding feature and the third speech decoding feature are aggregated, the first speech decoding feature, the second speech decoding feature and the third speech decoding feature are spliced into a longer feature vector to obtain a speech aggregation feature. The speech aggregation feature contains both the key speech information in the target speech data and the overall speech information in the target speech data. At the same time, the speech aggregation feature is substantially aligned with the target speech data (Mel frequency cepstrum feature).

[0122] In step S305 of some embodiments, the feature transformation network can be a neural network based on a transformer structure. Specifically, when the speech aggregation feature is transformed based on the feature transformation network, for each phoneme feature in the speech aggregation feature, attention calculation is first performed on the phoneme feature, and then classification probability calculation is performed on the attention calculation result of each phoneme feature according to a preset character table to determine the probability of each phoneme feature being each candidate character in the preset word table. The candidate character with the largest probability is taken as the predicted text character corresponding to the phoneme feature. Further, according to the feature sequence of each phoneme feature in the speech aggregation feature, the text characters corresponding to each phoneme feature are combined into a character sequence, and the character sequence is taken as the speech text content corresponding to the target speech data.

[0123] The advantage of this embodiment is that, in speech recognition, the joint decoding of the speech coding features by the first decoder, the second decoder and the third decoder can achieve automatic alignment of the generated speech aggregation feature (phoneme feature / character feature) and the original target speech data (acoustic feature) at the feature level, and can also achieve accurate extraction of the key feature information and complete feature information of the target speech data. When the feature decoding is performed by the feature transformation network, the text can be generated according to the overall phoneme feature information and the key local phoneme feature information of the target speech data, thereby improving the content accuracy and content integrity of the speech text content generated in speech recognition.

[0124] In step S104 of some embodiments, the speaker recognition network is used to perform speaker recognition on the target speech data and the speech coding features, to obtain a speaker voiceprint feature corresponding to the target speech data.

[0125] The speaker voiceprint feature is used to indicate the voiceprint characteristics of each speaker in the target speech data.

[0126] In the embodiments of the present application, the speaker recognition network includes a speaker encoder and a speaker decoder.

[0127] The speaker encoder is used to extract speaker-related feature information in the target speech data, wherein the speaker-related feature information includes but is not limited to the speaker's accent, voice frequency, timbre, etc.

[0128] The speaker decoder is used to project the speaker-related feature information extracted by the speaker encoder to a specific vector space, to obtain speaker embedding information corresponding to the target speech data. The speaker embedding information is used to indicate the representation of the voiceprint characteristics of each speaker in the target speech data in the specific vector space.

[0129] It should be noted that the speaker encoder and the speaker decoder in the embodiments of the present application can be neural network structures composed of multiple self-attention mechanisms and feedforward neural networks.

[0130] Please refer to Figure 4 In some embodiments, step S104 can include but is not limited to steps S401 to S403:

[0131] Step S401, performing feature splicing on the target speech data and the speech coding features to obtain target spliced features;

[0132] Step S402, performing speaker feature extraction on the target spliced features based on the speaker encoder to obtain speaker coding features;

[0133] Step S403, performing speaker recognition on the speaker coding features based on the speaker decoder to obtain a speaker voiceprint feature.

[0134] The steps S401 to S403 are described in detail as follows.

[0135] In step S401 of some embodiments, since the target speech data is in the form of mel-frequency cepstral coefficients, the speech coding feature is in the form of vector feature. Based on this, the target speech data and the speech coding feature are first mapped to the same feature dimension vector space to obtain the variable dimension speech feature corresponding to the target speech data and the variable dimension coding feature corresponding to the speech coding feature. Then, the variable dimension speech feature and the variable dimension coding feature in the same feature dimension are spliced into a longer feature vector to obtain the target splicing feature.

[0136] In step S402 of some embodiments, first, the target splicing feature is linearly projected by the query channel of the speaker encoder to obtain a query vector, linearly projected by the key channel of the speaker encoder to obtain a key vector, and linearly projected by the value channel of the speaker encoder to obtain a value vector, wherein the projection parameters of the key channel, the query channel and the value channel are different. Further, the query vector is multiplied by the transpose of the key vector to obtain a product result, and the product result is divided by the square difference of the feature dimension of the key vector to obtain a division result. Finally, the division result is normalized to obtain an attention weight, and the attention weight is multiplied by the value vector to obtain a speaker attention feature. The speaker attention feature is used to indicate the key feature information related to the speaker in the target speech data. Then, the speaker attention feature is nonlinearly transformed to realize feature mapping of the speaker attention feature to obtain a speaker encoding feature.

[0137] In step S403 of some embodiments, first, the speaker encoding feature is calculated by the speaker decoder to obtain a speaker representation feature, and the specific process is similar to the above step S402. To save space, it will not be described again. Then, the speaker representation feature is embedded by the speaker decoder to obtain a speaker voiceprint feature.

[0138] The advantage of this embodiment is that when performing speaker recognition on the target speech data, the speaker encoding feature can be obtained by using the speaker encoder to perform attention calculation and feedforward processing on the target splicing feature, which can capture the long-distance dependency relationship between various speech features in the target splicing feature, learn the complex speech information in the target speech data, and thus improve the feature accuracy and information effectiveness of the extracted speaker encoding feature. Further, the speaker voiceprint feature information in the speaker encoding feature can be obtained more conveniently by converting the speaker encoding feature into an embedded form using the speaker decoder, thereby improving the feature acquisition efficiency of the speaker voiceprint feature corresponding to the target speech data.

[0139] In step S105 of some embodiments, based on the speaker voiceprint features, each text character in the speech text content is classified to obtain target speech content generated by each speaker, wherein the target speech content is composed of multiple text characters.

[0140] The target speech content is used to indicate specific content expressed by a single speaker in the target speech data.

[0141] The text character refers to a text representation of a phoneme feature corresponding to each feature in the target speech data.

[0142] In the embodiments of the present application, the speaker voiceprint features include sub-voiceprint features corresponding to each text character in the speech text content.

[0143] The sub-voiceprint feature is used to indicate the voiceprint characteristics of the speaker expressing the speech content of a single text character in the target speech data. Since each feature in the target speech data corresponds to a phoneme feature, each phoneme feature corresponds to a text character, and each feature in the target speech data also corresponds to a speaker voiceprint feature. Therefore, according to the series of mapping relationships, it can be determined that each text character has a corresponding speaker voiceprint feature (sub-voiceprint feature).

[0144] Please refer to Figure 5 In some embodiments, step S105 can include but is not limited to steps S501 to S505:

[0145] Step S501, obtaining reference voiceprint features of each reference speaker object and reference object identifiers;

[0146] Step S502, for each text character, performing voiceprint similarity calculation on the sub-voiceprint feature and the reference voiceprint feature to obtain voiceprint similarity of the sub-voiceprint feature and each reference voiceprint feature;

[0147] Step S503, taking the reference object identifier corresponding to the reference voiceprint feature with the largest voiceprint similarity as the speaker identifier of the text character, wherein the speaker identifier is used to indicate the identity of the speaker;

[0148] Step S504, based on the speaker identifier of each text character, dividing text characters with the same speaker identifier into the same text character group;

[0149] Step S505, based on the text character group, determining target speech content generated by each speaker.

[0150] The steps S501 to S505 are described in detail below.

[0151] In step S501 of some embodiments, the reference speaker object refers to each object registered on a network platform or an application to which the speech recognition method is applied, and the reference voiceprint feature refers to the voiceprint feature extracted from the speech provided by the reference speaker object during registration. The reference voiceprint feature can indicate the identity of the reference speaker object from the voiceprint perspective. The reference object identifier is used to identify the identity of the reference speaker object, and the reference voiceprint features and the reference object identifiers of different reference speaker objects are different.

[0152] In the implementation of this embodiment, the reference voiceprint features and the reference object identifiers of each reference speaker object are collected and recorded in the server background when the reference speaker object is registered. Based on this, the reference voiceprint features and the reference object identifiers of each reference speaker object can be extracted from the server background log under the condition of authorized permission.

[0153] In step S502 of some embodiments, for each text character, the cosine similarity algorithm is used to perform voiceprint similarity calculation on the sub-voiceprint feature and the reference voiceprint feature, to obtain the voiceprint similarity between the sub-voiceprint feature and each reference voiceprint feature. The voiceprint similarity is used to indicate the voiceprint closeness between the sub-voiceprint feature and each reference voiceprint feature.

[0154] In step S503 of some embodiments, the greater the voiceprint similarity, the closer the sub-voiceprint feature and the reference voiceprint feature. Based on this, the reference voiceprint feature with the greatest voiceprint similarity is considered to come from the same speaker as the sub-voiceprint feature. Therefore, the reference object identifier corresponding to the reference voiceprint feature with the greatest voiceprint similarity is taken as the speaker identifier of the text character, and the speaker identifier is used to identify which reference speaker object the text character comes from.

[0155] In step S504 of some embodiments, since the text characters with the same speaker identifier are generated by the same speaker, based on this, the speaker identifiers of each text character are compared, and the text characters with the same speaker identifier are divided into the same text character group.

[0156] In step S505 of some embodiments, for each text character group, each text character of each text character group is combined and spliced to form a plurality of complete sentences, thereby obtaining the target speech content generated by each speaker. One text character group corresponds to the target speech content generated by one speaker.

[0157] The advantage of this embodiment is that the sub-voiceprint features of each text character are compared with the reference voiceprint features of each reference speaker collected in advance, and the reference object corresponding to the reference voiceprint feature with the highest voiceprint similarity is taken as the speaker identification of the text character, which can realize the conversion of the speech signal into text output and the identification of different speakers, so as to realize speech recognition, quickly switch between continuous speech inputs and accurately identify the text content corresponding to each speech signal and the real identity of the speaker producing the speech signal, thereby improving the speech recognition accuracy and comprehensiveness of speech recognition for multi-speaker speech.

[0158] Please refer to Figure 6 In some embodiments, the speech recognition network of the speech recognition model is trained by the following process:

[0159] In step S601, sample audio data and labeled text data corresponding to the sample audio data are obtained.

[0160] In step S602, the sample audio encoding features of the sample audio data are aligned based on a preset first original network to obtain first sample decoding features, and a first probability of generating the labeled text data based on the first sample decoding features is determined.

[0161] In step S603, phoneme feature recognition is performed on the sample audio encoding features based on the first original network to obtain second sample decoding features, and a second probability of generating the labeled text data based on the second sample decoding features is determined.

[0162] In step S604, feature mapping is performed on the sample audio encoding features based on the first original network to obtain third sample decoding features, and a third probability of generating the labeled text data based on the third sample decoding features is determined.

[0163] In step S605, the first original recognition network is trained based on the first probability, the second probability and the third probability to obtain the speech recognition network.

[0164] The steps S601 to S605 are described in detail below.

[0165] In step S601 of some embodiments, the sample audio data and the labeled text data corresponding to the sample audio data can be obtained from a preset audio database, or downloaded from a network platform or a multimedia device, or obtained by other means, without being limited thereto.

[0166] In step S602 of some embodiments, the first sample decoding feature is obtained by performing feature alignment on the sample audio encoding feature based on the preset first original network. The specific process is similar to step S301 described above. For brevity, details are not described herein.

[0167] Further, when determining the first probability of generating the labeled text data based on the first sample decoding feature, first, the first sample decoding feature is decoded to convert the first sample decoding feature from a feature sequence to a text sequence to obtain predicted text data corresponding to the sample audio data. The specific process is similar to step S305 described above. For brevity, details are not described herein.

[0168] Further, the predicted text data and the labeled text data are compared to calculate the similarity between the predicted text data and the labeled text data to obtain a text similarity, and the text similarity is taken as the first probability.

[0169] In step S603 of some embodiments, the second sample decoding feature is obtained by performing phoneme feature recognition on the sample audio encoding feature based on the first original network. The specific process is similar to step S302 described above. For brevity, details are not described herein.

[0170] Further, the specific process of determining the second probability of generating the labeled text data based on the second sample decoding feature is similar to the specific process of determining the first probability of generating the labeled text data based on the first sample decoding feature described above. For brevity, details are not described herein.

[0171] In step S604 of some embodiments, the third sample decoding feature is obtained by performing feature mapping on the sample audio encoding feature based on the first original network. The specific process is similar to step S303 described above. For brevity, details are not described herein.

[0172] Further, the specific process of determining the third probability of generating the labeled text data based on the third sample decoding feature is similar to the specific process of determining the first probability of generating the labeled text data based on the first sample decoding feature described above. For brevity, details are not described herein.

[0173] In step S605 of some embodiments, first, a first loss function is calculated based on the negative logarithm of the first probability. Then, a second loss function is calculated based on the negative logarithm of the second probability. Further, a third loss function is calculated based on the negative logarithm of the third probability. Then, a total loss function is obtained according to the weighted sum of the first loss function, the second loss function and the third loss function. Finally, the model parameters of the first original network are updated according to the output value of the total loss function at the current training round, and the above steps S601 to S605 are repeated to realize the iterative training of the first original network, until in a certain training round, the total loss function is less than a preset loss threshold, then the model parameters of the training round are taken as the final model parameters, and the first original network with the final model parameters is taken as the trained speech recognition network.

[0174] wherein the first loss function of the first original network at the current training round can be represented as formula (1):

[0175] L ctc =-logP ctc (Y|X) formula (1);

[0176] wherein L ctc is the first loss function, X refers to the sample audio data, Y refers to the labeled text data, and P ctc (Y|X) is the first probability of obtaining the labeled text data according to the output result of the first decoder by inputting the sample audio data into the first decoder.

[0177] The second loss function can be represented as formula (2):

[0178] L att =-logP att (Y|X) formula (2);

[0179] wherein L att is the second loss function, X refers to the sample audio data, Y refers to the labeled text data, and P att (Y|X) is the second probability of obtaining the labeled text data according to the output result of the second decoder by inputting the sample audio data into the second decoder.

[0180] The third loss function can be represented as formula (3):

[0181] L rnnt =-logP rnnt (Y|X) formula (3);

[0182] wherein L rnnt is the third loss function, X refers to the sample audio data, Y refers to the labeled text data, and P rnnt(Y|X) is a third probability of obtaining the labeled text data according to an output result of the third decoder by inputting the sample audio data into the third decoder.

[0183] The total loss function can be represented as shown in formula (4):

[0184] L = λ ctc L ctc + λ att L att + λ rnnt L rnnt Formula (4);

[0185] Wherein, L is a total loss function, λ ctc is a training weight of the first decoder, λ att is a training weight of the second decoder, and λ rnnt is a training weight of the third decoder. λ ctc , λ att , λ rmnt The specific values of λ

[0186] The advantage of this embodiment is that when training the speech recognition network of the speech recognition model, three different decoders (the first decoder CTC, the second decoder Attention, and the third decoder RNN-T) are 3D jointly modeled, and the performance of automatic speech recognition is improved through common training and optimization. During the training process, the model can simultaneously learn how to align the input and output sequences, focus on important parts, and decode using effective acoustic features. This joint modeling method helps improve the performance of the speech recognition system and enables accurate speech recognition in the case of irregular timing and complex background noise. This training method integrates the advantages and characteristics of different decoders, better handles overlapping speech, error correction, and task weighting, and thus improves the accuracy and robustness of the speech recognition model.

[0187] Please refer to Figure 7 In some embodiments, the speaker recognition network of the speech recognition model is trained by the following process:

[0188] Step S701, obtaining a plurality of sample speech frames and sample speaker voiceprint features corresponding to each sample speech frame;

[0189] Step S702, constructing a sample pair based on the sample speech frames and the sample speaker voiceprint features;

[0190] Step S703, inputting the positive sample pair into the preset second original network to perform speaker recognition, obtaining a first predicted voiceprint feature, and determining a first similarity between the first predicted voiceprint feature and the sample speaker voiceprint feature in the positive sample pair.

[0191] Step S704, inputting the negative sample pair into the second original network to perform speaker recognition, obtaining a second predicted voiceprint feature, and determining a second similarity between the second predicted voiceprint feature and the sample speaker voiceprint feature in the negative sample pair.

[0192] Step S705, training the second original network based on the first similarity and the second similarity of the plurality of sample speech frames, to obtain a speaker recognition network.

[0193] The steps S701 to S705 are described in detail below.

[0194] In step S701 of some embodiments, when obtaining a plurality of sample speech frames, the sample audio data of step S601 can be obtained first, and the sample audio data is regarded as a sample speech sequence for framing to obtain a plurality of speech frames, wherein each sample speech frame corresponds to a text character.

[0195] For example, a sample audio data is "Speaker A: What is your interest? Speaker B: Skiing; Speaker C: Swimming". Based on this, the speech coding features corresponding to the sample audio data have 11 features, the number of text characters of the text content obtained by the speech recognition network of the speech recognition model is 11, the number of sample speech frames is 11, the sample speaker voiceprint features corresponding to the first to seventh sample speech frames are the voiceprint features of speaker A, the sample speaker voiceprint features corresponding to the eighth to ninth sample speech frames are the voiceprint features of speaker B, and the sample speaker voiceprint features corresponding to the tenth to eleventh sample speech frames are the voiceprint features of speaker C.

[0196] In step S702 of some embodiments, the sample pair includes a positive sample pair composed of a sample speech frame and a sample speaker voiceprint feature corresponding to the sample speech frame, and a negative sample pair composed of a sample speech frame and a sample speaker voiceprint feature not corresponding to the sample speech frame.

[0197] In the implementation of this embodiment, when constructing the sample pair, the sample speech frame and the sample speaker voiceprint feature corresponding to the sample speech frame are constructed into the positive sample pair, and the sample speech frame and the sample speaker voiceprint feature of other sample speech frames are constructed into the negative sample pair.

[0198] In step S703 of some embodiments, first, the positive sample pair is input to the preset second original network. Then, the sample speech frames in the positive sample pair are subjected to speaker recognition by the second original network to obtain the first predicted voiceprint feature corresponding to the sample speech frames, and the specific process is similar to steps S401-S402 described above. Further, when determining the first similarity of the first predicted voiceprint feature and the sample speaker voiceprint feature in the positive sample pair, the process is similar to step S502 described above. For the sake of brevity, no longer described.

[0199] In step S704 of some embodiments, the specific process of step S704 is similar to step S703 described above. For the sake of brevity, no longer described.

[0200] In step S705 of some embodiments, first, for each sample speech frame, the natural constant is taken as the base number and the first similarity is taken as the exponent to perform exponential calculation to obtain a first value, and then the natural constant is taken as the base number and the second similarity is taken as the exponent to perform exponential calculation to obtain a second value, and the sum of the first value and the plurality of second values is taken as the denominator and the first value is taken as the numerator to obtain a first score. Further, the negative logarithm of the first score of all sample speech frames is summed to obtain the speaker loss function of the second original network at the current training round. Further, the second original network is trained according to the speaker loss function to obtain a speaker recognition network, and the specific process is similar to step S605 described above. For the sake of brevity, no longer described.

[0201] Wherein, the speaker loss function of the second original network at the current training round can be represented as formula (5):

[0202]

[0203] Wherein, L spk is the speaker loss function. u is the character index of the text character (i.e. speaker identification), e u is the first predicted voiceprint feature corresponding to the text character (sample speech frame) with speaker identification u u . d ′ is the sample speaker voiceprint feature corresponding to the text character (sample speech frame) with speaker identification u, and Φ(d) is the space composed of the speaker voiceprint features of all reference speakers. cos(e u ,d u ) is the first similarity, is the second similarity. o u refers to the speaker voiceprint feature (output vector set containing sub-voiceprint features corresponding to each text character) output by the second original network for the current speech frame sequence corresponding to the text character u. is a first fraction, is a first value. <cc>Refers to a complex set.

[0204] The advantage of this embodiment is that the speaker classification error corresponding to the sample speech frame will be minimized, that is, the correct probability of the speaker prediction of the sample frame speech is maximized as the training target, the contrast training method is introduced to train the speaker network of the speech recognition model, which can effectively handle the speaker change in the multi-speaker scene and provide more accurate speaker recognition results, thereby improving the accuracy of the speaker recognition of the speech recognition model.

[0205] Please refer to Figure 8 The embodiment of the application also provides a speech recognition device, which can implement the speech recognition method described above, and the device comprises:

[0206] The acquisition unit 801 is configured to acquire target speech data containing multiple speakers;

[0207] The encoding unit 802 is configured to perform encoding processing on the target speech data based on a preset speech recognition model to obtain speech coding features, wherein the speech recognition model comprises a speech recognition network and a speaker recognition network.

[0208] The content recognition unit 803 is configured to perform content recognition on the speech coding features based on the speech recognition network to obtain speech text content corresponding to the target speech data.

[0209] The speaker recognition unit 804 is configured to perform speaker recognition on the target speech data and the speech coding features based on the speaker recognition network to obtain speaker voiceprint features corresponding to the target speech data.

[0210] The character classification unit 805 is configured to classify each text character in the speech text content based on the speaker voiceprint features to obtain target speech content generated by each speaker, wherein the target speech content is composed of multiple text characters.

[0211] In some embodiments, the acquisition unit 801 is configured to:

[0212] Acquire original speech data containing multiple speakers;

[0213] Perform denoising processing on the original speech data to obtain denoised speech data;

[0214] Perform Fourier transform on the denoised speech data to obtain speech frequency domain data;

[0215] Perform Mel filtering on the speech frequency domain data to obtain the target speech data.

[0216] In some embodiments, the speech recognition network comprises a first decoder, a second decoder, a third decoder and a feature transformation network.

[0217] The content recognition unit 803 is configured to:

[0218] perform feature alignment on the speech coding features based on a first decoder to obtain first speech decoding features;

[0219] perform phoneme feature recognition on the speech coding features based on a second decoder to obtain second speech decoding features;

[0220] perform feature mapping on the speech coding features based on a third decoder to obtain third speech decoding features;

[0221] perform feature aggregation on the first speech decoding features, the second speech decoding features, and the third speech decoding features to obtain speech aggregated features;

[0222] perform feature transformation on the speech aggregated features based on a feature transformation network to obtain speech text content.

[0223] In some embodiments, the speaker recognition network comprises a speaker encoder and a speaker decoder;

[0224] The speaker recognition unit 804 is configured to:

[0225] perform feature splicing on the target speech data and the speech coding features to obtain target spliced features;

[0226] perform speaker feature extraction on the target spliced features based on the speaker encoder to obtain speaker coding features;

[0227] perform speaker recognition on the speaker coding features based on the speaker decoder to obtain speaker voiceprint features.

[0228] In some embodiments, the speaker voiceprint features comprise sub-voiceprint features corresponding to respective text characters in the speech text content;

[0229] The character classification unit 805 is configured to:

[0230] obtain reference voiceprint features of respective reference speaker objects and reference object identifiers;

[0231] for each text character, perform voiceprint similarity calculation on the sub-voiceprint features and the reference voiceprint features to obtain voiceprint similarities between the sub-voiceprint features and respective reference voiceprint features;

[0232] take a reference object identifier corresponding to a reference voiceprint feature with the largest voiceprint similarity as a speaker identifier of the text character, wherein the speaker identifier is used to indicate an identity of a speaker;

[0233] based on the speaker identifiers of the respective text characters, divide text characters with the same speaker identifier into a same text character group;

[0234] determine target speech content produced by each speaker based on the text character group.

[0235] In some embodiments, the speech recognition network is trained by the following process:

[0236] obtain sample audio data, and labeled text data corresponding to the sample audio data;

[0237] perform feature alignment on the sample audio encoding features of the sample audio data based on the preset first original network, to obtain first sample decoding features, and determine a first probability of generating the labeled text data based on the first sample decoding features;

[0238] perform phoneme feature recognition on the sample audio encoding features based on the first original network, to obtain second sample decoding features, and determine a second probability of generating the labeled text data based on the second sample decoding features;

[0239] perform feature mapping on the sample audio encoding features based on the first original network, to obtain third sample decoding features, and determine a third probability of generating the labeled text data based on the third sample decoding features;

[0240] train the first original recognition network based on the first probability, the second probability, and the third probability, to obtain the speech recognition network.

[0241] In some embodiments, the speaker recognition network is trained by the following process:

[0242] obtain a plurality of sample speech frames, and sample speaker voiceprint features corresponding to each sample speech frame;

[0243] construct sample pairs based on the sample speech frames and the sample speaker voiceprint features, the sample pairs including positive sample pairs composed of the sample speech frames and the sample speaker voiceprint features corresponding to the sample speech frames, and negative sample pairs composed of the sample speech frames and sample speaker voiceprint features not corresponding to the sample speech frames;

[0244] input the positive sample pairs into a preset second original network for speaker recognition, to obtain first predicted voiceprint features, and determine a first similarity between the first predicted voiceprint features and the sample speaker voiceprint features in the positive sample pairs;

[0245] input the negative sample pairs into the second original network for speaker recognition, to obtain second predicted voiceprint features, and determine a second similarity between the second predicted voiceprint features and the sample speaker voiceprint features in the negative sample pairs;

[0246] train the second original network based on the first similarity and the second similarity of the plurality of sample speech frames, to obtain the speaker recognition network.

[0247] The specific implementation of the voice recognition apparatus is basically the same as the specific embodiment of the voice recognition method described above, and thus will not be repeated here.

[0248] The embodiments of the present application also provide an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the voice recognition method. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0249] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0250] The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0251] The memory 902 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the voice recognition method of the embodiments of the present application.

[0252] The input / output interface 903 is used to realize information input and output.

[0253] The communication interface 904 is used to realize communication interaction between the device and other devices, and can realize communication through a wired manner (such as a USB, a network cable, etc.) or a wireless manner (such as a mobile network, WIFI, Bluetooth, etc.).

[0254] The bus 905 transmits information between various components (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0255] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 realize communication connection between each other inside the device through the bus 905.

[0256] The embodiment of the present application further provides a computer readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the voice recognition method.

[0257] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0258] The voice recognition method, the voice recognition device, the electronic equipment and the computer readable storage medium provided by the embodiment of the present application can obtain target voice data containing multiple speakers, encode the target voice data based on a preset voice recognition model to obtain voice coding features, convert the original voice signal into a series of voice coding features, and represent the key attributes (such as pitch, intensity and timbre) of the voice signal through the voice coding features. Further, the voice coding features are subjected to content recognition based on a voice recognition network to obtain voice text content corresponding to the target voice data, the voice coding features are mapped to the corresponding voice text content, so as to generate a text representation of the target voice data, and the voice content of the target voice data can be accurately recognized. Further, the target voice data and the voice coding features are subjected to speaker recognition based on a speaker recognition network to obtain speaker voiceprint features corresponding to the target voice data, and the voiceprint characteristics of the speaker of the target voice data can be captured. Finally, each text character in the voice text content is classified based on the speaker voiceprint features to obtain target voice content generated by each speaker, the target voice content generated by each speaker can be accurately recognized and distinguished, and the recognition accuracy of the voice containing multiple speakers can be improved.

[0259] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0260] Those skilled in the art can understand that, Figures 1-7 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown, or combine certain steps, or different steps.

[0261] The device embodiments described above are merely illustrative, and units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed over multiple network units. Some or all of the modules can be selected as needed to achieve the purposes of the embodiments.

[0262] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0263] The terms "first", "second", "third", "fourth" and the like in the description of the present application and in the claims of the above drawings, if any, are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0264] It should be understood that in the present application, "at least one" means one or more, "multiple" means two or more. "And / or", used to describe the association between the associated objects, means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents a "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0265] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0266] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0267] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0268] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part of the prior art that makes a contribution or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0269] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.< / cc>

Claims

1. A voice recognition method, characterized by, The voice recognition method comprises: obtaining target voice data containing multiple speakers; encoding the target voice data based on a preset voice recognition model to obtain voice coding features, wherein the voice recognition model comprises a voice recognition network and a speaker recognition network; performing content recognition on the voice coding features based on the voice recognition network to obtain voice text content corresponding to the target voice data; performing speaker recognition on the target voice data and the voice coding features based on the speaker recognition network to obtain speaker voiceprint features corresponding to the target voice data; classifying each text character in the voice text content based on the speaker voiceprint features to obtain target voice content generated by each speaker, wherein the target voice content is composed of multiple text characters; the speaker voiceprint features comprise sub-voiceprint features corresponding to each text character in the voice text content; the classification of each text character in the voice text content based on the speaker voiceprint features to obtain target voice content generated by each speaker comprises: obtaining reference voiceprint features and reference object identifiers of each reference speaker object; performing voiceprint similarity calculation on the sub-voiceprint features and the reference voiceprint features for each text character to obtain voiceprint similarity between the sub-voiceprint features and each reference voiceprint feature; taking the reference object identifier corresponding to the reference voiceprint feature with the largest voiceprint similarity as the speaker identifier of the text character, wherein the speaker identifier indicates the identity of the speaker; based on the speaker identifier of each text character, dividing the text characters with the same speaker identifier into the same text character group, and determining the target voice content generated by each speaker based on the text character group.

2. The method of claim 1, wherein, The voice recognition network comprises a first decoder, a second decoder, a third decoder and a feature transformation network; the content recognition of the voice coding features based on the voice recognition network to obtain the voice text content corresponding to the target voice data comprises: performing feature alignment on the voice coding features based on the first decoder to obtain first voice decoding features; performing phoneme feature recognition on the voice coding features based on the second decoder to obtain second voice decoding features; performing feature mapping on the voice coding features based on the third decoder to obtain third voice decoding features; performing feature aggregation on the first voice decoding features, the second voice decoding features and the third voice decoding features to obtain voice aggregation features; performing feature transformation on the voice aggregation features based on the feature transformation network to obtain the voice text content.

3. The method of claim 1, wherein, The speaker recognition network comprises a speaker encoder and a speaker decoder; the speaker recognition of the target voice data and the voice coding features based on the speaker recognition network to obtain the speaker voiceprint features corresponding to the target voice data comprises: concatenate the target speech data and the speech coding feature to obtain target concatenated features; extract speaker features from the target concatenated features based on the speaker encoder to obtain speaker encoding features; perform speaker recognition on the speaker encoding features based on the speaker decoder to obtain the speaker voiceprint features.

4. The method of claim 1, wherein, The speech recognition network is trained through the following process: obtain sample audio data and labeled text data corresponding to the sample audio data; align sample audio coding features of the sample audio data based on a preset first original network to obtain first sample decoding features, and determine a first probability of generating the labeled text data based on the first sample decoding features; perform phoneme feature recognition on the sample audio coding features based on the first original network to obtain second sample decoding features, and determine a second probability of generating the labeled text data based on the second sample decoding features; perform feature mapping on the sample audio coding features based on the first original network to obtain third sample decoding features, and determine a third probability of generating the labeled text data based on the third sample decoding features; train the first original recognition network based on the first probability, the second probability, and the third probability to obtain the speech recognition network.

5. The method of claim 1, wherein, The speaker recognition network is trained through the following process: obtain multiple sample speech frames and sample speaker voiceprint features corresponding to each sample speech frame; construct sample pairs based on the sample speech frames and the sample speaker voiceprint features, wherein the sample pairs include positive sample pairs composed of the sample speech frames and the sample speaker voiceprint features corresponding to the sample speech frames, and negative sample pairs composed of the sample speech frames and sample speaker voiceprint features not corresponding to the sample speech frames; input the positive sample pairs into a preset second original network to perform speaker recognition, obtain first predicted voiceprint features, and determine a first similarity between the first predicted voiceprint features and the sample speaker voiceprint features in the positive sample pairs; input the negative sample pairs into the second original network to perform speaker recognition, obtain second predicted voiceprint features, and determine a second similarity between the second predicted voiceprint features and the sample speaker voiceprint features in the negative sample pairs; train the second original network based on the first similarities and the second similarities of multiple sample speech frames to obtain the speaker recognition network.

6. The method according to any one of claims 1 to 5, characterized in that, The obtaining of the target speech data containing multiple speakers includes: obtain original speech data containing multiple speakers; perform denoising processing on the original speech data to obtain denoised speech data; perform Fourier transform on the denoised speech data to obtain speech frequency domain data; perform Mel filtering on the speech frequency domain data to obtain the target speech data.

7. A speech recognition apparatus characterized by comprising: The speech recognition device includes: an obtaining unit configured to obtain target speech data containing multiple speakers; The encoding unit is configured to encode the target voice data based on a preset voice recognition model to obtain voice encoding features, wherein the voice recognition model comprises a voice recognition network and a speaker recognition network. The content recognition unit is configured to recognize the content of the voice text based on the voice recognition network and the voice encoding features. The speaker recognition unit is configured to perform speaker recognition on the target voice data and the voice encoding features based on the speaker recognition network to obtain speaker voiceprint features corresponding to the target voice data. The character classification unit is configured to classify each text character in the voice text content based on the speaker voiceprint features to obtain target voice content generated by each speaker, wherein the target voice content comprises a plurality of text characters. The speaker voiceprint features comprise sub-voiceprint features corresponding to each text character in the voice text content. The classification of each text character in the voice text content based on the speaker voiceprint features to obtain target voice content generated by each speaker comprises: obtaining reference voiceprint features and reference object identifiers of each reference speaker object; performing voiceprint similarity calculation on the sub-voiceprint features and the reference voiceprint features for each text character to obtain voiceprint similarity between the sub-voiceprint features and each reference voiceprint feature; taking the reference object identifier corresponding to the reference voiceprint feature with the largest voiceprint similarity as the speaker identifier of the text character, wherein the speaker identifier is used to indicate the identity of the speaker; based on the speaker identifier of each text character, dividing the text characters with the same speaker identifier into the same text character group, and determining the target voice content generated by each speaker based on the text character group.

8. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the voice recognition method of any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the voice recognition method of any one of claims 1 to 6.

Images