Speech recognition model training method, speech recognition method and related equipment

Abstract

The embodiment of the invention provides a training method of a voice recognition model, a voice recognition method and related equipment. The training method of the voice recognition model comprises the following steps: acquiring a voice sample and a first voice recognition result for marking the voice sample; the first voice recognition result comprises a classification label of each frame of audio data in the voice sample, and each classification label represents a voice object participating in a conversation in the corresponding frame of audio data; performing feature extraction on the voice sample to obtain an audio feature of the voice sample; recognizing a voice object of each frame of audio data in the voice sample according to the audio features to obtain a second voice recognition result of the voice object; constructing a loss function according to the second voice recognition result and the first voice recognition result; and performing iterative training on the speech recognition model based on the loss function. Therefore, the recognition accuracy of the voice object can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a training method for a speech recognition model, a speech recognition method and related equipment. Background Art

[0002] With the development of electronic technology, the application of artificial intelligence is becoming more and more common. Artificial intelligence technology can achieve good performance in processing various speech recognition tasks, including speaker diarization tasks, speaker verification tasks, speech separation tasks, etc. Among them, the speaker diarization task refers to the process of determining who is speaking and when in a piece of audio or video. In short, when the speech data includes more than one speech object, the speaker diarization task can be used to identify each speech object in the speech data. Summary of the invention

[0003] The embodiments of the present application provide a speech recognition model training method, a speech recognition method and related equipment, which can improve the recognition accuracy of speech objects.

[0004] In a first aspect, an embodiment of the present application provides a method for training a speech recognition model, comprising: Acquire a speech sample and a first speech recognition result annotated with the speech sample; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, each classification label representing a speech object participating in the conversation in a corresponding frame of audio data; Extracting features from the speech sample to obtain audio features of the speech sample; According to the audio feature, identifying the voice object of each frame of audio data in the voice sample, and obtaining a second voice recognition result of the voice object; Constructing a loss function according to the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

[0005] In a second aspect, an embodiment of the present application provides a speech recognition method, comprising: Inputting the first speech data into a target speech recognition model to perform recognition processing on a speech object, and obtaining a speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the speech recognition model training method described in the first aspect; According to the speech recognition result of the speech object, the first speech data is divided into a plurality of time periods, and the speech object in each of the plurality of time periods is marked.

[0006] In a third aspect, an embodiment of the present application provides a training device for a speech recognition model, comprising: An acquisition unit, configured to acquire a speech sample and a first speech recognition result annotated on the speech sample; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, each classification label representing a speech object participating in a conversation in a corresponding frame of audio data; An extraction unit, used to extract features from the speech sample to obtain audio features of the speech sample; A first recognition unit, configured to recognize a speech object of each frame of audio data in the speech sample according to the audio feature, and obtain a second speech recognition result of the speech object; A construction unit, configured to construct a loss function according to the second speech recognition result and the first speech recognition result; A training unit is used to iteratively train the speech recognition model based on the loss function.

[0007] In a fourth aspect, an embodiment of the present application provides a speech recognition device, including: A second recognition unit is used to input the first speech data into a target speech recognition model to perform recognition processing on the speech object, and obtain a speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the training method of the speech recognition model described in the first aspect; A marking unit is used to divide the first voice data into a plurality of time periods according to the voice recognition result of the voice object, and mark the voice object in each of the plurality of time periods.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device comprising: a processor; and a memory configured to store computer-executable instructions, wherein when the computer-executable instructions are executed, the processor executes the training method of the speech recognition model as described in the first aspect or the speech recognition method as described in the second aspect.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the training method for a speech recognition model as described in the first aspect or the speech recognition method as described in the second aspect.

[0010] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the training method of the speech recognition model as described in the first aspect or the speech recognition method as described in the second aspect.

[0011] It can be seen that in an embodiment of the present application, first, a speech sample and a first speech recognition result that annotates the speech sample are obtained; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, and each classification label represents a speech object participating in the conversation in a corresponding frame of audio data; then, feature extraction is performed on the speech sample to obtain the audio features of the speech sample; then, based on the audio features, the speech object of each frame of audio data in the speech sample is identified to obtain a second speech recognition result of the speech object; next, a loss function is constructed based on the second speech recognition result and the first speech recognition result; finally, the speech recognition model is iteratively trained based on the loss function. Taking into account that in the case where a voice sample involves multiple voice objects participating in a conversation, each voice object may participate in the conversation at the same time. For example, three people speak at the same time. In one frame of audio data, the sounds of multiple voice objects participating in the conversation at the same time are mixed together. Each voice object may interfere with the recognition accuracy of other voice objects. In an embodiment of the present application, the first voice recognition result for annotating the voice sample includes a classification label for each frame of audio data. Each classification label represents a voice object participating in the conversation in the corresponding frame of audio data. In this way, even if there are multiple voice objects participating in the conversation at the same time in one frame of audio data, the one frame of audio data has only one classification label, which can be combined with the first classification label during the training process. The loss function is constructed based on the second speech recognition result, and the loss function can be used to guide the speech recognition model to learn how to map the one frame of audio data to an output category composed of multiple speech objects. An output category is the second speech recognition result of the speech object in the one frame of audio data. Furthermore, the sounds of multiple speech objects mixed together do not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution that guides the speech recognition model to learn how to map one frame of speech data to multiple output categories, each output category corresponding to a speech object, the embodiment of the present application can reduce the negative impact of the mutual interference of the sounds of each speech object uttered at the same time on the recognition of the speech object, and is more conducive to improving the recognition accuracy of the speech object. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor. Figure 1 A schematic diagram of an implementation environment for a method for training a speech recognition model provided in an embodiment of the present application; Figure 2 A processing flow chart of a method for training a speech recognition model provided in an embodiment of the present application; Figure 3 An example diagram of annotation information of an audio sample in a method for training a speech recognition model provided in an embodiment of the present application; Figure 4 A weight information diagram of a speech pre-training model in different tasks provided in an embodiment of the present application; Figure 5 A data flow diagram of another method for training a speech recognition model provided in an embodiment of the present application; Figure 6 A data flow diagram of another method for training a speech recognition model provided in an embodiment of the present application; Figure 7 A processing flow chart of a speech recognition method provided in an embodiment of the present application; Figure 8 A data flow diagram of another speech recognition method provided in an embodiment of the present application; Fig. 9 A schematic diagram of a training device for a speech recognition model provided in an embodiment of the present application; Fig.10 A schematic diagram of a speech recognition device provided in an embodiment of the present application; Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0014] This specification provides an embodiment of a method for training a recognition model: The speaker log task can be used to identify each voice object in the speech data. If there are only two voice objects in the speech data, that is, the speech data is a two-person conversation data, the two voice objects participating in the two-person conversation will often wait for the other party to finish speaking before speaking. Therefore, the situation where two people speak at the same time in the speech data is relatively rare, and the difficulty of identifying the voice object is relatively low.

[0015] In actual applications, voice data may contain more than two voice objects, and it is common for multiple people to speak at the same time. The more voice objects there are in a piece of voice data to be processed, the more difficult it is to identify the voice objects. The sounds of multiple voice objects speaking at the same time may interfere with each other, resulting in inaccurate recognition of the voice objects.

[0016] Therefore, in order to solve the above problems, an embodiment of the present application provides a method for training a speech recognition model.

[0017] The speech recognition model training method provided in one or more embodiments of this specification may be applicable to an implementation environment of the speech recognition model training method, which implementation environment at least includes a server 101 for model training.

[0018] Among them, server 101 can be a server, or a server cluster composed of several servers, or one or more cloud servers in a cloud computing platform, used for model training.

[0019] In this implementation environment, during the model training process, the server 101 first obtains a speech sample and a first speech recognition result that annotates the speech sample; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, and each classification label represents a speech object participating in the conversation in a corresponding frame of audio data; then, feature extraction is performed on the speech sample to obtain the audio feature of the speech sample; then, based on the audio feature, the speech object of each frame of audio data in the speech sample is identified to obtain a second speech recognition result of the speech object; next, a loss function is constructed based on the second speech recognition result and the first speech recognition result; finally, the speech recognition model is iteratively trained based on the loss function. Taking into account that in the case where a voice sample involves multiple voice objects participating in a conversation, each voice object may participate in the conversation at the same time. For example, three people speak at the same time. In one frame of audio data, the sounds of multiple voice objects participating in the conversation at the same time are mixed together. Each voice object may interfere with the recognition accuracy of other voice objects. In an embodiment of the present application, the first voice recognition result for annotating the voice sample includes a classification label for each frame of audio data. Each classification label represents a voice object participating in the conversation in the corresponding frame of audio data. In this way, even if there are multiple voice objects participating in the conversation at the same time in one frame of audio data, the one frame of audio data has only one classification label, which can be combined with the second classification label during the training process. The speech recognition result is used to construct a loss function, and then the loss function can be used to guide the speech recognition model to learn how to map the one frame of audio data to an output category composed of multiple speech objects, that is, the second speech recognition result of the speech object in the one frame of audio data. Furthermore, the sounds of multiple speech objects mixed together do not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution of guiding the speech recognition model to learn how to map one frame of speech data to multiple output categories, each output category corresponding to a speech object, the embodiment of the present application can reduce the negative impact of the mutual interference of the sounds of each speech object uttered at the same time on the recognition of the speech object, and is more conducive to improving the recognition accuracy of the speech object.

[0020] Figure 2 A processing flow chart of a method for training a speech recognition model provided in an embodiment of the present application. Figure 2 The training method of the recognition model provided in this embodiment specifically includes steps S202 to S208: Step S202, obtaining a speech sample and a first speech recognition result that annotates the speech sample; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, and each classification label represents a speech object participating in the conversation in a corresponding frame of audio data.

[0021] A voice sample is a specific example used to study and analyze a voice signal. It is usually a piece of audio data extracted from an actual voice conversation or recording. Specifically in the embodiment of the present application, the voice sample can be sample data in an audio format.

[0022] The number of voice samples may be one or more. Each voice sample used in this embodiment may contain at least two voice objects that make sounds. The voice object may be a person, a robot with conversational capabilities, or any other object that can communicate through natural language, etc.

[0023] The voice sample can be generated based on the conversation data, in which at least two voice objects participate in the conversation. For example, user 1 and user 2 discuss a life topic "weather" face to face, and conversation data can be generated during the discussion, and voice sample 1 can be generated using the conversation data. For another example, a customer asks a customer service agent for the specific details of an activity during a call, and conversation data can be generated during the consultation, and voice sample 2 can be generated using the conversation data.

[0024] Acquiring the voice sample may be acquiring conversation data, counting the number of voice objects in the conversation data, filtering and processing the conversation data according to the number of voice objects to obtain target conversation data, and treating each target conversation data as a voice sample.

[0025] In the process of filtering and processing the conversation data according to the number of voice objects to obtain the target conversation data, for each conversation data, the number of voice objects in the conversation data can be compared with the first number threshold and the second number threshold respectively. If the number of voice objects is greater than or equal to the first number threshold and less than or equal to the second number threshold, the conversation data is retained and the conversation data is used as a target conversation data. Otherwise, the conversation data is discarded.

[0026] The reason for filtering the conversation data according to the number of speech objects is that, on the one hand, the classification label of the speech sample is affected by the number of speech objects in the speech sample. To ensure the effectiveness of the subsequent classification label, the number of speech objects in each speech sample should not be too large. On the other hand, if the number of speech objects in the speech sample is too small, the possibility of multiple speech objects uttering at the same time is small, which may cause the samples corresponding to some classification labels to be sparse, which is not conducive to model training.

[0027] In addition, when filtering the conversation data according to the number of voice objects to obtain the target conversation data, for each conversation data, the number of voice objects in the conversation data may be compared with a first quantity threshold. If the number of voice objects is greater than or equal to the first quantity threshold, the conversation data is retained and the conversation data is used as a target conversation data. Otherwise, the conversation data is discarded.

[0028] Alternatively, in the process of filtering the conversation data according to the number of voice objects to obtain the target conversation data, for each conversation data, the number of voice objects in the conversation data is compared with a second quantity threshold; if the number of voice objects is less than or equal to the second quantity threshold, the conversation data is retained and the conversation data is used as a target conversation data; otherwise, the conversation data is discarded.

[0029] Optionally, the voice sample may be synthesized based on multiple audio segments, where each audio segment originates from a voice object. In specific implementation, the voice sample may be obtained by concatenating multiple audio segments to obtain the voice sample, or by cropping and concatenating multiple audio segments to obtain the voice sample.

[0030] For example, audio segments 1, 2, and 3 of user 1 are pre-recorded, and audio segments 4 and 5 of user 2 are also recorded. Audio segments 1, 2, 3, 4, and 5 are cropped and spliced ​​to obtain voice sample 3.

[0031] In a specific implementation, a method for generating a voice sample includes: obtaining first voice data and second voice data, the first voice data including voice data of at least two voice objects, and the second voice data including voice data of one voice object; generating virtual voice data of a virtual voice object, and expanding the virtual voice data into the second voice data to obtain expanded second voice data, the expanded second voice data including voice data of at least two voice objects; constructing a first voice sample based on the first voice data, and constructing a second voice sample based on the expanded second voice data; extracting voice samples from the first voice sample and the second voice sample according to a preset ratio to obtain a voice sample.

[0032] The use of ordinal numbers such as "first" and "second" in this specification is to facilitate the distinction between different technical features and has no actual meaning, and will not be repeated below.

[0033] Each first voice data includes voice data of at least two voice objects. The first voice data may include a portion of voice data in a general domain and a portion of voice data in a target domain.

[0034] The general domain is a general domain that involves all walks of life and has a relatively large scope. The target domain can be a predetermined specific domain that is different from the general domain, or the target domain can also be interpreted as a private domain. The private domain and the general domain are relative concepts. The private domain can focus on a specific industry and has a relatively small scope compared to the general domain, such as the customer service domain.

[0035] The first voice data may include conversation data occurring in various industries.

[0036] Exemplarily, the first voice data may be an open source speaker log data set. An open source speaker log data set refers to an open source data set whose data content is a speaker log. An open source data set refers to a data set that can be obtained, used, and shared for free.

[0037] Speaker logs usually refer to documents or databases that record the speech content of a specific speaker (such as a broadcaster, lecturer, conference host, etc.) within a certain period of time. This log can contain a variety of information, such as: Speech content: Record the specific speech content of the speaker, which can be a written record or an audio or video transcription.

[0038] Timestamp: Record the exact time of each speech to help track and organize information.

[0039] Occasion: Record the occasion or event where the speech takes place, such as a meeting, lecture, radio program, etc.

[0040] Subject: Record the topic of the speech or the issue of discussion.

[0041] Audience Feedback: This may also include feedback or interaction from the audience, if available.

[0042] The first voice data may also include conversation data occurring in a target business scenario. The target business scenario may be a specific scenario in a target field.

[0043] For example, the target domain is the call center domain, and the target business scenario is the consulting scenario, which specifically refers to the scenario in which the customer service agent provides consulting services to the customer during the call. The first voice data includes conversation data between the customer service agent and the customer.

[0044] Each second voice data includes voice data of one voice object.

[0045] The second voice data may include a data set for voiceprint recognition training in a general field, and each data set for voiceprint recognition training includes voice data of a voice object.

[0046] For example, the second voice data 1 is the voice data of user 1, the second voice data 2 is the voice data of user 2, the second voice data 3 is the voice data of user 3, and so on.

[0047] Generate virtual voice data of the virtual voice object, and expand the virtual voice data into the second voice data to obtain the expanded second voice data, wherein the expanded second voice data includes the voice data of at least two voice objects.

[0048] Generating virtual voice data for a virtual voice object may be performed by converting a preset template text into speech to obtain multiple candidate virtual voice data for the virtual voice object, randomly selecting one of the multiple candidate virtual voice data as the virtual voice data, or, based on the second voice data, determining one of the multiple candidate virtual voice data that matches the second voice data as the virtual voice data.

[0049] Generate virtual voice data of a virtual voice object, or perform speech-to-text processing on the second voice data to obtain a first text, input the first text into a language model for text generation processing to obtain a second text, perform text-to-speech processing on the second text to obtain virtual voice data, and so on.

[0050] This specification does not impose any special restrictions on the method of generating virtual voice data. Virtual voice data can be voice data including at least one voice object. The virtual here means that there is actually no conversation between two people, but in the subsequent steps, voice data including at least two voice objects are required to construct training samples. Therefore, by generating virtual voice data of virtual voice objects, it can be combined with the second voice data to simulate natural conversation.

[0051] Taking into account that the generation of virtual voice data is relatively flexible, the second voice data can be used as the main method in the process of simulating natural conversation. The corresponding virtual voice data can be determined based on the second voice data, and then the virtual voice data can be expanded to the second voice data to obtain the expanded second voice data. In this way, the second voice data before expansion is voice data involving only a single voice object, and the second voice data after expansion is conversation data including at least two voice objects.

[0052] The expansion method may be to splice the second voice data and the virtual voice data, or to cut them first and then splice them.

[0053] The expansion method may also be to construct the dialogue data according to the second voice data and one or more virtual voice data.

[0054] For example, the second voice data is converted into text to obtain text 1 corresponding to the second voice data, the text 1 is input into the large language model for text generation to obtain text 2, the text 2 is converted into speech to obtain virtual voice data, and the virtual voice data is spliced ​​together with the second voice data to obtain the expanded second voice data.

[0055] Constructing the first voice sample according to the first voice data may be performing sampling processing on the first voice data in a general field to obtain a portion of training data, performing sampling processing on the first voice data in a target field to obtain another portion of training data, and adding the two portions of training data to the training data set to obtain the first voice sample.

[0056] Constructing the second voice sample according to the expanded second voice data may be performed by sampling the expanded second voice data to obtain the second voice sample.

[0057] The expanded second voice data can be regarded as a simulated conversation data. For example, the expanded second voice data 1 includes a portion of the voice data of user 1 and a portion of the voice data of virtual user 2, and user 1 and virtual user 2 have not actually had a conversation.

[0058] Compared with the real conversation data, the virtual conversation data obtained by expanding the second voice data can be generated in batches efficiently, so that the amount of expanded second voice data is relatively large. Furthermore, the number of second voice samples in subsequent steps can be reduced through sampling, thereby flexibly controlling the ratio of the first voice sample to the second voice sample in the voice sample.

[0059] Extracting voice samples from the first voice sample and the second voice sample according to a preset ratio to obtain the voice sample can include: determining a first sample quantity of the first voice sample to be selected and a second sample quantity of the second voice sample to be selected according to a pre-configured total number of samples and a preset ratio parameter; randomly selecting a first sample quantity of first voice samples, and randomly selecting a second sample quantity of second voice samples; assembling the first sample quantity of first voice samples and the second sample quantity of second voice samples to obtain the voice sample.

[0060] For example, the preset ratio parameter is 8:2, where 8 corresponds to the first voice sample and 2 corresponds to the second voice sample. The total number of pre-configured samples is 100. According to the total number of samples and the preset ratio parameter, it can be determined that the number of first samples of the first voice sample to be selected is 80, and the number of second samples of the second voice sample to be selected is 20. 80 first voice samples are randomly selected from each first voice sample. 20 second voice samples are selected from each second voice sample. The 80 first voice samples and the 20 second voice samples are assembled to obtain a voice sample.

[0061] Extracting voice samples from the first voice sample and the second voice sample according to a preset ratio to obtain a voice sample may also include: determining a second number of second voice samples to be selected based on the number of first voice samples and a preset ratio parameter; randomly selecting a second number of second voice samples; and assembling the first voice sample and the second number of second voice samples to obtain a voice sample.

[0062] For example, the preset ratio parameter is 8:2, where 8 corresponds to the first voice sample and 2 corresponds to the second voice sample. The number of the first voice samples is 800. According to the number of the first voice samples and the preset ratio parameter, it is determined that the second number of the second voice samples to be selected is 200. 200 second training data are randomly selected from each second voice sample. The 800 first voice samples and the 200 second training data are assembled to obtain a voice sample.

[0063] The reason why the first voice sample and the second voice sample are assembled and processed is that the number of conversation data occurring in the target domain may not meet the training requirements of the machine learning model, that is, the number of first voice samples is too small. The first voice sample is a real conversation, that is, a real conversation, and the second voice sample is a simulated conversation. When the number of real conversations is too small, the number of samples can be expanded through simulated conversations, which is conducive to achieving better model training results.

[0064] Each speech sample can have corresponding annotation information. Annotation information usually refers to additional information added to a data set or document to explain, supplement, or highlight the content of the original data. This information can be text, symbols, labels, or other forms of annotations to help users better understand and use data. Annotation information is very important in many fields, especially in data science, machine learning, and natural language processing.

[0065] Specifically in the embodiment of the present application, the annotation information corresponding to each voice sample is the aforementioned first voice recognition result.

[0066] Obtaining the first speech recognition result by annotating the speech sample may be obtaining an artificially generated first speech recognition result, or may be classifying the speech sample through a pre-trained classification model, annotating the speech sample according to the classification result, and obtaining the first speech recognition result of the speech sample.

[0067] The first speech recognition result includes a classification label for each frame of audio data in the speech sample, and each classification label represents a speech object participating in the conversation in a corresponding frame of audio data.

[0068] In a specific implementation, the classification label is generated in the following manner: obtaining the voice objects participating in the conversation in the voice sample; constructing object groups based on the voice objects to obtain an object group set; in the object group set, the number of voice objects in each object group is within a preset range; generating a corresponding object identification group based on each object group in the object group set, and determining each object identification group as a classification label.

[0069] For each voice sample, the number of voice objects participating in the conversation in the voice sample can be represented by a first number, and the first number can be an integer greater than 1.

[0070] For example, voice sample 1 is the conversation data between user 1, user 2 and user 3 in a conference scene. It can be determined that the voice objects participating in the conversation include user 1, user 2 and user 3, so the first number of voice objects participating in the conversation is 3.

[0071] To obtain the voice objects participating in the conversation in the voice sample, the voice objects in each frame of audio data in the voice sample may be detected first and then deduplicated to obtain the voice objects participating in the conversation in the voice sample.

[0072] For example, a speech sample includes audio data from the first frame to the third frame. In the first frame of audio data of the speech sample, the speech objects detected therein include: object 1 and object 2. In the second frame of audio data of the speech sample, the speech objects detected therein include: object 1. In the third frame of audio data of the speech sample, the speech objects detected therein include: object 1 and object 3. The speech objects in the first frame of audio data, the speech objects in the second frame of audio data, and the speech objects in the third frame of audio data are deduplicated to obtain the speech objects participating in the conversation in the speech sample: object 1, object 2, and object 3.

[0073] At least one object group is generated according to the number of voice objects to obtain an object group set; in the object group set, the number of voice objects in each object group is within a preset range.

[0074] The preset range interval may be determined by a first quantity.

[0075] Specifically, the object group set may include one or more object groups, and the number of voice objects in each object group is greater than or equal to one and less than or equal to the first number.

[0076] An example is given in which the first number of voice objects participating in the conversation in the voice sample is 3: The voice objects participating in the conversation in voice sample 1 include: object 1, object 2, and object 3.

[0077] At least one object group is generated according to the number of voice objects, and multiple object groups are obtained as shown below: (c1) Object 1 (c2) Object 2 (c3) Object 3 (c4) Object 1 and Object 2 (c5) Object 1 and Object 3 (c6) Object 2 and Object 3 (c7) Object 1, Object 2, and Object 3 The above-mentioned object groups (c1)-(c7) together constitute an object group set. And the number of voice objects in each of the above-mentioned object groups (c1)-(c7) is greater than or equal to 1 and less than or equal to 3, which meets the above-mentioned requirement that the number of voice objects in each object group is greater than or equal to one and less than or equal to the first number.

[0078] A corresponding object identification group is generated according to each object group in the object group set, and each object identification group is determined as a classification label.

[0079] For example, a corresponding object identification group (001) is generated according to the object group (c1), and the object identification group is determined as the classification label d1; Generate a corresponding object identification group (002) according to the object group (c2), and determine the object identification group as the classification label d2; Generate a corresponding object identification group (003) according to the object group (c3), and determine the object identification group as the classification label d3; Generate a corresponding object identification group (001, 002) according to the object group (c4), and determine the object identification group as the classification label d4; Generate a corresponding object identification group (001, 003) according to the object group (c5), and determine the object identification group as the classification label d5; Generate a corresponding object identification group (002, 003) according to the object group (c6), and determine the object identification group as the classification label d6; A corresponding object identification group (001, 002, 003) is generated according to the object group (c7), and the object identification group is determined as the classification label d7.

[0080] In addition, a silence mark representing silence may also be used as a classification label.

[0081] After the classification label is determined, each frame of audio data in the speech sample can be annotated using the classification label to obtain a first speech recognition result of the speech sample.

[0082] Next, you can combine Figure 3 The first speech recognition result of the speech sample is exemplified. Figure 3 An example diagram of the annotation information of a speech sample in a method for training a speech recognition model provided in an embodiment of the present application.

[0083] Figure 3 The multi-category based label encoding is shown. The classification label set 304 includes: silence, speaker 1, speaker 2, speaker 3, speaker 1 & speaker 2, speaker 1 & speaker 3, speaker 2 & speaker 3, and all speakers.

[0084] Among them, "speaker 1 & speaker 2" can mean that speaker 1 and speaker 2 are two voice objects recognized at the same time. "speaker 1 & speaker 3" can mean that speaker 1 and speaker 3 are two voice objects recognized at the same time. "speaker 2 & speaker 3" can mean that speaker 2 and speaker 3 are two voice objects recognized at the same time. "all speakers" can mean that speaker 1, speaker 2 and speaker 3 are three voice objects recognized at the same time.

[0085] The time information 302 includes t1-t13, a total of 13 time points, each of which has a unique corresponding classification label. Among the 13 time points, each time point can be used to represent a frame of audio data in the speech sample.

[0086] Among them, the classification label at time point t1 is "silent"; the classification label at time points t2-t5 is "speaker 1"; the classification label at time points t6-t7 is "speaker 1 & speaker 2"; the classification label at time points t8-t9 is "speaker 2"; the classification label at time point t10 is "speaker 2 & speaker 3"; the classification label at time points t11-t12 is "speaker 3"; and the classification label at time point t13 is "silent".

[0087] In summary, the first speech recognition result of the speech sample includes: the classification label "silence" of the first frame audio data, the classification label "speaker 1" of the second frame audio data, the classification label "speaker 1" of the third frame audio data, ..., the classification label "silence" of the 13th frame audio data.

[0088] Step S204: extract features from the speech sample to obtain audio features of the speech sample. In some implementations, feature extraction is performed on a speech sample to obtain an audio feature of the speech sample, and the speech sample may be encoded based on a speech recognition model to obtain the audio feature of the speech sample.

[0089] In other implementations, during the process of extracting features from speech samples and obtaining audio features of the speech samples, the following operations may be performed: based on a speech recognition model, first coding information of the speech samples in N dimensions is obtained; based on the first coding information output by each coding layer and the weight parameters of each dimension, the audio features of the speech samples are determined.

[0090] The first encoding information of the speech sample in N dimensions may be obtained by encoding the speech sample through a speech recognition model. N may be an integer greater than 1.

[0091] Specifically, a round of encoding processing is performed on the speech sample through the speech recognition model to obtain the first encoded information of the first dimension; a round of encoding processing is performed on the first encoded information of the first dimension through the speech recognition model to obtain the first encoded information of the second dimension; ...; a round of encoding processing is performed on the first encoded information of the (N-1)th dimension through the speech recognition model to obtain the first encoded information of the Nth dimension.

[0092] The speech recognition model may include an encoding module, which may be composed of a speech pre-training model. The speech pre-training model may be a pre-training model trained with large-scale speech data and applicable to a variety of speech tasks. Exemplarily, the various speech tasks include at least one of the following: a speaker log task, a speaker verification task, a speech separation task, a speech recognition task, and the like.

[0093] The Speaker Diarization task is also known as voiceprint segmentation and clustering, and the main problem it solves is "who said what and when". That is, given a speech containing multiple people speaking alternately, the task needs to determine who was speaking at each time point. For example, for a call record between a customer and a customer service representative, the task needs to know which time periods the voice was spoken by the customer and which time periods the voice was spoken by the customer service representative.

[0094] The Speaker Verification task is to verify whether two audio clips are spoken by the same person.

[0095] The goal of the Speech Separation task is to separate a segment of speech containing multiple people and ensure that each output source contains only a segment of speech from one person.

[0096] The goal of the speech recognition task is to convert a piece of speech into text.

[0097] Pre-trained models are models that have been trained on a large amount of data for a specific task. These models can be used as a starting point for solving similar tasks. The characteristics of pre-trained models are as follows: Generality: Pre-trained models are usually trained on large and diverse datasets, so they can capture a wide range of linguistic or visual features.

[0098] Transfer learning: Pre-trained models can serve as the basis for transfer learning. Users can fine-tune the pre-trained model on a dataset for a specific task to adapt it to a new task.

[0099] Efficiency: Using pre-trained models can save a lot of training time and computing resources because most of the model's parameters have been optimized through pre-training.

[0100] Performance: Pretrained models often provide good initial performance, especially when there is limited data for the target task.

[0101] Exemplarily, the speech pre-training model can be a WavLM (Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing) model.

[0102] The WavLM model was obtained using the Denoising Masked Speech Modeling framework on 94,000 hours of unsupervised data. The pre-trained model trained on large-scale unsupervised data has very good generalization and can achieve good results by only fine-tuning on small-scale annotated data.

[0103] A speech pre-trained model can include one or more encoder layers. For example, one WavLM model has 12 Transformer encoder layers. Another WavLM model has 24 Transformer encoder layers. The Transformer Encoder Layer is a key component in the Transformer model, which is responsible for processing input data and generating corresponding feature representations.

[0104] The Transformer model consists of two parts: an encoder and a decoder. The encoder is responsible for converting the input sequence into an intermediate representation, and the decoder uses this intermediate representation to generate the output sequence. Both the encoder and the decoder are stacked with multiple identical layers, each of which contains two main sub-layers: Self-Attention mechanism: allows the model to take into account all words in the input sequence when processing each word, thereby capturing long-range dependencies between words.

[0105] Feed-Forward Neural Network: Performs further nonlinear transformation on the output of the self-attention layer.

[0106] Each sub-layer is followed by a residual connection and layer normalization to help the model train and generalize better.

[0107] The workflow of the Transformer encoding layer is as follows: (a1) Input processing: The input sequence is first converted into a vector representation through a word embedding layer, and then positional encoding is added to preserve the position information of the words in the sequence.

[0108] (a2) Self-attention mechanism: The input vector passes through the self-attention mechanism, and each word gets a new representation based on its relationship with other words in the sequence. The self-attention mechanism allows the model to establish direct connections between different positions, which is the key to the Transformer model's ability to effectively handle long-distance dependencies.

[0109] (a3) Feedforward Neural Network: The output of the self-attention layer is fed into a fully connected feedforward neural network, which applies the same nonlinear transformation to the vector at each position.

[0110] (a4) Residual connections and layer normalization: The output of each sub-layer passes through a residual connection and then undergoes layer normalization. Residual connections help solve the gradient vanishing problem in deep networks, while layer normalization helps stabilize the training process.

[0111] (a5) Output: After processing through multiple encoding layers, the final output is a series of feature vectors that capture rich information of the input sequence and can be used by the decoder to generate the output sequence.

[0112] In the process of obtaining the first coding information of the speech sample in N dimensions based on the speech recognition model, the speech sample can be input into the coding module of the speech recognition module, and the speech sample can be encoded through multiple coding layers in the coding module to obtain the first coding information of the speech sample in N dimensions.

[0113] In the process of obtaining the first coding information of the speech sample in N dimensions based on the speech recognition model, it is also possible to first perform feature extraction processing on the speech sample to obtain the Fbank feature of the speech sample, and then input the Fbank feature into the coding module of the speech recognition module, and encode the Fbank feature through multiple coding layers in the coding module to obtain the first coding information of the speech sample in N dimensions.

[0114] The definition of Fbank features is as follows: The human ear responds nonlinearly to the sound spectrum. Fbank is a front-end processing algorithm that processes audio in a similar way to the human ear, which can improve the performance of speech recognition. The general steps to obtain Fbank features of speech signals are: pre-emphasis, framing, windowing, short-time Fourier transform, mel filtering, etc.

[0115] In specific implementation, in the process of obtaining the first encoding information of the speech sample in N dimensions based on the speech recognition model, the following steps may be performed: (b1) Divide the speech sample into audio segments of fixed length Len.

[0116] Len is a preset value, for example, Len=10 seconds. If the total duration of the speech sample is not an integer multiple of Len, the insufficient part will be filled with silence. After the silence is filled, the speech sample includes K segmented audio segments. K is an integer greater than 0.

[0117] (b2) The size of the training batch is configured as b, and the number of iterations in each round of training is an integer of K / b.

[0118] In machine learning and deep learning, a training batch refers to a group of samples that are input into the model at one time during the model training process.

[0119] The configuration basis of the training batch can be the video memory size of the server that executes the model training, the parallel computing capability of the server, the memory size of the server, the convergence speed, etc.

[0120] (b3) Extract Fbank features from each batch of audio clips.

[0121] For example, b=64, 40-dimensional Fbank features can be extracted using 64 50-second audio clips, where the interval between two adjacent frames with a frame length of 25 milliseconds is 10 milliseconds, and 64*40*500 Fbank features can be obtained.

[0122] (b4) Input the Fbank features into the encoding module in the speech recognition model for encoding processing.

[0123] When the encoding module of the speech recognition model includes N encoding layers connected in series in sequence, the N encoding layers correspond one-to-one to N dimensions, and the first encoding information of the speech sample in the N dimensions can be obtained in the following way: obtain the data output by each of the N encoding layers, and use the data output by each encoding layer as the first encoding information corresponding to the encoding layer.

[0124] When the encoding module of the speech recognition model includes M encoding layers connected in series, and M is greater than N, the first encoding information of the speech sample in N dimensions can be obtained in the following way: obtain the data output by each of the M encoding layers, determine N target encoding layers according to a pre-configured set of encoding layers, and use the data output by each target encoding layer as the first encoding information corresponding to the target encoding layer.

[0125] The reason for configuring the coding layer set is that the speech recognition model in one or more embodiments of the present application can be used to perform the speaker log task, but the speech pre-training model of the coding module used to construct the speech recognition model is not a proprietary model designed for performing the speaker log task, but a general model that can be applied to a variety of speech tasks. Therefore, among the multiple coding layers included in the speech pre-training model, the first coding information output by some coding layers may have good performance when processing the speaker log task, but the first coding information output by another part of the coding layers is of little use when processing the speaker log task and can be approximately regarded as redundant data. Therefore, configuring the coding layer set is conducive to reducing redundant data and improving data processing efficiency in the process of performing the speaker log task.

[0126] In a specific implementation, a speech recognition model includes N speech coding layers; feature extraction is performed on a speech sample to obtain audio features of the speech sample, including: each speech coding layer encodes the speech sample to obtain the first coding information output by each speech coding layer; wherein, when i=1, the speech sample is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension; when i is greater than 1, the speech coding information of the i-1th dimension output by the i-1th speech coding layer is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension, where i is a positive integer less than or equal to N; based on the first coding information output by each coding layer and the weight parameter of each dimension, the audio features of the speech sample are determined.

[0127] N is an integer greater than 0. i is an integer greater than 0 and less than or equal to N.

[0128] In the process of performing multiple rounds of continuous encoding on speech samples, except for the last round of encoding, the first encoding information generated by each remaining round of encoding can be regarded as the intermediate data generated in the multiple rounds of encoding process. The intermediate data output by each round of encoding is taken out as the first encoding information of one dimension, and this intermediate data can also be used as the input data of the next round of encoding. In this way, the first encoding information of N dimensions obtained includes both (N-1) intermediate data and the output data of the last round of encoding.

[0129] In the speech recognition model, N speech coding layers can be connected in series in sequence. Then, according to the series connection order, the input of the first speech coding layer includes a speech sample, and the output of the first speech coding layer is the first coding information generated by performing a coding process on the speech sample. Except for the first speech coding layer, the input of each speech coding layer is the first coding information output by the previous speech coding layer, and the output of each speech coding layer is the first coding information generated by performing a coding process on the first coding information output by the previous speech coding layer.

[0130] In the N speech coding layers, each speech coding layer performs coding processing on the speech sample, and the specific processing process of obtaining the first coding information output by each speech coding layer is as follows: Inputting the speech sample into the first speech coding layer for a first coding process, obtaining the first coding information of the first speech coding layer and outputting it; Inputting the first coding information of the first speech coding layer into the second speech coding layer for a second coding process, obtaining the first coding information of the second speech coding layer and outputting it; … The first coding information of the (N-1)th speech coding layer is input into the Nth speech coding layer for Nth coding processing, and the first coding information of the Nth speech coding layer is obtained and output.

[0131] When i=1, the speech sample is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension; when i is greater than 1, the speech coding information of the i-1-th dimension output by the i-1-th speech coding layer is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension.

[0132] After the speech sample is input into the first speech coding layer for the first encoding process, the first encoding information output by the first speech coding layer can be obtained; after the first encoding information output by the first speech coding layer is input into the second speech coding layer for the second encoding process, the first encoding information output by the second speech coding layer can be obtained; after the first encoding information output by the second speech coding layer is input into the third speech coding layer for the third encoding process, the first encoding information output by the third speech coding layer can be obtained; ...; after the first encoding information output by the (N-1)th speech coding layer is input into the Nth speech coding layer for the Nth encoding process, the first encoding information output by the Nth speech coding layer can be obtained.

[0133] It should be noted that, among the N speech coding layers in the speech recognition model, the encoding methods adopted by each speech coding layer can be the same or different. However, considering that except for the first speech coding layer, the remaining (N-1) speech coding layers are all obtained by encoding the data output by the previous speech coding layer, therefore, the first encoding information output by each of the N speech encoders is different, and can be used to reflect the characteristics of the speech sample in different dimensions.

[0134] For example, in the process of generating the first coding information of the first dimension, the first speech coding layer performs a round of coding processing on the speech sample; in the process of generating the first coding information of the second dimension, the first speech coding layer performs a round of coding processing on the speech sample, and the second speech coding layer performs another round of coding processing on this basis; in the process of generating the first coding information of the third dimension, the first speech coding layer performs a round of coding processing on the speech sample, and the second speech coding layer performs another round of coding processing on this basis, and the third speech coding layer performs another round of coding processing on this basis… In addition, the reason why it is necessary to obtain the first encoding information output by each speech coding layer instead of only obtaining the first encoding information output by the last speech coding layer is that the N speech coding layers can be generated based on the speech pre-training model. The performance of each speech coding layer in the speech pre-training model is obviously different when processing different speech recognition tasks. Therefore, when processing different tasks, different weight information can be configured for each coding layer in the speech pre-training model.

[0135] The following can be combined Figure 4 The weight information of each encoding layer in the speech pre-training model when processing different speech tasks is exemplified. Figure 4 A weight information diagram of a speech pre-training model in different tasks provided in an embodiment of the present application.

[0136] like Figure 4 As shown, speech pre-training model 1 includes 12 coding layers, and each coding layer has different weight information for different tasks. Speech pre-training model 2 includes 24 coding layers, and each coding layer has different weight information for different tasks.

[0137] After performing N encoding processes on the speech sample at N speech coding layers, obtaining the first encoding information output by each speech coding layer, and obtaining the first encoding information of N dimensions, the audio features of the speech sample can be determined based on the first encoding information of the N dimensions and the weight parameters of each dimension.

[0138] In one or more embodiments of the present application, during the first training process of the speech recognition model, the weight parameter of each speech coding layer may be a custom configured weight initial value of the speech coding layer. During the kth training process of the speech recognition model, k is an integer greater than 1, and the weight parameter of each speech coding layer may be obtained by adjusting the weight parameter of the k-1th training according to the training loss of the k-1th training.

[0139] The speech recognition model can be an acoustic model based on deep learning whose parameters are to be trained. The speech recognition model can be used to identify and process speech objects in speech data. The speech objects can be people speaking in the speech data, or robots speaking in the speech data, etc. The speech recognition model can be used to perform speaker log tasks. Specifically, given a speech containing multiple people speaking alternately, the task needs to determine who is speaking at each time point.

[0140] In the case where the speech recognition model includes N speech coding layers, each speech coding layer has a weight parameter, which is one of the parameters to be trained of the speech recognition model. The N dimensions correspond one-to-one to the N weight parameters.

[0141] When the speech recognition model includes M speech coding layers and M is greater than N, each speech coding layer has a weight parameter. It should be noted that in the model training process of the speech recognition model, among the M weight parameters, only N of them belong to the parameters to be trained of the speech recognition model, and the remaining (MN) weight parameters can be configured to 0 and remain unchanged during the model training stage. The N weight parameters involved in the model training correspond one to one to the N dimensions. This is because in different speech recognition tasks, the first coding information of each dimension is different, and the first coding information of some dimensions may be regarded as redundant data, which is of no help to the subsequent speech recognition. Therefore, the weight parameters can be configured to 0 to reduce interference factors in the model training process.

[0142] Referring to the following formula (1), the audio features of the speech sample are determined based on the first coding information output by each coding layer and the weight parameter of each dimension. The first coding information output by each coding layer and the weight parameter of each dimension are weighted and summed to obtain the comprehensive coding information of the speech sample, and the comprehensive coding information is used as the audio feature of the speech sample.

[0143] (1) in, Refers to the comprehensive coding information, norm_weights is the weight parameter of the speech coding layer, is the first coded information output by the speech coding layer, the sum function represents summation, dim represents dimension, dim is used to specify the elements participating in weighted summation, when dim=1, the sum function is used to sum by column, each element in the column is the product of the weight parameter of a dimension and the first coded information output by the speech coding layer corresponding to the dimension ( ).

[0144] The speech recognition model may include multiple speech coding layers, and the first coding information output by each speech coding layer performs differently in different speech tasks. For example, the first coding information output by coding layer 1 performs extremely well in the speaker log task of the first scenario, but performs poorly in the voiceprint recognition task of the second scenario... Considering that multiple speech coding layers can be generated based on a speech pre-training model, and the speech pre-training model is not a proprietary model designed for the speaker log task, but a general-purpose model, therefore, in the first coding information output by each speech encoder, some features may have a great effect on the speaker log task, some features have a small effect, and some features have an effect between the first two. By configuring corresponding weight parameters for each dimension and establishing a corresponding relationship between the encoder and the dimension, comprehensive coding information that fully utilizes the feature extraction capabilities of each speech encoder of the speech pre-training model can be generated. The comprehensive coding information can be used as an audio feature of a speech sample, and each weight parameter can be learned and can be continuously adjusted and optimized during the model training process.

[0145] In some implementations, the speech recognition model may include a coding module, a linear layer, and a recognition model connected in series. The output of the coding module is the input of the linear layer, and the output of the linear layer is the input of the recognition module. The coding module may include multiple speech coding layers, which may be coding layers in a speech pre-training model.

[0146] The linear layer can be used to execute the step of "determining the audio features of the speech sample based on the first encoding information output by each encoding layer and the weight parameter of each dimension", the input of the linear layer is the first encoding information of the speech sample in N dimensions, and the output of the linear layer is the audio features of the speech sample. The recognition module can be used to execute the subsequent step S206.

[0147] The weight parameter of each dimension can be a learnable model parameter in the linear layer of the speech recognition model, that is, the weight parameter of each dimension can be continuously adjusted and optimized during the model training process of the speech recognition model.

[0148] In a specific implementation, the audio features of the speech sample are determined based on the first coding information output by each coding layer and the weight parameters of each dimension, including: for each dimension, comparing the weight parameter of the dimension with a preset weight threshold; if the comparison result indicates that the weight parameter of the dimension is less than the preset weight threshold, resetting the weight parameter of the dimension to a preset value; and performing weighted summation processing based on the weight parameter of each dimension after reset and the first coding information to obtain the audio features of the speech sample.

[0149] The preset weight threshold may be a custom configured weight parameter value with a relatively small value.

[0150] The weight parameter of each dimension is compared with the preset weight threshold. If the weight parameter is less than the preset weight threshold, the weight parameter is reset to the preset value; if the weight parameter is greater than or equal to the preset weight threshold, the value of the weight parameter remains unchanged.

[0151] Exemplarily, the preset weight threshold of each dimension can be set as needed, the sum of the preset weight threshold of each dimension is 1, and the preset value can be 0.

[0152] In the process of obtaining the audio features of the speech sample by weighted summing up the weight parameters of each dimension after the reset and the first coding information, there may be a part of the dimensions whose weight parameters are less than the preset weight threshold. For the convenience of distinction, this part of the dimensions can be called the first dimension, and the weight parameters of the first dimension need to be reset to 0; in the process of obtaining the audio features of the speech sample by weighted summing up the weight parameters of each dimension after the reset and the first coding information, there may be another part of the dimensions whose weight parameters are greater than or equal to the preset weight threshold. For the convenience of distinction, this other part of the dimensions can be called the second dimension, and the weight parameters of the second dimension do not need to be reset. The weight parameters of each dimension after the reset include the weight parameters of the second dimension and the weight parameters of the first dimension after the reset.

[0153] The audio features of the speech sample are obtained by performing weighted summation processing based on the weight parameters of each dimension after reset and the first coding information. The corresponding description part of the audio features of the speech sample can be determined by referring to the aforementioned first coding information based on the output of each coding layer and the weight parameters of each dimension.

[0154] For any dimension, if the value of the weight parameter of the dimension is too small, it means that the first encoding information of the speech sample in the dimension may be unnecessary for the speaker log task. The weight parameter can be reset to 0 before weighted calculation.

[0155] In addition, the weight parameter k1 before the reset can also be retained. In the process of performing the weighted calculation this time, the weight parameter k1 of the coding layer is replaced by 0. When the weight parameter of the coding layer is subsequently adjusted according to the training loss, it is adjusted based on k1.

[0156] Step S206: Identify the speech object of each frame of audio data in the speech sample according to the audio feature, and obtain a second speech recognition result of the speech object.

[0157] The voice object may be a person making the voice in the voice sample, a robot making the voice in the voice sample, or other objects making simulated human voice in the voice sample.

[0158] The second speech recognition result of the speech object may include multiple time periods and recognition sub-results of each time period. The multiple time periods are arranged in chronological order.

[0159] In multiple time periods, two adjacent time periods may be continuous, that is, the end time of the previous time period is the same as the start time of the next time period. For example, multiple time periods include: [T1, T2), [T2, T3), etc. Among them, the end time T2 of the time period [T1, T2) is the same as the start time T2 of the time period [T2, T3).

[0160] In multiple time periods, there may be a time interval between two adjacent time periods, that is, the end time of the previous time period is different from the start time of the next time period. For example, multiple time periods include: [T1, T2), [T3, T4), etc. Among them, the end time T2 of the time period [T1, T2) is different from the start time T3 of the time period [T3, T4).

[0161] The recognition sub-result of each time period corresponds to one or more speech objects, and the recognition sub-results of two adjacent time periods are different.

[0162] The recognition sub-result of each time period includes one or more speech objects uttered in the audio segment corresponding to the time period in the speech sample.

[0163] For example, multiple time periods include: [T1, T2), [T2, T3), and so on. The recognition sub-result of the time period [T1, T2) indicates that in the speech sample, only one speech object A appears in the audio segment corresponding to the time period [T1, T2), and the recognition sub-result of the time period [T2, T3) indicates that in the speech sample, speech object A and speech object B appear at the same time in the audio segment corresponding to the time period [T2, T3). It can be understood that in the time period [T1, T2), only speech object A speaks, and in the time period [T2, T3), while speech object A speaks, speech object B also makes a sound.

[0164] The multiple time periods and the recognition sub-results of each time period may be generated according to the speech object of each frame of audio data in the speech sample.

[0165] For example, the voice objects in the first frame of audio data, the second frame of audio data, and the third frame of audio data are the same and can all be represented by the object identification group (A, B). The voice object in the fourth frame of audio data is different from the voice object in the third frame of audio data. It can be determined that time period 1 includes the first frame of audio data, the second frame of audio data, and the third frame of audio data, and the recognition sub-result of time period 1 is (A, B).

[0166] According to the audio features, the speech object in the speech sample is identified to obtain the second speech recognition result of the speech object. The audio features may be input into the recognition module in the speech recognition model to perform recognition processing on the speech object to obtain the second speech recognition result of the speech sample.

[0167] According to the audio features, the speech object of each frame of audio data in the speech sample is identified to obtain a second speech recognition result of the speech object. Alternatively, the speech sample may be first divided and processed according to the audio features to obtain one or more audio segments. In the process of identifying the speech object of each audio segment, for each audio segment, the audio features are compared with each pre-stored audio feature in a preset feature set to obtain a comparison result. According to the comparison result, the speech object of the audio segment is determined to generate a second speech recognition result of the speech sample.

[0168] Each audio segment may include one or more frames of audio data. When an audio segment includes multiple frames of audio data, the speech objects of the multiple frames of audio data are the same. Identifying the speech object of each audio segment is equivalent to identifying the speech objects of the multiple frames of audio data in the audio segment at one time.

[0169] Exemplarily, the audio feature may include a voiceprint feature.

[0170] The speech sample is divided and processed according to the audio features to obtain one or more audio segments, which may be divided into one or more audio segments arranged in time order according to the voiceprint features, and the similarity of the voiceprint features of any two adjacent audio segments is less than a preset similarity threshold. For each audio segment, the audio feature is compared with each pre-stored audio feature in the preset feature set to obtain a comparison result, which may be compared with each candidate voiceprint feature in a pre-configured voiceprint library. If there is a candidate voiceprint feature identical to the voiceprint feature in the voiceprint library, the voice object corresponding to the candidate voiceprint feature is determined as the voice object in the audio segment.

[0171] In a specific implementation, a speech sample includes K frames of audio data, the number of audio features is K, and the K frames of audio data correspond one-to-one to the K audio features; based on the audio features, the speech object of each frame of audio data in the speech sample is identified to obtain a second speech recognition result of the speech object, including: encoding the K audio features to obtain first feature data of the K frames of audio data; decoding the first feature data of each frame of audio data to obtain second feature data describing the speech object of each frame of audio data; classifying the second feature data of each frame of audio data to obtain the speech object of each frame of audio data; and obtaining a second speech recognition result of the speech object based on the speech objects of the K frames of audio data.

[0172] The recognition module in the speech recognition model may include an encoding unit, a decoding unit and a classification unit connected in series, wherein the output of the encoding unit is the input of the decoding unit, and the output of the decoding unit is the input of the classification unit.

[0173] The encoding unit can use any encoder suitable for the speaker log task, for example, EEND (End-to-End Neural Diarization) and its improved series of encoders, the conformer structure for voiceprint recognition, Resnet (deep residual network) and its related variants, etc.

[0174] EEND is an end-to-end neural network model for speaker segmentation and clustering tasks in ASR (Automatic Speech Recognition), also known as speaker logging tasks. The EEND encoder refers to the part of the model that processes the input audio signal and extracts features.

[0175] The conformer structure is a deep learning architecture that combines a convolutional neural network and a Transformer model.

[0176] The speech sample may include K frames of audio data, the number of audio features is K, and the K frames of audio data correspond to the K audio features one by one. K is an integer greater than 1.

[0177] In a specific implementation, the encoding unit may encode K audio features to obtain first feature data of K frames of audio data. The first feature data may be feature data at the frame level describing the speech sample. The number of first feature data may be K. Each frame of audio data has a corresponding first feature data.

[0178] For example, a speech sample includes 100 frames of audio data, the number of audio features is 100, and the 100 frames of audio data correspond to the 100 audio features one by one. Through encoding processing, each audio feature can be converted into first feature data at a frame level.

[0179] The decoding unit may adopt any decoder suitable for the speaker log task, for example, LSTM (Long Short-Term Memory), EEND and its improved series decoder, etc.

[0180] In a specific implementation, the decoding unit may decode the first feature data of each frame of audio data to obtain second feature data describing the voice object of each frame of audio data. The second feature data may be frame-level feature data describing the voice object.

[0181] For example, the first feature data at the level of 100 frames is input into the decoding unit for decoding processing, and the second feature data at the level of 100 frames can be obtained.

[0182] The classification unit may adopt any classification layer, which may call a classification function to process input data. For example, the classification function may be a Softmax function.

[0183] In a specific implementation, the classification layer performs classification processing according to the second feature data of each frame of audio data to obtain a classification result of each frame of audio data, and the classification result may correspond to one of a plurality of preset classification labels. The plurality of preset classification labels may refer to the classification annotation information of the aforementioned voice sample. The classification result of each frame of audio data may be used to represent the voice object of each frame of audio data.

[0184] Classification processing is performed according to the second feature data to obtain a classification result. The classification function can be called to calculate the probability value of the classification result of the second feature data corresponding to each preset classification label, and the probability value is used as the classification result, or the classification result is determined according to the probability value.

[0185] For example, the probability value of the classification result of the second feature data calculated by calling the classification function corresponding to the classification label 1 is p1, the probability value of the classification result of the second feature data calculated by calling the classification function corresponding to the classification label 2 is p2, ..., the probability value of the classification result of the second feature data calculated by calling the classification function corresponding to the classification label 5 is p5, etc. Among them, if p1 is greater than or equal to the preset probability threshold, and p2-p5 are all less than the preset probability threshold, it can be determined that the classification result of the second feature data corresponds to the classification label 1.

[0186] Among multiple preset classification labels, some preset classification labels correspond to only one voice object, and other preset classification labels correspond to multiple voice objects. By predicting the possibility of occurrence of each situation through classification, the risk of confusing different voice objects when multiple people speak at the same time can be reduced.

[0187] Since the second feature data is frame-level feature data that describes the speech object, the classification result of each second feature data can reflect the classification result of the corresponding frame of the second feature data. Furthermore, the speech object of each frame of audio data in K frames of audio data can be added to the second speech recognition result of the speech object.

[0188] For example, the number of second feature data at the frame level is 10, and each of the 10 second feature data has a classification result: the classification result of the first second feature data is speech object 1; the classification result of the second second feature data is speech object 1; the classification result of the third second feature data includes speech object 1 and speech object 2;...; the classification result of the tenth second feature data includes speech object 1 and speech object 2.

[0189] From this we can see that K=10. Among the 10 frames of audio data, the voice object of the first frame of audio data is voice object 1; the voice object of the second frame of audio data is voice object 1; the voice object of the third frame of audio data includes voice object 1 and voice object 2; ...; the voice object of the tenth frame of audio data includes voice object 1 and voice object 2.

[0190] Furthermore, the voice object of the 10 frames of audio data may be added to the second voice recognition result of the voice object.

[0191] Take the example of a speech sample involving three speech objects ABC: the classification label of each frame of audio data can be one of the following: A, B, C, AB, AC, BC, ABC. Classification based on features means that for each frame of speech data, the feature of the frame of speech data is used to determine which of the above 7 categories the frame of data corresponds to. Assuming the classification result is BC, it means that the speech objects of the frame of audio data include B and C.

[0192] It should be emphasized that the training method of the speech recognition model provided in one or more embodiments of the present application is not to train the model to learn the features of a single speech object, and to identify whether A, B, or C are included in a frame of audio data. Instead, the training model learns how to distinguish between the features of only A making a sound, the features of only B making a sound, the features of AB making a sound at the same time, and other different situations. Therefore, there is no process of first identifying the speech object and then determining the classification result based on the speech object. Instead, it is directly classified according to the features. The classification result can indicate that there is only one speech object making a sound in the corresponding frame of speech data, or that multiple speech objects make sounds at the same time in the corresponding frame of speech data.

[0193] In addition, in the recognition module, the model parameters of the encoding unit and the model parameters of the decoding unit are both used as parameters to be trained, and can be continuously adjusted and optimized during the model training process of the speech recognition model.

[0194] Step S208, constructing a loss function according to the second speech recognition result and the first speech recognition result.

[0195] The first speech recognition result of the speech sample may include a classification label corresponding to each frame of audio data in the speech sample.

[0196] Constructing a loss function based on the second speech recognition result and the first speech recognition result may be constructing a loss function based on the classification result of the speech object of each frame of audio data in the speech sample and the classification label of each frame.

[0197] The loss function can be a cross entropy loss function or other custom loss function.

[0198] In a specific implementation, each classification label is a first object identification group, each object identification group includes a first identification and the number of first identifications is within a preset range; according to the audio features, the speech object of each frame of audio data in the speech sample is identified to obtain a second speech recognition result of the speech object, including: according to the audio features, the speech object of each frame of audio data is classified to obtain a second object identification group; a loss function is constructed according to the second speech recognition result and the first speech recognition result, including: a loss function is constructed according to the second object identification group of each frame of audio data and the first object identification group of each frame of audio data.

[0199] The classification label is used to represent the voice objects participating in the conversation in the corresponding frame of audio data. Specifically, each classification label is a first object identification group, and the voice objects in a frame of audio data can be represented in the form of a first object identification group. For example, the voice sample includes three objects ABC, including A, B, C, AB, AC, BC, ABC, a total of 7 classification labels, of which A, B, C, AB, AC, BC, ABC can be used to constitute 7 first object identification groups. A first identification represents a voice object, and each identification in the object identification group is a first identification. For example, A and C in the object identification group AC are both first identifications.

[0200] The maximum value of the preset range interval may be determined by the number of speech objects in each speech sample.

[0201] For example, the number of voice objects in voice sample 1 is 3, the number of voice objects in voice sample 2 is 2, and the number of voice objects in voice sample 3 is 4. Then, the maximum value of the preset range interval is 4, and the minimum value is 2.

[0202] The minimum value of the preset range may be 1. In this case, the classification label may further include a silence label indicating that the number of voice objects in a corresponding frame of audio data is 0.

[0203] Taking the maximum value of the preset range interval as 3 and the minimum value as 1 as an example, the classification labels are exemplified: the speech objects in each speech sample include object A, object B and object C, then classification label 1 is the first object identification group (A), classification label 2 is the first object identification group (B), classification label 3 is the first object identification group (C), classification label 4 is the first object identification group (A, B), classification label 5 is the first object identification group (B, C), classification label 6 is the first object identification group (A, C), and classification label 7 is the first object identification group (A, B, C). In addition, classification label 8 is a silent label S. The classification label of each frame of audio data in the speech sample is one of the above-mentioned classification labels 1 to 8.

[0204] For example, in speech sample 1, the classification label carried by the first frame of audio data is (A, B), the classification label carried by the second frame of audio data is (A, C), the classification label carried by the third frame of audio data is (A), the classification label carried by the fourth frame of audio data is S, and so on.

[0205] According to the audio features, the speech objects in each frame of audio data are classified to obtain a second object identification group.

[0206] The second object identification group of each frame of audio data is the classification result of each frame of audio data.

[0207] The second object identification group may be generated from the first object identification group.

[0208] For example, the classification labels include: a first object identification group (A), a first object identification group (B), and a first object identification group (A, B).

[0209] Then, the second object identification group of each frame of audio data may be one of the following: a second object identification group (A), a second object identification group (B), and a second object identification group (A, B).

[0210] It should be emphasized that in this implementation, there is no process of first identifying the speech object and then classifying it.

[0211] Take the example that the speech objects in the speech sample include A, B, and C: In this implementation, the speech recognition model is trained to learn, for a frame of audio data, what kind of features will be classified into the second object identification group (A), what kind of features will be classified into the second object identification group (A, B), what kind of features will be classified into the second object identification group (A, B, C), and so on.

[0212] The characteristics of A and B making sounds at the same time (that is, the mixed sound of A and B) are different from the characteristics of A making sounds alone, and the characteristics of the mixed sound of A and B are also different from the characteristics of B making sounds alone.

[0213] That is, when there are multiple voice objects speaking in a frame of audio data at the same time, the speech recognition model will not separately identify whether A exists in the frame of audio data, and whether B exists in the frame of audio data, but will classify this frame of audio data into the second object identification group (A, B) according to the audio features.

[0214] (A, B), (A, C), (B, C), (A, B, C), (A), (B), (C), these 7 categories are completely equal. Taking (A, B) as an example, during the model training process, it can be roughly regarded that the speech recognition model treats (A, B) as a completely new speech object that is different from (A) and (B) to learn its characteristics.

[0215] Therefore, the classification process is to identify the speech object, and the classification result may include one speech object or multiple speech objects.

[0216] In the model training process of the speech recognition model, the first object identification group can assist the speech recognition model in learning how to distinguish the features of multi-person mixed speech with different speech objects belonging to the mixed speech, for example, distinguishing (A, B) from (A, C), and the first object identification group can assist the speech recognition model in learning how to distinguish the features of a single person's speech from the features of multi-person mixed speech, for example, distinguishing (A) from (A, B). In this way, when a frame of audio data includes multiple speech objects, it is not necessary to identify each of the multiple speech objects separately, which can reduce the interference caused by each speech object in the multi-person mixed speech to the recognition of other speech objects.

[0217] In addition, after classifying the voice object of each frame of audio data according to the audio features, the classification result may also be a silence mark indicating that there is no voice object in the corresponding frame of audio data.

[0218] For example, in speech sample 1, the classification result of the first frame of audio data is the second object identification group (A, C), the classification result of the second frame of audio data is the second object identification group (A, C), the classification result of the third frame of audio data is the second object identification group (A), the classification result of the fourth frame of audio data is silence identification, and so on.

[0219] A loss function is constructed based on the second object identification group of each frame of audio data and the first object identification group of each frame of audio data. A cross entropy loss function is generated based on the second object identification group and the first object identification group for each frame of audio data, or any other predetermined loss function may be used.

[0220] In the process of recognizing speech objects, each speech object can be regarded as a category. For each frame of audio data, the predicted probability that the frame of audio data includes the speech object is output, and the predicted probability is compared with a threshold. If the predicted probability is greater than the threshold, the frame of audio data is deemed to include the speech object, otherwise the speech object is not included. In the case where multiple people may speak at the same time, the accuracy of this method will be affected by the threshold. If the threshold is set reasonably, the prediction accuracy will be high. If the threshold is unreasonable, it is easy to make recognition errors. However, the threshold of the same speech recognition model is likely to be different in different scenarios, which makes the migration ability of the model relatively poor. The prediction is relatively accurate in one scenario, but it will be inaccurate when used to process speech data in another scenario. For example, the speech sample includes three speech objects A, B, and C. In the kth frame of audio data, k is an integer greater than 0, the predicted probability of identifying A is p1, the predicted probability of identifying B is p2, and the predicted probability of identifying C is p3. p1, p2, and p3 are all values ​​between 0 and 1. If p1 is greater than P1, it is determined that the speech object of the kth frame of speech data includes A. If p2 is greater than P2, it is determined that the speech object of the kth frame of audio data includes B. If p3 is greater than P3, it is determined that the speech object of the kth frame of audio data includes C.

[0221] In the training method of the speech recognition model provided in the embodiment of the present application, multiple speech objects that sound simultaneously in the same frame of audio data can be regarded as one category. In this way, in the process of identifying the speech objects, there is only one classification result for each frame of audio data.

[0222] For example, the speech sample includes three objects ABC, and the classification labels include: A, B, C, AB, AC, BC, ABC, and a silent label S. The k-th frame of audio data is classified, k is a certificate greater than 0, and the classification result of the k-th frame of audio data is the second object identification group (B, C).

[0223] Compared with the above-mentioned technical solution of treating each speech object as a category, in the embodiment of the present application, all possibilities of one or more speech objects in a frame of speech data are listed, and each possibility is treated as a category. In this way, there is only one classification result for each frame of audio data. During the training process, the speech recognition model learns how to identify the speech object when there is only one speech object in a frame of audio data, and learns how to identify the number of speech objects when there are multiple speech objects in a frame of audio data at the same time, as well as who the multiple speech objects are. Therefore, the embodiment of the present application does not need to set a threshold and compare it with the threshold, which is more conducive to maintaining recognition accuracy when migrating the model to other scenarios.

[0224] In a specific implementation, the training method of the speech recognition model also includes: performing speech activity detection on the speech sample according to the audio features to obtain a first detection result; performing overlapping speech detection on the speech sample according to the audio features to obtain a second detection result; constructing a loss function according to the second speech recognition result and the first speech recognition result, including: constructing a loss function according to the second speech recognition result, the first detection result, the second detection result and the first speech recognition result.

[0225] VAD (Voice activity detection) is a technology used to identify whether there is active speech in a speech signal. It is mainly used in speech processing systems to help distinguish speech segments from non-speech segments (such as silence, background noise, etc.) in speech signals.

[0226] Overlapping speech detection may be detecting whether there are at least two speech objects uttering at the same time.

[0227] After encoding the audio features to obtain the first feature data of the speech sample, the first feature data can be feature data at the frame level that describes the speech sample. The first feature data can be used for speaker logging tasks, speech activity detection tasks, and overlapping speech detection tasks. Therefore, when it is necessary to perform speech object recognition processing, speech activity detection processing, and overlapping speech detection processing on the speech sample, the same encoding unit can be reused to encode the audio features to obtain intermediate encoding information, and then the intermediate encoding information can be decoded by different decoders to obtain decoded information, and the decoded information can be classified by different classification units.

[0228] According to the audio features, voice activity detection is performed on the voice sample to obtain a first detection result. The audio features may be encoded to obtain first feature data of the voice sample, the first feature data may be decoded to obtain voice activity features, and the voice activity features may be classified to obtain the first detection result.

[0229] According to the audio features, voice activity detection processing is performed in the voice sample to obtain a first detection result. Alternatively, the audio features are encoded to obtain first feature data of the voice sample, and the first feature data is classified to obtain a first detection result.

[0230] Since the first feature data is frame-level feature data, the first detection result corresponding to each first feature data is also a frame-level activity detection result.

[0231] The decoder for decoding the first feature data to obtain the voice activity feature may adopt a linear layer, or may adopt any decoder suitable for the voice activity detection task.

[0232] According to the audio features, the speech sample is subjected to overlapping speech detection processing to obtain a second detection result. The audio features may be encoded to obtain first feature data of the speech sample, the first feature data may be decoded to obtain overlapping speech features, and the overlapping speech features may be classified to obtain a second detection result.

[0233] Since the first feature data is frame-level feature data, the second detection result corresponding to each first feature data is also a frame-level overlapping speech detection result.

[0234] The decoder used to decode the first feature data to obtain the overlapping speech features may use a linear layer or any decoder suitable for the overlapping speech detection task.

[0235] The first speech recognition result of the speech sample may include time annotation information, category annotation information, and may also include speech activity annotation information and overlapping speech annotation information.

[0236] The speech activity annotation information may include one or more time periods in which speech signals exist in the speech sample, and the overlapping speech annotation information may include one or more time periods in which at least two speech objects make sounds.

[0237] Constructing a loss function based on the second speech recognition result, the first detection result, the second detection result and the first speech recognition result may be as follows: generating a first loss sub-function based on the time annotation information and the classification annotation information in the second speech recognition result and the first speech recognition result; generating a second loss sub-function based on the speech activity annotation information in the first detection result and the first speech recognition result; generating a third loss sub-function based on the overlapping speech annotation information in the second detection result and the first speech recognition result; and generating a loss function of the speech recognition model based on preset weight information, the first loss sub-function, the second loss sub-function and the third loss sub-function.

[0238] The first loss sub-function, the second loss sub-function, and the third loss sub-function may be a cross entropy loss function or other loss functions of a custom configuration.

[0239] In specific implementation, a weighted summation process can be performed based on the preset weight information, the first loss sub-function, the second loss sub-function and the third loss sub-function to obtain a comprehensive loss function, and the comprehensive loss function is used as the loss function of the speech recognition model. The preset weight information is a parameter to be trained during the training process of the speech recognition model, and the weight can be adjusted according to the prediction of the speaker log. For example, if the error rate of the overlapping sound detection part is high, the weight corresponding to the third loss sub-function is increased. If the prediction of whether there is a sound signal is poor, the weight corresponding to the third loss sub-function is appropriately increased.

[0240] The reason why the voice activity detection task, overlapping speech detection task and speaker log task are jointly trained is that in the process of executing the speaker log task, errors that are prone to occur include: misidentifying a frame of audio data as having one or more voice objects when there is no sound signal in the frame of audio data; misidentifying a frame of audio data as not having a sound signal when there is a sound signal from one or more voice objects in the frame of audio data; confusing the voice objects appearing in a frame of audio data. By executing the voice activity detection task, overlapping speech detection task and speaker log task at the same time, the first detection result and the second detection result can be used to assist in judging whether the recognition result is accurate. Therefore, during the training process, the model parameters are adjusted by the loss function, the voice activity detection task, overlapping speech detection task and speaker log task are optimized at the same time, and the speaker log error is directly minimized, which can achieve a complementary effect.

[0241] In one or more embodiments of the present application, the loss function of the speech recognition model may be constructed based on the second speech recognition result and the first speech recognition result; in one or more embodiments of the present application, the loss function of the speech recognition model may be constructed based on the second speech recognition result, the first detection result and the first speech recognition result; in one or more embodiments of the present application, the loss function of the speech recognition model may be constructed based on the second speech recognition result, the second detection result and the first speech recognition result; in one or more embodiments of the present application, the loss function of the speech recognition model may be constructed based on the second speech recognition result, the first detection result, the second detection result and the first speech recognition result. How the loss function of the speech recognition model is constructed can be flexibly configured according to actual needs.

[0242] In a specific implementation, the audio features include a first audio feature that describes the energy of the speech sample, a second audio feature that describes the frequency spectrum of the speech sample, and a third audio feature that describes the signal waveform of the speech sample; based on the audio features, speech activity detection is performed on the speech sample to obtain a first detection result, including: determining a first time period in which active speech exists in the speech sample based on the first audio feature, the second audio feature, and the third audio feature; determining a second time period in which active speech is missing based on the total time period of the speech sample and the first time period; and generating a first detection result based on the first time period and the second time period.

[0243] The first audio feature may be the energy of the speech sample, or other audio features that can directly or indirectly reflect the energy of the speech sample. For any speech data, the energy of the speech data refers to the energy of the speech signal within a certain time window. The energy of a speech segment with active speech is usually higher than that of a non-speech segment, so energy can be used as one of the references for speech activity detection.

[0244] The second audio feature may be a spectral feature of the speech sample, and illustratively, the spectral feature may be a formant, spectral flatness, etc. There is a clear difference between the spectral feature of a speech segment with active speech and the spectral feature of a non-speech segment, so the spectral feature can be used as one of the references for speech activity detection.

[0245] The third audio feature may be the zero-crossing rate of the speech sample, or other audio features that can directly or indirectly reflect the waveform characteristics of the speech sample. The zero-crossing rate refers to the number of times the signal waveform crosses the zero point per unit time. The zero-crossing rate energy of a speech segment with active speech is usually higher than that of a non-speech segment, so the zero-crossing rate can be used as one of the reference bases for speech activity detection.

[0246] In the process of determining a first time period in which active speech exists in a time period of a speech sample based on a first audio feature, a second audio feature, and a third audio feature, the following operations may be performed: for each frame of audio data in the speech sample, performing activity prediction processing on each frame of audio data according to the first audio feature to obtain a first prediction result; performing activity prediction processing on each frame of audio data according to the second audio feature to obtain a second prediction result; performing activity prediction processing on each frame of audio data according to the third audio feature to obtain a third prediction result; and generating a comprehensive prediction result for each frame of audio data based on the first prediction result, the second prediction result, and the third prediction result.

[0247] If the comprehensive prediction result indicates that there is active speech in a frame of audio data, a first mark is added to the frame of audio data. According to the first mark, a first time period in which there is active speech is determined in the speech sample, and each frame of audio data in the first time period carries the first mark.

[0248] Determining the second time period in which active speech is absent according to the time period of the speech sample and the first time period may be to determine the portion of the time period of the speech sample outside the first time period as the second time period.

[0249] A first detection result is generated according to the first time period and the second time period, wherein the first detection result indicates that active speech exists in the first time period and active speech does not exist in the second time period.

[0250] Step S210, iteratively training the speech recognition model based on the loss function.

[0251] The speech recognition model is iteratively trained based on the loss function to obtain the target speech recognition model. The specific implementation process of each model training can refer to steps S204-S206. After obtaining the speech recognition results corresponding to each model training, the training loss of the model training can be calculated based on the loss function, the second speech recognition result and the first speech recognition result. The training loss can be used to drive the adjustment of the model parameters of the speech recognition model.

[0252] Through iterative training, the loss can be continuously optimized and reduced until the training loss converges to less than or equal to the preset loss threshold.

[0253] The parameters to be trained of the speech recognition model include but are not limited to: weight parameters of each dimension, model parameters of the encoding unit in the recognition module, model parameters of the decoding unit in the recognition module, and so on.

[0254] In specific implementation, model training can also be divided into multiple stages, for example: In the first stage, the model parameters of multiple speech coding layers are frozen during the iterative training of the speech recognition model; in the second stage, the model parameters of multiple speech coding layers are unfrozen, and the model parameters of each speech coding layer are continuously adjusted and optimized during the iterative training of the speech recognition model. In the second stage, the ratio of the number of first speech samples constructed by real conversations to the second speech samples constructed by simulated conversations in the training samples can be adjusted, so as to use more first speech samples constructed by real conversations; in the third stage, only speech samples constructed by conversation data of the target business scenario are used for model training.

[0255] In addition, when the amount of dialogue data in the target business scenario is small, in order to avoid overfitting that reduces the training effect of the model, only the first two stages of model training mentioned above can be performed.

[0256] In addition, before the training of the machine learning model is started, it may be necessary to rely on manual experience and historical training data to configure the parameters of the model to be trained. For example, the model parameters of the machine learning model that performs similar tasks can be found from the historical training data, and the model parameters of the machine learning model to be trained can be configured based on the model parameters.

[0257] However, not all models can find corresponding historical training data, and the lack of human experience is also common. In this case, users may need to spend some time and energy in the parameter configuration process. Compared with directly using ready-made historical training data for parameter configuration, the overall model training time is longer and the model training efficiency is relatively low.

[0258] The training method of the speech recognition model provided by one or more embodiments of the present application can help the speech recognition model learn how to use the features of different dimensions of the speech sample to generate audio features suitable for speech recognition tasks to identify speech objects by establishing the corresponding relationship between each dimension and the speech coding layer, and continuously adjusting and optimizing the weight parameters of each dimension during the iterative training process. In this way, in the absence of historical training data as a reference, it is not necessary to pre-determine the importance of features of different dimensions and assign corresponding weights before model training, thereby improving the efficiency of model training. Moreover, each speech coding layer can be generated based on a pre-trained language model, which can be reused by different speech recognition tasks. Therefore, there is no need to pre-design a highly targeted coding module, and a general model can be used directly. In addition, the speech pre-training model can be a model that has been pre-trained with a large amount of unlabeled data, has very good generalization, and only needs to be fine-tuned on a small scale of labeled data to achieve a good training effect. Using the speech pre-training model for speaker log tasks is conducive to improving the efficiency of model training.

[0259] In such Figure 2In the illustrated embodiment, first, a speech sample and a first speech recognition result that annotates the speech sample are obtained; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, and each classification label represents a speech object participating in the conversation in a corresponding frame of audio data; then, feature extraction is performed on the speech sample to obtain audio features of the speech sample; then, based on the audio features, the speech object of each frame of audio data in the speech sample is identified to obtain a second speech recognition result of the speech object; next, a loss function is constructed based on the second speech recognition result and the first speech recognition result; finally, the speech recognition model is iteratively trained based on the loss function. Taking into account that in the case where a voice sample involves multiple voice objects participating in a conversation, each voice object may participate in the conversation at the same time. For example, three people speak at the same time. In one frame of audio data, the sounds of multiple voice objects participating in the conversation at the same time are mixed together. Each voice object may interfere with the recognition accuracy of other voice objects. In an embodiment of the present application, the first voice recognition result for annotating the voice sample includes a classification label for each frame of audio data. Each classification label represents a voice object participating in the conversation in the corresponding frame of audio data. In this way, even if there are multiple voice objects participating in the conversation at the same time in one frame of audio data, the one frame of audio data has only one classification label, which can be combined with the second classification label during the training process. The speech recognition result is used to construct a loss function, and then the loss function can be used to guide the speech recognition model to learn how to map the one frame of audio data to an output category composed of multiple speech objects, that is, the second speech recognition result of the speech object in the one frame of audio data. Furthermore, the sounds of multiple speech objects mixed together do not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution of guiding the speech recognition model to learn how to map one frame of speech data to multiple output categories, each output category corresponding to a speech object, the embodiment of the present application can reduce the negative impact of the mutual interference of the sounds of each speech object uttered at the same time on the recognition of the speech object, and is more conducive to improving the recognition accuracy of the speech object.

[0260] Based on the same technical concept, the embodiment of the present application also provides another method for training a speech recognition model. Figure 5 A data flow diagram of another method for training a speech recognition model provided in an embodiment of the present application.

[0261] like Figure 5As shown, the speech sample is input into the speech pre-training model 502 for encoding processing, and the first encoding information output by each speech encoding layer in the speech pre-training model 502 is obtained. For details, please refer to the corresponding description part of the aforementioned step S204. In the speech pre-training model 502, each speech encoding layer has a corresponding dimension. The first encoding information output by each speech encoding layer can be regarded as the first encoding information of the dimension corresponding to the speech coding layer of the speech sample. The dimension corresponding to the speech coding layer may be referred to as the coding layer dimension hereinafter.

[0262] The linear layer 504 with learnable weights generates comprehensive coding information of the speech sample according to the first coding information output by each speech coding layer and the weight parameters of each coding layer dimension, and uses the comprehensive coding information as the audio feature of the speech sample. For details, please refer to the corresponding description part of the aforementioned step S204.

[0263] The encoding unit 506, the decoding unit 508 and the classification unit 510 are used together to perform the following steps: according to the audio features, identify the voice object in the voice sample and obtain a second voice recognition result of the voice object. For details, please refer to the corresponding description part of the aforementioned step S206.

[0264] The second speech recognition result of the speech object can be regarded as the prediction result 512.

[0265] A loss function is constructed according to the prediction result 512 and the first speech recognition result 514 of the speech sample, and the speech recognition model is iteratively trained based on the loss function to obtain a target recognition model. For details, please refer to the corresponding description parts of the aforementioned steps S208 and S210.

[0266] The loss function may be a cross entropy loss 516 .

[0267] The cross entropy loss 516 may be used to drive the adjustment of the model parameters of the linear layer 504 , the encoding unit 506 , and the decoding unit 508 of the learnable weights.

[0268] Since the technical concept is the same, the description in this embodiment is relatively simple, and the relevant parts may refer to the corresponding description of the method embodiment provided above.

[0269] Based on the same technical concept, the embodiment of the present application also provides another method for training a speech recognition model. Figure 6 A data flow diagram of another method for training a speech recognition model provided in an embodiment of the present application.

[0270] like Figure 6As shown, the speech sample is input into the speech pre-training model 602 for encoding processing, and the first encoding information output by each speech encoding layer in the speech pre-training model 602 is obtained. For details, please refer to the corresponding description part of the aforementioned step S204.

[0271] The linear layer 604 with learnable weights generates comprehensive coding information of the speech audio sample according to the first coding information output by each speech coding layer and the weight parameters of each coding layer dimension, and uses the comprehensive coding information as the audio feature of the speech audio sample. For details, please refer to the corresponding description part of the aforementioned step S204.

[0272] The encoding unit 606, the speaker log decoding unit 608 and the speaker log classification unit 610 are used together to perform the following steps: according to the audio features, identify the speech object of each frame of audio data in the speech sample, and obtain a second speech recognition result 612 of the speech object. For details, please refer to the corresponding description part of the aforementioned step S206.

[0273] The encoding unit 606 , the voice activity detection decoding unit 614 and the voice activity detection classification unit 616 are used together to perform the following steps: performing voice activity detection processing on the audio sample according to the audio features to obtain a first detection result 618 .

[0274] The encoding unit 606 , the overlapping speech detection decoding unit 620 and the overlapping speech detection classification unit 622 are used together to perform the following steps: performing overlapping speech detection processing on the audio sample according to the audio features to obtain a second detection result 624 .

[0275] A loss function is constructed based on the speech recognition result 612, the first detection result 618, the second detection result 624, and the first speech recognition result 626 of the audio sample.

[0276] Since the technical concept is the same, the description in this embodiment is relatively simple, and the relevant parts may refer to the corresponding description of the method embodiment provided above.

[0277] One or more speech recognition method embodiments provided in this specification are as follows: In the above-mentioned embodiment, a method for training a speech recognition model is provided. Based on the same technical concept, an embodiment of the present application also provides a speech recognition method. Figure 7 A processing flow chart of a speech recognition method provided in an embodiment of the present application.

[0278] like Figure 7As shown, in step S702, the first speech data is input into the target speech recognition model to perform recognition processing of the speech object to obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the training method of the speech recognition model.

[0279] In this step, the training method of the speech recognition model refers to the training method of the speech recognition model provided by any one of the aforementioned method embodiments.

[0280] Step S704: divide the first speech data into a plurality of time periods according to the speech recognition result of the speech object, and mark the speech object in each of the plurality of time periods.

[0281] In some implementations, during the model training phase: According to the audio features, the voice object in the voice sample is identified to obtain the second voice recognition result of the voice object; according to the audio features, the audio sample is subjected to voice activity detection processing to obtain the first detection result; according to the audio features, the audio sample is subjected to overlapping voice detection processing to obtain the second detection result; according to the second voice recognition result, the first detection result, the second detection result and the first voice recognition result, a loss function is constructed, and the voice recognition model is iteratively trained based on the loss function to obtain the target voice recognition model. For details, please refer to Figure 2 The corresponding description part of step S204-step S210 in the embodiment.

[0282] During the model application phase: According to the audio features, the voice object in the first voice data is identified to obtain a voice recognition result of the voice object in the first voice data.

[0283] In summary, in some of the above-mentioned implementations, the voice activity detection task and the overlapping speech detection task are only used to assist the training of the speech recognition task during the model training phase. After the target speech recognition model is put into use, the voice activity detection task and the overlapping speech detection task are no longer performed.

[0284] In other implementations, during the model training phase: According to the audio features, the voice object in the voice sample is identified to obtain the second voice recognition result of the voice object; according to the audio features, the audio sample is subjected to voice activity detection processing to obtain the first detection result; according to the audio features, the audio sample is subjected to overlapping voice detection processing to obtain the second detection result; according to the second voice recognition result, the first detection result, the second detection result and the first voice recognition result, a loss function is constructed, and the voice recognition model is iteratively trained based on the loss function to obtain the target voice recognition model. For details, please refer to Figure 2 The corresponding description part of step S204-step S210 in the embodiment.

[0285] During the model application phase: According to the audio features, the voice object in the first voice data is identified to obtain the voice recognition result of the voice object in the first voice data; according to the audio features, the audio sample is subjected to voice activity detection processing to obtain a first detection result; according to the audio features, the audio sample is subjected to overlapping voice detection processing to obtain a second detection result; and the voice recognition result of the voice object is adjusted according to the first detection result and the second detection result.

[0286] In summary, in some of the other implementation methods mentioned above, the voice activity detection task and the overlapping voice detection task can be used to assist the training of the voice recognition task in the model training stage, and can also be used as reference information to improve the recognition accuracy of the voice recognition results after the target voice recognition model is put into use.

[0287] In addition, in different scenarios, you can configure the weight information of the first detection result, the second detection result, and the speech recognition result according to task requirements, so that the target speech recognition result obtained by the final weighted sum is more in line with the task requirements. You can also configure the specific processing logic for generating the target speech recognition result based on the first detection result, the second detection result, and the speech recognition result according to task requirements.

[0288] For example, in some scenarios, the task requirements state that no sound leakage is desired. In this case, for each frame of audio, if the first detection result of the frame of audio indicates the presence of a speech signal, but the speech recognition result of the target speech recognition model for the frame of audio is that there is no speech object, which contradicts the first detection result, then the frame of audio is considered to contain a speech object, and the speech object contained in the frame of audio is marked as the one with the largest predicted probability value corresponding to each preset classification label in the classification result.

[0289] For another example, in some scenarios, the purpose of identifying a voice object in a voice sample is to determine the audio segment in which the identified object speaks, and then subsequently voiceprint recognition processing can be performed based on the audio segment. Considering that overlapping voice and silence have a greater impact on subsequent voiceprint recognition, usually only audio segments in which a single voice object speaks are collected. For any frame of audio data, if at least one of the first detection result and the voice recognition result predicts that the frame of audio data is silent, then the frame of audio data is treated as silent data. If at least one of the second detection result and the voice recognition result predicts that at least two voice objects simultaneously speak in the frame of audio data, then the frame of audio data is treated as overlapping voice data. Then, the silent data and overlapping voice data in the first voice data can be removed to obtain an audio segment in which a single voice object speaks.

[0290] In an embodiment of the present application, first, the first voice data is input into a target voice recognition model for voice object recognition processing to obtain a voice recognition result of the voice object in the first voice data; the target voice recognition model is trained by a voice recognition model training method; according to the voice recognition result of the voice object, the first voice data is divided into multiple time periods, and the voice object in each of the multiple time periods is marked. Considering that in the case where a voice sample involves multiple voice objects participating in a conversation, each voice object may participate in the conversation at the same time. For example, three people speak at the same time. In a frame of audio data, the sounds of multiple voice objects participating in the conversation at the same time are mixed together. Each voice object may interfere with the recognition accuracy of other voice objects. In an embodiment of the present application, the first voice recognition result for annotating the voice sample includes a classification label for each frame of audio data. Each classification label represents a voice object participating in the conversation in the corresponding frame of audio data. In this way, even if there are multiple voice objects participating in the conversation at the same time in a frame of audio data, the frame of audio data has only one classification label. The classification label can be combined with the second classification label in the training process. The speech recognition result is used to construct a loss function, and then the loss function can be used to guide the speech recognition model to learn how to map the one frame of audio data to an output category composed of multiple speech objects, that is, the second speech recognition result of the speech object in the one frame of audio data. Furthermore, the sounds of multiple speech objects mixed together do not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution of guiding the speech recognition model to learn how to map one frame of speech data to multiple output categories, each output category corresponding to a speech object, the embodiment of the present application can reduce the negative impact of the mutual interference of the sounds of each speech object uttered at the same time on the recognition of the speech object, and is more conducive to improving the recognition accuracy of the speech object.

[0291] Since the technical concept is the same, the description in this embodiment is relatively simple, and the relevant parts may refer to the corresponding description of the method embodiment provided above.

[0292] Based on the same technical concept, the embodiment of the present application also provides another speech recognition method. Figure 8 A data flow diagram of another speech recognition method provided in an embodiment of the present application.

[0293] like Figure 8 As shown, the first speech data to be recognized is input into the speech pre-training model 802 for encoding processing, and the first encoding information output by each speech encoding layer in the speech pre-training model 802 is obtained. For details, please refer to the corresponding description parts of the aforementioned steps S204 and S702.

[0294] The linear layer 804 with learnable weights generates comprehensive coding information of the first voice data according to the weight parameters of each coding layer and the first coding information output by each coding layer, and uses the comprehensive coding information as the audio feature of the first voice data. For details, please refer to the corresponding description of the aforementioned step S204 and step S702. The encoding unit 806, the speaker log decoding unit 808, and the speaker log classification unit 810 are used together to perform the following steps: according to the audio feature, identify the voice object in the first voice data, and obtain the voice recognition result 812 of the voice object in the first voice data. For details, please refer to the corresponding description of the aforementioned step S206 and step S702.

[0295] The encoding unit 806 , the voice activity detection decoding unit 814 and the voice activity detection classification unit 816 are used together to perform the following steps: performing voice activity detection processing on the first voice data according to the audio features to obtain a first detection result 818 .

[0296] The encoding unit 806 , the overlapping speech detection decoding unit 820 and the overlapping speech detection classification unit 822 are used together to perform the following steps: performing overlapping speech detection processing on the first speech data according to the audio features to obtain a second detection result 824 .

[0297] The result fusion module 826 fuses the speech recognition result 812, the first detection result 818, and the second detection result 824 to obtain a target speech recognition result. The target speech recognition result can be regarded as a speech recognition result of the speech object in the first speech data obtained by adjusting the speech recognition result 812 based on the first detection result 818 and the second detection result 824.

[0298] Since the technical concept is the same, the description in this embodiment is relatively simple, and the relevant parts may refer to the corresponding description of the method embodiment provided above.

[0299] An embodiment of a training device for a speech recognition model provided in this specification is as follows: In the above-mentioned embodiment, a method for training a speech recognition model is provided. Based on the same technical concept, an embodiment of the present application also provides a device for training a speech recognition model, which is described below in conjunction with the accompanying drawings.

[0300] Fig. 9 A schematic diagram of a training device for a speech recognition model provided in an embodiment of the present application. This embodiment provides a training device 900 for a speech recognition model, comprising: An acquisition unit 902 is used to acquire a speech sample and a first speech recognition result annotated on the speech sample; the first speech recognition result includes a classification label of each frame of audio data in the speech sample, and each classification label represents a speech object participating in the conversation in a corresponding frame of audio data; An extraction unit 904 is used to extract features from the speech sample to obtain audio features of the speech sample; A first recognition unit 906 is used to recognize a speech object of each frame of audio data in the speech sample according to the audio feature, and obtain a second speech recognition result of the speech object; A construction unit 908, configured to construct a loss function according to the second speech recognition result and the first speech recognition result; The training unit 910 is used to iteratively train the speech recognition model based on the loss function.

[0301] In an embodiment of the present application, a training device for a speech recognition model includes: an acquisition unit for acquiring a speech sample and a first speech recognition result for annotating the speech sample; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, each classification label represents a speech object participating in a conversation in a corresponding frame of audio data; an extraction unit for performing feature extraction on the speech sample to obtain an audio feature of the speech sample; a first recognition unit for identifying the speech object of each frame of audio data in the speech sample based on the audio feature to obtain a second speech recognition result of the speech object; a construction unit for constructing a loss function based on the second speech recognition result and the first speech recognition result; and a training unit for iteratively training the speech recognition model based on the loss function. Taking into account that in the case where a voice sample involves multiple voice objects participating in a conversation, each voice object may participate in the conversation at the same time. For example, three people speak at the same time. In one frame of audio data, the sounds of multiple voice objects participating in the conversation at the same time are mixed together. Each voice object may interfere with the recognition accuracy of other voice objects. In an embodiment of the present application, the first voice recognition result for annotating the voice sample includes a classification label for each frame of audio data. Each classification label represents a voice object participating in the conversation in the corresponding frame of audio data. In this way, even if there are multiple voice objects participating in the conversation at the same time in one frame of audio data, the one frame of audio data has only one classification label, which can be combined with the second classification label during the training process. The speech recognition result is used to construct a loss function, and then the loss function can be used to guide the speech recognition model to learn how to map the one frame of audio data to an output category composed of multiple speech objects, that is, the second speech recognition result of the speech object in the one frame of audio data. Furthermore, the sounds of multiple speech objects mixed together do not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution of guiding the speech recognition model to learn how to map one frame of speech data to multiple output categories, each output category corresponding to a speech object, the embodiment of the present application can reduce the negative impact of the mutual interference of the sounds of each speech object uttered at the same time on the recognition of the speech object, and is more conducive to improving the recognition accuracy of the speech object.

[0302] An embodiment of a speech recognition device provided in this specification is as follows: In the above-mentioned embodiment, a speech recognition method is provided. Based on the same technical concept, the embodiment of the present application also provides a speech recognition device, which is described below in conjunction with the accompanying drawings.

[0303] Fig.10 A schematic diagram of a speech recognition device provided in an embodiment of the present application. This embodiment provides a speech recognition device 1000, including: The second recognition unit 1002 is used to input the first speech data into a target speech recognition model to perform recognition processing on the speech object, and obtain a speech recognition result of the speech object in the first speech data; the target speech recognition model is obtained by training using a speech recognition model training method; The marking unit 1004 is used to divide the first voice data into multiple time periods according to the voice recognition result of the voice object, and mark the voice object in each of the multiple time periods.

[0304] In an embodiment of the present application, a speech recognition device includes: a second recognition unit, which is used to input the first speech data into a target speech recognition model for speech object recognition processing to obtain a speech recognition result of the speech object in the first speech data; the target speech recognition model is obtained by training the speech recognition model; and a marking unit, which is used to divide the first speech data into multiple time periods according to the speech recognition result of the speech object, and to mark the speech object in each of the multiple time periods. Considering that in the case where a speech sample involves multiple speech objects participating in a conversation, each speech object may participate in the conversation at the same time. For example, three people speak at the same time. In a frame of audio data, the sounds of multiple speech objects participating in the conversation at the same time are mixed together. Each speech object may interfere with the recognition accuracy of other speech objects. In an embodiment of the present application, the first speech recognition result for marking the speech sample includes a classification label for each frame of audio data. Each classification label represents a speech object participating in the conversation in the corresponding frame of audio data. In this way, even if there are multiple speech objects participating in the conversation at the same time in a frame of audio data, the frame of audio data has only one classification label. The classification label can be combined with the second classification label in the training process. The speech recognition result is used to construct a loss function, and then the loss function can be used to guide the speech recognition model to learn how to map the one frame of audio data to an output category composed of multiple speech objects, that is, the second speech recognition result of the speech object in the one frame of audio data. Furthermore, the sounds of multiple speech objects mixed together do not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution of guiding the speech recognition model to learn how to map one frame of speech data to multiple output categories, each output category corresponding to a speech object, the embodiment of the present application can reduce the negative impact of the mutual interference of the sounds of each speech object uttered at the same time on the recognition of the speech object, and is more conducive to improving the recognition accuracy of the speech object.

[0305] Corresponding to the training method or speech recognition method of a speech recognition model described above, based on the same technical concept, an embodiment of the present application also provides an electronic device, which is used to execute the training method or speech recognition method of the speech recognition model provided above. Fig.11A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0306] like Fig.11 As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 1101 and memory 1102, and the memory 1102 may store one or more storage applications or data. Among them, the memory 1102 may be a short-term storage or a persistent storage. The application stored in the memory 1102 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 1101 may be configured to communicate with the memory 1102 and execute a series of computer executable instructions in the memory 1102 on the electronic device. The electronic device may also include one or more power supplies 1103, one or more wired or wireless network interfaces 1104, one or more input / output interfaces 1105, one or more keyboards 1106, etc.

[0307] In a specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the electronic device, and the one or more programs configured to be executed by one or more processors include the following computer executable instructions: Acquire a speech sample and a first speech recognition result annotated with the speech sample; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, each classification label representing a speech object participating in the conversation in a corresponding frame of audio data; Extracting features from the speech sample to obtain audio features of the speech sample; According to the audio feature, identifying the voice object of each frame of audio data in the voice sample, and obtaining a second voice recognition result of the voice object; Constructing a loss function according to the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

[0308] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented: Acquire a speech sample and a first speech recognition result annotated with the speech sample; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, each classification label representing a speech object participating in the conversation in a corresponding frame of audio data; Extracting features from the speech sample to obtain audio features of the speech sample; According to the audio feature, identifying the voice object of each frame of audio data in the voice sample, and obtaining a second voice recognition result of the voice object; Constructing a loss function according to the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

[0309] It should be noted that the aforementioned embodiment of the computer-readable storage medium in this specification and the embodiment of the training method of the speech recognition model in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned corresponding method, and the repeated parts will not be repeated.

[0310] In another specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the electronic device, and the one or more programs are configured to be executed by one or more processors, including computer executable instructions for performing the following: Inputting the first speech data into a target speech recognition model to perform recognition processing on a speech object, and obtaining a speech recognition result of the speech object in the first speech data; the target speech recognition model is obtained by training using a speech recognition model training method; According to the speech recognition result of the speech object, the first speech data is divided into a plurality of time periods, and the speech object in each of the plurality of time periods is marked.

[0311] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions. When the computer-executable instructions are executed by a processor, the following process is implemented: Inputting the first speech data into a target speech recognition model to perform recognition processing on a speech object, and obtaining a speech recognition result of the speech object in the first speech data; the target speech recognition model is obtained by training using a speech recognition model training method; According to the speech recognition result of the speech object, the first speech data is divided into a plurality of time periods, and the speech object in each of the plurality of time periods is marked.

[0312] It should be noted that the aforementioned embodiment of the computer-readable storage medium in this specification and the embodiment of the speech recognition method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned corresponding method, and the repeated parts will not be repeated.

[0313] Another embodiment of the present disclosure further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following process is implemented: Acquire a speech sample and a first speech recognition result annotated with the speech sample; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, each classification label representing a speech object participating in the conversation in a corresponding frame of audio data; Extracting features from the speech sample to obtain audio features of the speech sample; According to the audio feature, identifying the voice object of each frame of audio data in the voice sample, and obtaining a second voice recognition result of the voice object; Constructing a loss function according to the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

[0314] The computer program product in the embodiments of the present disclosure can implement the various processes of the above-mentioned speech recognition model training method embodiment and achieve the same effects and functions, which will not be repeated here.

[0315] Another embodiment of the present disclosure further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following process is implemented: Inputting the first speech data into a target speech recognition model to perform recognition processing on a speech object, and obtaining a speech recognition result of the speech object in the first speech data; the target speech recognition model is obtained by training using a speech recognition model training method; According to the speech recognition result of the speech object, the first speech data is divided into a plurality of time periods, and the speech object in each of the plurality of time periods is marked.

[0316] The computer program product in the embodiments of the present disclosure can implement the various processes of the above-mentioned speech recognition method embodiments and achieve the same effects and functions, which will not be repeated here.

[0317] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0318] It should be understood by those skilled in the art that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0319] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable electronic device to produce a machine, so that the instructions executed by the processor of the computer or other programmable electronic device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0320] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable electronic device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0321] These computer program instructions may also be loaded onto a computer or other programmable electronic device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0322] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0323] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0324] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0325] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0326] Embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0327] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0328] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A method for training a speech recognition model, characterized in that: include: Acquire a speech sample and a first speech recognition result of annotating the speech sample; The first speech recognition result includes a classification label for each frame of audio data in the speech sample, and each classification label represents a speech object participating in the conversation in a corresponding frame of audio data; Extracting features from the speech sample to obtain audio features of the speech sample; According to the audio feature, identifying the voice object of each frame of audio data in the voice sample, and obtaining a second voice recognition result of the voice object; Constructing a loss function according to the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

2. The method according to claim 1, characterized in that Each classification label is a first object identification group, each object identification group includes a first identification and the number of the first identifications is within a preset range; the identifying the speech object of each frame of audio data in the speech sample according to the audio feature to obtain a second speech recognition result of the speech object includes: Classifying the speech objects of each frame of audio data according to the audio features to obtain a second object identification group; The constructing a loss function according to the second speech recognition result and the first speech recognition result includes: A loss function is constructed based on the second object identification group of each frame of audio data and the first object identification group of each frame of audio data.

3. The method according to claim 1, characterized in that The classification labels are generated in the following way: Acquire the voice object participating in the conversation in the voice sample; Generate at least one object group according to the number of the voice objects to obtain an object group set; in the object group set, the number of the voice objects in each object group is within a preset range; A corresponding object identification group is generated according to each object group in the object group set, and each object identification group is determined as a classification label.

4. The method according to claim 1, characterized in that: Also includes: Performing voice activity detection on the voice sample according to the audio feature to obtain a first detection result; Performing overlapping speech detection on the speech sample according to the audio feature to obtain a second detection result; The constructing a loss function according to the second speech recognition result and the first speech recognition result includes: A loss function is constructed according to the second speech recognition result, the first detection result, the second detection result and the first speech recognition result.

5. The method according to claim 1, characterized in that The speech recognition model includes N speech coding layers; the feature extraction of the speech sample to obtain the audio features of the speech sample includes: Each of the speech coding layers performs coding processing on the speech sample to obtain first coding information output by each speech coding layer; Wherein, when i=1, the speech sample is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension; when i is greater than 1, the speech coding information of the i-1-th dimension output by the i-1-th speech coding layer is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension, and i is a positive integer less than or equal to N; The audio features of the speech sample are determined based on the first encoding information output by each of the N speech encoding layers and the weight parameter of each dimension.

6. The method according to claim 1, characterized in that The method for generating the speech sample comprises: Acquire first voice data and second voice data, wherein the first voice data includes voice data of at least two voice objects, and the second voice data includes voice data of one voice object; Generate virtual voice data of a virtual voice object, and expand the virtual voice data into the second voice data to obtain expanded second voice data, wherein the expanded second voice data includes voice data of at least two voice objects; Constructing a first voice sample according to the first voice data, and constructing a second voice sample according to the expanded second voice data; Speech sample extraction is performed on the first speech sample and the second speech sample according to a preset ratio to obtain the speech sample.

7. The method according to claim 1, characterized in that The speech sample includes K frames of audio data, the number of the audio features is K, and the K frames of audio data correspond to the K audio features one by one; and identifying the speech object of each frame of audio data in the speech sample according to the audio features to obtain a second speech recognition result of the speech object includes: Encoding the K audio features to obtain first feature data of the K frames of audio data; Decoding the first feature data of each frame of audio data to obtain second feature data describing the speech object of each frame of audio data; Performing classification processing according to the second feature data of each frame of audio data to obtain a speech object of each frame of audio data; Based on the voice object of the K frames of audio data, a second voice recognition result of the voice object is obtained.

8. The method according to claim 5, characterized in that The determining the audio feature of the speech sample based on the first encoding information output by each encoding layer and the weight parameter of each dimension includes: For each dimension, comparing the weight parameter of the dimension with a preset weight threshold; If the comparison result indicates that the weight parameter of the dimension is less than the preset weight threshold, the weight parameter of the dimension is reset to a preset value; The audio features of the speech sample are obtained by performing a weighted summation based on the reset weight parameter of each dimension and the first coding information.

9. The method according to claim 4, characterized in that The audio features include a first audio feature describing the energy of the speech sample, a second audio feature describing the frequency spectrum of the speech sample, and a third audio feature describing the signal waveform of the speech sample; The performing voice activity detection on the voice sample according to the audio feature to obtain a first detection result includes: Determine a first time period in which active speech exists in the speech sample according to the first audio feature, the second audio feature, and the third audio feature; Determine a second time period in which active speech is missing based on the total time period of the speech sample and the first time period; The first detection result is generated according to the first time period and the second time period.

10. A speech recognition method, characterized in that: include: Inputting the first speech data into a target speech recognition model to perform speech object recognition processing, and obtaining a speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the speech recognition model training method described in any one of claims 1 to 9; According to the speech recognition result of the speech object, the first speech data is divided into a plurality of time periods, and the speech object in each of the plurality of time periods is marked.

11. An electronic device, characterized in that: include: processor; And, a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the training method for a speech recognition model as described in any one of claims 1 to 9 or the speech recognition method as described in claim 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, which, when executed by a processor, implement the training method for a speech recognition model as described in any one of claims 1 to 9 or the speech recognition method as described in claim 10.

13. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the training method of the speech recognition model as described in any one of claims 1 to 9 or the speech recognition method as described in claim 10.

Images