Speech recognition model training method, speech recognition method, and related device

Abstract

The embodiment of the application provides a speech recognition model training method, a speech recognition method and related equipment. The speech recognition model training method comprises the following steps: obtaining a speech sample and a first speech recognition result of marking the speech sample; the first speech recognition result comprises a classification label of each frame of audio data in the speech sample, and each classification label represents a speech object participating in a conversation in a corresponding frame of audio data; performing feature extraction on the speech sample to obtain audio features of the speech sample; identifying the speech object of each frame of audio data in the speech sample according to the audio features to obtain a second speech recognition result of the speech object; constructing a loss function according to the second speech recognition result and the first speech recognition result; and iteratively training the speech recognition model based on the loss function. In this way, the recognition accuracy of the speech object can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a training method for a speech recognition model, a speech recognition method, and related equipment. Background Technology

[0002] With the development of electronic technology, the application of artificial intelligence is becoming increasingly widespread. AI technology performs well in handling various speech recognition tasks, including speaker digitization, speaker verification, and speech separation. The speaker digitization task, in particular, refers to the process of determining who speaks and when in an audio or video clip. Simply put, when the speech data includes more than one speech object, the speaker digitization task can be used to identify the individual speech objects within the speech data. Summary of the Invention

[0003] This application provides a method for training a speech recognition model, a speech recognition method, and related equipment, which can improve the accuracy of speech object recognition.

[0004] In a first aspect, embodiments of this application provide a method for training a speech recognition model, comprising: Acquire speech samples and a first speech recognition result annotated with the speech samples; the first speech recognition result includes a classification label for each frame of audio data in the speech samples, and each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data; Feature extraction is performed on the speech samples to obtain the audio features of the speech samples; Based on the audio features, identify the speech object in each frame of audio data in the speech sample, and obtain the second speech recognition result of the speech object; A loss function is constructed based on the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

[0005] Secondly, embodiments of this application provide a speech recognition method, including: The first speech data is input into the target speech recognition model for speech object recognition processing to obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the speech recognition model training method described in the first aspect; Based on the speech recognition results of the speech objects, the first speech data is divided into multiple time periods, and the speech objects in each of the multiple time periods are marked.

[0006] Thirdly, embodiments of this application provide a training apparatus for a speech recognition model, comprising: The acquisition unit is used to acquire speech samples and a first speech recognition result annotated with the speech samples; the first speech recognition result includes a classification label for each frame of audio data in the speech samples, and each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data. An extraction unit is used to extract features from speech samples to obtain the audio features of the speech samples; The first recognition unit is used to recognize the speech object in each frame of audio data in the speech sample based on the audio features, and to obtain the second speech recognition result of the speech object; The construction unit is used to construct a loss function based on the second speech recognition result and the first speech recognition result; The training unit is used to iteratively train the speech recognition model based on the loss function.

[0007] Fourthly, embodiments of this application provide a voice recognition device, including: The second recognition unit is used to input the first speech data into the target speech recognition model for speech object recognition processing, and obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the speech recognition model training method described in the first aspect; The tagging unit is used to divide the first speech data into multiple time periods based on the speech recognition result of the speech object, and to tag the speech object in each of the multiple time periods.

[0008] Fifthly, embodiments of this application provide an electronic device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform a training method for a speech recognition model as described in the first aspect or a speech recognition method as described in the second aspect.

[0009] In a sixth aspect, embodiments of this application provide a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the training method for the speech recognition model as described in the first aspect or the speech recognition method as described in the second aspect.

[0010] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the training method for the speech recognition model as described in the first aspect or the speech recognition method as described in the second aspect.

[0011] As can be seen, in this embodiment, firstly, speech samples and first speech recognition results annotated on the speech samples are obtained; the first speech recognition result includes classification labels for each frame of audio data in the speech sample, each classification label representing a speech object participating in the dialogue in the corresponding frame of audio data; then, feature extraction is performed on the speech samples to obtain audio features of the speech samples; next, based on the audio features, the speech objects in each frame of audio data in the speech samples are identified to obtain second speech recognition results for the speech objects; next, a loss function is constructed based on the second speech recognition result and the first speech recognition result; finally, the speech recognition model is iteratively trained based on the loss function. Considering that when a speech sample involves multiple speech objects participating in a dialogue, these speech objects may participate in the dialogue simultaneously, for example, three people speaking at the same time. In a single frame of audio data, the voices of multiple speech objects participating in the dialogue simultaneously are mixed together, and each speech object may interfere with the recognition accuracy of other speech objects. In this embodiment, the first speech recognition result for annotating the speech sample includes a classification label for each frame of audio data. Each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data. In this way, even if there are multiple speech objects participating in the dialogue simultaneously in a frame of audio data, that frame of audio data will only have one classification label. This classification label can be combined with the first... The loss function is constructed from the second speech recognition result. This loss function can then guide the speech recognition model to learn how to map a frame of audio data to an output category composed of multiple speech objects. An output category is the second speech recognition result of the speech objects in the frame of audio data. Furthermore, the mixing of the sounds of multiple speech objects does not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution that guides the speech recognition model to learn how to map a frame of speech data to multiple output categories, with each output category corresponding to a speech object, the embodiments of this application can reduce the negative impact of the mutual interference of the sounds of the simultaneous speech objects on the recognition of speech objects, and is more conducive to improving the recognition accuracy of speech objects. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 A schematic diagram illustrating the implementation environment of a training method for a speech recognition model provided in this application embodiment; Figure 2 A flowchart illustrating a training method for a speech recognition model provided in this application embodiment; Figure 3 An example diagram of the annotation information of audio samples in a training method for a speech recognition model provided in this application embodiment; Figure 4 A weight information graph of a speech pre-trained model in different tasks is provided in the embodiments of this application; Figure 5 Data flow diagram of another speech recognition model training method provided in the embodiments of this application; Figure 6 A data flow diagram for a training method of another speech recognition model provided in an embodiment of this application; Figure 7 A flowchart illustrating a speech recognition method provided in an embodiment of this application; Figure 8 Data flow diagram of another speech recognition method provided in the embodiments of this application; Figure 9 A schematic diagram of a training device for a speech recognition model provided in an embodiment of this application; Figure 10 A schematic diagram of a speech recognition device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this application.

[0014] This specification provides an example of a training method for a recognition model: Speaker log tasks can be used to identify individual speech objects in speech data. When the speech data includes only two speech objects, i.e., when the speech data is a two-person dialogue, the two speech objects participating in the dialogue will often wait for the other to finish speaking before speaking. Therefore, the situation where two people speak at the same time is relatively rare, and the difficulty of speech object recognition is relatively low.

[0015] In practical applications, speech data is likely to include more than two speech objects, and it is common for multiple people to speak simultaneously. The more speech objects there are in a speech data set to be processed, the more difficult it becomes to recognize and process those speech objects. The voices of multiple speech objects speaking at the same time may interfere with each other, leading to inaccurate recognition of the speech objects.

[0016] Therefore, in order to solve the above problems, this application provides a method for training a speech recognition model.

[0017] The training method for a speech recognition model provided in one or more embodiments of this specification is applicable to the implementation environment of a speech recognition model training method, which includes at least a server 101 for performing model training.

[0018] Server 101 can be a single server, a server cluster consisting of several servers, or one or more cloud servers in a cloud computing platform, used for model training.

[0019] In this implementation environment, during model training, server 101 first acquires speech samples and first speech recognition results annotated with the speech samples. The first speech recognition result includes classification labels for each frame of audio data in the speech samples, with each classification label representing a speech object participating in the dialogue in the corresponding frame of audio data. Then, feature extraction is performed on the speech samples to obtain audio features. Next, based on the audio features, the speech objects in each frame of audio data in the speech samples are identified to obtain second speech recognition results for the speech objects. Following this, a loss function is constructed based on the second speech recognition result and the first speech recognition result. Finally, the speech recognition model is iteratively trained based on the loss function. Considering that when a speech sample involves multiple speech objects participating in a dialogue, these speech objects may participate in the dialogue simultaneously, for example, three people speaking at the same time. In a single frame of audio data, the voices of multiple speech objects participating in the dialogue simultaneously are mixed together, and each speech object may interfere with the recognition accuracy of other speech objects. In this embodiment, the first speech recognition result for annotating the speech sample includes a classification label for each frame of audio data. Each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data. In this way, even if there are multiple speech objects participating in the dialogue simultaneously in a frame of audio data, that frame of audio data will only have one classification label. This classification label can be combined with a second classification label during training. The speech recognition results are used to construct a loss function, which can then guide the speech recognition model to learn how to map a frame of audio data to an output category composed of multiple speech objects. This output category is the second speech recognition result of the speech objects in the frame of audio data. Furthermore, the mixing of the sounds of multiple speech objects does not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution that guides the speech recognition model to learn how to map a frame of speech data to multiple output categories, with each output category corresponding to a speech object, the embodiments of this application can reduce the negative impact of the mutual interference of the sounds of the speech objects speaking at the same time on the recognition of speech objects, and is more conducive to improving the recognition accuracy of speech objects.

[0020] Figure 2 A flowchart illustrating a training method for a speech recognition model provided in this application embodiment. (Refer to...) Figure 2 The training method for the recognition model provided in this embodiment specifically includes steps S202 to S208: Step S202: Obtain speech samples and a first speech recognition result annotated with the speech samples; the first speech recognition result includes a classification label for each frame of audio data in the speech samples, and each classification label represents the speech object participating in the dialogue in the corresponding frame of audio data.

[0021] A speech sample is a specific instance used for studying and analyzing speech signals. It is typically a segment of audio data extracted from actual speech conversations or recordings. Specifically, in the embodiments of this application, a speech sample can be sample data in audio format.

[0022] The number of speech samples can be one or more. In this embodiment, each speech sample may contain at least two speaking objects. The speaking objects can be humans, robots with dialogue capabilities, or any other objects that can communicate through natural language, etc.

[0023] Voice samples can be generated based on dialogue data in which at least two voice entities are involved in the conversation. For example, User 1 and User 2 are discussing a daily topic, "the weather," face-to-face. This discussion generates dialogue data, which can be used to generate voice sample 1. As another example, a customer inquires about the specific details of an event during a call. This inquiry generates dialogue data, which can be used to generate voice sample 2.

[0024] Obtaining speech samples can involve acquiring dialogue data, counting the number of speech objects in the dialogue data, filtering the dialogue data according to the number of speech objects, obtaining target dialogue data, and treating each target dialogue data as a speech sample.

[0025] In the process of filtering dialogue data according to the number of voice objects to obtain target dialogue data, for each dialogue data, the number of voice objects in the dialogue data can be compared with the first number threshold and the second number threshold respectively. If the number of voice objects is greater than or equal to the first number threshold and less than or equal to the second number threshold, the dialogue data is retained and used as a target dialogue data; otherwise, the dialogue data is discarded.

[0026] The reason for filtering dialogue data based on the number of speech objects is that, on the one hand, the classification label of a speech sample is affected by the number of speech objects in the speech sample. In order to ensure the effectiveness of subsequent classification label usage, the number of speech objects in each speech sample should not be too large. On the other hand, if the number of speech objects in a speech sample is too small, the possibility of multiple speech objects speaking at the same time is small, which may lead to sparse samples corresponding to some classification labels, which is not conducive to model training.

[0027] Alternatively, during the process of filtering dialogue data according to the number of voice objects to obtain target dialogue data, for each dialogue data, the number of voice objects in the dialogue data is compared with a first quantity threshold. If the number of voice objects is greater than or equal to the first quantity threshold, the dialogue data is retained and used as a target dialogue data; otherwise, the dialogue data is discarded.

[0028] Alternatively, during the process of filtering dialogue data according to the number of voice objects to obtain target dialogue data, for each dialogue data, the number of voice objects in the dialogue data is compared with a second quantity threshold. If the number of voice objects is less than or equal to the second quantity threshold, the dialogue data is retained and used as a target dialogue data; otherwise, the dialogue data is discarded.

[0029] Optionally, the speech sample can be synthesized based on multiple audio segments, each originating from a speech object. In specific implementations, obtaining a speech sample can be achieved by concatenating multiple audio segments, or by trimming and concatenating multiple audio segments.

[0030] For example, audio segments 1, 2, and 3 for user 1 were pre-recorded, and audio segments 4 and 5 for user 2 were also pre-recorded. Audio segments 1, 2, 3, 4, and 5 were then trimmed and spliced ​​to obtain speech sample 3.

[0031] In one specific implementation, the method for generating speech samples includes: acquiring first speech data and second speech data, wherein the first speech data includes speech data of at least two speech objects and the second speech data includes speech data of one speech object; generating virtual speech data of a virtual speech object and expanding the virtual speech data into the second speech data to obtain expanded second speech data, wherein the expanded second speech data includes speech data of at least two speech objects; constructing a first speech sample based on the first speech data and constructing a second speech sample based on the expanded second speech data; and extracting speech samples from the first speech sample and the second speech sample according to a preset ratio to obtain speech samples.

[0032] The use of ordinal numbers such as "first" and "second" in this specification is for the purpose of distinguishing different technical features and has no actual meaning. They will not be elaborated on further below.

[0033] Each first speech data set includes speech data from at least two speech objects. The first speech data set may include a portion of general domain speech data and a portion of target domain speech data.

[0034] A general domain is a broad, encompassing various industries. A target domain, on the other hand, can be a pre-defined, specific domain, distinct from the general domain, or it can be interpreted as a private domain. The concepts of private and general domains are relative. A private domain may focus on a specific industry and have a relatively smaller scope compared to a general domain, such as customer service.

[0035] The first voice data can include dialogue data that occurs in various industries.

[0036] For example, the first speech data could be an open-source speaker log dataset. An open-source speaker log dataset refers to an open-source dataset whose data content is speaker logs. An open-source dataset refers to a collection of data that can be freely obtained, used, and shared.

[0037] A speaker log typically refers to a document or database that records the content of a specific speaker's (such as an announcer, speaker, or conference moderator) speech over a specific period of time. Such a log can contain various types of information, such as: Speech content: Record the specific content of the speaker's speech, which can be a written record, or a transcription of audio or video.

[0038] Timestamps: Record the specific time of each speech, which helps in tracking and organizing information.

[0039] Occasion: Record the occasion or activity in which the speech takes place, such as a meeting, a speech, or a radio program.

[0040] Topic: The topic of the speech or the issue discussed.

[0041] Audience feedback: If applicable, this may also include audience feedback or interaction.

[0042] The first voice data may also include dialogue data that occurs within the target business scenario. The target business scenario can be a specific scenario within the target domain.

[0043] For example, the target domain is the call center domain, and the target business scenario is the consultation scenario, specifically referring to the scenario where customer service agents provide consultation services to customers during a call. The first voice data includes the dialogue data between customer service agents and customers.

[0044] Each second speech data point includes the speech data of a speech object.

[0045] The second speech data may include a general domain dataset for training voiceprint recognition, with each dataset for voiceprint recognition training including speech data of a speech object.

[0046] For example, second voice data 1 is the voice data of user 1, second voice data 2 is the voice data of user 2, second voice data 3 is the voice data of user 3, and so on.

[0047] Virtual voice data of virtual voice objects is generated, and the virtual voice data is expanded into the second voice data to obtain the expanded second voice data, which includes the voice data of at least two voice objects.

[0048] The virtual voice data for generating a virtual voice object can be obtained by converting a preset template text into speech, resulting in multiple candidate virtual voice data for the virtual voice object. One of the multiple candidate virtual voice data is then randomly selected as the virtual voice data. Alternatively, based on the second voice data, one of the multiple candidate virtual voice data that matches the second voice data is determined as the virtual voice data.

[0049] The virtual voice data that generates the virtual voice object can also be obtained by performing speech-to-text processing on the second voice data to obtain the first text, inputting the first text into a language model for text generation processing to obtain the second text, performing text-to-speech processing on the second text to obtain the virtual voice data, and so on.

[0050] This specification does not impose any special restrictions on the method of generating virtual speech data. Virtual speech data can be speech data that includes at least one speech object. Here, "virtual" means that there is actually no dialogue between two people, but subsequent steps require speech data including at least two speech objects to construct training samples. Therefore, virtual speech data generated by virtual speech objects can be combined with the second speech data to simulate natural dialogue.

[0051] Considering the flexibility in generating virtual voice data, the second voice data can be used as the main source during the simulation of natural dialogue. Based on the second voice data, the corresponding virtual voice data can be determined, and then the virtual voice data can be expanded into the second voice data to obtain the expanded second voice data. In this way, the second voice data before expansion is voice data involving only a single voice object, while the expanded second voice data is dialogue data including at least two voice objects.

[0052] The expansion method can be to splice the second voice data and the virtual voice data, or to cut and then splice them.

[0053] Another method of expansion is to construct dialogue data based on the second voice data and one or more virtual voice data.

[0054] For example, the second speech data is converted to text to obtain text 1 corresponding to the second speech data. This text 1 is then input into a large language model to generate text 2. This text 2 is then converted to speech to obtain virtual speech data. This virtual speech data is then concatenated with the second speech data to obtain the expanded second speech data.

[0055] Constructing a first speech sample based on the first speech data can be achieved by sampling the first speech data in a general domain to obtain a portion of training data, sampling the first speech data in the target domain to obtain another portion of training data, and adding the two portions of training data to the training dataset to obtain the first speech sample.

[0056] Constructing a second speech sample based on the expanded second speech data can be achieved by sampling the expanded second speech data to obtain the second speech sample.

[0057] The expanded second voice data can be considered as simulated dialogue data. For example, the expanded second voice data 1 includes a portion of the voice data of user 1 and a portion of the voice data of virtual user 2, even though user 1 and virtual user 2 have not actually had a conversation.

[0058] Compared with real dialogue data, virtual dialogue data obtained by expanding the second speech data can be generated in batches with high efficiency, resulting in a relatively large number of expanded second speech data. Furthermore, sampling can reduce the number of second speech samples in subsequent steps, thereby flexibly controlling the ratio of first speech samples to second speech samples in the speech samples.

[0059] The process of extracting speech samples from the first speech sample and the second speech sample according to a preset ratio to obtain speech samples may include: determining the number of first samples and the number of second samples to be selected based on the pre-configured total number of samples and the preset ratio parameter; randomly selecting the number of first samples and the number of second samples; and assembling the number of first samples and the number of second samples to obtain speech samples.

[0060] For example, the preset ratio parameter is 8:2, where 8 corresponds to the first speech sample and 2 corresponds to the second speech sample. The pre-configured total number of samples is 100. Based on the total number of samples and the preset ratio parameter, the number of first speech samples to be selected is 80, and the number of second speech samples to be selected is 20. 80 first speech samples are randomly selected from each set of first speech samples. 20 second speech samples are selected from each set of second speech samples. The 80 first speech samples and 20 second speech samples are then assembled to obtain the speech samples.

[0061] The process of extracting speech samples from the first speech sample and the second speech sample according to a preset ratio to obtain speech samples may also include: determining a second number of second speech samples to be selected based on the number of first speech samples and the preset ratio parameter; randomly selecting a second number of second speech samples; and assembling the first speech sample and the second number of second speech samples to obtain speech samples.

[0062] For example, the preset ratio parameter is 8:2, where 8 corresponds to the first speech sample and 2 corresponds to the second speech sample. The number of first speech samples is 800. Based on the number of first speech samples and the preset ratio parameter, the number of second speech samples to be selected is determined to be 200. 200 second training data are randomly selected from each of the second speech samples. The 800 first speech samples and 200 second training data are then assembled to obtain the speech samples.

[0063] The reason for assembling the first and second speech samples is that the amount of dialogue data occurring in the target domain may not be sufficient to meet the training needs of the machine learning model; that is, the number of first speech samples is too small. The first speech sample is a real dialogue that has actually occurred, while the second speech sample is a simulated dialogue. When the number of real dialogues is too small, simulating dialogues can expand the number of samples, which is beneficial for achieving better model training results.

[0064] Each speech sample can have corresponding annotation information. Annotation information typically refers to additional information added to a dataset or document to explain, supplement, or highlight the content of the original data. This information can be text, symbols, labels, or other forms of annotation, designed to help users better understand and use the data. Annotation information is crucial in many fields, especially in data science, machine learning, and natural language processing.

[0065] Specifically, in the embodiments of this application, the annotation information corresponding to each speech sample is the aforementioned first speech recognition result.

[0066] Obtaining the first speech recognition result of the annotated speech sample can be either obtaining a manually generated first speech recognition result, or classifying the speech sample using a pre-trained classification model, and then annotating the speech sample based on the classification result to obtain the first speech recognition result of the speech sample.

[0067] The first speech recognition result includes the classification label of each frame of audio data in the speech sample. Each classification label represents the speech object participating in the dialogue in the corresponding frame of audio data.

[0068] In one specific implementation, the classification labels are generated as follows: obtain the voice objects participating in the dialogue in the voice sample; construct object groups based on the voice objects to obtain a set of object groups; in the set of object groups, the number of voice objects in each object group is within a preset range; generate a corresponding object identifier group based on each object group in the set of object groups, and determine each object identifier group as a classification label.

[0069] For each speech sample, the number of speech objects participating in the dialogue in that speech sample can be represented by a first number, which can be an integer greater than 1.

[0070] For example, voice sample 1 is dialogue data between user 1, user 2 and user 3 in a meeting scenario. It can be determined that the voice objects participating in the dialogue include user 1, user 2 and user 3. Therefore, the first number of voice objects participating in the dialogue is 3.

[0071] To obtain the speech objects involved in the dialogue in a speech sample, one can first detect the speech objects in each frame of audio data in the speech sample and then remove duplicates to obtain the speech objects involved in the dialogue in the speech sample.

[0072] For example, a speech sample includes audio data frames 1 through 3. In the first frame of the audio data, the detected speech objects are: object 1 and object 2. In the second frame of the audio data, the detected speech object is object 1. In the third frame of the audio data, the detected speech objects are object 1 and object 3. After deduplicating the speech objects in the first, second, and third frames of the audio data, the speech objects involved in the dialogue in the speech sample are obtained as: object 1, object 2, and object 3.

[0073] At least one object group is generated based on the number of speech objects, resulting in a set of object groups; in the set of object groups, the number of speech objects in each object group is within a preset range.

[0074] The preset range can be determined by the first quantity.

[0075] Specifically, the object set may include one or more object groups, and the number of speech objects in each object group is greater than or equal to one and less than or equal to a first number.

[0076] Let's take an example where the initial number of voice objects participating in the dialogue in the voice sample is 3: The voice objects participating in the dialogue in voice sample 1 include: object 1, object 2, and object 3.

[0077] At least one object group is generated based on the number of speech objects, resulting in multiple object groups as shown below: (c1) Object 1 (c2) Object 2 (c3) Object 3 (c4) Object 1 and Object 2 (c5) Object 1 and Object 3 (c6) Object 2 and Object 3 (c7) Object 1, Object 2, and Object 3 The aforementioned object groups (c1)-(c7) together constitute the object group set. Furthermore, the number of speech objects in each of the aforementioned object groups (c1)-(c7) is greater than or equal to 1 and less than or equal to 3, which satisfies the aforementioned requirement that the number of speech objects in each object group is greater than or equal to one and less than or equal to the first quantity.

[0078] For each object group in the object group set, generate a corresponding object identifier group, and determine each object identifier group as a category label.

[0079] For example, generate a corresponding object identifier group (001) based on the object group (c1), and determine the object identifier group as the category label d1; Generate a corresponding object identifier group (002) based on the object group (c2), and determine the object identifier group as the category label d2; Generate a corresponding object identifier group (003) based on the object group (c3), and determine the object identifier group as the category label d3; Generate a corresponding object identifier group (001, 002) based on the object group (c4), and determine the object identifier group as the category label d4; Generate a corresponding object identifier group (001, 003) based on the object group (c5), and determine the object identifier group as the category label d5; Generate a corresponding object identifier group (002, 003) based on the object group (c6), and determine the object identifier group as the category label d6; Generate a corresponding object identifier group (001, 002, 003) based on the object group (c7), and determine the object identifier group as the category label d7.

[0080] In addition, the mute symbol, which represents silence, can be used as a classification label.

[0081] After determining the classification labels, each frame of audio data in the speech sample can be labeled using the classification labels to obtain the first speech recognition result of the speech sample.

[0082] Next, we can combine... Figure 3 The first speech recognition result of the speech sample is illustrated by example. Figure 3 An example diagram of the annotation information of speech samples in a speech recognition model training method provided in an embodiment of this application.

[0083] Figure 3 The label encoding based on multiple categories is shown. The category label set 304 includes: Mute, Speaker 1, Speaker 2, Speaker 3, Speaker 1 & Speaker 2, Speaker 1 & Speaker 3, Speaker 2 & Speaker 3, and All Speakers.

[0084] Here, "Speaker 1 & Speaker 2" indicates that Speaker 1 and Speaker 2 are two speech objects identified at the same time. "Speaker 1 & Speaker 3" indicates that Speaker 1 and Speaker 3 are two speech objects identified at the same time. "Speaker 2 & Speaker 3" indicates that Speaker 2 and Speaker 3 are two speech objects identified at the same time. "All Speakers" indicates that Speaker 1, Speaker 2, and Speaker 3 are three speech objects identified at the same time.

[0085] The time information 302 includes 13 time points, t1-t13, each with a unique corresponding classification label. Each of these 13 time points can be used to represent a frame of audio data in a speech sample.

[0086] The classification label for time point t1 is "mute"; the classification label for time points t2-t5 is "speaker 1"; the classification label for time points t6-t7 is "speaker 1 & speaker 2"; the classification label for time points t8-t9 is "speaker 2"; the classification label for time point t10 is "speaker 2 & speaker 3"; the classification label for time points t11-t12 is "speaker 3"; and the classification label for time point t13 is "mute".

[0087] In summary, the first speech recognition results of the speech samples include: the classification label of the first frame of audio data is "mute", the classification label of the second frame of audio data is "speaker 1", the classification label of the third frame of audio data is "speaker 1", ..., the classification label of the thirteenth frame of audio data is "mute".

[0088] Step S204: Extract features from the speech samples to obtain the audio features of the speech samples. In some implementations, feature extraction is performed on speech samples to obtain audio features of the speech samples. This can be done by encoding the speech samples based on a speech recognition model to obtain the audio features of the speech samples.

[0089] In other implementations, the following operations can be performed during the process of extracting features from speech samples to obtain the audio features of the speech samples: obtaining the first encoding information of the speech samples in N dimensions based on the speech recognition model; and determining the audio features of the speech samples based on the first encoding information output by each encoding layer and the weight parameters of each dimension.

[0090] The first encoded information of a speech sample in N dimensions can be obtained by encoding the speech sample through a speech recognition model. N can be an integer greater than 1.

[0091] Specifically, the speech sample is encoded using a speech recognition model to obtain the first encoded information of the first dimension; the first encoded information of the first dimension is encoded using the speech recognition model to obtain the first encoded information of the second dimension; ...; the first encoded information of the (N-1)th dimension is encoded using the speech recognition model to obtain the first encoded information of the Nth dimension.

[0092] A speech recognition model may include an encoding module, which may consist of a pre-trained speech model. The pre-trained speech model may be a pre-trained model trained on large-scale speech data and applicable to various speech tasks. For example, various speech tasks include at least one of the following: speaker logging task, speaker verification task, speech separation task, speech recognition task, etc.

[0093] Speaker digitization, also known as voiceprint segmentation clustering or speaker segmentation clustering, primarily addresses the question of "when and who is saying what." In other words, given a speech recording of multiple speakers taking turns speaking, this task requires determining who is speaking at each point in time. For example, in a customer service call log, the task needs to identify which time periods are occupied by the customer and which by the customer service representative.

[0094] The speaker verification task is primarily to verify whether two audio recordings are spoken by the same person.

[0095] The goal of the speech separation task is to separate a speech containing multiple people, ensuring that each source of the output contains only a speech from one person.

[0096] The goal of speech recognition is to convert speech into text.

[0097] Pre-trained models are models that have been trained on a large amount of data for a specific task. These models can be used as a starting point for solving similar tasks. The characteristics of pre-trained models are as follows: Generality: Pre-trained models are typically trained on large-scale, diverse datasets, so they are able to capture a wide range of linguistic or visual features.

[0098] Transfer learning: Pre-trained models can serve as the foundation for transfer learning. Users can adapt pre-trained models to new tasks by fine-tuning them on task-specific datasets.

[0099] Efficiency: Using pre-trained models can save a lot of training time and computing resources because most of the model's parameters have already been optimized through pre-training.

[0100] Performance: Pre-trained models typically offer better initial performance, especially when the target task data is limited.

[0101] For example, the speech pre-training model can be a WavLM (Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing) model.

[0102] The WavLM model was derived using the Denoising Masked Speech Modeling framework on 94,000 hours of unsupervised data. Pre-trained models trained on large-scale unsupervised data exhibit excellent generalization capabilities and can achieve good results with only fine-tuning on a small amount of labeled data.

[0103] Speech pre-trained models can include one or more encoding layers. For example, some WavLM models have 12 Transformer encoding layers, while others have 24. The Transformer Encoder Layer is a key component of the Transformer model, responsible for processing the input data and generating the corresponding feature representations.

[0104] The Transformer model consists of two parts: an encoder and a decoder. The encoder is responsible for converting the input sequence into an intermediate representation, and the decoder uses this intermediate representation to generate the output sequence. Both the encoder and decoder are composed of multiple identical layers stacked together, each layer containing two main sublayers: Self-attention mechanism: allows the model to consider all words in the input sequence when processing each word, thereby capturing long-distance dependencies between words.

[0105] Feed-Forward Neural Network: Performs further nonlinear transformations on the output of the self-attention layer.

[0106] Each sublayer is followed by a residual connection and layer normalization to help the model train and generalize better.

[0107] The workflow of the Transformer encoding layer is as follows: (a1) Input processing: The input sequence is first converted into a vector representation through a word embedding layer, and then positional encoding is added to preserve the positional information of the words in the sequence.

[0108] (a2) Self-attention mechanism: The input vector passes through the self-attention mechanism, and each word obtains a new representation based on its relationship with other words in the sequence. The self-attention mechanism allows the model to establish direct connections between different positions, which is the key to the Transformer model's ability to effectively handle long-distance dependencies.

[0109] (a3) Feedforward Neural Network: The output of the self-attention layer is fed into a fully connected feedforward neural network that performs the same nonlinear transformation on the vector at each position.

[0110] (a4) Residual Connections and Layer Normalization: The output of each sublayer is connected via residual connections and then normalized. Residual connections help solve the vanishing gradient problem in deep networks, while layer normalization helps stabilize the training process.

[0111] (a5) Output: After processing through multiple coding layers, the final output is a series of feature vectors that capture rich information of the input sequence and can be used by the decoder to generate the output sequence.

[0112] In the process of obtaining the first encoded information of a speech sample in N dimensions based on a speech recognition model, the speech sample can be input into the encoding module of the speech recognition module. The speech sample is then encoded through multiple encoding layers in the encoding module to obtain the first encoded information of the speech sample in N dimensions.

[0113] In the process of obtaining the first encoded information of a speech sample in N dimensions based on a speech recognition model, the speech sample can first be processed by feature extraction to obtain the Fbank feature of the speech sample. Then, the Fbank feature is input into the encoding module of the speech recognition module. The Fbank feature is encoded by multiple encoding layers in the encoding module to obtain the first encoded information of the speech sample in N dimensions.

[0114] The definition of Fbank features is as follows: The human ear's response to the sound spectrum is non-linear. Fbank is a front-end processing algorithm that processes audio in a manner similar to the human ear, thereby improving speech recognition performance. The general steps for obtaining Fbank features of a speech signal are: pre-emphasis, framing, windowing, short-time Fourier transform, and MEL filtering.

[0115] In practice, the following steps can be performed when obtaining the first encoded information of a speech sample in N dimensions based on a speech recognition model: (b1) Segment the speech samples into audio segments of fixed length Len.

[0116] Len is a preset value, for example, Len = 10 seconds. If the total duration of the speech sample is not an integer multiple of Len, the shortfall will be padded with silence. After padded silence, the speech sample consists of K segmented audio segments. K is an integer greater than 0.

[0117] (b2) Configure the size of the training batch to b, and the number of iterations in each training round is an integer of K / b.

[0118] In machine learning and deep learning, a training batch refers to a set of samples that are input into the model all at once during the model training process.

[0119] The configuration of training batches can be based on factors such as the size of the GPU memory of the server executing model training, the parallel computing capability of the server, the size of the server's memory, the convergence speed, and so on.

[0120] (b3) Extract Fbank features from each batch of audio segments.

[0121] For example, with b=64, 40-dimensional Fbank features can be extracted using 64 audio segments of 50 seconds each. The interval between two adjacent frames with a frame length of 25 milliseconds is 10 milliseconds, which can yield 64*40*500 Fbank features.

[0122] (b4) Input the Fbank features into the encoding module in the speech recognition model for encoding processing.

[0123] In a speech recognition model where the encoding module consists of N encoding layers connected in series, each of the N encoding layers corresponds one-to-one with one of the N dimensions. The first encoding information of the speech sample in the N dimensions can be obtained as follows: obtain the output data of each of the N encoding layers and use the output data of each encoding layer as the first encoding information corresponding to that encoding layer.

[0124] When the encoding module of the speech recognition model includes M encoding layers connected in series, and M is greater than N, the first encoding information of the speech sample in N dimensions can be obtained as follows: obtain the output data of each encoding layer in the M encoding layers, determine N target encoding layers according to the pre-configured encoding layer set, and use the output data of each target encoding layer as the first encoding information corresponding to that target encoding layer.

[0125] The reason for configuring a set of encoding layers is that, in one or more embodiments of this application, the speech recognition model can be used to perform speaker logging tasks. However, the speech pre-training model of the encoding module used to construct the speech recognition model is not a proprietary model designed for performing speaker logging tasks, but a general model applicable to a variety of speech tasks. Therefore, among the multiple encoding layers included in the speech pre-training model, some of the first encoding information output by the encoding layers may perform well when processing speaker logging tasks, while the first encoding information output by other encoding layers may be of little use when processing speaker logging tasks and can be approximated as redundant data. Therefore, configuring a set of encoding layers is beneficial to reduce redundant data and improve the data processing efficiency during the execution of speaker logging tasks.

[0126] In one specific implementation, the speech recognition model includes N speech coding layers; feature extraction is performed on the speech samples to obtain the audio features of the speech samples, including: each speech coding layer encodes the speech samples to obtain the first coding information output by each speech coding layer; wherein, when i=1, the speech sample is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension; when i is greater than 1, the speech coding information of the (i-1)-th dimension output by the (i-1)-th speech coding layer is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension, where i is a positive integer less than or equal to N; based on the first coding information output by each coding layer and the weight parameters of each dimension, the audio features of the speech samples are determined.

[0127] N is an integer greater than 0. i is an integer greater than 0 and less than or equal to N.

[0128] In the process of continuous multi-round encoding of speech samples, except for the last round of encoding, the first encoded information generated in each remaining round of encoding can be regarded as the intermediate data generated in the multi-round encoding process. The intermediate data output by each round of encoding is taken out as the first encoded information of one dimension. This intermediate data can also be used as the input data for the next round of encoding. In this way, the first encoded information of N dimensions includes both (N-1) intermediate data and the output data of the last round of encoding.

[0129] In a speech recognition model, N speech coding layers can be connected in series. In the order of the series, the input of the first speech coding layer includes the speech sample, and the output of the first speech coding layer is the first coded information generated by encoding the speech sample once. Except for the first speech coding layer, the input of each speech coding layer is the first coded information output by the previous speech coding layer, and the output of each speech coding layer is the first coded information generated by encoding the first coded information output by the previous speech coding layer once.

[0130] In N speech coding layers, each speech coding layer encodes speech samples to obtain the first coded information output by each speech coding layer. The specific processing steps are as follows: The speech sample is input into the first speech coding layer for the first coding process, and the first coding information of the first speech coding layer is obtained and output. The first encoding information of the first speech coding layer is input into the second speech coding layer for a second encoding process, and the first encoding information of the second speech coding layer is obtained and output. ... The first encoding information of the (N-1)th speech coding layer is input into the Nth speech coding layer for the Nth encoding process to obtain the first encoding information of the Nth speech coding layer and output it.

[0131] When i=1, the speech sample is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension; when i is greater than 1, the speech coding information of the (i-1)-th dimension output by the (i-1)-th speech coding layer is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension.

[0132] After inputting a speech sample into the first speech coding layer for the first coding process, the first coding information output by the first speech coding layer can be obtained; after inputting the first coding information output by the first speech coding layer into the second speech coding layer for the second coding process, the first coding information output by the second speech coding layer can be obtained; after inputting the first coding information output by the second speech coding layer into the third speech coding layer for the third coding process, the first coding information output by the third speech coding layer can be obtained; ...; after inputting the first coding information output by the (N-1)th speech coding layer into the Nth speech coding layer for the Nth coding process, the first coding information output by the Nth speech coding layer can be obtained.

[0133] It should be noted that in the N speech coding layers of the speech recognition model, the coding methods adopted by each speech coding layer can be the same or different. However, considering that except for the first speech coding layer, the remaining (N-1) speech coding layers are all obtained by encoding the data output by the previous speech coding layer, the first coding information output by each of the N speech encoders is different and can be used to reflect the features of the speech sample in different dimensions.

[0134] For example, in the generation of the first encoded information in the first dimension, the first speech coding layer performs one round of coding processing on the speech sample; in the generation of the first encoded information in the second dimension, the first speech coding layer performs one round of coding processing on the speech sample, and the second speech coding layer performs another round of coding processing on top of that; in the generation of the first encoded information in the third dimension, the first speech coding layer performs one round of coding processing on the speech sample, the second speech coding layer performs another round of coding processing on top of that, and the third speech coding layer performs another round of coding processing on top of that… Furthermore, the reason for obtaining the first coding information output by each speech coding layer, rather than just the first coding information output by the last speech coding layer, is that the N speech coding layers can be generated based on a speech pre-trained model. The performance of each speech coding layer in the speech pre-trained model varies significantly when handling different speech recognition tasks. Therefore, different weight information can be configured for each coding layer in the speech pre-trained model when handling different tasks.

[0135] The following can be combined Figure 4 This example illustrates the weight information of each coding layer in a speech pre-trained model when processing different speech tasks. Figure 4 This application provides a weight information graph of a speech pre-trained model in different tasks.

[0136] like Figure 4 As shown, speech pre-trained model 1 includes 12 encoding layers, with each layer having different weights for different tasks. Speech pre-trained model 2 includes 24 encoding layers, with each layer having different weights for different tasks.

[0137] The speech sample is encoded N times through N speech coding layers to obtain the first coding information output by each speech coding layer. After obtaining the first coding information in N dimensions, the audio features of the speech sample can be determined based on the first coding information in N dimensions and the weight parameters of each dimension.

[0138] In one or more embodiments of this application, during the first training process of the speech recognition model, the weight parameters of each speech coding layer can be custom-configured initial weight values ​​for that speech coding layer. During the k-th training process of the speech recognition model, where k is an integer greater than 1, the weight parameters of each speech coding layer can be obtained by adjusting the weight parameters of the (k-1)-th training process based on the training loss of the (k-1)-th training process.

[0139] A speech recognition model can be a deep learning-based acoustic model with one set of parameters to be trained. It can be used to identify speech objects in speech data, such as a person speaking, a robot speaking, etc. Specifically, given a speech dataset containing multiple speakers taking turns speaking, the task requires determining who is speaking at each point in time.

[0140] In a speech recognition model comprising N speech coding layers, each speech coding layer has a weight parameter, which is one of the parameters to be trained in the speech recognition model. There is a one-to-one correspondence between the N dimensions and the N weight parameters.

[0141] In a speech recognition model comprising M speech coding layers, where M is greater than N, each speech coding layer has one weight parameter. It's important to note that during model training, only N of the M weight parameters are actually used for training; the remaining (MN) weight parameters can be configured to 0 and remain unchanged throughout the training phase. The N weight parameters involved in model training correspond one-to-one with the N dimensions. This is because the first encoded information of each dimension may differ in different speech recognition tasks, and some of the first encoded information might be considered redundant data, offering no benefit to subsequent speech recognition. Therefore, configuring the weight parameters to 0 reduces interference factors during model training.

[0142] Referring to the following formula (1), the audio features of the speech sample are determined based on the first coding information output by each coding layer and the weight parameters of each dimension. This can be achieved by performing a weighted summation based on the first coding information output by each coding layer and the weight parameters of each dimension to obtain the comprehensive coding information of the speech sample, which is then used as the audio features of the speech sample.

[0143] (1) in, This refers to the comprehensive coding information; norm_weights are the weight parameters of the speech coding layer. This is the first encoded information output by the speech coding layer. The `sum` function represents summation, and `dim` represents the dimension. `dim` is used to specify the elements participating in the weighted summation. When `dim=1`, the `sum` function is used to sum by column, where each element in the column is the product of the weight parameter of one dimension and the first encoded information output by the speech coding layer corresponding to that dimension. ).

[0144] Speech recognition models can include multiple speech coding layers, and the first encoded information output by each speech coding layer performs differently in different speech tasks. For example, the first encoded information output by coding layer 1 performs extremely well in the speaker log task in the first scenario, but poorly in the speaker recognition task in the second scenario, and so on. Considering that multiple speech coding layers can be generated based on a speech pre-trained model, and that this speech pre-trained model is not a proprietary model designed for the speaker log task, but a general-purpose model, some features in the first encoded information output by each speech encoder may be very useful for the speaker log task, some features may be very useful, and some features may be in between. By configuring corresponding weight parameters for each dimension and establishing the correspondence between the encoder and the dimension, comprehensive encoded information that fully utilizes the feature extraction capabilities of each speech encoder in the speech pre-trained model can be generated. This comprehensive encoded information can be used as the audio features of speech samples, and each weight parameter can be learned and continuously adjusted and optimized during model training.

[0145] In some implementations, a speech recognition model may include an encoding module, a linear layer, and a recognition model connected in series. The output of the encoding module is the input of the linear layer, and the output of the linear layer is the input of the recognition module. The encoding module may include multiple speech coding layers, which may be coding layers in a pre-trained speech model.

[0146] The linear layer can be used to perform the step "determine the audio features of the speech sample based on the first coding information output by each coding layer and the weight parameters of each dimension". The input of the linear layer is the first coding information of the speech sample in N dimensions, and the output of the linear layer is the audio features of the speech sample. The recognition module can be used to perform the subsequent step S206.

[0147] The weight parameters for each dimension can be learnable model parameters in the linear layers of the speech recognition model, meaning that the weight parameters for each dimension can be continuously adjusted and optimized during the training process of the speech recognition model.

[0148] In one specific implementation, the audio features of a speech sample are determined based on the first coding information output by each coding layer and the weight parameters of each dimension. This includes: comparing the weight parameters of each dimension with a preset weight threshold for each dimension; if the comparison result indicates that the weight parameters of the dimension are less than the preset weight threshold, then resetting the weight parameters of the dimension to a preset value; and performing a weighted summation process based on the reset weight parameters of each dimension and the first coding information to obtain the audio features of the speech sample.

[0149] The preset weight threshold can be a custom-configured, relatively small weight parameter value.

[0150] The weight parameter of each dimension is compared with the preset weight threshold. If the weight parameter is less than the preset weight threshold, the weight parameter is reset to the preset value; if the weight parameter is greater than or equal to the preset weight threshold, the value of the weight parameter remains unchanged.

[0151] For example, the preset weight threshold for each dimension can be set as needed, and the sum of the preset weight thresholds for each dimension is 1, and the preset value can be 0.

[0152] In the process of obtaining the audio features of a speech sample by performing a weighted summation based on the reset weight parameters of each dimension and the first encoding information, some dimensions may have weight parameters less than a preset weight threshold. To facilitate differentiation, these dimensions can be referred to as the first dimension, and their weight parameters need to be reset to 0. In the same process, another set of dimensions may have weight parameters greater than or equal to a preset weight threshold. To facilitate differentiation, this other set of dimensions can be referred to as the second dimension, and its weight parameters do not need to be reset. The reset weight parameters of each dimension include both the weight parameters of the second dimension and the reset weight parameters of the first dimension.

[0153] The audio features of the speech sample are obtained by performing a weighted summation process based on the weight parameters of each dimension after the reset and the first encoding information. The corresponding description of the audio features of the speech sample can be determined by referring to the first encoding information output by each encoding layer and the weight parameters of each dimension mentioned above.

[0154] If the weight parameter of any dimension is too small, it means that the first encoded information of the speech sample in that dimension may be unnecessary for the speaker log task. The weight parameter can be reset to 0 before weighted calculation.

[0155] Alternatively, the weight parameter k1 before the reset can be retained. During the current weighted calculation, k1 can be replaced with 0. When adjusting the weight parameter of the encoding layer based on the training loss later, the adjustment will be based on k1.

[0156] Step S206: Based on the audio features, identify the speech object in each frame of audio data in the speech sample to obtain the second speech recognition result of the speech object.

[0157] The voice object can be a person making a sound in the voice sample, a robot making a sound in the voice sample, or other objects making simulated human voices in the voice sample.

[0158] The second speech recognition result of the speech object can include multiple time periods and recognition sub-results for each time period. The multiple time periods are arranged in chronological order.

[0159] Within multiple time periods, two adjacent time periods may be consecutive, meaning the end time of one time period is the same as the start time of the next. For example, multiple time periods include: [T1, T2), [T2, T3), etc. In this case, the end time T2 of time period [T1, T2) is the same as the start time T2 of time period [T2, T3).

[0160] Within multiple time periods, there may be a time interval between two adjacent time periods, meaning that the end time of one time period is different from the start time of the next time period. For example, multiple time periods include: [T1, T2), [T3, T4), etc. Among them, the end time T2 of time period [T1, T2) is different from the start time T3 of time period [T3, T4).

[0161] Each time period's recognition sub-result corresponds to one or more speech objects, and the recognition sub-results of two adjacent time periods are different.

[0162] The recognition sub-result for each time period includes one or more speech objects that speak in the audio segment corresponding to that time period in the speech sample.

[0163] For example, multiple time periods include: [T1, T2), [T2, T3), etc. The recognition result for time period [T1, T2) indicates that in the speech sample, only one speech object A appears in the audio segment corresponding to time period [T1, T2). The recognition result for time period [T2, T3) indicates that in the speech sample, both speech object A and speech object B appear in the audio segment corresponding to time period [T2, T3). This can be understood as follows: in time period [T1, T2), only speech object A speaks; in time period [T2, T3), while speech object A is speaking, speech object B also makes a sound.

[0164] Multiple time periods and the recognition sub-results for each time period can be generated from the speech object of each frame of audio data in the speech sample.

[0165] For example, if the speech objects in the first, second, and third frames of audio data are the same, they can all be represented by the object identifier group (A, B). If the speech objects in the fourth frame of audio data are different from those in the third frame of audio data, then it can be determined that time period 1 includes the first, second, and third frames of audio data, and the recognition sub-result of time period 1 is (A, B).

[0166] Based on audio features, the speech objects in the speech samples are identified, and the second speech recognition result of the speech objects is obtained. Alternatively, the audio features can be input into the recognition module of the speech recognition model to perform recognition processing on the speech objects, thereby obtaining the second speech recognition result of the speech samples.

[0167] Based on audio features, the speech object in each frame of audio data in the speech sample is identified, and the second speech recognition result of the speech object is obtained. Alternatively, the speech sample can be divided according to audio features to obtain one or more audio segments. In the process of identifying the speech object of each audio segment, the audio features of each audio segment are compared with each pre-stored audio feature in the preset feature set to obtain the comparison result. Based on the comparison result, the speech object of the audio segment is determined to generate the second speech recognition result of the speech sample.

[0168] Each audio segment may include one or more frames of audio data. When an audio segment includes multiple frames of audio data, the speech objects in those multiple frames are the same. Identifying the speech object in each audio segment is equivalent to identifying the speech objects in multiple frames of audio data within that audio segment at once.

[0169] For example, audio features may include voiceprint features.

[0170] The speech samples are segmented based on audio features to obtain one or more audio segments. This can be done by dividing the speech samples into one or more audio segments arranged in chronological order based on voiceprint features, where the similarity of the voiceprint features between any two adjacent audio segments is less than a preset similarity threshold. For each audio segment, the audio features are compared with each pre-stored audio feature in a preset feature set to obtain the comparison result. This can be done by comparing the voiceprint features with each candidate voiceprint feature in a pre-configured voiceprint database. If a candidate voiceprint feature with the same voiceprint feature exists in the voiceprint database, the speech object corresponding to that candidate voiceprint feature is identified as the speech object in that audio segment.

[0171] In one specific implementation, the speech sample includes K frames of audio data, with K audio features, and each of the K frames of audio data corresponds one-to-one with one of the K audio features. Based on the audio features, the speech object in each frame of audio data in the speech sample is identified to obtain a second speech recognition result for the speech object. This includes: encoding the K audio features to obtain first feature data of the K frames of audio data; decoding the first feature data of each frame of audio data to obtain second feature data describing the speech object of each frame of audio data; classifying the second feature data of each frame of audio data to obtain the speech object of each frame of audio data; and obtaining the second speech recognition result of the speech object based on the speech object of the K frames of audio data.

[0172] The recognition module in a speech recognition model can include an encoding unit, a decoding unit, and a classification unit connected in series. The output of the encoding unit is the input of the decoding unit, and the output of the decoding unit is the input of the classification unit.

[0173] The encoding unit can employ any encoder suitable for speaker logging tasks, such as EEND (End-to-End Neural Diarization) and its improved series of encoders, the Conformer architecture for speaker recognition, ResNet (Deep Residual Network) and its related variants, etc.

[0174] EEND is an end-to-end neural network model used for speaker segmentation and clustering tasks in ASR (Automatic Speech Recognition), also known as speaker logging. The EEND encoder refers to the part of the model that processes the input audio signal and extracts features.

[0175] The conformer architecture is a deep learning architecture that combines convolutional neural networks and Transformer models.

[0176] A speech sample can include K frames of audio data, with K audio features. There is a one-to-one correspondence between the K frames of audio data and the K audio features. K is an integer greater than 1.

[0177] In practice, the encoding unit can encode K audio features to obtain the first feature data of K frames of audio data. This first feature data can be frame-level feature data describing a speech sample. The number of first feature data can be K. Each frame of audio data has one corresponding first feature data.

[0178] For example, a speech sample consists of 100 frames of audio data, with 100 audio features, and each of the 100 frames of audio data corresponds one-to-one with one of the 100 audio features. Through encoding, each audio feature can be converted into a frame-level first feature data.

[0179] The decoding unit can employ any decoder suitable for speaker logging tasks, such as LSTM (Long Short-Term Memory), EEND and its improved series of decoders, etc.

[0180] In practice, the decoding unit can decode the first feature data of each frame of audio data to obtain the second feature data describing the speech object of each frame of audio data. The second feature data can be frame-level feature data describing the speech object.

[0181] For example, by inputting 100 frames of first feature data into the decoding unit for decoding processing, 100 frames of second feature data can be obtained.

[0182] The classification unit can use any classification layer, which can call a classification function to process the input data. For example, the classification function can be the Softmax function.

[0183] In practice, the classification layer performs classification processing based on the second feature data of each frame of audio data to obtain the classification result of each frame. This classification result can correspond to one of multiple preset classification labels. These multiple preset classification labels can refer to the classification annotation information of the aforementioned speech samples. The classification result of each frame of audio data can be used to represent the speech object of each frame of audio data.

[0184] The classification process is performed based on the second feature data to obtain the classification result. This can be achieved by calling a classification function to calculate the probability value of the classification result of the second feature data corresponding to each preset classification label, and using the probability value as the classification result, or by determining the classification result based on the probability value.

[0185] For example, calling the classification function calculates the probability value p1 for the classification result of the second feature data corresponding to category label 1, p2 for the classification result of the second feature data corresponding to category label 2, and so on, and the probability value p5 for the classification result of the second feature data corresponding to category label 5, and so on. Where p1 is greater than or equal to a preset probability threshold, and p2-p5 are all less than the preset probability threshold, then it can be determined that the classification result of the second feature data corresponds to category label 1.

[0186] With multiple preset category labels, some preset category labels correspond to only one speech object, while others correspond to multiple speech objects. By classifying and predicting the probability of each situation occurring, the risk of confusing different speech objects when multiple people speak at the same time can be reduced.

[0187] Since the second feature data is frame-level feature data describing the speech object, the classification result of each second feature data can reflect the classification result of the corresponding frame. Therefore, the speech object of each frame of audio data in K frames of audio data can be added to the second speech recognition result of the speech object.

[0188] For example, there are 10 second feature data at the frame level, and each of these 10 second feature data has a classification result: the classification result of the first second feature data is speech object 1; the classification result of the second second feature data is speech object 1; the classification result of the third second feature data includes speech object 1 and speech object 2; ...; the classification result of the tenth second feature data includes speech object 1 and speech object 2.

[0189] Therefore, K=10. In the 10 frames of audio data, the speech object of the first frame of audio data is speech object 1; the speech object of the second frame of audio data is speech object 1; the speech object of the third frame of audio data includes speech object 1 and speech object 2; ...; the speech object of the tenth frame of audio data includes speech object 1 and speech object 2.

[0190] Furthermore, the speech object of 10 frames of audio data can be added to the second speech recognition result of the speech object.

[0191] Taking a speech sample involving three speech objects (A, B, C) as an example: the classification label for each frame of audio data can be one of the following: A, B, C, AB, AC, BC, or ABC. Feature-based classification means that for each frame of speech data, the features of that frame determine which of the above seven categories the frame corresponds to. For example, if the classification result is BC, then the speech objects in that frame of audio data include B and C.

[0192] It is important to emphasize that the training method for the speech recognition model provided in one or more embodiments of this application does not train the model to learn the features of a single speech object, identifying whether a frame of audio data includes A, B, or C respectively. Instead, it trains the model to learn how to distinguish between different situations, such as features where only A is audible, features where only B is audible, and features where A and B are audible simultaneously. Therefore, there is no process of first identifying speech objects and then determining the classification result based on the speech objects; instead, classification is directly based on features. This classification result can indicate that only one speech object is audible in the corresponding frame of speech data, or that multiple speech objects are audible simultaneously in the corresponding frame of speech data.

[0193] In addition, in the recognition module, the model parameters of the encoding unit and the decoding unit are used as training parameters, which can be continuously adjusted and optimized during the training process of the speech recognition model.

[0194] Step S208: Construct a loss function based on the second speech recognition result and the first speech recognition result.

[0195] The first speech recognition result of the speech sample can include the classification label corresponding to each frame of audio data in the speech sample.

[0196] The loss function is constructed based on the second speech recognition result and the first speech recognition result. It can be constructed based on the classification result of the speech object in each frame of audio data in the speech sample and the classification label of each frame.

[0197] The loss function can be the cross-entropy loss function or other custom loss functions.

[0198] In one specific implementation, each classification label is a first object identifier group, and each object identifier group includes a first identifier and the number of first identifiers is within a preset range; based on audio features, the speech object in each frame of audio data in the speech sample is identified to obtain a second speech recognition result of the speech object, including: classifying the speech object in each frame of audio data according to audio features to obtain a second object identifier group; constructing a loss function based on the second speech recognition result and the first speech recognition result, including: constructing a loss function based on the second object identifier group of each frame of audio data and the first object identifier group of each frame of audio data.

[0199] Classification labels are used to represent the speech objects participating in the dialogue within a corresponding frame of audio data. Specifically, each classification label is a first object identifier group, and thus, speech objects in a frame of audio data can be represented in the form of first object identifier groups. For example, if a speech sample includes three objects, A, B, and C, then A, B, C, AB, AC, BC, and ABC are included, for a total of seven classification labels. A, B, C, AB, AC, BC, and ABC can be used to form seven first object identifier groups. One first identifier represents one speech object, and each identifier in the object identifier group is the first identifier. For example, A and C in the object identifier group AC are both first identifiers.

[0200] The maximum value of the preset range can be determined by the number of speech objects in each speech sample.

[0201] For example, if there are 3 speech objects in speech sample 1, 2 speech objects in speech sample 2, and 4 speech objects in speech sample 3, then the maximum value of the preset range is 4 and the minimum value is 2.

[0202] The minimum value within the preset range can be 1. In this case, the classification labels can also include a silence label indicating that the number of speech objects in the corresponding frame of audio data is 0.

[0203] Taking a preset range with a maximum value of 3 and a minimum value of 1 as an example, the classification labels are illustrated as follows: Each speech sample contains speech objects A, B, and C. Therefore, classification label 1 is the first object identifier group (A), classification label 2 is the first object identifier group (B), classification label 3 is the first object identifier group (C), classification label 4 is the first object identifier group (A, B), classification label 5 is the first object identifier group (B, C), classification label 6 is the first object identifier group (A, C), and classification label 7 is the first object identifier group (A, B, C). Additionally, classification label 8 is the silence label S. The classification label for each frame of audio data in the speech sample is one of the above classification labels 1 to 8.

[0204] For example, in speech sample 1, the classification label of the first frame of audio data is (A, B), the classification label of the second frame of audio data is (A, C), the classification label of the third frame of audio data is (A), the classification label of the fourth frame of audio data is S, and so on.

[0205] Based on audio features, the speech objects in each frame of audio data are classified to obtain a second object identifier group.

[0206] The second object identifier group of each frame of audio data is the classification result of each frame of audio data.

[0207] The second object identifier group can be generated from the first object identifier group.

[0208] For example, the category labels include: first object identifier group (A), first object identifier group (B), and first object identifier group (A, B).

[0209] Therefore, the second object identifier group for each frame of audio data can be one of the following: second object identifier group (A), second object identifier group (B), and second object identifier group (A, B).

[0210] It should be emphasized that in this implementation, there is no process of first recognizing the speech object and then classifying it.

[0211] Let's take the example of speech objects A, B, and C in the speech sample: In this implementation, the speech recognition model is trained to learn which features will be classified into the second object identifier group (A), which features will be classified into the second object identifier group (A, B), which features will be classified into the second object identifier group (A, B, C), and so on, for a frame of audio data.

[0212] The characteristics of A and B sounding simultaneously (i.e., the mixed sound of A and B) are different from the characteristics of A sounding alone, and the characteristics of the mixed sound of A and B are also different from the characteristics of B sounding alone.

[0213] In other words, when multiple speech objects are speaking in a frame of audio data at the same time, the speech recognition model will not identify whether A exists in the frame of audio data separately, and whether B exists in the frame of audio data separately. Instead, it will classify the frame of audio data into the second object identification group (A, B) based on the audio features.

[0214] The seven categories (A, B), (A, C), (B, C), (A, B, C), (A), (B), and (C) are completely equal. Taking (A, B) as an example, during the model training process, it can be approximated that the speech recognition model treats (A, B) as a completely new speech object that is different from (A) and different from (B) to learn its features.

[0215] Therefore, the classification process involves recognizing speech objects. The classification result may include one speech object or multiple speech objects.

[0216] During the training process of the speech recognition model, the first object identifier group can help the speech recognition model learn how to distinguish the features of different multi-person mixed speech objects belonging to mixed speech, for example, distinguishing (A, B) from (A, C). Furthermore, the first object identifier group can help the speech recognition model learn how to distinguish the features of single-person speech from the features of multi-person mixed speech, for example, distinguishing (A) from (A, B). In this way, when a frame of audio data includes multiple speech objects, it is not necessary to identify each of these multiple speech objects separately, which can reduce the interference caused by each speech object in multi-person mixed speech on the recognition of other speech objects.

[0217] In addition, after classifying the speech objects in each frame of audio data according to audio features, the classification result can also be a silence indicator indicating that there are no speech objects in the corresponding frame of audio data.

[0218] For example, in speech sample 1, the classification result of the first frame of audio data is the second object identifier group (A, C), the classification result of the second frame of audio data is the second object identifier group (A, C), the classification result of the third frame of audio data is the second object identifier group (A), the classification result of the fourth frame of audio data is the silence identifier, and so on.

[0219] The loss function is constructed based on the second object identifier group and the first object identifier group of each frame of audio data. This can be a cross-entropy loss function generated for each frame of audio data based on the second object identifier group and the first object identifier group, or any other predetermined loss function can be used.

[0220] In the process of speech object recognition, each speech object can be treated as a category. For each frame of audio data, the predicted probability of including the speech object in that frame is output. This predicted probability is compared with a threshold. If the predicted probability is greater than the threshold, the speech object is considered to be included in that frame; otherwise, it is not. However, the accuracy of this method is affected by the threshold when multiple people may speak simultaneously. A well-set threshold leads to high prediction accuracy, while an unsuitable threshold easily leads to recognition errors. Furthermore, the threshold for the same speech recognition model may differ in different scenarios, resulting in relatively poor transferability of the model. A prediction that is accurate in one scenario may be inaccurate when processing speech data from another scenario. For example, a speech sample includes three speech objects: A, B, and C. In the k-th frame of audio data, k is an integer greater than 0. The predicted probability of identifying A is p1, the predicted probability of identifying B is p2, and the predicted probability of identifying C is p3. p1, p2, and p3 are all values ​​between 0 and 1. If p1 is greater than P1, then the speech object in the k-th frame of audio data is determined to include A. If p2 is greater than P2, then the speech object in the k-th frame of audio data is determined to include B. If p3 is greater than P3, then the speech object in the k-th frame of audio data is determined to include C.

[0221] In the training method of the speech recognition model provided in the embodiments of this application, multiple speech objects that speak simultaneously in the same frame of audio data can be classified as a category. In this way, during the process of recognizing speech objects, there is only one classification result for each frame of audio data.

[0222] For example, a speech sample includes three objects: A, B, C, AB, AC, BC, ABC, and a silence label S. Classifying the k-th frame of audio data (where k is greater than 0), the classification result for the k-th frame is the second object identifier group (B, C).

[0223] Compared to the aforementioned technical solutions that treat each speech object as a category, this embodiment lists all possibilities of one or more speech objects in a frame of speech data, with each possibility as a category. In this way, each frame of audio data has only one classification result. During training, the speech recognition model learns how to identify a speech object when there is only one speech object in a frame of audio data, and also learns how to identify the number of speech objects when there are multiple speech objects in a frame of audio data, as well as who each of the multiple speech objects is. Therefore, this embodiment does not require setting a threshold and comparing it with a threshold, which is more conducive to maintaining recognition accuracy when the model is transferred to other scenarios.

[0224] In one specific implementation, the training method for the speech recognition model further includes: performing speech activity detection on the speech samples based on audio features to obtain a first detection result; performing overlapping speech detection on the speech samples based on audio features to obtain a second detection result; and constructing a loss function based on the second speech recognition result and the first speech recognition result, including: constructing a loss function based on the second speech recognition result, the first detection result, the second detection result, and the first speech recognition result.

[0225] VAD (Voice Activity Detection) is a technique used to identify the presence of active speech in a speech signal. It is primarily used in speech processing systems to help distinguish between speech segments and non-speech segments (such as silence, background noise, etc.) in a speech signal.

[0226] Overlapping speech detection can detect whether at least two speech objects are speaking at the same time.

[0227] After encoding the audio features to obtain the first feature data of the speech sample, this first feature data can be frame-level feature data describing the speech sample. This first feature data can be used for speaker logging tasks, speech activity detection tasks, and overlapping speech detection tasks. Therefore, when speech samples need to be processed separately for speech object recognition, speech activity detection, and overlapping speech detection, the same encoding unit can be reused to encode the audio features to obtain intermediate encoded information. This intermediate encoded information can then be decoded using different decoders to obtain decoded information, and the decoded information can be classified using different classification units.

[0228] Based on audio features, speech activity detection processing is performed on the speech samples to obtain a first detection result. This can be achieved by encoding the audio features to obtain the first feature data of the speech samples, decoding the first feature data to obtain speech activity features, and classifying the speech activity features to obtain the first detection result.

[0229] Based on audio features, speech activity detection processing is performed on the speech samples to obtain the first detection result. Alternatively, the audio features can be encoded to obtain the first feature data of the speech samples, and the first feature data can be classified to obtain the first detection result.

[0230] Since the first feature data is frame-level feature data, the first detection result corresponding to each first feature data is also a frame-level activity detection result.

[0231] The decoder used to decode the first feature data to obtain the speech activity features can be a linear layer or any decoder suitable for the speech activity detection task.

[0232] Based on audio features, overlapping speech detection processing is performed on the speech samples to obtain a second detection result. This can be achieved by encoding the audio features to obtain the first feature data of the speech samples, decoding the first feature data to obtain overlapping speech features, and classifying the overlapping speech features to obtain the second detection result.

[0233] Since the first feature data is frame-level feature data, the second test result corresponding to each first feature data is also a frame-level overlapping speech detection result.

[0234] The decoder used to decode the first feature data to obtain overlapping speech features can be a linear layer or any decoder suitable for overlapping speech detection tasks.

[0235] The first speech recognition result of the speech sample may include time labeling information, classification labeling information, and may also include speech activity labeling information and overlapping speech labeling information.

[0236] Specifically, speech activity annotation information can include one or more time periods in the speech sample where a speech signal exists. Overlapping speech annotation information can include one or more time periods where at least two speech objects are uttering.

[0237] The loss function is constructed based on the second speech recognition result, the first detection result, the second detection result, and the first speech recognition result. This can be achieved by generating a first loss sub-function based on the time annotation information and classification annotation information in the second and first speech recognition results; generating a second loss sub-function based on the speech activity annotation information in the first detection and first speech recognition results; generating a third loss sub-function based on the overlapping speech annotation information in the second detection and first speech recognition results; and finally generating the loss function of the speech recognition model based on preset weight information, the first loss sub-function, the second loss sub-function, and the third loss sub-function.

[0238] The first, second, and third loss functions can be the cross-entropy loss function or other custom-configured loss functions.

[0239] In practice, a weighted summation can be performed based on preset weight information, the first loss function, the second loss function, and the third loss function to obtain a comprehensive loss function, which is then used as the loss function of the speech recognition model. The preset weight information serves as the training parameters during the speech recognition model's training process. The weights can be adjusted based on the speaker log predictions. For example, if the error rate is high in overlapping sound detection, the weight corresponding to the third loss function can be increased; conversely, if the prediction of the presence or absence of sound signals is poor, the weight corresponding to the third loss function can be appropriately increased.

[0240] The reason for jointly training the speech activity detection, overlapping speech detection, and speaker logging tasks is to account for common errors that may occur during the speaker logging task: misidentifying a frame of audio data containing no sound signal as containing one or more speech objects; misidentifying a frame of audio data containing sound signals originating from one or more speech objects as not containing any sound signal; and confusing speech objects appearing in a frame of audio data. By simultaneously executing the speech activity detection, overlapping speech detection, and speaker logging tasks, the first and second detection results can be used to help determine the accuracy of the recognition results. Therefore, during training, adjusting the model parameters through a loss function, simultaneously optimizing the speech activity detection, overlapping speech detection, and speaker logging tasks, and directly minimizing speaker logging errors, can achieve a synergistic effect.

[0241] In one or more embodiments of this application, the loss function of the speech recognition model may be constructed based on the second speech recognition result and the first speech recognition result; in one or more embodiments of this application, the loss function of the speech recognition model may be constructed based on the second speech recognition result, the first detection result, and the first speech recognition result; in one or more embodiments of this application, the loss function of the speech recognition model may be constructed based on the second speech recognition result, the second detection result, and the first speech recognition result; in one or more embodiments of this application, the loss function of the speech recognition model may be constructed based on the second speech recognition result, the first detection result, the second detection result, and the first speech recognition result. The construction of the loss function of the speech recognition model can be flexibly configured according to actual needs.

[0242] In one specific implementation, the audio features include a first audio feature describing the energy of the speech sample, a second audio feature describing the spectrum of the speech sample, and a third audio feature describing the signal waveform of the speech sample. Based on the audio features, speech activity detection is performed on the speech sample to obtain a first detection result, including: determining a first time period in which active speech exists in the speech sample based on the first, second, and third audio features; determining a second time period in which active speech is missing based on the total time period of the speech sample and the first time period; and generating the first detection result based on the first and second time periods.

[0243] The first audio feature can be the energy of the speech sample, or other audio features that can directly or indirectly reflect the energy level of the speech sample. For any given speech data, the energy of the speech data refers to the energy level of the speech signal within a certain time window. Speech segments with active speech usually have higher energy than non-speech segments, so energy can be used as one of the reference criteria for speech activity detection.

[0244] The second audio feature can be the spectral features of the speech sample. For example, spectral features can be formants, spectral flatness, etc. The spectral features of speech segments with active speech are significantly different from those of non-speech segments. Therefore, spectral features can be used as one of the reference bases for speech activity detection.

[0245] The third audio feature can be the zero-crossing rate of the speech sample, or other audio features that can directly or indirectly reflect the waveform characteristics of the speech sample. The zero-crossing rate refers to the number of times the signal waveform crosses a zero point per unit time. The zero-crossing rate energy of speech segments with active speech is usually higher than that of non-speech segments, so the zero-crossing rate can be used as one of the reference criteria for speech activity detection.

[0246] In determining the first time period containing active speech within a speech sample based on the first, second, and third audio features, the following operations can be performed: For each frame of audio data in the speech sample, perform activity prediction processing on each frame of audio data based on the first audio feature to obtain a first prediction result; perform activity prediction processing on each frame of audio data based on the second audio feature to obtain a second prediction result; perform activity prediction processing on each frame of audio data based on the third audio feature to obtain a third prediction result; and generate a comprehensive prediction result for each frame of audio data based on the first, second, and third prediction results.

[0247] If the comprehensive prediction results indicate that there is active speech in a frame of audio data, then a first marker is added to the frame of audio data. Based on the first marker, a first time period in which active speech exists is determined in the speech sample. Each frame of audio data in the first time period carries the first marker.

[0248] Based on the time period of the speech sample and the first time period, the second time period with missing active speech can be determined. This can be achieved by defining the portion of the speech sample outside the first time period as the second time period.

[0249] Based on the first time period and the second time period, a first detection result is generated, which indicates that there is active speech in the first time period and no active speech in the second time period.

[0250] Step S210: Iteratively train the speech recognition model based on the loss function.

[0251] The speech recognition model is iteratively trained based on the loss function to obtain the target speech recognition model. The specific implementation process of each model training can be referred to steps S204-S206. After obtaining the speech recognition result corresponding to each model training, the training loss of the model training can be calculated based on the loss function, the second speech recognition result and the first speech recognition result. The training loss can be used to drive the adjustment of the model parameters of the speech recognition model.

[0252] Through iterative training, the loss can be continuously optimized and reduced until the training loss converges to less than or equal to the preset loss threshold.

[0253] The training parameters of a speech recognition model include, but are not limited to: weight parameters for each dimension, model parameters for the encoding units in the recognition module, and model parameters for the decoding units in the recognition module, etc.

[0254] In practice, model training can be divided into multiple stages, for example: In the first stage, the model parameters of multiple speech coding layers are frozen during the iterative training of the speech recognition model. In the second stage, the model parameters of multiple speech coding layers are unfrozen, and the model parameters of each speech coding layer are continuously adjusted and optimized during the iterative training of the speech recognition model. In addition, the second stage can adjust the ratio of the number of first speech samples constructed from real dialogues to the number of second speech samples constructed from simulated dialogues in the training samples, so as to use more first speech samples constructed from real dialogues. In the third stage, only speech samples constructed from dialogue data of the target business scenario are used for model training.

[0255] In addition, if the amount of dialogue data in the target business scenario is small, in order to avoid overfitting and reducing the training effect of the model, only the first two stages of model training mentioned above can be performed.

[0256] Additionally, before training a machine learning model begins, it may be necessary to rely on human experience and historical training data to configure the parameters of the model to be trained. For example, the model parameters of machine learning models that perform similar tasks can be found from historical training data, and the model parameters of the machine learning model to be trained can be configured based on these model parameters.

[0257] However, not all models have corresponding historical training data available, and the lack of human experience is also common. In such cases, users may need to spend some time and effort on parameter configuration. Compared with directly using existing historical training data for parameter configuration, the overall model training time is longer and the model training efficiency is relatively low.

[0258] The speech recognition model training method provided in one or more embodiments of this application, by establishing the correspondence between various dimensions and speech coding layers, and by continuously adjusting and optimizing the weight parameters of each dimension during iterative training, helps the speech recognition model learn how to utilize the features of different dimensions of speech samples to generate audio features suitable for speech recognition tasks. In this way, in the absence of historical training data as a reference, it is not necessary to predetermine the importance of features in different dimensions and assign corresponding weights before model training, thus improving model training efficiency. Furthermore, each speech coding layer can be generated based on a pre-trained language model, which can be reused for different speech recognition tasks. Therefore, there is no need to pre-design a highly targeted coding module; a general model can be used directly. In addition, the speech pre-trained model can be a model pre-trained on a large amount of unlabeled data, exhibiting excellent generalization ability. Good training results can be achieved with only fine-tuning on a small scale of labeled data. Using the speech pre-trained model for speaker log tasks is beneficial for improving model training efficiency.

[0259] In such Figure 2In the illustrated embodiment, firstly, speech samples and a first speech recognition result annotated with the speech samples are obtained; the first speech recognition result includes a classification label for each frame of audio data in the speech sample, where each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data; then, feature extraction is performed on the speech samples to obtain audio features of the speech samples; next, based on the audio features, the speech objects in each frame of audio data in the speech samples are identified to obtain a second speech recognition result for the speech objects; next, a loss function is constructed based on the second speech recognition result and the first speech recognition result; finally, the speech recognition model is iteratively trained based on the loss function. Considering that when a speech sample involves multiple speech objects participating in a dialogue, these speech objects may participate in the dialogue simultaneously, for example, three people speaking at the same time. In a single frame of audio data, the voices of multiple speech objects participating in the dialogue simultaneously are mixed together, and each speech object may interfere with the recognition accuracy of other speech objects. In this embodiment, the first speech recognition result for annotating the speech sample includes a classification label for each frame of audio data. Each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data. In this way, even if there are multiple speech objects participating in the dialogue simultaneously in a frame of audio data, that frame of audio data will only have one classification label. This classification label can be combined with a second classification label during training. The speech recognition results are used to construct a loss function, which can then guide the speech recognition model to learn how to map a frame of audio data to an output category composed of multiple speech objects. This output category is the second speech recognition result of the speech objects in the frame of audio data. Furthermore, the mixing of the sounds of multiple speech objects does not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution that guides the speech recognition model to learn how to map a frame of speech data to multiple output categories, with each output category corresponding to a speech object, the embodiments of this application can reduce the negative impact of the mutual interference of the sounds of the speech objects speaking at the same time on the recognition of speech objects, and is more conducive to improving the recognition accuracy of speech objects.

[0260] Based on the same technical concept, this application also provides another method for training a speech recognition model. Figure 5 Data flow diagram of another speech recognition model training method provided in the embodiments of this application.

[0261] like Figure 5As shown, speech samples are input into the speech pre-training model 502 for encoding processing, and the first encoding information output by each speech encoding layer in the speech pre-training model 502 is obtained. For details, please refer to the corresponding explanation of step S204 above. In the speech pre-training model 502, each speech encoding layer has a corresponding dimension. The first encoding information output by each speech encoding layer can be regarded as the first encoding information of the speech sample in the dimension corresponding to that speech encoding layer. In the following text, the dimension corresponding to the speech encoding layer can be simply referred to as the encoding layer dimension.

[0262] The linear layer 504 with learnable weights generates comprehensive coding information of speech samples based on the first coding information output by each speech coding layer and the weight parameters of each coding layer dimension. The comprehensive coding information is used as the audio features of the speech samples. For details, please refer to the corresponding description of step S204 above.

[0263] The encoding unit 506, the decoding unit 508, and the classification unit 510 work together to perform the following steps: based on audio features, identify speech objects in speech samples and obtain a second speech recognition result of the speech objects, as detailed in the description of step S206 above.

[0264] The second speech recognition result of the speech object can be regarded as the prediction result 512.

[0265] A loss function is constructed based on the prediction result 512 and the first speech recognition result 514 of the speech sample. The speech recognition model is then iteratively trained based on the loss function to obtain the target recognition model. For details, please refer to the corresponding explanations of steps S208 and S210 above.

[0266] The loss function can be cross-entropy loss 516.

[0267] The cross-entropy loss 516 can be used to drive the adjustment of model parameters for the linear layer 504, the encoding unit 506, and the decoding unit 508 with learnable weights.

[0268] Since the technical concept is the same, the description in this embodiment is relatively simple. For the relevant parts, please refer to the corresponding descriptions of the method embodiments provided above.

[0269] Based on the same technical concept, this application also provides another method for training a speech recognition model. Figure 6 Data flow diagram of another speech recognition model training method provided in the embodiments of this application.

[0270] like Figure 6As shown, the speech samples are input into the speech pre-training model 602 for encoding processing, and the first encoding information output by each speech encoding layer in the speech pre-training model 602 is obtained. For details, please refer to the corresponding description of step S204 above.

[0271] The linear layer 604 with learnable weights generates comprehensive coding information of speech audio samples based on the first coding information output by each speech coding layer and the weight parameters of each coding layer dimension. The comprehensive coding information is used as the audio features of the speech audio samples. For details, please refer to the corresponding description of step S204 above.

[0272] The encoding unit 606, the speaker log decoding unit 608, and the speaker log classification unit 610 are used together to perform the following steps: based on the audio features, identify the speech object of each frame of audio data in the speech sample, and obtain the second speech recognition result 612 of the speech object. For details, please refer to the corresponding description of the aforementioned step S206.

[0273] The encoding unit 606, the speech activity detection decoding unit 614, and the speech activity detection classification unit 616 are used together to perform the following steps: based on the audio features, perform speech activity detection processing on the audio sample to obtain the first detection result 618.

[0274] The encoding unit 606, the overlapping speech detection decoding unit 620, and the overlapping speech detection classification unit 622 are used together to perform the following steps: based on the audio features, the audio samples are processed to perform overlapping speech detection to obtain a second detection result 624.

[0275] A loss function is constructed based on the speech recognition result 612, the first detection result 618, the second detection result 624, and the first speech recognition result 626 of the audio sample.

[0276] Since the technical concept is the same, the description in this embodiment is relatively simple. For the relevant parts, please refer to the corresponding descriptions of the method embodiments provided above.

[0277] The following are examples of one or more speech recognition methods provided in this specification: In the above embodiments, a method for training a speech recognition model is provided. Based on the same technical concept, this application also provides a speech recognition method. Figure 7 This is a flowchart illustrating a speech recognition method provided in an embodiment of this application.

[0278] like Figure 7As shown, in step S702, the first speech data is input into the target speech recognition model for speech object recognition processing to obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the speech recognition model training method.

[0279] In this step, the training method of the speech recognition model refers to the training method of the speech recognition model provided in any of the aforementioned method embodiments.

[0280] Step S704: Based on the speech recognition results of the speech objects, the first speech data is divided into multiple time periods, and the speech objects in each time period are marked.

[0281] In some implementations, during the model training phase: Based on audio features, speech objects in speech samples are identified, yielding a second speech recognition result for each speech object. Based on audio features, speech activity detection is performed on the audio samples, yielding a first detection result. Based on audio features, overlapping speech detection is performed on the audio samples, yielding a second detection result. A loss function is constructed based on the second speech recognition result, the first detection result, the second detection result, and the first speech recognition result. The speech recognition model is then iteratively trained based on this loss function to obtain the target speech recognition model. For details, please refer to [reference needed]. Figure 2 The corresponding descriptions of steps S204-S210 in the embodiments.

[0282] During the model application phase: Based on audio features, the speech objects in the first speech data are identified, and the speech recognition results of the speech objects in the first speech data are obtained.

[0283] In summary, in the above implementation methods, the speech activity detection task and the overlapping speech detection task are only used to assist the training of the speech recognition task during the model training phase. After the target speech recognition model is put into application, the speech activity detection task and the overlapping speech detection task are no longer executed.

[0284] In other implementations, during the model training phase: Based on audio features, speech objects in speech samples are identified, yielding a second speech recognition result for each speech object. Based on audio features, speech activity detection is performed on the audio samples, yielding a first detection result. Based on audio features, overlapping speech detection is performed on the audio samples, yielding a second detection result. A loss function is constructed based on the second speech recognition result, the first detection result, the second detection result, and the first speech recognition result. The speech recognition model is then iteratively trained based on this loss function to obtain the target speech recognition model. For details, please refer to [reference needed]. Figure 2 The corresponding descriptions of steps S204-S210 in the embodiments.

[0285] During the model application phase: Based on audio features, identify speech objects in the first speech data to obtain speech recognition results for the speech objects in the first speech data; based on audio features, perform speech activity detection processing on the audio samples to obtain a first detection result; based on audio features, perform overlapping speech detection processing on the audio samples to obtain a second detection result; adjust the speech recognition results of the speech objects based on the first and second detection results.

[0286] In summary, among the other implementation methods mentioned above, the speech activity detection task and the overlapping speech detection task can be used to assist the training of the speech recognition task during the model training stage, or they can be used as reference information to improve the recognition accuracy of the speech recognition results after the target speech recognition model is put into application.

[0287] In addition, in different scenarios, the weight information of the first detection result, the second detection result, and the speech recognition result can be configured according to the task requirements, so that the final weighted summation of the target speech recognition result is more in line with the task requirements. Alternatively, the specific processing logic for generating the target speech recognition result based on the first detection result, the second detection result, and the speech recognition result can be configured according to the task requirements.

[0288] For example, in some scenarios, the task requires that no sound leakage should occur. For each frame of audio, if the first detection result of the audio frame indicates the presence of a speech signal, but the speech recognition model of the target audio frame indicates that there is no speech object, which contradicts the first detection result, then the audio frame is considered to contain a speech object, and the speech object contained in the audio frame is labeled as the one with the highest predicted probability value among the preset classification labels in the classification results.

[0289] For example, in some scenarios, the purpose of identifying speech objects in speech samples is to determine the audio segments in which the objects are speaking, so that subsequent voiceprint recognition processing can be performed based on these audio segments. Considering that overlapping speech and silence have a significant impact on subsequent voiceprint recognition, usually only the audio segments in which a single speech object is speaking are collected. For any frame of audio data, if at least one of the first detection result and the speech recognition result predicts that the frame of audio data is silent, then the frame of audio data is considered silent data. If at least one of the second detection result and the speech recognition result predicts that at least two speech objects are speaking simultaneously in the frame of audio data, then the frame of audio data is considered overlapping speech data. Thus, the silent data and overlapping speech data in the first speech data can be removed to obtain the audio segments in which a single speech object is speaking.

[0290] In this embodiment, firstly, the first speech data is input into a target speech recognition model for speech object recognition processing to obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained using a speech recognition model training method; based on the speech recognition result of the speech object, the first speech data is divided into multiple time periods, and the speech objects in each time period are labeled. Considering that when a speech sample involves multiple speech objects participating in a dialogue, each speech object may participate in the dialogue simultaneously, for example, three people speaking at the same time. In a frame of audio data, the voices of multiple speech objects participating in the dialogue simultaneously are mixed together, and each speech object may interfere with the recognition accuracy of other speech objects. In this embodiment, the first speech recognition result labeled with the speech sample includes a classification label for each frame of audio data. Each classification label represents the speech object participating in the dialogue in the corresponding frame of audio data. In this way, even if there are multiple speech objects participating in the dialogue simultaneously in a frame of audio data, that frame of audio data only has one classification label. This classification label can be combined with a second... The speech recognition results are used to construct a loss function, which can then guide the speech recognition model to learn how to map a frame of audio data to an output category composed of multiple speech objects. This output category is the second speech recognition result of the speech objects in the frame of audio data. Furthermore, the mixing of the sounds of multiple speech objects does not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution that guides the speech recognition model to learn how to map a frame of speech data to multiple output categories, with each output category corresponding to a speech object, the embodiments of this application can reduce the negative impact of the mutual interference of the sounds of the speech objects speaking at the same time on the recognition of speech objects, and is more conducive to improving the recognition accuracy of speech objects.

[0291] Since the technical concept is the same, the description in this embodiment is relatively simple. For the relevant parts, please refer to the corresponding descriptions of the method embodiments provided above.

[0292] Based on the same technical concept, this application also provides another speech recognition method. Figure 8 Data flow diagram of another speech recognition method provided in the embodiments of this application.

[0293] like Figure 8 As shown, the first speech data to be recognized is input into the speech pre-training model 802 for encoding processing, and the first encoding information output by each speech encoding layer in the speech pre-training model 802 is obtained. For details, please refer to the corresponding descriptions of steps S204 and S702 above.

[0294] The linear layer 804 with learnable weights generates comprehensive coding information for the first speech data based on the weight parameters of each coding layer and the first coding information output by each coding layer. This comprehensive coding information is then used as the audio feature of the first speech data. For details, please refer to the corresponding descriptions of steps S204 and S702 above. The coding unit 806, the speaker log decoding unit 808, and the speaker log classification unit 810 jointly perform the following steps: based on the audio features, they identify the speech objects in the first speech data, obtaining the speech recognition result 812 of the speech objects in the first speech data. For details, please refer to the corresponding descriptions of steps S206 and S702 above.

[0295] The encoding unit 806, the speech activity detection decoding unit 814, and the speech activity detection classification unit 816 are used together to perform the following steps: based on the audio features, perform speech activity detection processing on the first speech data to obtain the first detection result 818.

[0296] The encoding unit 806, the overlapping speech detection decoding unit 820, and the overlapping speech detection classification unit 822 are used together to perform the following steps: based on the audio features, the first speech data is processed for overlapping speech detection to obtain the second detection result 824.

[0297] The result fusion module 826 fuses the speech recognition result 812, the first detection result 818, and the second detection result 824 to obtain the target speech recognition result. This target speech recognition result can be regarded as the speech recognition result of the speech object in the first speech data obtained after adjusting the speech recognition result 812 based on the first detection result 818 and the second detection result 824.

[0298] Since the technical concept is the same, the description in this embodiment is relatively simple. For the relevant parts, please refer to the corresponding descriptions of the method embodiments provided above.

[0299] The following is an example of a training device for a speech recognition model provided in this specification: In the above embodiments, a method for training a speech recognition model is provided. Based on the same technical concept, this application also provides a device for training a speech recognition model, which will be described below with reference to the accompanying drawings.

[0300] Figure 9 This is a schematic diagram of a speech recognition model training device provided in an embodiment of this application. This embodiment provides a speech recognition model training device 900, including: The acquisition unit 902 is used to acquire speech samples and a first speech recognition result annotated with the speech samples; the first speech recognition result includes a classification label for each frame of audio data in the speech samples, and each classification label represents the speech object participating in the dialogue in the corresponding frame of audio data. The extraction unit 904 is used to extract features from the speech samples to obtain the audio features of the speech samples; The first recognition unit 906 is used to recognize the speech object in each frame of audio data in the speech sample according to the audio features, and to obtain the second speech recognition result of the speech object; Construction unit 908 is used to construct a loss function based on the second speech recognition result and the first speech recognition result; Training unit 910 is used to iteratively train the speech recognition model based on the loss function.

[0301] In this embodiment, the training apparatus for the speech recognition model includes: an acquisition unit, used to acquire speech samples and a first speech recognition result annotated with the speech samples; the first speech recognition result includes a classification label for each frame of audio data in the speech samples, each classification label representing a speech object participating in the dialogue in the corresponding frame of audio data; an extraction unit, used to extract features from the speech samples to obtain audio features of the speech samples; a first recognition unit, used to recognize the speech object in each frame of audio data in the speech samples based on the audio features, to obtain a second speech recognition result for the speech object; a construction unit, used to construct a loss function based on the second speech recognition result and the first speech recognition result; and a training unit, used to iteratively train the speech recognition model based on the loss function. Considering that when a speech sample involves multiple speech objects participating in a dialogue, these speech objects may participate in the dialogue simultaneously, for example, three people speaking at the same time. In a single frame of audio data, the voices of multiple speech objects participating in the dialogue simultaneously are mixed together, and each speech object may interfere with the recognition accuracy of other speech objects. In this embodiment, the first speech recognition result for annotating the speech sample includes a classification label for each frame of audio data. Each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data. In this way, even if there are multiple speech objects participating in the dialogue simultaneously in a frame of audio data, that frame of audio data will only have one classification label. This classification label can be combined with a second classification label during training. The speech recognition results are used to construct a loss function, which can then guide the speech recognition model to learn how to map a frame of audio data to an output category composed of multiple speech objects. This output category is the second speech recognition result of the speech objects in the frame of audio data. Furthermore, the mixing of the sounds of multiple speech objects does not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution that guides the speech recognition model to learn how to map a frame of speech data to multiple output categories, with each output category corresponding to a speech object, the embodiments of this application can reduce the negative impact of the mutual interference of the sounds of the speech objects speaking at the same time on the recognition of speech objects, and is more conducive to improving the recognition accuracy of speech objects.

[0302] The following is an embodiment of a speech recognition device provided in this specification: In the above embodiments, a speech recognition method is provided. Based on the same technical concept, this application also provides a speech recognition device, which will be described below with reference to the accompanying drawings.

[0303] Figure 10 This is a schematic diagram of a speech recognition device provided in an embodiment of this application. This embodiment provides a speech recognition device 1000, including: The second recognition unit 1002 is used to input the first speech data into the target speech recognition model for speech object recognition processing, and obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by a speech recognition model training method; The tagging unit 1004 is used to divide the first speech data into multiple time periods based on the speech recognition result of the speech object, and to tag the speech object in each of the multiple time periods.

[0304] In this embodiment, the speech recognition device includes: a second recognition unit, configured to input first speech data into a target speech recognition model for speech object recognition processing, and obtain speech recognition results of the speech objects in the first speech data; the target speech recognition model is trained using a speech recognition model training method; and a labeling unit, configured to divide the first speech data into multiple time periods based on the speech recognition results of the speech objects, and to label the speech objects in each of the multiple time periods. Considering that when a speech sample involves multiple speech objects participating in a dialogue, each speech object may participate in the dialogue simultaneously, for example, three people speaking at the same time. In a frame of audio data, the voices of multiple speech objects participating in the dialogue simultaneously are mixed together, and each speech object may interfere with the recognition accuracy of other speech objects. In this embodiment, the first speech recognition result for labeling the speech sample includes a classification label for each frame of audio data. Each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data. Thus, even if multiple speech objects participate in the dialogue simultaneously in a frame of audio data, that frame of audio data only has one classification label. This classification label can be combined with a second... The speech recognition results are used to construct a loss function, which can then guide the speech recognition model to learn how to map a frame of audio data to an output category composed of multiple speech objects. This output category is the second speech recognition result of the speech objects in the frame of audio data. Furthermore, the mixing of the sounds of multiple speech objects does not affect the recognition accuracy of the speech recognition model. Therefore, compared with the technical solution that guides the speech recognition model to learn how to map a frame of speech data to multiple output categories, with each output category corresponding to a speech object, the embodiments of this application can reduce the negative impact of the mutual interference of the sounds of the speech objects speaking at the same time on the recognition of speech objects, and is more conducive to improving the recognition accuracy of speech objects.

[0305] Corresponding to the speech recognition model training method or speech recognition method described above, based on the same technical concept, this application embodiment also provides an electronic device for executing the speech recognition model training method or speech recognition method provided above. Figure 11This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0306] like Figure 11 As shown, electronic devices can vary considerably due to differences in configuration or performance. They may include one or more processors 1101 and memory 1102, with memory 1102 storing one or more application programs or data. Memory 1102 can be temporary or persistent storage. The application programs stored in memory 1102 may include one or more modules (not shown), each module including a series of computer-executable instructions within the electronic device. Furthermore, processor 1101 may be configured to communicate with memory 1102, executing the series of computer-executable instructions stored in memory 1102 on the electronic device. The electronic device may also include one or more power supplies 1103, one or more wired or wireless network interfaces 1104, one or more input / output interfaces 1105, one or more keyboards 1106, etc.

[0307] In one specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following: Acquire speech samples and a first speech recognition result annotated with the speech samples; the first speech recognition result includes a classification label for each frame of audio data in the speech samples, and each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data; Feature extraction is performed on the speech samples to obtain the audio features of the speech samples; Based on the audio features, identify the speech object in each frame of audio data in the speech sample, and obtain the second speech recognition result of the speech object; A loss function is constructed based on the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

[0308] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed by a processor, implement the following process: Acquire speech samples and a first speech recognition result annotated with the speech samples; the first speech recognition result includes a classification label for each frame of audio data in the speech samples, and each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data; Feature extraction is performed on the speech samples to obtain the audio features of the speech samples; Based on the audio features, identify the speech object in each frame of audio data in the speech sample, and obtain the second speech recognition result of the speech object; A loss function is constructed based on the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

[0309] It should be noted that the embodiments of computer-readable storage media described above in this specification and the embodiments of training methods for speech recognition models described above in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0310] In another specific embodiment, the electronic device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for use in the electronic device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following: The first speech data is input into the target speech recognition model for speech object recognition processing to obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the speech recognition model training method. Based on the speech recognition results of the speech objects, the first speech data is divided into multiple time periods, and the speech objects in each of the multiple time periods are marked.

[0311] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed by a processor, implement the following process: The first speech data is input into the target speech recognition model for speech object recognition processing to obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the speech recognition model training method. Based on the speech recognition results of the speech objects, the first speech data is divided into multiple time periods, and the speech objects in each of the multiple time periods are marked.

[0312] It should be noted that the embodiments of computer-readable storage media described above in this specification and the embodiments of speech recognition methods described above in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0313] Another embodiment of this disclosure also provides a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the following process: Acquire speech samples and a first speech recognition result annotated with the speech samples; the first speech recognition result includes a classification label for each frame of audio data in the speech samples, and each classification label represents a speech object participating in the dialogue in the corresponding frame of audio data; Feature extraction is performed on the speech samples to obtain the audio features of the speech samples; Based on the audio features, identify the speech object in each frame of audio data in the speech sample, and obtain the second speech recognition result of the speech object; A loss function is constructed based on the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

[0314] The computer program product in this disclosure embodiment can implement the various processes of the above-described speech recognition model training method embodiment and achieve the same effect and function, which will not be repeated here.

[0315] Another embodiment of this disclosure also provides a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the following process: The first speech data is input into the target speech recognition model for speech object recognition processing to obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the speech recognition model training method. Based on the speech recognition results of the speech objects, the first speech data is divided into multiple time periods, and the speech objects in each of the multiple time periods are marked.

[0316] The computer program product in this disclosure embodiment can implement the various processes of the above-described speech recognition method embodiment and achieve the same effect and function, which will not be repeated here.

[0317] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0318] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, this specification can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0319] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable electronic device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable electronic device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0320] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable electronic device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0321] These computer program instructions may also be loaded onto a computer or other programmable electronic device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0322] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0323] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0324] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0325] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0326] The embodiments of this application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0327] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0328] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A method for training a speech recognition model, characterized in that, include: Acquire speech samples and a first speech recognition result annotated with the speech samples; The first speech recognition result includes a classification label for each frame of audio data in the speech sample. Each classification label is a first object identifier group, which is used to identify the speech object participating in the dialogue in each frame of audio data and the number of the speech object. The speech sample includes multiple audio data frames containing multiple speech objects participating in the dialogue. Based on the speech recognition model, the audio features of each frame of audio data in the speech sample are obtained. as well as, Based on the audio features of each frame of audio data and each pre-stored audio feature in the preset feature set, the speech object in each frame of audio data in the speech sample is identified, and a second speech recognition result of the speech object is obtained. The second speech recognition result is a second object identifier group. The preset feature set includes a first pre-stored audio feature, which is a feature when multiple speech objects speak at the same time. The first pre-stored audio feature corresponds to the multiple speech objects. A loss function is constructed based on the second speech recognition result and the first speech recognition result; The speech recognition model is iteratively trained based on the loss function.

2. The method according to claim 1, characterized in that, Each first object identifier group includes a first identifier, and the number of the first identifiers is within a preset range; the step of identifying the speech object in each frame of audio data in the speech sample based on the audio features of each frame of audio data and each pre-stored audio feature in the preset feature set, and obtaining the second speech recognition result of the speech object, includes: Based on the audio features of each frame of audio data and each pre-stored audio feature in the preset feature set, the speech objects of each frame of audio data are classified to obtain a second object identifier group; The step of constructing a loss function based on the second speech recognition result and the first speech recognition result includes: A loss function is constructed based on the second object identifier group and the first object identifier group of each frame of audio data.

3. The method according to claim 1, characterized in that, The category labels are generated in the following way: Obtain the voice objects participating in the dialogue from the voice samples; At least one object group is generated based on the number of the voice objects, resulting in a set of object groups; in the set of object groups, the number of the voice objects in each object group is within a preset range. For each object group in the object group set, a corresponding object identifier group is generated, and each object identifier group is determined as a category label.

4. The method according to claim 1, characterized in that, Also includes: Based on the audio features, speech activity detection is performed on the speech sample to obtain a first detection result; Based on the audio features, overlapping speech detection is performed on the speech samples to obtain a second detection result; The step of constructing a loss function based on the second speech recognition result and the first speech recognition result includes: A loss function is constructed based on the second speech recognition result, the first detection result, the second detection result, and the first speech recognition result.

5. The method according to claim 1, characterized in that, The speech recognition model includes N speech coding layers; the feature extraction of the speech samples to obtain the audio features of the speech samples includes: Each speech coding layer encodes the speech sample to obtain the first encoded information output by each speech coding layer. Where i=1, the speech sample is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension; when i is greater than 1, the speech coding information of the (i-1)-th dimension output by the (i-1)-th speech coding layer is input into the i-th speech coding layer to obtain the speech coding information of the i-th dimension, where i is a positive integer less than or equal to N. Based on the first coding information output by each of the N speech coding layers and the weight parameters of each dimension, the audio features of the speech sample are determined.

6. The method according to claim 1, characterized in that, The method for generating the speech samples includes: Acquire first voice data and second voice data, wherein the first voice data includes voice data of at least two voice objects and the second voice data includes voice data of one voice object; Virtual voice data of virtual voice objects is generated, and the virtual voice data is expanded into the second voice data to obtain expanded second voice data, wherein the expanded second voice data includes voice data of at least two voice objects; A first speech sample is constructed based on the first speech data, and a second speech sample is constructed based on the expanded second speech data; The first speech sample and the second speech sample are extracted according to a preset ratio to obtain the speech sample.

7. The method according to claim 1, characterized in that, The speech sample includes K frames of audio data, and the number of audio features is K, with each of the K frames of audio data corresponding to one of the K audio features. The process of identifying the speech object in each frame of audio data in the speech sample and obtaining the second speech recognition result of the speech object includes: The K audio features are encoded to obtain the first feature data of the K frames of audio data; The first feature data of each frame of audio data is decoded to obtain the second feature data describing the speech object of each frame of audio data. The second feature data of each frame of audio data is classified to obtain the speech object of each frame of audio data. Based on the speech object of the K frames of audio data, a second speech recognition result of the speech object is obtained.

8. The method according to claim 5, characterized in that, The step of determining the audio features of the speech sample based on the first coding information output by each coding layer and the weight parameters of each dimension includes: For each dimension, the weight parameter of that dimension is compared with a preset weight threshold; If the comparison result indicates that the weight parameter of the dimension is less than the preset weight threshold, then the weight parameter of the dimension is reset to the preset value. The audio features of the speech sample are obtained by weighted summation of the weight parameters of each dimension after reset and the first encoding information.

9. The method according to claim 4, characterized in that, The audio features include a first audio feature describing the energy of the speech sample, a second audio feature describing the spectrum of the speech sample, and a third audio feature describing the signal waveform of the speech sample. The step of performing speech activity detection on the speech sample based on the audio features to obtain a first detection result includes: Based on the first audio feature, the second audio feature, and the third audio feature, a first time period in which active speech exists in the speech sample is determined. Based on the total time period of the speech samples and the first time period, a second time period with missing active speech is determined; The first detection result is generated based on the first time period and the second time period.

10. A speech recognition method, characterized in that, include: The first speech data is input into the target speech recognition model for speech object recognition processing to obtain the speech recognition result of the speech object in the first speech data; the target speech recognition model is trained by the speech recognition model training method of any one of claims 1-9. Based on the speech recognition results of the speech objects, the first speech data is divided into multiple time periods, and the speech objects in each of the multiple time periods are marked.

11. An electronic device, characterized in that, include: processor; And a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform a training method for a speech recognition model as described in any one of claims 1 to 9 or a speech recognition method as described in claim 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store computer-executable instructions, which, when executed by a processor, implement the training method of the speech recognition model as described in any one of claims 1 to 9 or the speech recognition method as described in claim 10.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the training method for the speech recognition model as described in any one of claims 1 to 9 or the speech recognition method as described in claim 10.

Images