Speech recognition model training method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to a method and device for training a speech recognition model, equipment and a storage medium. The method comprises the steps that a voice sample set is acquired, the voice sample set comprises a first group of voice samples and a second group of voice samples, the time length of the first group of voice samples is smaller than a first threshold value, and the time length of the second group of voice samples is larger than a second threshold value; and training a speech recognition model using the speech sample set and the corresponding text information to at least adjust parameters of a speech encoding unit in the speech recognition model, the speech recognition model comprising a speech encoding unit configured to generate a speech encoded representation of the speech content and a decoding unit configured to decode the speech encoded representation of the speech content. The decoding unit is configured to generate a speech recognition result based on the speech encoded representation. In this way, the voice recognition capability of the model for the long voice content can be improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for training speech recognition models. Background Technology

[0002] In recent years, with the rapid development of machine learning technology, speech recognition models based on machine learning have been widely used to improve the efficiency of processing speech content. However, existing speech recognition models cannot meet people's needs for processing speech content. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for training a speech recognition model is provided. The method includes: acquiring a speech sample set, the speech sample set including a first group of speech samples and a second group of language samples, wherein the duration of the first group of speech samples is less than a first threshold, and the duration of the second group of speech samples is greater than a second threshold; and training a speech recognition model using the speech sample set and corresponding text information to at least adjust the parameters of a speech coding unit in the speech recognition model, the speech recognition model including a speech coding unit and a decoding unit, the speech coding unit being configured to generate a speech-coded representation of speech content, and the decoding unit being configured to generate a speech recognition result based on the speech-coded representation.

[0004] In a second aspect of this disclosure, an apparatus for training a speech recognition model is provided. The apparatus includes: an acquisition module configured to acquire a speech sample set, the speech sample set including a first set of speech samples and a second set of language samples, wherein the duration of the first set of speech samples is less than a first threshold, and the duration of the second set of speech samples is greater than a second threshold; and a training module configured to train the speech recognition model using the speech sample set and corresponding text information, to at least adjust the parameters of a speech coding unit in the speech recognition model, the speech recognition model including a speech coding unit and a decoding unit, the speech coding unit being configured to generate a speech-coded representation of speech content, and the decoding unit being configured to generate a speech recognition result based on the speech-coded representation.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0010] Figure 2 A flowchart illustrating an example process for training a speech recognition model according to some embodiments of the present disclosure is shown;

[0011] Figure 3 Example structures of speech recognition models according to some embodiments of the present disclosure are shown;

[0012] Figure 4 A schematic structural block diagram of an example apparatus for training a speech recognition model according to some embodiments of the present disclosure is shown; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0018] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0019] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0020] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as an input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0021] As mentioned above, with the rapid development of machine learning technology, speech recognition models based on machine learning are widely used to improve the efficiency of speech recognition. However, existing speech recognition models have limited capabilities, only able to process specific speech content and failing to meet the needs of users processing diverse speech content. Especially for speech content of varying lengths, existing speech recognition models exhibit low accuracy and poor recognition performance.

[0022] Embodiments of this disclosure propose a scheme for training a speech recognition model. According to this scheme, a speech sample set can be obtained, comprising a first set of speech samples and a second set of language samples, wherein the duration of the first set of speech samples is less than a first threshold, and the duration of the second set of speech samples is greater than a second threshold; and a speech recognition model is trained using the speech sample set and corresponding text information to at least adjust the parameters of the speech coding unit in the speech recognition model. The speech recognition model includes a speech coding unit and a decoding unit, wherein the speech coding unit is configured to generate a speech-coded representation of the speech content, and the decoding unit is configured to generate a speech recognition result based on the speech-coded representation.

[0023] In this way, the embodiments of this disclosure can train a speech recognition model based on two sets of speech samples associated with different thresholds, so that the trained speech recognition model can be adapted to the recognition of speech content of different durations, thereby improving the efficiency and accuracy of speech recognition.

[0024] Furthermore, compared to simply breaking down long speech content into multiple short speech segments for recognition, the embodiments of this disclosure can utilize richer contextual information to achieve the recognition of long speech content, thereby improving the accuracy of speech recognition.

[0025] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0026] Example Environment

[0027] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110.

[0028] In this example environment 100, electronic device 110 can run an application 120 that supports user interface interaction. Application 120 can be any suitable type of application for user interface interaction, examples of which may include, but are not limited to, voice applications or other applications related to voice recognition. User 140 can interact with application 120 via electronic device 110 and / or its attached devices.

[0029] exist Figure 1 In environment 100, if application 120 is active, electronic device 110 can use application 120 to present interface 150 for supporting interface interaction.

[0030] In some embodiments, electronic device 110 communicates with server 130 to provide services to application 120. Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry).

[0031] Server 130 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server 130 may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. Server 130 can provide backend services for applications 120 that support virtual scenarios in electronic devices 110.

[0032] A communication connection can be established between server 130 and electronic device 110. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections; the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, server 130 and electronic device 110 can achieve signaling interaction through the communication connection between them.

[0033] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0034] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0035] Example process

[0036] Figure 2 A flowchart illustrating an example process 200 for training a speech recognition model according to some embodiments of the present disclosure is shown. Process 200 can be implemented at electronic device 110. Reference is made below. Figure 1 To describe process 200.

[0037] like Figure 2 As shown in box 210, the electronic device 110 can acquire a speech sample set, which may include a first set of speech samples and a second set of language samples. The duration of the first set of speech samples may be less than a first threshold, and the duration of the second set of speech samples may be greater than a second threshold.

[0038] In some embodiments, the first threshold may be the same as the second threshold, or the first threshold may be less than the second threshold. This disclosure is not intended to limit the specific size of the first and second thresholds. In some scenarios, speech samples with a duration less than the first threshold may also be referred to as short speech samples, and speech samples with a duration greater than the second threshold may also be referred to as long speech samples.

[0039] The following will be referenced Figure 3 The illustrated speech recognition model 300 is used to describe an example process of training a speech recognition model according to an embodiment of the present disclosure.

[0040] Figure 3 An example framework of a speech recognition model 300 according to some embodiments of the present disclosure is shown. Figure 3 As shown, the speech recognition model 300 may include a speech encoding unit 310, a conversion unit 315, and a decoding unit 320.

[0041] In some embodiments, reference Figure 3 The speech encoding unit 310 can generate a speech encoded representation (or speech feature sequence) corresponding to the speech content. For example, the speech encoding unit 310 can generate a speech encoded representation corresponding to the speech sample set 305. For example, the speech encoding unit 310 can be implemented as an audio encoder.

[0042] In some embodiments, continue to refer to Figure 3 The conversion unit 315 can convert the speech encoded representation into speech features for provision to the decoding unit 320. As an example, the conversion unit 315 can be implemented based on a modality converter.

[0043] In some embodiments, continue to refer to Figure 3 The decoding unit 320 can generate a speech recognition result based on the speech features generated by the conversion unit 320. As an example, the decoding unit 320 can be a language model. By utilizing a language model as the decoding unit 320, embodiments of this disclosure can leverage the long text modeling capabilities of the language model to better process the relevant contextual information of long speech content, thereby improving the accuracy of speech recognition.

[0044] In other scenarios, the decoding unit 320 may also generate the speech recognition result directly based on the speech encoding representation output by the speech encoding unit 320. Accordingly, the conversion unit 315 may be omitted from the speech recognition model 300.

[0045] In some embodiments, such as Figure 3 As shown, a language model is used as an example of the decoding unit 320, which generates speech recognition results, for example, based on next token prediction (NTP).

[0046] like Figure 3 As shown, <sos>Indicates the start of a sentence; <eos>Indicates the end of a sentence. As an example, the decoding unit 320 can predict the next token to be output based on the input token sequence. As an example, the output token can correspond to a character or a word.

[0047] Take Figure 3 As an example, after determining that the first output token is "今", the interface unit 320 can determine that the next token to be output is "天" based on the updated token sequence including "今", thereby implementing NTP-based speech recognition.

[0048] In some embodiments, such as Figure 3 As shown, the token sequence input to the decoding unit 320 may further include a hint item 325. The hint item 325 can be used to instruct the decoding unit 320 to perform a speech recognition task.

[0049] In some embodiments, the electronic device 110 can first use training speech data to pre-train the speech coding unit 310 in the speech recognition model 300. For example, the electronic device 110 can pre-train the speech coding unit 310 in the speech recognition model 300 through a self-supervised learning (SSL) process.

[0050] In some embodiments, during the pre-training process, the electronic device 110 can generate a first feature sequence of the training speech samples, for example, spectral features. Further, the electronic device 110 can generate a second feature sequence by masking at least part of the first sequence feature. As an example, the electronic device 110 can randomly mask the features corresponding to at least part of the moments of the first sequence feature to obtain the second feature sequence.

[0051] In some embodiments, the electronic device 110 can use the speech coding unit 310 to be trained to process the second feature sequence to generate first label information. As an example, when the electronic device 110 uses the speech coding unit 310 to process the second feature sequence, it can encode the features and predict the features at the masked positions to obtain the first label information.

[0052] In some embodiments, the electronic device 110 can obtain second label information by comparing the first feature sequence with a preset codebook. As an example, the preset codebook can include a set of preset feature representations. As an example, the electronic device 110 can obtain a set of indexes matching the first feature sequence as the second label information based on the preset codebook.

[0053] In some embodiments, the electronic device 110 can obtain a comparison result based on comparing the first tag information and the second tag information. Further, the electronic device 110 can adjust the parameters of the speech coding unit 310 based on the comparison result. In this way, the trained speech coding unit 310 can have a strong predictive ability for discontinuous (e.g., partially missing) speech content.

[0054] In some embodiments, in order to support the speech recognition model 300 in processing speech samples of various durations, especially longer durations, the electronic device 110 may use a mixture of speech samples of different durations to train the speech recognition model.

[0055] In some embodiments, the electronic device 110 may construct a speech sample set 305 using a first set of speech samples and a second set of speech samples with different durations. In some scenarios, the first set of speech samples may also be referred to as "short speech samples," and the second set of speech samples may also be referred to as "long speech samples." It should be understood that the threshold for distinguishing between "short speech samples" and "long speech samples" can be appropriately set according to the actual situation, and the embodiments of this disclosure do not limit this.

[0056] In some embodiments, the second set of samples may include multiple speech samples corresponding to multiple preset time lengths. For example, the second set of samples may be obtained by averaging samples over a preset time range greater than a second threshold. For example, the second threshold may be 0.2h, and the preset time range may be from 0.2h to 3h. Further, for example, a preset step size of 0.2h may be used, then the multiple preset time lengths may include 0.2h, 0.4h, ..., 2.8h, 3h. Further, the second set of samples may include a predetermined number of samples corresponding to each preset time length. For example, approximately 100 samples may be sampled for each preset time length. It should be noted that this is merely an illustrative example, and the specific values ​​of the second threshold, preset time range, preset step size, and multiple preset time lengths are not limited in this disclosure.

[0057] In some embodiments, the electronic device 110 may also acquire text information associated with the speech sample set 305. As an example, the text information may serve as annotation information corresponding to the speech sample set 305.

[0058] In box 220, electronic device 110 can use speech sample set 305 and corresponding text information to train speech recognition model to at least adjust the parameters of speech coding unit 310 in speech recognition model.

[0059] In some embodiments, the electronic device 110 can determine the target loss for the speech recognition model based at least on the text information corresponding to the speech sample set 305 and the speech recognition results. Furthermore, the electronic device 110 can adjust the parameters involved in the speech recognition model based on the target loss.

[0060] In some embodiments, the speech recognition result may include a prediction result obtained by the decoding unit 320 based on the next token prediction. The electronic device 110 may also determine the target loss based on the prediction result and text information.

[0061] In some embodiments, during the training of the speech recognition model, the electronic device 110 may also adjust the parameters of the conversion unit 315 based on the speech sample set 305 and the corresponding text information. As an example, the electronic device 110 may adjust the parameters of the conversion unit 315 by comparing the text information corresponding to the speech sample set 305 with the speech recognition result produced by the speech recognition model. In some embodiments, when the decoding unit 320 includes a language model, such loss includes the loss incurred by the language model in performing token prediction.

[0062] In some embodiments, the electronic device 110 may also fix the parameters of the decoding unit 320 when training the speech recognition model based on the speech sample set 305 and the corresponding text information. It is understood that fixing the parameters of the decoding unit 320 can prevent the speech recognition model training process from affecting the decoding unit 320.

[0063] In some embodiments, the electronic device 110 can also fine-tune the parameters of the decoding unit 320 when training the speech recognition model based on the speech sample set 305 and the corresponding text information. As an example, the electronic device 110 can set a small learning rate for the parameters of the decoding unit 320 when training the speech recognition model to achieve fine-tuning of the decoding unit 320.

[0064] In some embodiments, the electronic device 110 may also adjust the parameters of the fine-tuning module associated with the decoding unit 320 when training the speech recognition model based on the speech sample set 305 and the corresponding text information. As an example, the electronic device 110 may adjust the parameters of the low-rank adaptation (LoRA) module associated with the decoding unit 320 accordingly.

[0065] In some embodiments, the electronic device 110 can train a speech recognition model through multiple training stages using the speech sample set 305 and the corresponding text information. Specifically, the process of training the speech recognition model using mixed samples described above can correspond to the second training stage.

[0066] In some embodiments, the electronic device 110 may train a speech recognition model using a first set of speech training samples during a first training phase. Thus, through training in the first training phase, a speech recognition model with strong processing capabilities for speech content with a duration less than a first threshold can be obtained.

[0067] In some embodiments, as referenced above Figure 2 As described, the electronic device 110 can train a speech recognition model in the second training phase using a mixture of the first and second sets of speech samples. As an example, the mixture of the first and second sets of speech samples enables the speech recognition model to handle speech content of varying durations. Thus, the speech recognition model can handle speech content of different durations.

[0068] In some embodiments, the electronic device 110 may determine the recognition performance of the speech recognition model based on the speech sample set 305 used to train the speech recognition model and / or on tests of the speech recognition model's recognition capabilities. As an example, it may be determined that the speech recognition model achieves good recognition efficiency and / or accuracy for speech content of a target time length.

[0069] In some embodiments, the electronic device 110 may acquire target speech content to be processed. Further, in response to the target speech content's length (e.g., time length) exceeding a third threshold, the electronic device 110 may split the target speech content into multiple speech segments based on the target time length. Further, the electronic device 110 may utilize a trained speech recognition model to process the multiple speech segments to generate a speech generation result for the target speech content.

[0070] As an example, electronic device 110 can use a trained speech recognition model to process multiple speech segments, obtaining multiple speech recognition results. Further, electronic device 110 can concatenate the multiple speech recognition results according to the corresponding order of the multiple speech segments to obtain the speech generation result of the target speech content. This disclosure does not limit the specific value of the third threshold. Thus, for speech content with a relatively long duration (e.g., several hours to tens of hours), segmenting the speech content based on the recognition performance of the speech recognition model can improve the efficiency and accuracy of the model in processing the speech content.

[0071] Based on the training process described above, embodiments of this disclosure can adjust the parameters of each unit (or module) in the speech recognition model based on speech samples of different durations to obtain a trained speech recognition model. When processing speech content based on the speech recognition model, the speech content can be divided into multiple speech segments based on an appropriate target duration for processing, depending on the performance of the speech recognition model.

[0072] Furthermore, in the speech recognition stage, the embodiments of this disclosure can adaptively segment speech content that is too long, thereby improving the training efficiency of the speech recognition model and its ability to recognize speech content.

[0073] Example devices and equipment

[0074] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an example apparatus 400 for training a speech recognition model according to certain embodiments of the present disclosure is shown. Apparatus 400 may be implemented as or included in electronic device 110. Various modules / components in apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0075] like Figure 4 As shown, the device 400 includes an acquisition module 410 configured to acquire a speech sample set, the speech sample set including a first set of speech samples and a second set of language samples, the duration of the first set of speech samples being less than a first threshold, and the duration of the second set of speech samples being greater than a second threshold; and a training module 420 configured to train a speech recognition model using the speech sample set and corresponding text information, so as to at least adjust the parameters of the speech coding unit in the speech recognition model, the speech recognition model including a speech coding unit and a decoding unit, the speech coding unit being configured to generate a speech coding representation of speech content, and the decoding unit being configured to generate a speech recognition result based on the speech coding representation.

[0076] In some embodiments, the acquisition module 410 is further configured to include a second set of speech samples that correspond to a plurality of preset time lengths.

[0077] In some embodiments, the training module 420 is further configured to: train a speech recognition model using a speech sample set and corresponding text information, including: adjusting the parameters of a pre-trained speech coding unit in the speech recognition model based on the speech sample set and text information, wherein the speech coding unit is pre-trained based on training speech data.

[0078] In some embodiments, the training module 420 is further configured to: pre-train the speech coding unit based on a self-supervised training process, the self-supervised training process including: generating a first feature sequence of training speech samples; generating a second feature sequence by masking at least a portion of the first feature sequence; processing the second feature sequence using the speech coding unit to generate first label information; and adjusting the parameters of the speech coding unit based on a comparison of the first label information and the second label information, the second label information being generated based on a comparison of the first feature sequence and a preset codebook.

[0079] In some embodiments, the training module 420 is further configured such that the speech recognition model further includes a conversion unit configured to convert the speech encoded representation into speech features processed by the decoding unit.

[0080] In some embodiments, the training module 420 is further configured to: train the speech recognition model using the speech sample set and the corresponding text information, further comprising: adjusting the parameters of the conversion unit based on the speech sample set and the text information.

[0081] In some embodiments, the training module 420 is further configured to: train the speech recognition model using a speech sample set and corresponding text information, including: fixing the parameters of the decoding unit; fine-tuning the parameters of the decoding unit; or adjusting the parameters of the fine-tuning module associated with the decoding unit.

[0082] In some embodiments, the training module 420 is further configured to: train the speech recognition model using a set of speech samples and corresponding text information, including: in a first training phase, training the speech recognition model using a first set of speech samples; and in a second training phase, training the speech recognition model using a mixture of the first set of speech samples and the second set of speech samples.

[0083] In some embodiments, the apparatus 400 further includes a processing module configured to: acquire target speech content to be processed; in response to the length of the target speech content being greater than a third threshold, split the target speech content into multiple speech segments based on a target time length; and process the multiple speech segments using a trained speech recognition model to generate a speech generation result for the target speech content.

[0084] In some embodiments, the processing module is further configured such that the target duration is determined based on the recognition performance of the speech recognition model for speech content of different durations.

[0085] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Electronic devices 110.

[0086] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0087] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0088] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0089] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0090] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0091] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0092] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0093] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0094] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0095] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0096] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.< / eos> < / sos>

Claims

1. A method for training a speech recognition model, comprising: Obtain a speech sample set, which includes a first set of speech samples and a second set of language samples. The duration of the first set of speech samples is less than a first threshold, and the duration of the second set of speech samples is greater than a second threshold. as well as Using the speech sample set and corresponding text information, a speech recognition model is trained to at least adjust the parameters of the speech coding unit in the speech recognition model. The speech recognition model includes the speech coding unit and a decoding unit. The speech coding unit is configured to generate a speech coding representation of the speech content, and the decoding unit is configured to generate a speech recognition result based on the speech coding representation.

2. The method according to claim 1, wherein the second group of speech samples includes multiple speech samples corresponding to multiple preset time lengths.

3. The method according to claim 1, wherein training the speech recognition model using the speech sample set and corresponding text information comprises: Based on the speech sample set and the text information, the parameters of the pre-trained speech coding unit in the speech recognition model are adjusted, wherein the speech coding unit is pre-trained based on training speech data.

4. The method according to claim 3, wherein the speech coding unit is pre-trained based on a self-supervised training process, the self-supervised training process comprising: Generate the first feature sequence of the training speech samples; A second feature sequence is generated by masking at least a portion of the first feature sequence; The second feature sequence is processed using the speech coding unit to generate first label information; as well as Based on the comparison of the first tag information and the second tag information, the parameters of the speech coding unit are adjusted. The second tag information is generated based on the comparison of the first feature sequence and the preset codebook.

5. The method of claim 1, wherein the speech recognition model further comprises a conversion unit configured to convert the speech encoded representation into speech features processed by the decoding unit.

6. The method according to claim 5, wherein training the speech recognition model using the speech sample set and corresponding text information further comprises: The parameters of the conversion unit are adjusted based on the speech sample set and the text information.

7. The method according to claim 1, wherein training the speech recognition model using the speech sample set and corresponding text information further comprises: The parameters of the decoding unit are fixed; Fine-tune the parameters of the decoding unit; or Adjust the parameters of the fine-tuning module associated with the decoding unit.

8. The method according to claim 1, wherein training the speech recognition model using the speech sample set and corresponding text information further comprises: In the first training phase, the speech recognition model is trained using the first set of speech samples; as well as In the second training phase, the speech recognition model is trained using a mixture of the first set of speech samples and the second set of speech samples.

9. The method according to claim 1, further comprising: Obtain the target speech content to be processed; In response to the target speech content having a length greater than a third threshold, the target speech content is split into multiple speech segments based on the target time length; as well as The trained speech recognition model is used to process the multiple speech segments to generate a speech generation result for the target speech content.

10. The method according to claim 9, wherein the target time length is determined based on the recognition performance of the speech recognition model for speech content of different time lengths.

11. An apparatus for training a speech recognition model, comprising: The acquisition module is configured to acquire a speech sample set, which includes a first set of speech samples and a second set of language samples. The duration of the first set of speech samples is less than a first threshold, and the duration of the second set of speech samples is greater than a second threshold. as well as The training module is configured to train a speech recognition model using the speech sample set and corresponding text information, so as to at least adjust the parameters of the speech coding unit in the speech recognition model. The speech recognition model includes the speech coding unit and the decoding unit. The speech coding unit is configured to generate a speech coding representation of the speech content, and the decoding unit is configured to generate a speech recognition result based on the speech coding representation.

12. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 10.

Images